Skip to main content

Enabling End-to-End TLS for Vault HA with Integrated Storage on EKS

· 4 min read
Saintmalik

When you are preparing your vault environment for production, you would want to implement the end-to-end tls setup as stated in the hashicorp vault production-ready documentation.

To make sure communications within your vault and its replicas are encrypted and secure if you have the HA(High Availability) on.

In this guide, I will work you through the process for those using vault on eks, let's jump into it

Prerequisites

You have your eks cluster up and you have installed vault, you have kubectl and OpenSSL installed on your local environment.

Creating the TLS Certificate

  1. You have to create the private key, do that with the following command
openssl genrsa -out vault.key 2048

The above command will create an RSA private key for you.

  1. You have to prepare a the CSR( Certificate Signing Request) configurations, here you put all the important vault DNS
 cat > vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = app_serving
req_extensions = v3_req
[ app_serving ]
O = system:nodes
CN = system:node:*.vault.svc.cluster.local
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.vault-internal
DNS.2 = *.vault-internal.vault.svc.cluster.local
DNS.3 = *.vault
DNS.4 = vault
DNS.5 = vault-active.vault.svc.cluster.local
IP.1 = 127.0.0.1
EOF
  1. With the above configuration file, you can generate the CSR file using the following command
openssl req -new -key vault.key -out vault.csr -config vault-csr.conf
  1. Now that you have the CSR created, you have to issue it to EKS, so you can generate and sign your vault certificates using EKS CA(Certificate Authority).

Create the csr yaml file with the following command

cat <<EOF >vault-csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: vault.svc
spec:
signerName: beta.eks.amazonaws.com/app-serving
request: $(cat vault.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF

Notice the signerName we are using here is beta.eks.amazonaws.com/app-serving instead of kubernetes.io/kubelet-serving according to other guides.

Yes, this is because from EKS v1.22 or later, you have to use a custom signer, else you would see that your CSR would be approved but not signed.

  1. It's time to apply the CSR yaml file to eks by running the following command
kubectl apply -f vault-csr.yaml
  1. Once the CSR has been deployed to your EKS, let's approve the serving cert.
kubectl certificate approve vault.svc
  1. Time to check if the cert has been approved and signed
vault get csr vault.svc

You should see an output just like this if everything goes well

NAME       AGE         SIGNERNAME                                             REQUESTOR          CONDITION
vault.svc 1m beta.eks.amazonaws.com/app-serving kubernetes-admin Approved, Issued
  1. Since everything went well, let's retrieve the certificate that was generated and signed for us by the EKS CA
kubectl get csr vualt.svc -o jsonpath='{.status.certificate}'| base64 -d > vault.crt
  1. You should have vault.key and vault.crt file by now, let's retrieve the EKS CA, since that's what we used in generating and signing vault.crt.
kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > vault.ca
  1. It would be great to verify the cert with the CA to check if there is any error, to avoid wasting so much time debugging your vault for the cause of errors like Bad Certificate.
openssl verify -verbose -CAfile vault.ca vault.crt

if it goes well, you should see the output vault.crt: OK

  1. Now that we are done, let's create the TLS secret in Kubernetes
kubectl create secret generic vault-tls \
-n vault \
--from-file=vault.key=vault.key \
--from-file=vault.crt=vault.crt \
--from-file=vault.ca=vault.ca
  1. if the certs tls secret creation goes well, time to mount them back into your vault override configurations.
global:
tlsDisable: false
server:
extraEnvironmentVars:
VAULT_CACERT: /vault/tls/vault.ca
VAULT_TLSCERT: /vault/tls/vault.crt
VAULT_TLSKEY: /vault/tls/vault.key
volumes:
- name: tls
secret:
secretName: vault-tls
volumeMounts:
- name: tls
mountPath: "/vault/tls"
readOnly: true
ha:
enabled: true
raft:
enabled: true
setNodeId: true
config: |
ui = true
disable_mlock = true
listener "tcp" {
tls_disable = false
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/tls/vault.crt"
tls_key_file = "/vault/tls/vault.key"
tls_client_ca_file = "/vault/tls/vault.ca"
}
storage "raft" {
path = "/vault/data"

service_registration "kubernetes" {}
}

Well, that's it, folks! I hope you find this piece insightful and helpful.

Okay, thats all

Comments