Using Hashicorp Vault on Kubernetes
I wanted to deploy hashicorp vault on a kubernetes clusters just to see how the integration works out. Here are some of the steps I took to deploy vault in my kubernetes cluster.
Prereqs
Here are some tools I need to install prior to running through the setup.
Install helm
First we need to install helm, the setup is covered in Installing Helm:
curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
Setting up Vault
There are a bunch of steps, so let’s break them down into sections:
Installing Vault in K8S
Most instructions are available at Vault on Kubernetes Deployment Guide. First we need to add the helm repo:
> helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
Then we can check out the latest version of package:
> helm search repo hashicorp/vault
NAME CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.16.0 1.8.2 Official HashiCorp Vault Chart
Now we can use helm to get all the k8s manifests:
> helm template vault hashicorp/vault --set "injector.enabled=false"
After I got all resources templated out, I used kustomize to apply them:
> k apply -k base
serviceaccount/vault created
clusterrolebinding.rbac.authorization.k8s.io/vault-server-binding created
configmap/vault-config created
service/vault created
statefulset.apps/vault created
Post Install Configuration of Vault
Now let’s initialze the vault:
> kubectl exec --stdin=true --tty=true vault-0 -- vault operator init
Unseal Key 1: KEPop9G9Qi+3ypNks1KqpE4qydVf1TB/5aDd/NDVMeI7
Unseal Key 2: H5QLGcEiOxcXuynA6zilQs6DCiad0z6AHz4Gtz6/vcLx
Unseal Key 3: +/GXkOC2W9Ne+SEywnBJWStT7uwUXzAJFhpOJIhksdVt
Unseal Key 4: uhHsX82gzcJovLCnsYxj099uMvZK1u67hoXHuKePqiEN
Unseal Key 5: N4wBz1BEG4NhPtho5MDusyTIwGK3RbgfBWajvEeJjXZN
Initial Root Token: s.qiz9McA66XL2U0YAc5FXqL0M
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
Next we need to unseal the vault:
> kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce c6fef280-a3c5-1d6f-f89f-48a097a13f0f
Version 1.8.2
Storage Type file
HA Enabled false
> kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce c6fef280-a3c5-1d6f-f89f-48a097a13f0f
Version 1.8.2
Storage Type file
HA Enabled false
> kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.8.2
Storage Type file
Cluster Name vault-cluster-2ba9762b
Cluster ID 4d5e7c9e-356c-38f1-34e6-6ba813ada808
HA Enabled false
After that all the pods should be in a READY state:
> kubectl get pods -l app=vault
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 33m
Creating a secret in vault
First we have to authenticate with the root token, which was provided when we unsealed the vault:
> kubectl exec -it vault-0 -- /bin/sh
/ $ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.qiz9McA66XL2U0YAc5FXqL0M
token_accessor 6LJVwnIE5W0WhJxrp2bLrm6M
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
Now let’s enable the kv engine:
/ $ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/
Now let’s create a simple secret:
vault kv put secret/app-pass password="coolio"
Now let’s confirm it’s set:
/ $ vault kv get secret/app-pass
====== Metadata ======
Key Value
--- -----
created_time 2021-09-29T17:57:33.485749067Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
password coolio
Not too shabby :)
Exposing Secrets to Pods
There are a couple of ways of accomplishing this. The first one is using the vault agent. Basically you have a side car container running next to your application container and it takes care of presenting the secret to the application container. From Vault Agent with Kubernetes here is a nice overview:
We can also use the CSI provider and the guide is at Mount Vault Secrets through Container Storage Interface (CSI) Volume. Also here is a nice diagram from Retrieve HashiCorp Vault Secrets with Kubernetes CSI:
And lasty we can call the API directly using client libraries (Nice example with python is seen at How to Store Your Secrets Using Vault and Python)
Vault CSI Driver in K8S
Let’s try to use the CSI driver to get a secret into a pod, steps are covered Mount Vault Secrets through Container Storage Interface (CSI) Volume. First we need to install the necessary system components into K8S, we can do that by using helm to generate the templates:
helm template vault hashicorp/vault --set "injector.enabled=false" \
--set "csi.enabled=true"
And then to apply your custom configs with kustomize:
> k apply -k base
serviceaccount/vault-csi-provider created
clusterrole.rbac.authorization.k8s.io/vault-csi-provider-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/vault-csi-provider-clusterrolebinding created
daemonset.apps/vault-csi-provider created
At this point you should see your daemonset deploy all the pods across all the nodes:
> k get pods -l app=vault-csi
NAME READY STATUS RESTARTS AGE
vault-csi-provider-5dtd9 1/1 Running 0 92s
vault-csi-provider-cfvb8 1/1 Running 0 92s
vault-csi-provider-sdn7r 1/1 Running 0 92s
Now to install the CSI driver (this is covered in The Secrets Store CSI Driver Book):
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm template secrets-store-csi-driver/secrets-store-csi-driver \
--namespace kube-system
And here are the resources deployed:
> k apply -k base
customresourcedefinition.apiextensions.k8s.io/secretproviderclasses.secrets-store.csi.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io created
serviceaccount/secrets-store-csi-driver created
clusterrole.rbac.authorization.k8s.io/secretproviderclasses-role created
clusterrolebinding.rbac.authorization.k8s.io/secretproviderclasses-rolebinding created
daemonset.apps/secrets-store-csi-driver created
csidriver.storage.k8s.io/secrets-store.csi.k8s.io created
Let’s make sure they are running:
> k get pods -n kube-system -l app=secrets-store-csi-driver
NAME READY STATUS RESTARTS AGE
secrets-store-csi-driver-2rzqq 3/3 Running 0 3m8s
secrets-store-csi-driver-ns42p 3/3 Running 0 3m8s
secrets-store-csi-driver-zdhb8 3/3 Running 0 3m8s
Now that we have the system components deployed, let’s move on to the next steps.
Enable K8S Authentication
Vault has integration with K8S authentication, all we need to do is point to the k8s token and the CA:
> kubectl exec -it vault-0 -- /bin/sh
/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $ vault write auth/kubernetes/config \
> issuer="https://kubernetes.default.svc.cluster.local" \
> token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
> kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
> kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config
Next we need to create a policy to all to read the secret and also assign that policy to a Kubernetes service account, so that a pod running as that service account can read that secret:
/ $ vault policy write internal-app - <<EOF
> path "secret/data/app-pass" {
> capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: internal-app
/ $ vault write auth/kubernetes/role/app \
> bound_service_account_names=app-sa \
> bound_service_account_namespaces=default \
> policies=internal-app \
> ttl=20m
Success! Data written to: auth/kubernetes/role/app
Now we have to create a CSI configuration to link the vault secret to the K8S secret:
cat > spc-vault-app.yaml <<EOF
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: vault-app
spec:
provider: vault
parameters:
vaultAddress: "http://vault.default:8200"
roleName: "app"
objects: |
- objectName: "app-password"
secretPath: "secret/data/app-pass"
secretKey: "password"
EOF
> k apply -f spc-vault-app.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/vault-app created
Now onto the latest steps.
Create a Pod with the Secret Mounted
First let’s create the service account that we allowed read capablity to our vault secret:
> kubectl create serviceaccount app-sa
serviceaccount/app-sa created
Then for the pod manifest which runs as our service account and mounts the secret:
cat > app-pod.yaml <<EOF
kind: Pod
apiVersion: v1
metadata:
name: app
spec:
serviceAccountName: app-sa
containers:
- image: busybox
name: webapp
command: ["/bin/sh"]
args: ["-c", "while true; do sleep 300;done"]
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-app"
EOF
> k apply -f app-pod.yaml
pod/app created
And now for the final test:
> k exec -it app -- cat /mnt/secrets-store/app-password
coolio
Phew, I am glad that worked out.