I wanted to configure Cert Manager to automatically renew a Wild Card SSL Certificate and I also wanted to get notified when the renewal occurred. I ran into a couple of options for the notifications component:

  1. k8s-notify from Redhat
    1. This looked really good it just didn’t support plain web hooks, but other than really cool project
  2. eventrouter from heptio/VMware
    1. This also looked cool, but it didn’t look like it could filter out events it would just forward everything over to a sink (which definitely has it’s use case, check out How Grafana Labs Effectively Pairs Loki and Kubernetes Events)
  3. kubernetes-event-exporter from opsgenie
    1. This was actually my second choice, but I just really like the ease of use of the other ones
  4. Kubernetes Event Exporter from caicloud
    1. This was actually an interesting project, where it would create metrics for prometheus to scrape from the kubernetes events, which I thought was an interesting thought
  5. Kubewatch from bitnami
    1. This looked great, but it seemed like it was supporting specific type of events like pods changes and deployments, but wasn’t generic enough to support any resource change.
  6. BotKube from infracloudio
    1. For some reason this one looked cool, so I decided to try it out.

I settled on using BotKube since it seemed the most flexible. Since I was playing around with tekton, I also decided to create a Pipeline to automatically send a message to slack when the certificate is updated. So let’s get into the configuration one by one.

Cert Manager

Installation

The Cert Manager components are documented very well. From that same doc, here is a nice illustration of what it supports:

cert-man-componts.png

The installation is covered in Installing with regular manifests, so I just ran the following:

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml

And after the install was finished I had the following CRDs:

> kubectl get crd | grep -i cert-manager
certificaterequests.cert-manager.io                   2020-05-12T01:58:25Z
certificates.cert-manager.io                          2020-05-12T01:58:25Z
challenges.acme.cert-manager.io                       2020-05-12T01:58:25Z
clusterissuers.cert-manager.io                        2020-05-12T01:58:25Z
issuers.cert-manager.io                               2020-05-12T01:58:25Z
orders.acme.cert-manager.io                           2020-05-12T01:58:25Z

And the following pods running:

> k get pods -n cert-manager 
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7cb75cf6b4-gxjp6              1/1     Running   0          4d5h
cert-manager-cainjector-759496659c-vgq5k   1/1     Running   0          4d5h
cert-manager-webhook-7c75b89bf6-8gpvn      1/1     Running   0          5d1h

Let’s move to the next steps.

Creating a ClusterIssuer

Next you can define an Issuer, I was using Let’s Encrypt in the past so I decided to keep that. Here are the instructions on configuring an ACME Issuer. I ended up creating the following configuration:

> cat cluster-issuer.yaml 
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
#    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: "YOUR_EMAIL"
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: issuer-account-key
    solvers:
    - dns01:
        cloudflare:
          email: "YOUR_EMAIL"
          apiKeySecretRef:
            name: cloudflare-api-key-secret
            key: api-key

This defines which server to use and also which solver.

DNS01 Solver

I knew I wanted to get a Wild Card SSL Certificate and I decided to use the dns01 challenge to accomplish that. I also decided to use cloudflare for my DNS provider and the configuration for that is covered in cloudflare. I created a private key which contained my API key:

> cat cf-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-key: YOU_CLOUDFLARE_KEY

Then creating the secret and issuer worked out:

> k apply -f cf-secret.yaml
> k apply -f cluster-issuer.yaml

And I saw my clusterissuer created:

> k get clusterissuers                                       
NAME               READY   AGE
letsencrypt-prod   True    5d1h

And ready to process new certificates:

> k get clusterissuers letsencrypt-prod -o json | jq .status
{
  "acme": {
    "lastRegisteredEmail": "YOUR_EMAIL",
    "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/85910270"
  },
  "conditions": [
    {
      "lastTransitionTime": "2020-05-12T02:23:08Z",
      "message": "The ACME account was registered with the ACME server",
      "reason": "ACMEAccountRegistered",
      "status": "True",
      "type": "Ready"
    }
  ]
}

Now let’s move on to the next steps.

Creating a Certificate

Now that we have the clusterIssuer and we defined our solver let’s request a Certificate. I ended up creating the following config:

> cat cert.yaml 
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: wild-YOUR_DOMAIN
#  namespace: cert-manager
spec:
  secretName: wild-YOUR_DOMAIN
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "YOUR_DOMAIN"
    - "*.YOUR_DOMAIN"

Then applying that configuration will automatically trigger a request process and you will see a challenge created:

> k get challenges                      
NAME                                                STATE     DOMAIN    AGE
wild-YOUR_DOMAIN-1876214257-1906930536-3362969436   pending   YOUR_DOMAIN  6s
wild-YOUR_DOMAIN-1876214257-1906930536-3864605829             YOUR_DOMAIN   6s

And you can describe the challenge to see how far along it is in the process:

> k describe challenge wild-YOUR_DOMAIN-1876214257-1906930536-3362969436 | tail 
Status:
  Presented:   false
  Processing:  true
  Reason:      Waiting for DNS Record
  State:       pending
Events:
  Type     Reason        Age                 From          Message
  ----     ------        ----                ----          -------
  Normal   Started       10m                 cert-manager  Challenge scheduled for processing

And you will also see orders in a pending state:

> k get orders
NAME                                     STATE     AGE
wild-YOUR_DOMAIN-1876214257-1906930536   pending   6m33s

And if you check out the events you will see the following:

> k get events --sort-by='.metadata.creationTimestamp' -A -w
default     68s         Normal    GeneratedKey        certificate/wild-YOUR_DOMAIN                                                  Generated a new private key
default     68s         Normal    OrderCreated        certificaterequest/wild-YOUR_DOMAIN-1876214257                                Created Order resource default/wild-YOUR_DOMAIN-1876214257-1906930536
default     68s         Normal    Requested           certificate/wild-YOUR_DOMAIN                                                  Created new CertificateRequest resource "wild-YOUR_DOMAIN-1876214257"
default     67s         Normal    Created             order/wild-YOUR_DOMAIN-1876214257-1906930536                                  Created Challenge resource "wild-YOUR_DOMAIN-1876214257-1906930536-3864605829" for domain "YOUR_DOMAIN"
default     67s         Normal    Created             order/wild-YOUR_DOMAIN-1876214257-1906930536                                  Created Challenge resource "wild-YOUR_DOMAIN-1876214257-1906930536-3362969436" for domain "YOUR_DOMAIN"
default     66s         Normal    Started             challenge/wild-YOUR_DOMAIN-1876214257-1906930536-3362969436                   Challenge scheduled for processing

After it’s done the challenges will be gone, and the order will become valid:

> k get challenges
No resources found in default namespace.
> k get orders
NAME                                     STATE     AGE
wild-YOUR_DOMAIN-1392782545-2299396756   valid     4d5h

If you are interested you can also dig up the original CertificateRequest:

> k get CertificateRequests        
NAME                          READY   AGE
wild-YOUR_DOMAIN-1392782545   True    4d5h

And at this point you will see the secret that you specified with the TLS certificate:

> k get secret wild-YOUR_DOMAIN -o json | jq .data
{
  "ca.crt": "",
  "tls.crt": "LS0t...",
  "tls.key": "LS0..."
}

Super Cool. BTW there is also a super nice guide that covers the steps really well here: Installing cert-manager on Kubenetes with CloudFlare DNS - Update.

Botkube

I decided to try out BotKube, from their documentation here is the architecture:

botkube-arch.png

It’s pretty nifty, it basically monitors all the events from the kube-apiserver and is then able to send notifications based on the criteria that you define.

BotKube Install

The install process is covered in BotKube > Installation > Slack. First install the App in slack:

botkube-app-in-slack.png

Then invite the @BotKube to your channel:

add-botkube-slack.png

Then download the manifest file:

> wget -q https://raw.githubusercontent.com/infracloudio/botkube/v0.10.0/deploy-all-in-one.yaml

You can apply that to install for now:

> k apply -f deploy-all-in-one.yaml

Now let’s configure it

BotKube Configuration

First let’s configure our slack settings:

> cat secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: botkube-communication-secret
  namespace: botkube
  labels:
    app: botkube
type: Opaque
stringData:
  comm_config.yaml: |
    # Communication settings
    communications:
      # Settings for Slack
      slack:
        enabled: true
        channel: 'general'
        token: 'YOUR_TOKEN'
        notiftype: short

And also what to monitor:

> cat cm.yaml 
# Configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: botkube-configmap
  namespace: botkube
  labels:
    app: botkube
data:
  resource_config.yaml: |
    ## Resources you want to watch
    resources:
      - name: secret
        namespaces:
          include:
            - default
        events:
          - all
      - name: deployment
        namespaces:
          include:
            - all
        events:
          - create
          - update
          - delete
          - error
        updateSetting:
          includeDiff: true
          fields:
            - spec.template.spec.containers[*].image
            - status.availableReplicas
    recommendations: true

    # Setting to support multiple clusters
    settings:
      clustername: k8s
      allowkubectl: true
      restrictAccess: false
      #kubectl:
      #  enabled: true
      #  defaultNamespace: default
      #  restrictAccess: false
      configwatcher: true
      upgradeNotifier: true

I decided to keep an eye on the deployments as well, but for the next section we only need to monitor secrets. After you apply both:

k apply -f secret.yaml cm.yaml

You should see the pod running:

> k -n botkube get po            
NAME                       READY   STATUS    RESTARTS   AGE
botkube-655f987c58-wwkn7   1/1     Running   1          17m

Using BotKube

After the install and configurations are finished, you will be able to get the status of the Bot in slack:

botkube-ping-slack.png

If you enabled the kubectl options you can also run kubectl commands:

botkube-slack-kctl.png

If you update the configmap, you will also see the following notification:

botkube-slack-deploy-update.png

And this will confirm the deployment section is working.

Tekton

Let’s break this down into a couple of sections. As I kept playing around with tekton I ran into this pretty awesome flow diagram from their documentation (CI with Tekton), it kind helped me put all the pieces together:

tekton-ci-flow-diag.png

Here is the directory structure that I had:

> tree
.
├── pipeline
│   ├── cert-pipeline.yaml
│   └── condition.yaml
├── tasks
│   ├── cert-task.yaml
│   └── slack-secret.yaml
└── trigger
    ├── eventlistener.yaml
    ├── secret-test.yaml
    ├── triggerbinding.yaml
    └── triggertemplate.yaml

Let’s get into these one by one.

Event Listener

First let’s create an event listener, to which botkube can post to:

> cat eventlistener.yaml
---
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: bot-listener
spec:
  triggers:
    - name: bot-trigger
      interceptors:
        - cel:
            filter: "body.meta.kind == 'Secret'"
            overlays:
            - key: extensions.secret_name
              expression: "body.meta.name"
      bindings:
        - name: bot-pipeline-binding
      template:
        name: bot-pipeline-template

I did end up using a CEL Interceptor since I wanted to ignore the deployment updates. We will see the full body of the message from botkube to find out how to generate the filter. Next here is the triggerbinding:

> cat triggerbinding.yaml 
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
  name: bot-pipeline-binding
spec:
  params:
  - name: body
    value: $(body)
  - name: secret_name
    value: $(body.extensions.secret_name)

And lastly here is the triggertemplate:

> cat triggertemplate.yaml 
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: botkube-pipeline-template
spec:
  params:
  - name: body
    description: "Body of BotKube Post (For testing)"
    default: "Test body"
  - name: secret_name
    description: "Secret Name Changed"
    default: "Me"
  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      generateName: cert-pr-
    spec:
      pipelineRef:
        name: cert-pipeline
      params:
        - name: secret_name
          value: $(params.secret_name)
      workspaces:
      - name: shared-workspace
        emptyDir: {}

This will kick off the cert-pipeline with a pipelinerun.

Tekton Task

Here is task I created:

> cat tasks/cert-task.yaml 
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: cert-task
spec:
  workspaces:
    - name: shared
      description: shared workspace
  params:
  - name: secret_name
    description: "secret name of the tls cert"
    type: string
  steps:
  - name: get-pem-files
    image: gcr.io/cloud-builders/kubectl
    script: |
      #!/bin/bash
      SECRET_NAME="$(params.secret_name)"
      KUBECTL="/builder/google-cloud-sdk/bin/kubectl"
      SHARED_DIR="$(workspaces.shared.path)"
      CERT_BASE="${SHARED_DIR}/${SECRET_NAME}"
      CERT_FILE="${CERT_BASE}.crt"
      KEY_FILE="${CERT_BASE}.key"
      BASE64="/usr/bin/base64"
      echo "Getting Certs from Secrets"
      # run command
      ${KUBECTL} get secret -n default ${SECRET_NAME} -o jsonpath="{.data['tls\.crt']}" | ${BASE64} -d > ${CERT_FILE}
      ${KUBECTL} get secret -n default ${SECRET_NAME} -o jsonpath="{.data['tls\.key']}" | ${BASE64} -d > ${KEY_FILE}

  - name: send-mesg
    image: ellerbrock/alpine-bash-curl-ssl
    script: |
      #!/bin/bash
      SECRET_NAME="$(params.secret_name)"
      SHARED_DIR="$(workspaces.shared.path)"
      CERT_BASE="${SHARED_DIR}/${SECRET_NAME}"
      CERT_FILE="${CERT_BASE}.crt"
      KEY_FILE="${CERT_BASE}.key"
      SUBJECT="Let's Encrypt Cert Updated"
      SLACK_MSG="${SHARED_DIR}/msg"
      CURL="/usr/bin/curl"
      OPENSSL="/usr/bin/openssl"
      SED="/bin/sed"
      # Create new slack message
      # Add cert info to the message
      ${OPENSSL} x509 -in ${CERT_FILE} -noout -issuer > ${SLACK_MSG}
      ${OPENSSL} x509 -in ${CERT_FILE} -noout -subject >> ${SLACK_MSG}
      ${OPENSSL} x509 -in ${CERT_FILE} -noout -dates >> ${SLACK_MSG}
      # prepare the mesg for slack
      while IFS= read -r line; do
        body="$body$line\n"
      done < ${SLACK_MSG}
      escapedText=$(echo $body | ${SED} 's/"/\"/g' | ${SED} "s/'/\'/g")
      esSubject=$(echo $SUBJECT | ${SED} 's/"/\"/g' | ${SED} "s/'/\'/g")

      # create JSON payload
      json="{\"blocks\": [{\"type\": \"section\", \"text\": { \"type\": \"mrkdwn\", \"text\": \"*${esSubject}*\n$escapedText\"}}]}"

      # fire off slack message post
      ${CURL} -s -d "payload=$json" "${SLACK_URL}"

    env:
    - name: SLACK_URL
      valueFrom:
        secretKeyRef:
          name: webhook-secret
          key: url

I specified a workspace to keep track of where that data is shared between the steps. And, here the secret for the slack URL:

> cat slack-secret.yaml 
kind: Secret
apiVersion: v1
metadata:
  name: webhook-secret
stringData:
  url: YOUR_SLACK_URL

Next we can define a condition for our pipeline, which makes sure we only start our pipeline, only if the cert secret changes:

> cat pipeline/condition.yaml 
apiVersion: tekton.dev/v1alpha1
kind: Condition
metadata:
  name: check-cert-secret
spec:
  params:
    - name: secret_name
      type: string
    - name: domain_name
      type: string
  check:
    image: ubuntu
    script: |
      #!/bin/bash
      DOMAIN="$(params.domain_name)"
      SECRET_NAME="wild-${DOMAIN//\./-}"
      # troubleshooting
      if [[ $(params.secret_name) =~ ${SECRET_NAME} ]]; then
        echo "The $(params.secret_name) secret changed, proceeding"
        exit 0
      else
        echo "The $(params.secret_name) secret changed but we are waiting for ${SECRET_NAME} secret, quitting"
        exit 1
      fi

And finally here is the pipeline:

> cat pipeline/cert-pipeline.yaml 
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: cert-pipeline
spec:
  workspaces:
    - name: shared-workspace
      description: shared space between tasks
  params:
    - name: secret_name
      description: "The Secret that changed"
    - name: domain_name
      description: "Domain Name we are monitoring"
      default: "test.domain"
  tasks:
    - name: print-cert-info
      conditions:
      - conditionRef: check-cert-secret
        params:
          - name: secret_name
            value: "$(params.secret_name)"
          - name: domain_name
            value: "$(params.domain_name)"
      taskRef:
        name: cert-task
      workspaces:
        - name: shared
          workspace: shared-workspace
      params:
        - name: secret_name
          value: "$(params.secret_name)"

You can probably store the domain in a secret but for my testing I left that out.

Test out the Pipeline

I created a test secret to test with, which just had a CA cert. Then I created the cert:

> k apply -f trigger/secret-test.yaml

Next let’s get the eventlistener:

> k get svc -l eventlistener=botkube-listener
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
el-botkube-listener   ClusterIP   10.96.158.176   <none>        8080/TCP   76m

Now let’s post a sample payload:

> curl -X POST -d '{"meta": {"kind": "Secret", "name": "wild-test-domain"}}' 10.96.158.176:8080
{"eventListener":"botkube-listener","namespace":"default","eventID":"hvcdf"}

If all is well, we should see the pipelinerun finish:

> tkn pr list
NAME                 STARTED         DURATION    STATUS
cert-pr-hktxd        1 minute ago    1 minute    Succeeded
> tkn pr logs cert-pr-hktxd
[print-cert-info : get-pem-files] Getting Certs from Secrets

[print-cert-info : send-mesg] ok

And if you check out the taskruns you should see those complete as well. There will be two, one for the task and one for the condition check:

> tkn tr list
NAME                                                              STARTED         DURATION    STATUS
cert-pr-hktxd-print-cert-info-z5j78                               2 minutes ago   1 minute    Succeeded
cert-pr-hktxd-print-cert-info-z5j78-check-cert-secret-0-djvzn     2 minutes ago   9 seconds   Succeeded
> tkn tr logs cert-pr-hktxd-print-cert-info-z5j78-check-cert-secret-0-djvzn
[condition-check-check-cert-secret] The wild-test-domain secret changed, proceeding

And if you check out your slack channel, you should see something like this:

slack-webhook-updated-cert.png

I am glad to see all the components come together. I just added the following section to the BotKube to enable it to send POSTs to the Tekton Event listener:

> cat secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: botkube-communication-secret
  namespace: botkube
  labels:
    app: botkube
type: Opaque
stringData:
  comm_config.yaml: |
    # Communication settings
    communications:
      # Settings for Slack
      slack:
        enabled: true
        channel: 'general'
        token: 'YOUR_SLACK_TOKEN'
        notiftype: short
      # Settings for Webhook
      webhook:
        enabled: true
        url: 'http://el-botkube-listener.default.svc.cluster.local:8080'