Cert Manager and Letsencrypt

This article will quickly go through how to create wildcard certificates and automatically renew them with Lets Encrypt and Cert-Manager

Cert-Manager

cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.

It can issue certificates from a variety of supported sources, including Let's Encrypt, HashiCorp Vault, and Venafi as well as private PKI.

It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry. link

Install Cert-Manager

I prefer the Helm way so lets add the cert-manager helm chart:

1helm repo add jetstack https://charts.jetstack.io
2helm repo update

Then we need to deploy cert-manager. This can be done out-of-the-box with the commands given from the official docs (this also installed the necessary CRDs):

1helm install \
2  cert-manager jetstack/cert-manager \
3  --namespace cert-manager \
4  --create-namespace \
5  --version v1.9.1 \
6  --set installCRDs=true

Or if you need to customize some settings, as I needed to do, I used this command:

1helm install -f /path/to/cert-manager.values.yaml cert-manager jetstack/cert-manager   --namespace cert-manager   --version v1.9.1   --set installCRDs=true --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=xx.xx.xx.xx:53\,xx.xx.xx:53}'

The above command takes care of the cert-manager installation including the necessary CRDs, but it will also adjust the DNS servers Cert-Manager will use to verify the ownership of my domain.

DNS01 - Wildcard certificate

In this post I will go with wildcard certificate creation. I find it easier to use instead of having a separate cert for everthing I do, as long as they are within the same subdomain. So if I have my services in *.example.com they can use the same certificate. But if I have services in *.int.example.com I can not use the same certificate as LetsEncrypt certificates dont support that. Then you need to create a separate wildcard cert for each subdomain. But Cert-manager will handle that for you very easy.

The offiicial Cert-Manager supported DNS01 providers are:

• ACMEDNS
• Akamai
• AzureDNS
• CloudFlare
• Google
• Route53
• DigitalOcean
• RFC2136

There is also an option to use Webhooks. I did try that as my previous DNS registrar were not on the DNS01 supported list. I did not succeed with using the webhook approach. It could be an issue with the specific webhooks I used or even with my registrar so I decided to migrate over to CloudFlare which is on the supported list, "out of the box".

Issuer - CloudFlare and LetsEncrypt

The first we need to do is to create a secret for Cert-Manager to use when "interacting" with CloudFlare. I went with API Token. So head over to your CloudFlare control panel and create a token for Cert Manager like this:

Here is the permissions:

Now use the tokens to create your secret:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: cloudflare-api-token-secret
5  namespace: cert-manager
6type: Opaque
7stringData:
8  api-token:

Apply it kubect apply -f name.of.yaml

Now create your issuer. LetsEncrypt have two repos, one called staging and one production. Start out with staging until everything works so you dont hit the LetsEncrypt limit. In regards to this I created two issuers, one for staging and one for production. When everything was working and I have verified the certificates etc I deployed the certs using the prod-issuer.

Issuer-staging:

 1apiVersion: cert-manager.io/v1
 2kind: ClusterIssuer
 3metadata:
 4  name: letsencrypt-staging
 5spec:
 6  acme:
 7    # ACME Server
 8    # prod : https://acme-v02.api.letsencrypt.org/directory
 9    # staging : https://acme-staging-v02.api.letsencrypt.org/directory
10    server: https://acme-staging-v02.api.letsencrypt.org/directory
11    # ACME Email address
12    email: xxx.xxx@xxx.xxx
13    privateKeySecretRef:
14      name: letsencrypt-key-staging # staging or production
15    solvers:
16    - dns01:
17        cloudflare:
18          apiTokenSecretRef:
19            name: cloudflare-api-token-secret ## created and applied above
20            key: api-token

Issuer-production:

 1apiVersion: cert-manager.io/v1
 2kind: ClusterIssuer
 3metadata:
 4  name: letsencrypt-prod
 5  namespace: cert-manager
 6spec:
 7  acme:
 8    # ACME Server
 9    # prod : https://acme-v02.api.letsencrypt.org/directory
10    # staging : https://acme-staging-v02.api.letsencrypt.org/directory
11    server: https://acme-v02.api.letsencrypt.org/directory
12    # ACME Email address
13    email: xxx.xxx@xxx.xxx
14    privateKeySecretRef:
15      name: letsencrypt-key-prod # staging or production
16    solvers:
17    - dns01:
18        cloudflare:
19          apiTokenSecretRef:
20            name: cloudflare-api-token-secret # created and applied above
21            key: api-token

Request certificate

Now that the groundwork for cert-manager has been setup, its time to "print" some certificates. Prepare your yamls for both the staging key and production key.

Wildcard-staging:

 1apiVersion: cert-manager.io/v1
 2kind: Certificate
 3metadata:
 4  name: name-tls-test
 5  namespace: namespace-you-want-the-cert-in
 6spec:
 7  secretName: name-tls-staging
 8  issuerRef:
 9    name: letsencrypt-staging
10    kind: ClusterIssuer
11  duration: 2160h # 90d
12  renewBefore: 720h # 30d before SSL will expire, renew it
13  dnsNames:
14    - "*.example.com"

Wildcard-production:

 1apiVersion: cert-manager.io/v1
 2kind: Certificate
 3metadata:
 4  name: name-tls-production
 5  namespace: namespace-you-want-the-cert-in
 6spec:
 7  secretName: name-tls-prod
 8  issuerRef:
 9    name: letsencrypt-prod
10    kind: ClusterIssuer
11  duration: 2160h # 90d
12  renewBefore: 720h # 30d before SSL will expire, renew it
13  dnsNames:
14    - "*.example.com"

Apply the staging request first. Check your certificate status with this command:

1$ kubectl get certificate -n namespace-you-wrote 
2NAME               READY   SECRET                AGE
3name-tls-staging   True    name-tls-staging      8d

Please note that it can take a couple of minutes before the certificate is ready. This applies for production also.

If everything went well, delete your staging certificate and apply your production certificate with the production yaml. Thats it. Now Cert-Manager will take care of updating your certificate for your, sit back and enjoy your applications with your always up to date certificates.

Troubleshooting tips, commands

If something should fail there is a couple of commands you can use to figure out whats going on.

1$ kubectl get issuer
2$ kubectl get clusterissuer
3$ kubectl describe issuer 
4$ kubectl describe clusterissuer
5$ kubectl describe certificaterequest
6$ kubectl describe order
7$ kubectl get challenges
8$ kubectl describe challenges

For more detailed explanation go here