Kubernetes, nginx-ingress and Let's Encrypt with cert-manager - Ghost blog on Azure Kubernetes Service

In this post, we're building a simple Ghost blog with HTTPS provided by Let's Encrypt. This will also renew itself. This is initially based on a guide made by Elton Stoneman, but we'll be diving into nginx-ingress, Let's Encrypt and cert-manager as well.

Before starting you'll need a couple of things. I'm doing this on a Windows 10 machine with Git Bash.

Azure CLI
Docker & Kubernetes
Helm (that I installed from Chocolatey) & Tiller

Creating and pushing images to an azure registry is quite simple and I followed this guide. Our azure resource group is called "Blog". This is a basic script to push image to your newly created registry.

#!/bin/bash

docker-compose build

date=`date +%Y%m%d`
docker image tag blog.azurecr.io/blog:ghost "blog.azurecr.io/blog:ghost-${date}"

docker image push "blog.azurecr.io/blog:ghost-${date}"

Connecting Kubernetes to an Azure Container Registry

To make your azure kubernetes cluster to pull images from your azure container registry, you need to add a docker registry to kubernetes image pull secrets.

$ az acr show --name blog --query loginServer --output tsv
blog.azurecr.io

$ az acr show --name blog --query id --output tsv
/subscriptions/subscriptionkey-guid-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/Blog/providers/Microsoft.ContainerRegistry/registries/blog

$ az ad sp create-for-rbac --name acr-service-principal --role acrpull --scopes /subscriptions/subscriptionkey-guid-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/Blog/providers/Microsoft.ContainerRegistry/registries/blog --query password --output tsv
Changing "acr-service-principal" to a valid URI of "http://acr-service-principal", which is the required format used for service principal names
Retrying role assignment creation: 1/36
Retrying role assignment creation: 2/36
Retrying role assignment creation: 3/36
userid-guid-xxxx-xxxx-xxxxxxxxxxxx

$ az ad sp show --id http://acr-service-principal --query appId --output tsv
password-guid-xxxx-xxxx-xxxxxxxxxxxx

$ kubectl create secret docker-registry acr-auth --docker-server blog.azurecr.io --docker-username userid-guid-xxxx-xxxx-xxxxxxxxxxxx --docker-password password-guid-xxxx-xxxx-xxxxxxxxxxxx --docker-email karl@example.com
secret "acr-auth" created

Deploying and exposing the docker image with Azure Kubernetes Service

Now we can create a deployment for the ghost image that you have pushed to our registry.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: blog-ghost
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: blog-ghost
    spec:
      containers:
      - name: words
        image: blog.azurecr.io/blog:ghost-20181229
        imagePullPolicy: Always
        env: 
        - name: url
          value: "https://blog.karlsolgard.net"
        ports:
        - containerPort: 2368
          name: ghost
      imagePullSecrets:
      - name: acr-auth

A few things happening here. We pull the image from our custom registry with the acr-auth secret. env is used to inject environment variables for the ghost-image.
Deploy this using kubectl apply -f blog.yaml. For the sake of simplicity, add these .yaml files in a deployment folder and use kubectl apply -f <folder>/ to deploy multiple files.

Now it's time to add nginx-ingress. We'll get this from Helm. You'll also need Tiller, the server portion of Helm, that typically runs inside of your Kubernetes cluster. But for development, it can also be run locally, and configured to talk to a remote Kubernetes cluster. Azure runs a "Role-Based Access Control" or RBAC for short. This means that you have to create a service account for Tiller. To link Tiller to Azure, run the following commands.

$ helm init --service-account tiller --upgrade

$ kubectl create serviceaccount --namespace kube-system tiller
serviceaccount "tiller" created

$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io "tiller-cluster-rule" created

kube-system is the namespace for objects created by the Kubernetes system. Typically, this would contain pods like kube-dns, kube-proxy, kubernetes-dashboard and stuff like fluentd, heapster, ingresses and so on. Source

In a file called helm-rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

Deploy this and install the ingress to the cluster.

$ helm install --name nginx-ingress --set rbac.create=false --set rbac.createRole=false --set rbac.createClusterRole=false --namespace kube-system --set controller.replicaCount=2 stable/nginx-ingress

Now we can add a deployment for the ingress to expose the site to a public IP-address.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blog-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: blog.karlsolgard.net
    http:
      paths:
      - path: /
        backend:
          serviceName: ghost
          servicePort: 2368

Here we have an ingress set up. The ghost app is now pointed towards the root of the site and listens to the exposed port. If we deploy this we should get an external IP we can use to access the site with.

$ kubectl get service -l app=nginx-ingress --namespace kube-system
NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
nginx-ingress-controller        LoadBalancer   **.**.**.**    **.**.**.**      80:31642/TCP,443:31398/TCP      
nginx-ingress-default-backend   ClusterIP      **.**.**.**    <none>           80/TCP

The site is now deployed and exposed with nginx-ingress!

HTTPS with Let's Encrypt and renewal with cert-manager

Let's Encrypt is a free, automated and open Certificate Authority. And that is pretty cool. Let's add this to our AKS cluster. All will be handled by a little tool called cert-manager. We want to issue our certificates for production. Create two new yml files.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: tls-secret
spec:
  secretName: tls-secret
  dnsNames:
  - blog.karlsolgard.net
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - blog.karlsolgard.net
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: test@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}

These configurations will issue a certificate that nginx-ingress can read and evaluate. Now we need to install cert-manager from helm.

$ helm install --name cert-manager --namespace kube-system --set rbac.create=false --set ingressShim.extraArgs='{--default-issuer-name=letsencrypt-prod,--default-issuer-kind=ClusterIssuer}' --set createCustomResource=false stable/cert-manager

The ingress now needs a couple of new entries to trust the issuer. Change the nginx-ingress file to look like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blog-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - blog.karlsolgard.net
    secretName: tls-secret
  rules:
  - host: blog.karlsolgard.net
    http:
      paths:
      - path: /
        backend:
          serviceName: ghost
          servicePort: 2368

Do one final deploy and your site should have HTTPS. Remember to set your DNS to the ingress public IP-address.

I recommend reading through the following articles for further knowledge on how to set up cert-manager and nginx-ingress:

I also recommend reading the official AKS docs. It's quite well documented.

How to improve: Use and persist mysql database and content

As of right now the sqlite is bundled with the ghost image. I know it's possible to persist databases through volumes and claims in Kubernetes. Ghost also supports mysql. Another problem is uploading images and making the files distributed. This is something I would want to do in the future and might be a subject in a future blog post.