Deploying on Kubernetes #11: Annotations

This is the eleventh in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to…

Apr 13, 2018

This is the eleventh in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve as a tutorial (there are many out there already), but rather to discuss some of the approaches we take.

Assumptions

To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.

Necessary Background

So far we’ve been able:

Extending the Kubernetes API

As awesome as Kubernetes is, there are things that it cannot do. Things like:

Monitoring
Disk Backups
Vulnerability Scanning

as well as 1000 other use cases. There are various ways to extend the Kubernetes API:

Annotations: Adding additional arbitrary key / value pairs to existing Kubernetes objects
Custom Resource Definitions: Adding new Kubernetes API properties.

As cool as Custom Resource Definitions (CRDs) are, we’re going to leave them for another time. However, for a practical application of these take a look at JetStack’s kube-cert-manager project.

Annotations

As mentioned, annotations are arbitrary key / value pairs that are added to the existing Kubernetes definitions. One common use case is that of service discovery for the Prometheus time series database.

The starter chart actually comes with such annotations. They’re already applied to our fleet chart:

$ kubectl get pods kolide-fleet-fleet-6d65997cb6-4wzc7 --output=yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    #
    # These are annotations
    #
    prometheus.io/path: /metrics
    prometheus.io/port: "9102"
    prometheus.io/scheme: http
    prometheus.io/scrape: "false"
  creationTimestamp: 2018-04-10T16:01:40Z
  generateName: kolide-fleet-fleet-6d65997cb6-

They can be appended to the metadata block of any object. Then, other applications can query the API for these objects, and read the key / value pairs for additional information that they require.

As general background, the Prometheus time series database works by making HTTP(S) connections to a service at a supplied endpoint, collecting metrics in a specific format and storing those metrics on a given timestamp in it’s database.

In the case of Prometheus, the annotations are used as follows:

prometheus.io/path: /metrics: Which endpoint to make the HTTP request to
prometheus.io/port: “9102”: On which port to make the HTTP request
prometheus.io/scheme: http: Whether to use HTTP or HTTPS for this request
prometheus.io/scrape: “false”: Whether to enable scraping this resource

Annotations can be used for a wide variety of tasks. At the minute, the starter chart has helpfully included a set of Prometheus annotations; Prometheus being the canonical monitoring server in use at Sitewards. However, given that we are submitting this chart to the public repositories, we want to support annotations in whatever mechanism the user would like to provide.

Abusing annotations for fun and configuration changes

Annotations are quite handy in that they help us solve a problem that we’re facing with configuration. The problem is essentially that, although Kubernetes will update configuration in a container, it does not tell the application the configuration is updated! There are various ways in discussion to solve this, but it’s not natively supported.

Consider the following set of actions:

$ helm upgrade --install stable/kolide-fleet

# Deployment success! Hang on I want to change the fleet jwt key:

$ helm upgrade --install stable/kolide-fleet --set="fleet.jwt.key=1234abcd"

In the case of the above, the JWT key will never be updated! Fleet has successfully loaded the first time, and even though the configuration has changed on disk fleet has no way of knowing that there is new configuration available.

However, we can abuse helm and annotations to trigger an update to our pods every configuration change. As documented in the “Helm Tips and Tricks” article, we can add the following annotation to trigger a deployment with configuration change update:

checksum/config: {{ include (print $.Chart.Name "/templates/secret.yaml") . | sha256sum }}

This works as follows:

Read the file templates/secret
Pass it to the pipe
Hash (sha256) the input

Thus: every change to our secret.yaml file will be an update to this annotation! This, being part of the pod spec, will trigger a rotation of pods.

This looks like:

# templates/deployment.yaml:20-24

    metadata:
      annotations:
        checksum.helm.kubernetes.io/secret: {{ include (print $.Chart.Name "/templates/secret.yaml") . | sha256sum }}
        checksum.helm.kubernetes.io/configmap: {{ include (print $.Chart.Name "/templates/configmap.yaml") . | sha256sum }}

Triggering the update with:

$ helm upgrade --install kolide-fleet .

Triggers the pod rotation. But updating it again does not trigger the rotation. It is only updated as the configuration changes, and a restart is required.

We can see the hashes on our pod object:

$ kubectl describe pod kolide-fleet-fleet-5c8fd4d7d4-wwz7p
Name:           kolide-fleet-fleet-5c8fd4d7d4-wwz7p
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Fri, 13 Apr 2018 18:06:05 +0200
Labels:         app=kolide-fleet-fleet
                pod-template-hash=1749808380
                release=kolide-fleet
Annotations:    checksum.helm.kubernetes.io/configmap=9c6537e04747e9446108943eb174b85e5f150921b87e765695aaded3bf708ad9
                checksum.helm.kubernetes.io/secret=d6879f097839afec1d84160c1d98e9cba079bda2ad663f4881747cf22d26e769

Success! However, there are more uses for annotations just yet.

Delegating (the rest of the) choice

Because the possibilities of annotations are so vague, we do not want to imply what users should implement. Instead, we can simply delegate that option to the user.

Helm, the package management tool we’re using, has a function called toYaml. It does exactly what it implies — takes values, and expresses them as yaml. We can then simply supply an option to the user to add annotation as they wish, and test our own annotations through the same mechanism.

First, let’s add the spec to the Values.yaml file such that it’s clear where the annotations should go. We’ll only be allowing arbitrary annotations to the pod and the service at this stage:

# values.yaml:56-62

pod:
  annotations: {}
    ## Attach arbitrary key → value pairs for the pod definition. For example, 
    # prometheus.io/path: /metrics
    # prometheus.io/port: "8080"
    # prometheus.io/scheme: https
    # prometheus.io/scrape: "true"

# values.yaml:80,104-110
service:
  annotations: {}
    ## Attack arbitrary key → value pairs for the pod definition. For example
    # prometheus.io/probe: true 
    # prometheus.io/path: /healthz
    # prometheus.io/scheme: https
    # prometheus.io/port: 443

Leaving the values commented out makes it easy for reference later, and using the {} character leaves the annotations block defined, but empty.

From here, it’s simply a matter of adding our toYaml filter to the right spot in the deployment:

# templates/deployment.yaml:20-24
    metadata:
      annotations:
{{- if .Values.pod.annotations }}
{{ toYaml .Values.pod.annotations | indent 8 }}
{{ end }}

and our service:

# templates/service.yaml:4-8

metadata:
  annotations:
{{- if .Values.pod.annotations }}
{{ toYaml .Values.service.annotations | indent 4 }}
{{ end }}

The {{ blocks are all at the start of the line so that they are indented correctly.

Then, we can add our annotations to another values.yaml file and include it in our deployment:

$ cat <<EOF > values.annotations.yaml
---
pod:
  annotations:
    k8s.littleman.co/example: "Awesome! Look at our annotation"

service:
  annotations:
    k8s.littleman.co/example: "Awesome! Another annotation"
EOF

Running the upgrade command will trigger the update:

$ helm upgrade --install kolide-fleet --values values.yaml --values values.annotations.yaml .

and we can see our annotation!

$ kubectl describe pod kolide-fleet-fleet-6fbd759d9d-plnzv

Name:           kolide-fleet-fleet-6fbd759d9d-plnzv
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Fri, 13 Apr 2018 18:27:15 +0200
Labels:         app=kolide-fleet-fleet
                pod-template-hash=2968315858
                release=kolide-fleet
Annotations:    checksum.helm.kubernetes.io/configmap=9c6537e04747e9446108943eb174b85e5f150921b87e765695aaded3bf708ad9
                checksum.helm.kubernetes.io/secret=d6879f097839afec1d84160c1d98e9cba079bda2ad663f4881747cf22d26e769
                k8s.littleman.co/example=Awesome! Look at our annotation
Status:         Running

In Conclusion

Annotations are a super handy mechanism to extend the default properties of Kubernetes. Many applications are configured to query objects based on their annotations, such as Prometheus, various backup tools etc.

As usual, you can see the changes here:

AD-HOC feat (Annotations); Introduce customisable annotations · andrewhowdencom/charts@7768556
This commit introduces the ability to customise annotations on both the service and the deployment specification. This…github.com

Only a couple more to go! We should cover resource limits and tidying up for public submission. ❤ hasn’t it been fun!

The next draft is:

__NOWHERE_I_HAVENT_WRITTEN_IT_YET__

Simple, Beautiful Software Development

Discussion about this post