Deploying on Kubernetes #4: The deployment object

This is the fourth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to…

This is the fourth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve as a tutorial (there are many out there already), but rather to discuss some of the approaches we take.

Assumptions

To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.

Necessary Background

So far we’ve been able:

1. Define Requirements

2. Create the helm chart to manage the resources

3. Add the MySQL and Redis dependencies

Next, we need to create a deployment object to actually release the software onto the cluster!

Managing Deployments

Deployment (noun)

the action of bringing resources into effective action.
 — “the rapid deployment of high-speed cable Internet services to consumers”

Software deployment turns out to be a reasonably tricky problem, particularly for custom developed software. We must:

1. Create a new version of the software

2. Create a clean upgrade path between the versions

3. Deploy that version of the software somewhere in parallel or on top of the existing software

4. (if required) delete the old version of the software

5. Repeat

This process can repeat many times a day, and is always inherently risky. It’s possible that:

• The application didn’t build correctly

• The application is a critical bug that wasn’t caught during QA

• The application is incompatible with something else stored in the production environment

• The application is having a bad day

There are now several primitives to handle this process, but (obviously) the one I prefer is Kubernetes deploying Containers

Containers

Part of the solution to making software deployment a reliable, repeatable process is solved by “containers”. From the Docker website:

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.

Phrase another way, Docker is everything that you need packaged up and stored in something like a tarball for later running. It solves some of the above problems, such as:

• The application didn’t build correctly (it can be tested)

• The application is incompatible with something else stored in the production environment (it’s packaged in the container — it is compatible by design)

This heavily mitigates some of the risks of managing production software. But still, there are parts left missing. Those parts are solved by Kubernetes

Kubernetes

From the Kubernetes website:

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

Broadly, Kubernetes is software that you install across a bunch of machines to treat it as one giant logical machine. This is incredibly freeing, as it solves the other hard problems of software management such as:

• Managing the transition between software versions

• Scaling the software deployment

• Handling the failure of the software, hardware, network or other components between

Kubernetes creates an extremely reliable, resilient service by combining a number of machines with containers, and a logical system to control them.

The Deployment Object

From our earlier posts, we know that we have deployed MySQL and Redis onto a Kubernetes cluster. But we haven’t yet deployed an actual instance of kolide/fleet . That’s the next step! Hooray!

The deployment object is super large. So, we’ll first be examining it whole, then evaluating it section by section. Let’s have a look at the Redis deployment. First, we see what deployments are in the cluster:

$ kubectl get deployments

NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kolide-fleet-mysql   1         1         1            1           22h
kolide-fleet-redis   1         1         1            1           22h

Then, we get the deployment we are interested with out:

$ kubectl get deployment kolide-fleet-redis --output=yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-03-27T16:25:08Z
  generation: 1
  labels:
    app: kolide-fleet-redis
    chart: redis-1.1.21
    heritage: Tiller
    release: kolide-fleet
  name: kolide-fleet-redis
  namespace: default
  resourceVersion: "889"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/kolide-fleet-redis
  uid: 69f8140c-31db-11e8-81e0-080027c1d0f5
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kolide-fleet-redis
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kolide-fleet-redis
    spec:
      containers:
      - env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              key: redis-password
              name: kolide-fleet-redis
        - name: REDIS_DISABLE_COMMANDS
          value: FLUSHDB,FLUSHALL
        image: bitnami/redis:4.0.9-r0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - redis-cli
            - ping
          failureThreshold: 3
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kolide-fleet-redis
        ports:
        - containerPort: 6379
          name: redis
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - redis-cli
            - ping
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami
          name: redis-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      terminationGracePeriodSeconds: 30
      volumes:
      - name: redis-data
        persistentVolumeClaim:
          claimName: kolide-fleet-redis
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-03-27T16:25:08Z
    lastUpdateTime: 2018-03-27T16:25:08Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Wow. There’s a tonne of stuff going on there. We’re going to approach all of it, but one step at a time. However, it’s worth knowing at this point that there are two types of information there:

• Information we added. For example:

image: bitnami/redis:4.0.9-r0
        imagePullPolicy: IfNotPresent

• Information that was added by the kubernetes admin. For example:

- lastTransitionTime: 2018-03-27T16:25:08Z
    lastUpdateTime: 2018-03-27T16:25:08Z

They use the same object to record their data. How cool is that! It means that we can see the current status of our deployment in the format that we used to create the deployment. Anyway, let’s start creating our own deployment object.

My own personal deployment

The deployment specification I will be starting with is the one from the starter chart. I’m not going to paste it in it’s entirety as it’s .. rather large, but instead approach it in sections. You can check out the full spec on GitHub. If you know about each section — skip it.

Metadata

An example of this section is below:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  labels:
    app: {{ template "fleet.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    heritage: "{{ .Release.Service }}"
    release: "{{ .Release.Name }}"
  name: {{ template "fleet.fullname" . }}

Let’s dive in a little deeper

apiVersion: "apps/v1"

Kubernetes provides many different primitives to work with — about 25 at least count, and growing. Some of these primitives are newer than others, and some might only be used in some circumstances. Additionally, administrators may which to disable certain APIs or enable certain others. Third parties might wish to extend the API.

The apiVersion node allows us to specify which part of the Kuberntes API that we want to use. The apps/v1 API is used for the management of applications.

kind: "Deployment"

Within the apiVersion there are certain kinds of objects. The kind is what dictates what Kubernetes will do with the particular configuration, as well as the kind of specification that it needs to met. All Kinds are documented in the OpenAPI specification of the Kubernetes API.

metadata:
  labels:
    app: {{ template "fleet.fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    heritage: "{{ .Release.Service }}"
    release: "{{ .Release.Name }}"
  name: {{ template "fleet.fullname" . }}

The metadata node contains two objects of note:

• Labels

• Name

The name is fairly self explanatory — it’s the primary name for that object. It’s the reference used when you create, update or delete that object.

However, labels are more interesting. A label is how an object is selected. The most trivial example of this is using the kubectl tool to select all objects from a particular release:

$ kubectl get all --selector app=kolide-fleet-mysql
NAME                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/kolide-fleet-mysql   1         1         1            1           23h

NAME                               DESIRED   CURRENT   READY     AGE
rs/kolide-fleet-mysql-6c859797b4   1         1         1         23h

NAME                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/kolide-fleet-mysql   1         1         1            1           23h

NAME                               DESIRED   CURRENT   READY     AGE
rs/kolide-fleet-mysql-6c859797b4   1         1         1         23h

NAME                                     READY     STATUS    RESTARTS   AGE
po/kolide-fleet-mysql-6c859797b4-gf6lk   1/1       Running   1          23h

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
svc/kolide-fleet-mysql   ClusterIP   10.104.173.216   <none>        3306/TCP   23h

Our earlier realease has labeled a whole series of objects with app: kolide-fleet-mysql . So we can easily query them with the command above.

Additionally, this is our first view of :

"{{ .Chart.Name }}-{{ .Chart.Version }}"

This is what gives helm it’s power. We will cover it later with replicas — one thing at a time!

(Deployment) Spec

The spec node is the meat of the deployment. It is where the behaviour of the object is defined.

There is little more to be said of it, save that it simply the “god” property for a bunch of sub-properties:

spec:
  replicas: {{ default 2 .Values.deployment.replicas }}
  selector:
    matchLabels:
    # ... lots of other properties we'll approach shortly

Replicas

spec:
  #
  # Spec illustrated for orientation
  #
  replicas: {{ default 2 .Values.deployment.replicas }}

Replicas is fairly straight forward. How many of a particular container do you want to run? 5? 10? 200?

Kubernetes can scale to extremely large deployments. Well written applications can also, though at that scale there may be some logic required to ensure that the routing is efficient and that routing trees do not get overloaded (perhaps segment your application geographically? anyways)

However, as mentioned earlier, we also have the helm {{}} syntax at play now. This syntax comes from the golang text template engine and is designed to make rendering text templates easier. The functions are either defined there, or in the sprig template library.

The values come from a file in the helm chart called Values.yml. In our starter chart, it comes pre-populated with a whole stack of variables — including the ones relevant here:

deployment:
  replicas: 1

The dot notation is the object path in the Values.yml file. So, in it’s entirety the above expression is:

1. Use the default value 2

2. Unless there is a value set in the Values.yml file. Then use that

We will not be extensively covering all properties — just a general guide as we go on. There is already excellent documentation on this topic, and I do not wish to repeat it.

Selector

From the Kubernetes docs:

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.

Within our deployment it looks like:

selector:
    matchLabels:
      app: {{ template "fleet.fullname" . }}
      release: "{{ .Release.Name }}"

What’s important to understand is that with a deployment, we are creating a resource that creates further resources. A “deployment” object does nothing by itself — it’s a specification for creating further specifications.

As we saw earlier, Kubernetes records both the current state of the object and the object definition together, in one object. This, combined, means that the above is required.

A deployment will create a resource called a Replica Set, which will in turn create resources called Pods. The Replica Set will create a make sure that there are the ${REPLICAS} number of pods running that match the above selector.

So, it’s how the deployment knows which pods it owns.

Strategy

The strategy is what happens in the case of updates to the deployment specification. It looks like this:

strategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1

There are two types of update strategies:

1. Rolling Update, in which if there are many replicas of a container running they will be replaced incrementally

2. Recreate, in which all existing pods are killed before a new release is started.

In the case of kolide/fleet, we do not know if there are database transitions or other stateful management between releases. We do not want to get into a situation in which some pods have some config, but others do not and thus corrupt state — thus, the safest option is Recreate.

AD-HOC refactor (Deployment): Change update strategy to recreate · andrewhowdencom/charts@d05eedd
Though the kolide application does not store any state directly attached to it's canonical service (instead…github.com

Template

template:
    metadata:
       # ... metadataa
    spec:

As mentioned earlier, the deployment is an object that manages the creation or destruction of other objects. The template object is as it sounds — a template for the pods that will be created.

In and of itself, the template object is not super interesting.

(Pod) Metadata

template:
    metadata:
      labels:
        app: {{ template "fleet.fullname" . }}
        release: "{{ .Release.Name }}"
      annotations:
       # ..

The pod metadata follows exactly the same role as the deployment metadata, with one notable exception: annotations!

(Pod) Spec + Containers

The pod specification the primary definition for how run containers on Kubernetes. A minimal spec looks like:

spec:
     containers:
        - name: fleet
          image: {{ .Values.pod.fleet.image | quote }}
          ports:
            - containerPort: 8080
              protocol: "TCP"
              name: "http"

A pod consists of {$N} containers, all of whom share the same loopback interface (so they can communicate across localhost). In this case, there is only the one container — kolide/fleet.

Much of the values are self explanatory:

• name is the name of the container,

• image is the name of the (usually Docker) container that will be used

• ports is the ports that expect to be open

The ports are given a name for easy reference in other parts of the deployment spec, such as liveness probes. However, this is a topic for another day!

Restart Policy

restartPolicy: "Always"

Occasionally, software will fail for miscellaneous reasons. This configuration determines what happens in that scenario — will it be restarted, or will it simply be left alone?

A sane default in this case is Always. It is how init daemons traditionally behave with managed software.

AD-HOC feat (deployment): Fill out the RestartPolicy object with "Alw... ·…
ays" There are various reasons that software will fail in a production environment: - OOM - SegFault - Application…github.com

In conclusion

The deployment object is suuper complex, but it is also among the most important specifications to become accustomed to as we’re developing applications for Kubernetes.

Additionally, astute follers will note that I have not covered everything here — notable things missing were:

• Annotations

• Liveness Probes

• Resource Usage

• Security Context

Etc. Some of these are still needed to be “production worthy” and some are not needed at all in this case.

Future posts will approach these topics separately — for now, we have a deployable unit of software!

Find the next post in this series here:

https://medium.com/@andrewhowdencom/deploying-on-kubernetes-4-application-configuration-ce6dd7cdb750