Discover more from Simple, Beautiful Software Development
Deploying on Kubernetes #4: The deployment object
This is the fourth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to…
This is the fourth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve as a tutorial (there are many out there already), but rather to discuss some of the approaches we take.
Assumptions
To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.
Necessary Background
So far we’ve been able:
Next, we need to create a deployment object to actually release the software onto the cluster!
Managing Deployments
Deployment (noun)
the action of bringing resources into effective action.
— “the rapid deployment of high-speed cable Internet services to consumers”
Software deployment turns out to be a reasonably tricky problem, particularly for custom developed software. We must:
Create a new version of the software
Create a clean upgrade path between the versions
Deploy that version of the software somewhere in parallel or on top of the existing software
(if required) delete the old version of the software
Repeat
This process can repeat many times a day, and is always inherently risky. It’s possible that:
The application didn’t build correctly
The application is a critical bug that wasn’t caught during QA
The application is incompatible with something else stored in the production environment
The application is having a bad day
There are now several primitives to handle this process, but (obviously) the one I prefer is Kubernetes deploying Containers
Containers
Part of the solution to making software deployment a reliable, repeatable process is solved by “containers”. From the Docker website:
A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
Phrase another way, Docker is everything that you need packaged up and stored in something like a tarball for later running. It solves some of the above problems, such as:
The application didn’t build correctly (it can be tested)
The application is incompatible with something else stored in the production environment (it’s packaged in the container — it is compatible by design)
This heavily mitigates some of the risks of managing production software. But still, there are parts left missing. Those parts are solved by Kubernetes
Kubernetes
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
Broadly, Kubernetes is software that you install across a bunch of machines to treat it as one giant logical machine. This is incredibly freeing, as it solves the other hard problems of software management such as:
Managing the transition between software versions
Scaling the software deployment
Handling the failure of the software, hardware, network or other components between
Kubernetes creates an extremely reliable, resilient service by combining a number of machines with containers, and a logical system to control them.
The Deployment Object
From our earlier posts, we know that we have deployed MySQL and Redis onto a Kubernetes cluster. But we haven’t yet deployed an actual instance of kolide/fleet . That’s the next step! Hooray!
The deployment object is super large. So, we’ll first be examining it whole, then evaluating it section by section. Let’s have a look at the Redis deployment. First, we see what deployments are in the cluster:
$ kubectl get deploymentsNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kolide-fleet-mysql 1 1 1 1 22h
kolide-fleet-redis 1 1 1 1 22hThen, we get the deployment we are interested with out:
$ kubectl get deployment kolide-fleet-redis --output=yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2018-03-27T16:25:08Z
generation: 1
labels:
app: kolide-fleet-redis
chart: redis-1.1.21
heritage: Tiller
release: kolide-fleet
name: kolide-fleet-redis
namespace: default
resourceVersion: "889"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/kolide-fleet-redis
uid: 69f8140c-31db-11e8-81e0-080027c1d0f5
spec:
replicas: 1
selector:
matchLabels:
app: kolide-fleet-redis
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: kolide-fleet-redis
spec:
containers:
- env:
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
key: redis-password
name: kolide-fleet-redis
- name: REDIS_DISABLE_COMMANDS
value: FLUSHDB,FLUSHALL
image: bitnami/redis:4.0.9-r0
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- redis-cli
- ping
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: kolide-fleet-redis
ports:
- containerPort: 6379
name: redis
protocol: TCP
readinessProbe:
exec:
command:
- redis-cli
- ping
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /bitnami
name: redis-data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
runAsUser: 1001
terminationGracePeriodSeconds: 30
volumes:
- name: redis-data
persistentVolumeClaim:
claimName: kolide-fleet-redis
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2018-03-27T16:25:08Z
lastUpdateTime: 2018-03-27T16:25:08Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1Wow. There’s a tonne of stuff going on there. We’re going to approach all of it, but one step at a time. However, it’s worth knowing at this point that there are two types of information there:
Information we added. For example:
image: bitnami/redis:4.0.9-r0
imagePullPolicy: IfNotPresentInformation that was added by the kubernetes admin. For example:
- lastTransitionTime: 2018-03-27T16:25:08Z
lastUpdateTime: 2018-03-27T16:25:08ZThey use the same object to record their data. How cool is that! It means that we can see the current status of our deployment in the format that we used to create the deployment. Anyway, let’s start creating our own deployment object.
My own personal deployment
The deployment specification I will be starting with is the one from the starter chart. I’m not going to paste it in it’s entirety as it’s .. rather large, but instead approach it in sections. You can check out the full spec on GitHub. If you know about each section — skip it.
Metadata
An example of this section is below:
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
labels:
app: {{ template "fleet.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
name: {{ template "fleet.fullname" . }}Let’s dive in a little deeper
apiVersion: "apps/v1"Kubernetes provides many different primitives to work with — about 25 at least count, and growing. Some of these primitives are newer than others, and some might only be used in some circumstances. Additionally, administrators may which to disable certain APIs or enable certain others. Third parties might wish to extend the API.
The apiVersion node allows us to specify which part of the Kuberntes API that we want to use. The apps/v1 API is used for the management of applications.
kind: "Deployment"Within the apiVersion there are certain kinds of objects. The kind is what dictates what Kubernetes will do with the particular configuration, as well as the kind of specification that it needs to met. All Kinds are documented in the OpenAPI specification of the Kubernetes API.
metadata:
labels:
app: {{ template "fleet.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
name: {{ template "fleet.fullname" . }}The metadata node contains two objects of note:
Labels
Name
The name is fairly self explanatory — it’s the primary name for that object. It’s the reference used when you create, update or delete that object.
However, labels are more interesting. A label is how an object is selected. The most trivial example of this is using the kubectl tool to select all objects from a particular release:
$ kubectl get all --selector app=kolide-fleet-mysql
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kolide-fleet-mysql 1 1 1 1 23hNAME DESIRED CURRENT READY AGE
rs/kolide-fleet-mysql-6c859797b4 1 1 1 23hNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kolide-fleet-mysql 1 1 1 1 23hNAME DESIRED CURRENT READY AGE
rs/kolide-fleet-mysql-6c859797b4 1 1 1 23hNAME READY STATUS RESTARTS AGE
po/kolide-fleet-mysql-6c859797b4-gf6lk 1/1 Running 1 23hNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kolide-fleet-mysql ClusterIP 10.104.173.216 <none> 3306/TCP 23hOur earlier realease has labeled a whole series of objects with app: kolide-fleet-mysql . So we can easily query them with the command above.
Additionally, this is our first view of :
"{{ .Chart.Name }}-{{ .Chart.Version }}"This is what gives helm it’s power. We will cover it later with replicas — one thing at a time!
(Deployment) Spec
The spec node is the meat of the deployment. It is where the behaviour of the object is defined.
There is little more to be said of it, save that it simply the “god” property for a bunch of sub-properties:
spec:
replicas: {{ default 2 .Values.deployment.replicas }}
selector:
matchLabels:
# ... lots of other properties we'll approach shortlyReplicas
spec:
#
# Spec illustrated for orientation
#
replicas: {{ default 2 .Values.deployment.replicas }}Replicas is fairly straight forward. How many of a particular container do you want to run? 5? 10? 200?
Kubernetes can scale to extremely large deployments. Well written applications can also, though at that scale there may be some logic required to ensure that the routing is efficient and that routing trees do not get overloaded (perhaps segment your application geographically? anyways)
However, as mentioned earlier, we also have the helm {{}} syntax at play now. This syntax comes from the golang text template engine and is designed to make rendering text templates easier. The functions are either defined there, or in the sprig template library.
The values come from a file in the helm chart called Values.yml. In our starter chart, it comes pre-populated with a whole stack of variables — including the ones relevant here:
deployment:
replicas: 1The dot notation is the object path in the Values.yml file. So, in it’s entirety the above expression is:
Use the default value 2
Unless there is a value set in the Values.yml file. Then use that
We will not be extensively covering all properties — just a general guide as we go on. There is already excellent documentation on this topic, and I do not wish to repeat it.
Selector
From the Kubernetes docs:
A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
Within our deployment it looks like:
selector:
matchLabels:
app: {{ template "fleet.fullname" . }}
release: "{{ .Release.Name }}"What’s important to understand is that with a deployment, we are creating a resource that creates further resources. A “deployment” object does nothing by itself — it’s a specification for creating further specifications.
As we saw earlier, Kubernetes records both the current state of the object and the object definition together, in one object. This, combined, means that the above is required.
A deployment will create a resource called a Replica Set, which will in turn create resources called Pods. The Replica Set will create a make sure that there are the ${REPLICAS} number of pods running that match the above selector.
So, it’s how the deployment knows which pods it owns.
Strategy
The strategy is what happens in the case of updates to the deployment specification. It looks like this:
strategy:
type: "RollingUpdate"
rollingUpdate:
maxSurge: 1
maxUnavailable: 1There are two types of update strategies:
Rolling Update, in which if there are many replicas of a container running they will be replaced incrementally
Recreate, in which all existing pods are killed before a new release is started.
In the case of kolide/fleet, we do not know if there are database transitions or other stateful management between releases. We do not want to get into a situation in which some pods have some config, but others do not and thus corrupt state — thus, the safest option is Recreate.
AD-HOC refactor (Deployment): Change update strategy to recreate · andrewhowdencom/charts@d05eedd
Though the kolide application does not store any state directly attached to it's canonical service (instead…github.com
Template
template:
metadata:
# ... metadataa
spec:
As mentioned earlier, the deployment is an object that manages the creation or destruction of other objects. The template object is as it sounds — a template for the pods that will be created.
In and of itself, the template object is not super interesting.
(Pod) Metadata
template:
metadata:
labels:
app: {{ template "fleet.fullname" . }}
release: "{{ .Release.Name }}"
annotations:
# ..The pod metadata follows exactly the same role as the deployment metadata, with one notable exception: annotations!
(Pod) Spec + Containers
The pod specification the primary definition for how run containers on Kubernetes. A minimal spec looks like:
spec:
containers:
- name: fleet
image: {{ .Values.pod.fleet.image | quote }}
ports:
- containerPort: 8080
protocol: "TCP"
name: "http"A pod consists of {$N} containers, all of whom share the same loopback interface (so they can communicate across localhost). In this case, there is only the one container — kolide/fleet.
Much of the values are self explanatory:
nameis the name of the container,imageis the name of the (usually Docker) container that will be usedportsis the ports that expect to be open
The ports are given a name for easy reference in other parts of the deployment spec, such as liveness probes. However, this is a topic for another day!
Restart Policy
restartPolicy: "Always"Occasionally, software will fail for miscellaneous reasons. This configuration determines what happens in that scenario — will it be restarted, or will it simply be left alone?
A sane default in this case is Always. It is how init daemons traditionally behave with managed software.
AD-HOC feat (deployment): Fill out the RestartPolicy object with "Alw... ·…
ays" There are various reasons that software will fail in a production environment: - OOM - SegFault - Application…github.com
In conclusion
The deployment object is suuper complex, but it is also among the most important specifications to become accustomed to as we’re developing applications for Kubernetes.
Additionally, astute follers will note that I have not covered everything here — notable things missing were:
Annotations
Liveness Probes
Resource Usage
Security Context
Etc. Some of these are still needed to be “production worthy” and some are not needed at all in this case.
Future posts will approach these topics separately — for now, we have a deployable unit of software!
Find the next post in this series here:
https://medium.com/@andrewhowdencom/deploying-on-kubernetes-4-application-configuration-ce6dd7cdb750