Deploying on Kubernetes #6: Application Secrets
This is the sixth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve…
This is the sixth in a series of blog posts that hope to detail the journey deploying a service on Kubernetes. It’s purpose is not to serve as a tutorial (there are many out there already), but rather to discuss some of the approaches we take.
Assumptions
To read this it’s expected that you’re familiar with Docker, and have perhaps played with building docker containers. Additionally, some experience with docker-compose is perhaps useful, though not immediately related.
Necessary Background
So far we’ve been able:
Secrets
There is some information that is inherently sensitive. When managing authentication applications must be able to uniquely prove that they’re authorised to access a given resource (username/password), or otherwise prove their identity (PKI). Should that information be compromised, the guarantees inherent in providing identity to a limited set of users are lost.
Kubernetes provides a primitive that is designed especially for secret management. It is very similar to ConfigMap as created previously, exception:
It’s a different object, so access can be limited via Role Based Access Control (RBAC)
The information is encoded in base64, so odd secret types (for example, pgp keys) can be embedded easily
It is possible to encrypt them such that only applications with the required identity can read them.
The secrets we need
In the previous post, we added a large amount of configuration to the fleet binary. However, we deliberately omitted secret information:
# templates/configmap.yaml:14-18
14 mysql:
address: {{ default "kolide-fleet-mysql:3306" .Values.fleet.mysql.address }}
username: {{ default "kolide" .Values.fleet.mysql.username }}
# Handled as a secret in an environment variable
# password: kolide
This information should be handled as “safely” as possible. The configuration (represented as dot notation) is:
mysql.password
redis.password
auth.jwt_key
Reusing existing secrets
Both MySQL and Redis already have provision for managing secrets. Indeed, they automatically generate secrets when the applications are loaded. Unfortunately, it is a little difficult to reach those secrets. Additionally, there are problems with those secrets as time goes on — with each release, they are regenerated. This might be fine if something takes care of updating those secrets, but in doing preliminary testing this does not appear to be the case.
However, we can store these secrets in something like thevalues.yml
file where they will reused consistently.
Stubbing Configuration
First, we need to stub the values for these secrets. Earlier, we added redis
and mysql
as dependencies to this chart. We are able to influence the configuration in those charts in our chart, simply by adding the variables to our charts values.yml
and prefixing it with the appropriate package name:
# values.yml:6-12
mysql:
# This is a required value.
mysqlPassword: ""
redis:
# This is a required value.
redisPassword: ""
The passwords are currently empty. That’s fine — we don’t want to add the same passwords to the default chart, which we will later submit to kubernetes/charts
for all to use. We’ll come to that shortly.
We now need to reference this configuration in our secret object:
# templates/secret.yml:1-11
---
apiVersion: "v1"
kind: "Secret"
metadata:
labels:
app: {{ template "fleet.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
name: {{ template "fleet.fullname" . }}
data:
We add the configuration as follows:
# templates/secret.yml:11-14
data:
fleet.mysql.username: {{ required "A valid MySQL Username is required" .Values.mysql.mysqlUser | b64enc }}
fleet.mysql.password: {{ required "A valid MySQL Password is required" .Values.mysql.mysqlPassword | b64enc }}
fleet.auth.jwt_key: {{ required "A valid JWT key is required" .Values.fleet.auth.jwt_key | b64enc }}
There’s a few things to note there:
All the secrets are wrapped in a
required
function. It allows us to denote to the user that this will automatically handled — please take care of it.All secrets are base64 encoded, as denoted by the specification
Perfect! Now, we need to actually supply some secrets. Luckily, helm lets us do this with multiple values.yml files:
$ cat <<EOF > values.secret.yaml
---
mysql:
mysqlUser: "kolide"
mysqlPassword: $(pwgen 32 1)
redis:
redisPassword: $(pwgen 32 1)
fleet:
auth:
jwt_key: $(pwgen 32 1)
EOF
(Presuming you have the pwgen
utility installed) This will create a file with the appropriate passwords. How you handle this is up to you.
Once the file is created, we can tell helm
to use both the normal values.yml
file and the additional values.secret.yml
file as source values for this chart:
$ helm upgrade --install kolide-fleet --values values.yaml --values values.secret.yaml .
Awesome! Our values are being used… well, sortof. They’re in the configuration, but not being consumed yet by the application.
Consuming Configuration
Kolide/Fleet
allows specifying configuration via environment variables. We’ll take advantage of that to inject our secret information.
We need to edit the deployment object:
# templates/deployment.yml:50-68
containers:
- name: fleet
env:
- name: "KOLIDE_AUTH_JWT_KEY"
valueFrom:
secretKeyRef:
name: {{ template "fleet.fullname" . }}
key: "fleet.auth.jwt_key"
- name: "KOLIDE_MYSQL_USERNAME"
valueFrom:
secretKeyRef:
name: {{ template "fleet.fullname" . }}
key: "fleet.mysql.username"
- name: "KOLIDE_MYSQL_PASSWORD"
valueFrom:
secretKeyRef:
name: {{ template "fleet.fullname" . }}
key: "fleet.mysql.password"
image: {{ .Values.pod.fleet.image | quote }}
The configuration above injects the environment variables into the fleet container, which should be consumed by the fleet application.
In summary
That’s it! If we check the fleet logs, we can see it’s successfully able to connect to MySQL:
$ kubectl logs kolide-fleet-fleet-7c5f4999d7-9j6bt
Using config file: /etc/fleet/config.yml
################################################################################
# ERROR:
# Your Fleet database is not initialized. Fleet cannot start up.
#
# Run `fleet prepare db` to initialize the database.
################################################################################
Hmm. Looks like it’s not setup. Let’s cover that in the next chapter:
https://medium.com/@andrewhowdencom/deploying-on-kubernetes-7-application-installation-f9ce876c51a4