Snooping on container traffic in docker-compose

A debugging tip to assist in the container sharp stick pokery

Jan 18, 2019

A debugging tip to assist in the container sharp stick pokery

Docker compose is great. It allows you to take a bunch of random programs and mash them all together reliably into a distributed, networked application and then poke parts of that application with a stick. It’s particularly useful for developers of web based applications of some kind, where there’s usually:

A data store (MySQL or something similar)
A cache
Block storage
Other

That the application is expected to access in production. However, one of the less than excellent parts of docker-compose is it’s essentially magical network setup. Sometimes, you want to see what’s happening between those two network bits.

The good news is, you can! It’s just not so obvious.

Necessary Background

Containers aren’t so magical, and the network side of them is not so magical either. Each container in a docker-compose network gets its own IP, and each docker-compose network gets its own subnet. The host has access to this subnet to allow the magic that port-forwards docker containers.

In docker-compose, the host network acts as the “gateway” for all traffic — it all passes through the host ${SOMEWHERE}. The tricky part is just understanding where.

Poking the container with sticks

Given the containers:

$ docker ps

# Many details are omitted for Mediums short code windows
NAMES
docker-compose_keycloak.local_1
docker-compose_mysql.keycloak.local_1
docker-compose_api.faq.local_1

We need to fetch an IP

$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-compose_keycloak.local_1
172.20.0.5

The gateway is traditionally at xxx.xxx.xxx.1, a pattern that also applies with Docker. So, we can look up the interface on our machine assigned an ip of 172.20.0.1 :

$ ip addr | grep 172.20.0.1
    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-faaa09172475

The br-faaa09172475 at the end of the line is the interface on our host that our containers talk through. So, we can just dump all traffic from that!

$ sudo tcpdump --interface=br-faaa09172475

... ${CRAP} ...

.M.@.@................b.aZ.M.....b<.....
.gR.E\.WHTTP/1.1 200 OK
Connection: keep-alive
Cache-Control: no-cache, must-revalidate, no-transform, no-store
Content-Type: application/json
Content-Length: 2335
Date: Fri, 18 Jan 2019 15:36:23 GMT

... ${MORE CRAP} ...

Nice! Here we see a GET request from golang's OIDC libary by CoreOS. But it doesn’t matter — look at MySQL, or HTTP, or Redis if you’re feeling masochistic.

Major Caveat

The following instructions only work on Linux directly. You can also probably use them on Mac or Windows, but you’ll need to access the VM that Docker is running in, and then find a way to run tcpdump