Using Google Container Registry with Marathon

We recently were tasked with setting up a container management solution on Google Cloud Engine (GCE). After standing up Mesos and Marathon on CoreOS, our initial tests worked fine and we deployed several docker apps just fine to Marathon. We ran into a snag, however, when switching from a public registry to a private Google Container Registry.

Private Registries with Marathon

The process for setting up private registries with Marathon is pretty straightforward and well-documented, so we began to follow the prescribed steps. We generated a gzipped configuration file, put it in a well-known location on the mesos slave machines, and added the URI to the files array in the Marathon JSON. Old hat. The only sticking point involved the first step, which creates a .docker directory via docker login. Google prefers you use the gcloud command-line tool in order to authenticate with the container registry, so we ran:

This produced the directory we were expecting, which we gzipped and shipped to mesos slave containers running on CoreOS. Unfortunately, Marathon deployments were failing immediately, an indicator of registry connection problem, so it was back to the drawing board. We couldn't find any explicit documentation regarding using Marathon with GCR, but a generic search led us to this article, which mentions using GCR with third-party orchestration tools and directed us to the advanced authentication documentation.

We quickly discovered that the reason our first approach was failing is that gcloud docker login authenticates using a temporary token, which had expired by the time we started trying to deploy apps on Marathon. Imagine how perplexing it would have been had we successfully deployed Marathon tasks before the token expired! Below is a quick walkthrough of what's outlined in the documentation linked above.

Configuring GCE to play nicely with Marathon

Create a service account

In GCE land, service accounts are special accounts designed to be used when server-to-server communication requires some form of authentication. If you're comfortable with AWS, this would be like setting up an IAM credential with limited access to provide valid credentials to an application. The important detail is that service accounts have long-lived access tokens that don't expire unless you tell them to. This is perfect for authenticating against the private GCR and reusing the resulting config on our Mesos slaves.

Go to the credentials page and choose your project.
Click "Add Credentials" and select "Service Account"
Create a JSON key pair. It should automatically download.

Log in with the service account credentials

The process of authenticating against GCR without using the gcloud CLI is a bit funny looking. Google could care less about the username you enter, but cares a lot about the password, since it's the whole JSON config file you had to download in the last step. One way of doing this is:

Windows users, you'll have to do something slightly different, but do not fear: there are PowerShell instructions in the documentation.

Now we have a valid config we can gzip and distribute to our slaves! The rest of the process is exactly how the Marathon docs lays it out, just pick it up from step 1 number 4.

Testing

To test, I pushed a custom nginx image to the private registry:

and submitted a Marathon application that referenced that image. Don't forget to add the file in the URIs field!

Loading it in the browser show that the task starts up just fine and nginx is serving traffic.