Using Google Container Registry with Marathon
We recently were tasked with setting up a container management solution on Google Cloud Engine (GCE). After standing up Mesos and Marathon on CoreOS, our initial tests worked fine and we deployed several docker apps just fine to Marathon. We ran into a snag, however, when switching from a public registry to a private Google Container Registry.
Private Registries with Marathon
The process for setting up private registries with Marathon is pretty
so we began to follow the prescribed steps. We generated a gzipped
configuration file, put it in a well-known location on the mesos slave
machines, and added the URI to the files array in the Marathon JSON. Old
hat. The only sticking point involved the first step, which creates a
.docker directory via
docker login. Google prefers you use the
gcloud command-line tool in order to authenticate with the container
registry, so we ran:
This produced the directory we were expecting, which we gzipped and shipped to mesos slave containers running on CoreOS. Unfortunately, Marathon deployments were failing immediately, an indicator of registry connection problem, so it was back to the drawing board. We couldn't find any explicit documentation regarding using Marathon with GCR, but a generic search led us to this article, which mentions using GCR with third-party orchestration tools and directed us to the advanced authentication documentation.
We quickly discovered that the reason our first approach was failing is
gcloud docker login authenticates using a temporary token, which
had expired by the time we started trying to deploy apps on Marathon.
Imagine how perplexing it would have been had we successfully deployed
Marathon tasks before the token expired! Below is a quick walkthrough
of what's outlined in the documentation linked above.
Configuring GCE to play nicely with Marathon
Create a service account
In GCE land, service accounts are special accounts designed to be used when server-to-server communication requires some form of authentication. If you're comfortable with AWS, this would be like setting up an IAM credential with limited access to provide valid credentials to an application. The important detail is that service accounts have long-lived access tokens that don't expire unless you tell them to. This is perfect for authenticating against the private GCR and reusing the resulting config on our Mesos slaves.
- Go to the credentials page and choose your project.
- Click "Add Credentials" and select "Service Account"
- Create a JSON key pair. It should automatically download.
Log in with the service account credentials
The process of authenticating against GCR without using the
CLI is a bit funny looking. Google could care less about the username
you enter, but cares a lot about the password, since it's the whole
JSON config file you had to download in the last step. One way of doing
Windows users, you'll have to do something slightly different, but do not fear: there are PowerShell instructions in the documentation.
Now we have a valid config we can gzip and distribute to our slaves! The rest of the process is exactly how the Marathon docs lays it out, just pick it up from step 1 number 4.
To test, I pushed a custom nginx image to the private registry:loading
and submitted a Marathon application that referenced that image. Don't forget to add the file in the URIs field!loading
Loading it in the browser show that the task starts up just fine and nginx is serving traffic.