This spring, there was a some movement on the Illumos/SmartOS front in implementing features to better support running LX zones with Linux variants. Since Docker images (generally) run on Linux underpinnings, support for running Docker images on SmartOS are dependent upon this support working correctly.

For those familiar with Triton, you know that Triton can run Docker directly as part of its standard configuration. But, for those of us who don't run Triton, but do run SmartOS, there are some steps that can be taken to use Docker images under SmartOS.

To provide a concrete example, I'm going to use GitLab as the example in this article.

Docker on SmartOS, the harder way

I'm completely stealing that line (and adapting some of the content) from a 2016 blog post by Jasper Lievisse Adriaanse of the same name.

His summary of using Docker on SmartOS was the best resource that I found for creating virtual machines using vmadm.

Getting docker images on SmartOS

Docker Hub is "the world's largest library... for container images" and as such is basically where you want to go to get your docker images.

To get access to Docker Hub, you need to add the source to imgadm:

# imgadm sources --add-docker-hub

As noted in the post, imgadm avail doesn't work against Docker Hub, so you'll need to search there manually or get it directly from another source. Once you know what docker image you need, you can add it using imgadm import.

# imgadm import gitlab/gitlab-ee:latest

As is common with docker, some of the items in the image descriptor can be left off, most notably :latest can be omitted and the latest tag will be used by default.

Once you have the images loaded, you can see them weeded out from the rest of your images by using:

# imgadm list --docker
UUID                                  REPOSITORY              TAG     IMAGE_ID      CREATED
a001c571-b91a-a5b0-c251-0514a0d4a174  gitlab/gitlab-ce        latest  sha256:42486  2021-06-07T19:28:36Z
9080e799-d964-782e-e369-87d339e50798  gitlab/gitlab-ee        latest  sha256:1f383  2021-06-07T19:36:13Z

Reasoning through the requirements

One disadvantage of using SmartOS natively for docker in comparison to using docker on Linux is that there isn't a docker control daemon to set things up for you. As such, you'll need to dig in a bit to the requirements in order to make sure you have all of the right sittings to get up and running.

You'll need to take a look at the docker parameters. Some of these parameters are baked in to the images during the build phase and others are usually shown in command line arguments in the instructions to run the code. As is frequently the case, there are some of each to pay attention to in gitlab.

The instructions for running gitlab in docker (as of 2021-06-11) call for the following docker command line:

sudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 22:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  gitlab/gitlab-ee:latest

Let's look at the key parameters here:

In addition to the parameters on the command line, there are also parameters inherent in the docker container that we need to propagate to the SmartOS JSON.

We can see the key information in the JSON that comes with the docker images by using

# imgadm info <uuid>

which will output the json for the image.

The key section to look for is the tags section, which in this version of gitlab-ee contains:

"tags": {
  "docker:repo": "gitlab/gitlab-ee",
  "docker:id": "sha256:1f38337b3401d2536562e4323999233b665aa41a2e6ef2c7509a0b938e53d94d",
  "docker:architecture": "amd64",
  "docker:tag:latest": true,
  "docker:config": {
    "Cmd": [
      "/assets/wrapper"
    ],
    "Entrypoint": null,
    "Env": [
      "PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "LANG=C.UTF-8",
      "TERM=xterm"
    ],
    "WorkingDir": ""
  }

Again, we'll look at the key parameters of the docker:config object:

It's important to note that the "dockery" configuration elements are all strings, so you need to appropriately quote them to get them in the internal_metadata portion of your json for vmadm.

Setting up the storage

Your specific mileage may vary. In many cases, images may run only with ephemeral storage, in which case, you have nothing to do for filesystems, but in this example case, we have three specific mount points: /etc/gitlab, /var/log/gitlab, and /var/opt/gitlab. In our SmartOS systems, we use a separate data pool (usuallly spinning rust, in contrast to the zones pool, which is all SSD), so you'll see that in the example. Further, we have a naming standard for volumes that requires the FQDN and then the mount point.

Once you've figured out what zfs zones you need, create them using zfs create. I'll leave that as an exercise for the reader.

Constructing the vmadm json

As is frequently the case (I may address doing this in ansible at a later date, but so far these are all bespoke), you want to have a json file containing the parameters of the new zone, so that you can pass them along using vmadm create -f x.json.

Now, we need to put together what we know from the information we've gathered so far:

1. Start with a template LX zone
2. Make sure brand is lx and docker is true
3. Set the docker image UUID in image_uuid
4. Set up your network as required (see the gitlab note below for an understanding of why there are two interfaces)
5. Configure your file systems based on the required mount points
6. Put the operative docker information in the internal_metadata section
7. You will need to put an owner_uuid in the json because it is theoretically required by the firewall code.

{
	"alias": "gitlab.example.com",
	"hostname": "gitlab.example.com",
	"image_uuid": "9080e799-d964-782e-e369-87d339e50798",
	"owner_uuid": "f834f98a-cac8-11eb-8ca3-cbddba9a698b",
	"nics": [
		{
		 "nic_tag": "vlan",
		 "ips": [
			"X.X.X.X/24"
		 ],
		 "gateways": ["X.X.X.1"],
		 "vlan_id": 100
		},
		{
		 "nic_tag": "vlan",
		 "ips": [
			"Y.Y.Y.Y/24"
		 ],
		 "vlan_id": 200
		}

	],
       "filesystems": [
          {
	      "source": "/data/gitlab.example.com/config",
	      "target": "/etc/gitlab",
	      "type": "lofs"
	    },
	  {
	      "source": "/data/gitlab.example.com/logs",
	      "target": "/var/log/gitlab",
	      "type": "lofs"
	  },
	  {
	      "source": "/data/gitlab.example.com/data",
	      "target": "/var/opt/gitlab",
	      "type": "lofs"
	  }
	  ],

	"brand": "lx",
	"docker": true,
	"kernel_version": "5.4.0",
	"max_physical_memory": 8192,
	"maintain_resolvers": true,
	"resolvers": [
		"8.8.8.8"
	],
	"quota":100,
	"internal_metadata": {
		"docker:cmd": "[\"/assets/wrapper\"]",
		"docker:env" :
		"[ \"PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:⏎
			/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",⏎
		 	\"LANG=C.UTF-8\",⏎
		 	\"TERM=xterm\" ]"
	}
}

NOTE: I've split the docker:env line above for readibility. You'll need to keep that as a single line for it to work correctly. Remember that all of the docker items are strings, so they can't be split up. I've marked the returns for readibilty with ⏎ above.

Once this is prepared, you should be able to bring the zone up with

# vmadm create -f myzone.json

Networking or Protecting your container from the internet

One important thing to note about docker containers is that they all assume they're not accessible from the internet. Under normal circumstances they're running on a loopback or maybe an internal network, but certainly not on the internet. Depending on how your systems are set up, this may be an issue, as internal services are not protected using any normal process. You may get lucky and the folks who build the Docker container may have used loopback for everything not going off-container, but don't count on it.

This protection is appropriate for SmartOS built-in firewall system, which is controlled by fwadm from the global zone. Generally speaking, you want to firewall everything except the ports that were specifically mentioned in the publish argument to the docker command.

This is where the owner_id comes in. It turns out this is necessary to run the firewall in the global zone. No big deal, and you can add it later using vmadm set before you enable the firewall if you forgot to do so prior to pulling the system up.

1. Start the firewall for your container

   # fwadm start 85effadf-f4d3-63ca-cec4-8549b8797f75
   

2. Enable access to your ports

   fwadm add -e --desc 'allow docker ports' -O 5a2e68b0-ca8d-11eb-944f-8b6840c190dc⏎
    "FROM any ⏎
   	TO vm 85effadf-f4d3-63ca-cec4-8549b8797f75 ⏎
   	ALLOW tcp (PORT 80 AND PORT 443 AND PORT 22)"
   

   (again, I'm using the counter-intuitive ⏎ to mean you should not put a return there)

   Should be self-explanatory, but the value after vm is the current zone's UUID, The value after -O is the owner UUID.

Debugging

In the end, the vms here are just LX zones running on SmartOS, so you can still access their filesystems and you can execute commands on them using zlogin (if you're careful).

• Look in /zones/${UUID}/logs/stdio.log for stdio of the docker environment
• Start a shell using zlogin -i ${UUID} /native/usr/vm/sbin/dockerexec /bin/sh
• You can run an arbitrary command in the container using zlogin -i ${UUID} /native/usr/vm/sbin/dockerexec, the /bin/sh is just a specifically useful example

Additional hints

Here are some practical hints that I have developed by getting some of these images (and others) running in SmartOS:

• The gitlab binaries appear to really want a private interface in order to look for other nodes in a potential cluster. As such, I found I needed to add a second, private network interface so that it didn't get confused. This was made clear by error messages thrown by the startup code.

• Some Linux calls still don't work exactly the same in LX zones. This seems to be particularly The case with process and system information gathering. In the case of gitlab, the CE version ran with few changes to the configuration; but the EE version required reducing the worker count to "0" (actually 1 worker, but that's the semaphore). This is another place to look when debugging LX zones in general, and docker images in particular.

Additional links

• Run Docker images on SmartOS
• dockerinit source