Portable Services Walkthrough (Go Edition)
A few months ago I posted a blog story with a walkthrough of systemd
example service given was written in C, and the image was built with
mkosi. In this blog story I'd
like to revisit the exercise, but this time focus on a different
aspect: modern programming languages like Go and Rust push users a lot
more towards static linking of libraries than the usual dynamic
linking preferred by C (at least in the way C is used by traditional
Static linking means we can greatly simplify image building: if we don't have to link against shared libraries during runtime we don't have to include them in the portable service image. And that means pretty much all need for building an image from a Linux distribution of some kind goes away as we'll have next to no dependencies that would require us to rely on a distribution package manager or distribution packages. In fact, as it turns out, we only need as few as three files in the portable service image to be fully functional.
So, let's have a closer look how such an image can be put together. All of the following is available in this git repository.
A Simple Go Service
Let's start with a simple Go service, an HTTP service that simply counts how often a page from it is requested. Here are the sources: main.go — note that I am not a seasoned Go programmer, hence please be gracious.
The service implements systemd's socket activation protocol, and thus
can receive bound TCP listener sockets from systemd, using the
$LISTEN_FDS environment variables.
The service will store the counter data in the directory indicated in
$STATE_DIRECTORY environment variable, which happens to be an
environment variable current systemd versions set based on the
setting in service files.
Two Simple Unit Files
When a service shall be managed by systemd a unit file is
required. Since the service we are putting together shall be socket
activatable, we even have two:
(the description of the service binary itself) and
(the description of the sockets to listen on for the service).
These units are not particularly remarkable: the
primarily contains the command line to invoke and a
setting to make sure the service when invoked gets its own private
state directory under
/var/lib/ (and the
environment variable is set to the resulting path). The
simply lists 8080 as TCP/IP port to listen on.
An OS Description File
OS images (and that includes portable service images) generally should
file. Usually, that is provided by the distribution. Since we are
building an image without any distribution let's write our own
version of such a
on we can use the
portablectl inspect command to have a look at this
metadata of our image.
Putting it All Together
The four files described above are already every file we need to build
our image. Let's now put the portable service image together. For that
I've written a
contains two relevant rules: the first one builds the static binary
from the Go program sources. The second one then puts together a
squashfs file system combining the following:
- The compiled, statically linked service binary
- The two systemd unit files
- A couple of empty directories such as
/dev/and so on that need to be over-mounted with the respective kernel API file system. We need to create them as empty directories here since Linux insists on directories to exist in order to over-mount them, and since the image we are building is going to be an immutable read-only image (
squashfs) these directories cannot be created dynamically when the portable image is mounted.
- Two empty files
/etc/machine-idthat can be over-mounted with the same files from the host.
And that's already it. After a quick
make we'll have our portable
portable-walkthrough-go.raw and are ready to go.
Trying it out
Let's now attach the portable service image to our host system:
# portablectl attach ./portable-walkthrough-go.raw (Matching unit files with prefix 'portable-walkthrough-go'.) Created directory /etc/systemd/system.attached. Created directory /etc/systemd/system.attached/portable-walkthrough-go.socket.d. Written /etc/systemd/system.attached/portable-walkthrough-go.socket.d/20-portable.conf. Copied /etc/systemd/system.attached/portable-walkthrough-go.socket. Created directory /etc/systemd/system.attached/portable-walkthrough-go.service.d. Written /etc/systemd/system.attached/portable-walkthrough-go.service.d/20-portable.conf. Created symlink /etc/systemd/system.attached/portable-walkthrough-go.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf. Copied /etc/systemd/system.attached/portable-walkthrough-go.service. Created symlink /etc/portables/portable-walkthrough-go.raw → /home/lennart/projects/portable-walkthrough-go/portable-walkthrough-go.raw.
The portable service image is now attached to the host, which means we can now go and start it (or even enable it):
# systemctl start portable-walkthrough-go.socket
Let's see if our little web service works, by doing an HTTP request on port 8080:
# curl localhost:8080 Hello! You are visitor #1!
Let's try this again, to check if it counts correctly:
# curl localhost:8080 Hello! You are visitor #2!
Nice! It worked. Let's now stop the service again, and detach the image again:
# systemctl stop portable-walkthrough-go.service portable-walkthrough-go.socket # portablectl detach portable-walkthrough-go Removed /etc/systemd/system.attached/portable-walkthrough-go.service. Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d/10-profile.conf. Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d/20-portable.conf. Removed /etc/systemd/system.attached/portable-walkthrough-go.service.d. Removed /etc/systemd/system.attached/portable-walkthrough-go.socket. Removed /etc/systemd/system.attached/portable-walkthrough-go.socket.d/20-portable.conf. Removed /etc/systemd/system.attached/portable-walkthrough-go.socket.d. Removed /etc/portables/portable-walkthrough-go.raw. Removed /etc/systemd/system.attached.
And there we go, the portable image file is detached from the host again.
A Couple of Notes
Of course, this is a simplistic example: in real life services will be more than one compiled file, even when statically linked. But you get the idea, and it's very easy to extend the example above to include any additional, auxiliary files in the portable service image.
The service is very nicely sandboxed during runtime: while it runs as regular service on the host (and you thus can watch its logs or do resource management on it like you would do for all other systemd services), it runs in a very restricted environment under a dynamically assigned UID that ceases to exist when the service is stopped again.
Originally I wanted to make the service not only socket activatable but also implement exit-on-idle, i.e. add a logic so that the service terminates on its own when there's no ongoing HTTP connection for a while. I couldn't figure out how to do this race-freely in Go though, but I am sure an interested reader might want to add that? By combining socket activation with exit-on-idle we can turn this project into an excercise of putting together an extremely resource-friendly and robust service architecture: the service is started only when needed and terminates when no longer needed. This would allow to pack services at a much higher density even on systems with few resources.
While the basic concepts of portable services have been around since systemd 239, it's best to try the above with systemd 241 or newer since the portable service logic received a number of fixes since then.
A low-level document introducing Portable Services is shipped along with systemd.
Please have a look at the blog story from a few months ago that did something very similar with a service written in C.