For those living under a rock, the videos from everybody's favourite Userspace Linux Conference All Systems Go! 2017 are now available online.
The videos for my own two talks are available here:
Synchronizing Images with casync (Slides)
Containers without a Container Manager, with systemd (Slides)
Of course, this is the stellar work of the CCC VOC folks, who are hard to beat when it comes to videotaping of community conferences.
The GNOME.Asia Summit 2017 organizers invited to me to speak at their conference in Chongqing/China, and it was an excellent event! Here's my brief report:
Because we arrived one day early in Chongqing, my GNOME friends Sri, Matthias, Jonathan, David and I started our journey with an excursion to the Dazu Rock Carvings, a short bus trip from Chongqing, and an excellent (and sometimes quite surprising) sight. I mean, where else can you see a buddha with 1000+ hands, and centuries old, holding a cell Nexus 5 cell phone? Here's proof:
The GNOME.Asia schedule was excellent, with various good talks, including some about Flatpak, Endless OS, rpm-ostree, Blockchains and more. My own talk was about The Path to a Fully Protected GNOME Desktop OS Image (Slides available here). In the hallway track I did my best to advocate casync to whoever was willing to listen, and I think enough were ;-). As we all know attending conferences is at least as much about the hallway track as about the talks, and GNOME.Asia was a fantastic way to meet the Chinese GNOME and Open Source communities.
The day after the conference the organizers of GNOME.Asia organized a Chongqing day trip. A particular highlight was the ubiqutious hot pot, sometimes with the local speciality: fresh pig brain.
Here some random photos from the trip: sights, food, social event and more.
I'd like to thank the GNOME Foundation for funding my trip to GNOME.Asia. And that's all for now. But let me close with an old chinese wisdom:
The Trials Of A Long Journey Always Feeling, Civilized Travel Pass Reputation.
TL;DR: systemd now can do per-service IP traffic accounting, as well as access control for IP address ranges.
Last Friday we released systemd 235. I already blogged about its Dynamic User feature in detail, but there's one more piece of new functionality that I think deserves special attention: IP accounting and access control.
Before v235 systemd already provided per-unit resource management hooks for a number of different kinds of resources: consumed CPU time, disk I/O, memory usage and number of tasks. With v235 another kind of resource can be controlled per-unit with systemd: network traffic (specifically IP).
Three new unit file settings have been added in this context:
IPAccounting= is a boolean setting. If enabled for a unit, all IP
traffic sent and received by processes associated with it is counted
both in terms of bytes and of packets.
IPAddressDeny= takes an IP address prefix (that means: an IP
address with a network mask). All traffic from and to this address will be
prohibited for processes of the service.
IPAddressAllow= is the matching positive counterpart to
IPAddressDeny=. All traffic matching this IP address/network mask
combination will be allowed, even if otherwise listed in
IPAddressDeny=.
The three options are thin wrappers around kernel functionality
introduced with Linux 4.11: the control group eBPF hooks. The actual
work is done by the kernel, systemd just provides a number of new
settings to configure this facet of it. Note that cgroup/eBPF is
unrelated to classic Linux firewalling,
i.e. NetFilter/iptables. It's up to you whether you use one or the
other, or both in combination (or of course neither).
Let's have a closer look at the IP accounting logic mentioned
above. Let's write a simple unit
/etc/systemd/system/ip-accounting-test.service:
[Service]
ExecStart=/usr/bin/ping 8.8.8.8
IPAccounting=yes
This simple unit invokes the
ping(8) command to
send a series of ICMP/IP ping packets to the IP address 8.8.8.8 (which
is the Google DNS server IP; we use it for testing here, since it's
easy to remember, reachable everywhere and known to react to ICMP
pings; any other IP address responding to pings would be fine to use,
too). The IPAccounting= option is used to turn on IP accounting for
the unit.
Let's start this service after writing the file. Let's then have a
look at the status output of systemctl:
# systemctl daemon-reload
# systemctl start ip-accounting-test
# systemctl status ip-accounting-test
● ip-accounting-test.service
Loaded: loaded (/etc/systemd/system/ip-accounting-test.service; static; vendor preset: disabled)
Active: active (running) since Mon 2017-10-09 18:05:47 CEST; 1s ago
Main PID: 32152 (ping)
IP: 168B in, 168B out
Tasks: 1 (limit: 4915)
CGroup: /system.slice/ip-accounting-test.service
└─32152 /usr/bin/ping 8.8.8.8
Okt 09 18:05:47 sigma systemd[1]: Started ip-accounting-test.service.
Okt 09 18:05:47 sigma ping[32152]: PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
Okt 09 18:05:47 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=29.2 ms
Okt 09 18:05:48 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=28.0 ms
This shows the ping command running — it's currently at its second
ping cycle as we can see in the logs at the end of the output. More
interesting however is the IP: line further up showing the current
IP byte counters. It currently shows 168 bytes have been received, and
168 bytes have been sent. That the two counters are at the same value
is not surprising: ICMP ping requests and responses are supposed to
have the same size. Note that this line is shown only if
IPAccounting= is turned on for the service, as only then this data
is collected.
Let's wait a bit, and invoke systemctl status again:
# systemctl status ip-accounting-test
● ip-accounting-test.service
Loaded: loaded (/etc/systemd/system/ip-accounting-test.service; static; vendor preset: disabled)
Active: active (running) since Mon 2017-10-09 18:05:47 CEST; 4min 28s ago
Main PID: 32152 (ping)
IP: 22.2K in, 22.2K out
Tasks: 1 (limit: 4915)
CGroup: /system.slice/ip-accounting-test.service
└─32152 /usr/bin/ping 8.8.8.8
Okt 09 18:10:07 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=260 ttl=59 time=27.7 ms
Okt 09 18:10:08 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=261 ttl=59 time=28.0 ms
Okt 09 18:10:09 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=262 ttl=59 time=33.8 ms
Okt 09 18:10:10 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=263 ttl=59 time=48.9 ms
Okt 09 18:10:11 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=264 ttl=59 time=27.2 ms
Okt 09 18:10:12 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=265 ttl=59 time=27.0 ms
Okt 09 18:10:13 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=266 ttl=59 time=26.8 ms
Okt 09 18:10:14 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=267 ttl=59 time=27.4 ms
Okt 09 18:10:15 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=268 ttl=59 time=29.7 ms
Okt 09 18:10:16 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=269 ttl=59 time=27.6 ms
As we can see, after 269 pings the counters are much higher: at 22K.
Note that while systemctl status shows only the byte counters,
packet counters are kept as well. Use the low-level systemctl show
command to query the current raw values of the in and out packet and
byte counters:
# systemctl show ip-accounting-test -p IPIngressBytes -p IPIngressPackets -p IPEgressBytes -p IPEgressPackets
IPIngressBytes=37776
IPIngressPackets=449
IPEgressBytes=37776
IPEgressPackets=449
Of course, the same information is also available via the D-Bus
APIs. If you want to process this data further consider talking proper
D-Bus, rather than scraping the output of systemctl show.
Now, let's stop the service again:
# systemctl stop ip-accounting-test
When a service with such accounting turned on terminates, a log line
about all its consumed resources is written to the logs. Let's check
with journalctl:
# journalctl -u ip-accounting-test -n 5
-- Logs begin at Thu 2016-08-18 23:09:37 CEST, end at Mon 2017-10-09 18:17:02 CEST. --
Okt 09 18:15:50 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=603 ttl=59 time=26.9 ms
Okt 09 18:15:51 sigma ping[32152]: 64 bytes from 8.8.8.8: icmp_seq=604 ttl=59 time=27.2 ms
Okt 09 18:15:52 sigma systemd[1]: Stopping ip-accounting-test.service...
Okt 09 18:15:52 sigma systemd[1]: Stopped ip-accounting-test.service.
Okt 09 18:15:52 sigma systemd[1]: ip-accounting-test.service: Received 49.5K IP traffic, sent 49.5K IP traffic
The last line shown is the interesting one, that shows the accounting data. It's actually a structured log message, and among its metadata fields it contains the more comprehensive raw data:
# journalctl -u ip-accounting-test -n 1 -o verbose
-- Logs begin at Thu 2016-08-18 23:09:37 CEST, end at Mon 2017-10-09 18:18:50 CEST. --
Mon 2017-10-09 18:15:52.649028 CEST [s=89a2cc877fdf4dafb2269a7631afedad;i=14d7;b=4c7e7adcba0c45b69d612857270716d3;m=137592e75e;t=55b1f81298605;x=c3c9b57b28c9490e]
PRIORITY=6
_BOOT_ID=4c7e7adcba0c45b69d612857270716d3
_MACHINE_ID=e87bfd866aea4ae4b761aff06c9c3cb3
_HOSTNAME=sigma
SYSLOG_FACILITY=3
SYSLOG_IDENTIFIER=systemd
_UID=0
_GID=0
_TRANSPORT=journal
_PID=1
_COMM=systemd
_EXE=/usr/lib/systemd/systemd
_CAP_EFFECTIVE=3fffffffff
_SYSTEMD_CGROUP=/init.scope
_SYSTEMD_UNIT=init.scope
_SYSTEMD_SLICE=-.slice
CODE_FILE=../src/core/unit.c
_CMDLINE=/usr/lib/systemd/systemd --switched-root --system --deserialize 25
_SELINUX_CONTEXT=system_u:system_r:init_t:s0
UNIT=ip-accounting-test.service
CODE_LINE=2115
CODE_FUNC=unit_log_resources
MESSAGE_ID=ae8f7b866b0347b9af31fe1c80b127c0
INVOCATION_ID=98a6e756fa9d421d8dfc82b6df06a9c3
IP_METRIC_INGRESS_BYTES=50880
IP_METRIC_INGRESS_PACKETS=605
IP_METRIC_EGRESS_BYTES=50880
IP_METRIC_EGRESS_PACKETS=605
MESSAGE=ip-accounting-test.service: Received 49.6K IP traffic, sent 49.6K IP traffic
_SOURCE_REALTIME_TIMESTAMP=1507565752649028
The interesting fields of this log message are of course
IP_METRIC_INGRESS_BYTES=, IP_METRIC_INGRESS_PACKETS=,
IP_METRIC_EGRESS_BYTES=, IP_METRIC_EGRESS_PACKETS= that show the
consumed data.
The log message carries a message
ID
that may be used to quickly search for all such resource log messages
(ae8f7b866b0347b9af31fe1c80b127c0). We can combine a search term for
messages of this ID with journalctl's -u switch to quickly find
out about the resource usage of any invocation of a specific
service. Let's try:
# journalctl -u ip-accounting-test MESSAGE_ID=ae8f7b866b0347b9af31fe1c80b127c0
-- Logs begin at Thu 2016-08-18 23:09:37 CEST, end at Mon 2017-10-09 18:25:27 CEST. --
Okt 09 18:15:52 sigma systemd[1]: ip-accounting-test.service: Received 49.6K IP traffic, sent 49.6K IP traffic
Of course, the output above shows only one message at the moment, since we started the service only once, but a new one will appear every time you start and stop it again.
The IP accounting logic is also hooked up with
systemd-run,
which is useful for transiently running a command as systemd service
with IP accounting turned on. Let's try it:
# systemd-run -p IPAccounting=yes --wait wget https://cfp.all-systems-go.io/en/ASG2017/public/schedule/2.pdf
Running as unit: run-u2761.service
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 878ms
IP traffic received: 231.0K
IP traffic sent: 3.7K
This uses wget to download the
PDF version of the 2nd day
schedule
of everybody's favorite Linux user-space conference All Systems Go!
2017 (BTW, have you already booked your
ticket? We are very close to
selling out, be quick!). The IP traffic this command generated was
231K ingress and 4K egress. In the systemd-run command line two
parameters are important. First of all, we use -p IPAccounting=yes
to turn on IP accounting for the transient service (as above). And
secondly we use --wait to tell systemd-run to wait for the service
to exit. If --wait is used, systemd-run will also show you various
statistics about the service that just ran and terminated, including
the IP statistics you are seeing if IP accounting has been turned on.
It's fun to combine this sort of IP accounting with interactive transient units. Let's try that:
# systemd-run -p IPAccounting=1 -t /bin/sh
Running as unit: run-u2779.service
Press ^] three times within 1s to disconnect TTY.
sh-4.4# dnf update
…
sh-4.4# dnf install firefox
…
sh-4.4# exit
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 5.297s
IP traffic received: …B
IP traffic sent: …B
This uses systemd-run's --pty switch (or short: -t), which opens
an interactive pseudo-TTY connection to the invoked service process,
which is a bourne shell in this case. Doing this means we have a full,
comprehensive shell with job control and everything. Since the shell
is running as part of a service with IP accounting turned on, all IP
traffic we generate or receive will be accounted for. And as soon as
we exit the shell, we'll see what it consumed. (For the sake of
brevity I actually didn't paste the whole output above, but truncated
core parts. Try it out for yourself, if you want to see the output in
full.)
Sometimes it might make sense to turn on IP accounting for a unit that
is already running. For that, use systemctl set-property
foobar.service IPAccounting=yes, which will instantly turn on
accounting for it. Note that it won't count retroactively though: only
the traffic sent/received after the point in time you turned it on
will be collected. You may turn off accounting for the unit with the
same command.
Of course, sometimes it's interesting to collect IP accounting data
for all services, and turning on IPAccounting=yes in every single
unit is cumbersome. To deal with that there's a global option
DefaultIPAccounting=
available which can be set in /etc/systemd/system.conf.
So much about IP accounting. Let's now have a look at IP access
control with systemd 235. As mentioned above, the two new unit file
settings, IPAddressAllow= and IPAddressDeny= maybe be used for
that. They operate in the following way:
If the source address of an incoming packet or the destination
address of an outgoing packet matches one of the IP addresses/network
masks in the relevant unit's IPAddressAllow= setting then it will be
allowed to go through.
Otherwise, if a packet matches an IPAddressDeny= entry configured
for the service it is dropped.
If the packet matches neither of the above it is allowed to go through.
Or in other words, IPAddressDeny= implements a blacklist, but
IPAddressAllow= takes precedence.
Let's try that out. Let's modify our last example above in order to get a transient service running an interactive shell which has such an access list set:
# systemd-run -p IPAddressDeny=any -p IPAddressAllow=8.8.8.8 -p IPAddressAllow=127.0.0.0/8 -t /bin/sh
Running as unit: run-u2850.service
Press ^] three times within 1s to disconnect TTY.
sh-4.4# ping 8.8.8.8 -c1
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=27.9 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 27.957/27.957/27.957/0.000 ms
sh-4.4# ping 8.8.4.4 -c1
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
^C
--- 8.8.4.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
sh-4.4# ping 127.0.0.2 -c1
PING 127.0.0.1 (127.0.0.2) 56(84) bytes of data.
64 bytes from 127.0.0.2: icmp_seq=1 ttl=64 time=0.116 ms
--- 127.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.116/0.116/0.116/0.000 ms
sh-4.4# exit
The access list we set up uses IPAddressDeny=any in order to define
an IP white-list: all traffic will be prohibited for the session,
except for what is explicitly white-listed. In this command line, we
white-listed two address prefixes: 8.8.8.8 (with no explicit network
mask, which means the mask with all bits turned on is implied,
i.e. /32), and 127.0.0.0/8. Thus, the service can communicate with
Google's DNS server and everything on the local loop-back, but nothing
else. The commands run in this interactive shell show this: First we
try pinging 8.8.8.8 which happily responds. Then, we try to ping
8.8.4.4 (that's Google's other DNS server, but excluded from this
white-list), and as we see it is immediately refused with an Operation
not permitted error. As last step we ping 127.0.0.2 (which is on the
local loop-back), and we see it works fine again, as expected.
In the example above we used IPAddressDeny=any. The any
identifier is a shortcut for writing 0.0.0.0/0 ::/0, i.e. it's a
shortcut for everything, on both IPv4 and IPv6. A number of other
such shortcuts exist. For example, instead of spelling out
127.0.0.0/8 we could also have used the more descriptive shortcut
localhost which is expanded to 127.0.0.0/8 ::1/128, i.e. everything
on the local loopback device, on both IPv4 and IPv6.
Being able to configure IP access lists individually for each unit is
pretty nice already. However, typically one wants to configure this
comprehensively, not just for individual units, but for a set of units
in one go or even the system as a whole. In systemd, that's possible
by making use of
.slice
units (for those who don't know systemd that well, slice units are a
concept for organizing services in hierarchical tree for the purpose of
resource management): the IP access list in effect for a unit is the
combination of the individual IP access lists configured for the unit
itself and those of all slice units it is contained in.
By default, system services are assigned to
system.slice,
which in turn is a child of the root slice
-.slice. Either
of these two slice units are hence suitable for locking down all
system services at once. If an access list is configured on
system.slice it will only apply to system services, however, if
configured on -.slice it will apply to all user processes of the
system, including all user session processes (i.e. which are by
default assigned to user.slice which is a child of -.slice) in
addition to the system services.
Let's make use of this:
# systemctl set-property system.slice IPAddressDeny=any IPAddressAllow=localhost
# systemctl set-property apache.service IPAddressAllow=10.0.0.0/8
The two commands above are a very powerful way to first turn off all IP communication for all system services (with the exception of loop-back traffic), followed by an explicit white-listing of 10.0.0.0/8 (which could refer to the local company network, you get the idea) but only for the Apache service.
After playing around a bit with this, let's talk about use-cases. Here are a few ideas:
The IP access list logic can in many ways provide a more modern replacement for the venerable TCP Wrapper, but unlike it it applies to all IP sockets of a service unconditionally, and requires no explicit support in any way in the service's code: no patching required. On the other hand, TCP wrappers have a number of features this scheme cannot cover, most importantly systemd's IP access lists operate solely on the level of IP addresses and network masks, there is no way to configure access by DNS name (though quite frankly, that is a very dubious feature anyway, as doing networking — unsecured networking even – in order to restrict networking sounds quite questionable, at least to me).
It can also replace (or augment) some facets of IP firewalling,
i.e. Linux NetFilter/iptables. Right now, systemd's access lists are
of course a lot more minimal than NetFilter, but they have one major
benefit: they understand the service concept, and thus are a lot more
context-aware than NetFilter. Classic firewalls, such as NetFilter,
derive most service context from the IP port number alone, but we live
in a world where IP port numbers are a lot more dynamic than they used
to be. As one example, a BitTorrent client or server may use any IP
port it likes for its file transfer, and writing IP firewalling rules
matching that precisely is hence hard. With the systemd IP access list
implementing this is easy: just set the list for your BitTorrent
service unit, and all is good.
Let me stress though that you should be careful when comparing NetFilter with systemd's IP address list logic, it's really like comparing apples and oranges: to start with, the IP address list logic has a clearly local focus, it only knows what a local service is and manages access of it. NetFilter on the other hand may run on border gateways, at a point where the traffic flowing through is pure IP, carrying no information about a systemd unit concept or anything like that.
It's a simple way to lock down distribution/vendor supplied system
services by default. For example, if you ship a service that you know
never needs to access the network, then simply set IPAddressDeny=any
(possibly combined with IPAddressAllow=localhost) for it, and it
will live in a very tight networking sand-box it cannot escape
from. systemd itself makes use of this for a number of its services by
default now. For example, the logging service
systemd-journald.service, the login manager systemd-logind or the
core-dump processing unit systemd-coredump@.service all have such a
rule set out-of-the-box, because we know that neither of these
services should be able to access the network, under any
circumstances.
Because the IP access list logic can be combined with transient
units, it can be used to quickly and effectively sandbox arbitrary
commands, and even include them in shell pipelines and such. For
example, let's say we don't trust our
curl implementation (maybe it
got modified locally by a hacker, and phones home?), but want to use
it anyway to download the the slides of my most recent casync
talk in order to
print it, but want to make sure it doesn't connect anywhere except
where we tell it to (and to make this even more fun, let's minimize
privileges further, by setting
DynamicUser=yes):
# systemd-resolve 0pointer.de
0pointer.de: 85.214.157.71
2a01:238:43ed:c300:10c3:bcf3:3266:da74
-- Information acquired via protocol DNS in 2.8ms.
-- Data is authenticated: no
# systemd-run --pipe -p IPAddressDeny=any \
-p IPAddressAllow=85.214.157.71 \
-p IPAddressAllow=2a01:238:43ed:c300:10c3:bcf3:3266:da74 \
-p DynamicUser=yes \
curl http://0pointer.de/public/casync-kinvolk2017.pdf | lp
So much about use-cases. This is by no means a comprehensive list of what you can do with it, after all both IP accounting and IP access lists are very generic concepts. But I do hope the above inspires your fantasy.
IP accounting and IP access control are primarily concepts for the
local administrator. However, As suggested above, it's a very good
idea to ship services that by design have no network-facing
functionality with an access list of IPAddressDeny=any (and possibly
IPAddressAllow=localhost), in order to improve the out-of-the-box
security of our systems.
An option for security-minded distributions might be a more radical
approach: ship the system with -.slice or system.slice configured
to IPAddressDeny=any by default, and ask the administrator to punch
holes into that for each network facing service with systemctl
set-property … IPAddressAllow=…. But of course, that's only an
option for distributions willing to break compatibility with what was
before.
A couple of additional notes:
IP accounting and access lists may be mixed with socket activation. In this case, it's a good idea to configure access lists and accounting for both the socket unit that activates and the service unit that is activated, as both units maintain fully separate settings. Note that IP accounting and access lists configured on the socket unit applies to all sockets created on behalf of that unit, and even if these sockets are passed on to the activated services, they will still remain in effect and belong to the socket unit. This also means that IP traffic done on such sockets will be accounted to the socket unit, not the service unit. The fact that IP access lists are maintained separately for the kernel sockets created on behalf of the socket unit and for the kernel sockets created by the service code itself enables some interesting uses. For example, it's possible to set a relatively open access list on the socket unit, but a very restrictive access list on the service unit, thus making the sockets configured through the socket unit the only way in and out of the service.
systemd's IP accounting and access lists apply to IP sockets only,
not to sockets of any other address families. That also means that
AF_PACKET (i.e. raw) sockets are not covered. This means it's a good
idea to combine IP access lists with RestrictAddressFamilies=AF_UNIX
AF_INET
AF_INET6
in order to lock this down.
You may wonder if the per-unit resource log message and
systemd-run --wait may also show you details about other types or
resources consumed by a service. The answer is yes: if you turn on
CPUAccounting= for a service, you'll also see a summary of consumed
CPU time in the log message and the command output. And we are
planning to hook-up IOAccounting= the same way too, soon.
Note that IP accounting and access lists aren't entirely free. systemd inserts an eBPF program into the IP pipeline to make this functionality work. However, eBPF execution has been optimized for speed in the last kernel versions already, and given that it currently is in the focus of interest to many I'd expect to be optimized even further, so that the cost for enabling these features will be negligible, if it isn't already.
IP accounting is currently not recursive. That means you cannot use a slice unit to join the accounting of multiple units into one. This is something we definitely want to add, but requires some more kernel work first.
You might wonder how the
PrivateNetwork=
setting relates to IPAccessDeny=any. Superficially they have similar
effects: they make the network unavailable to services. However,
looking more closely there are a number of
differences. PrivateNetwork= is implemented using Linux network
name-spaces. As such it entirely detaches all networking of a service
from the host, including non-IP networking. It does so by creating a
private little environment the service lives in where communication
with itself is still allowed though. In addition using the
JoinsNamespaceOf=
dependency additional services may be added to the same environment,
thus permitting communication with each other but not with anything
outside of this group. IPAddressAllow= and IPAddressDeny= are much
less invasive. First of all they apply to IP networking only, and can
match against specific IP addresses. A service running with
PrivateNetwork= turned off but IPAddressDeny=any turned on, may
enumerate the network interfaces and their IP configured even though
it cannot actually do any IP communication. On the other hand if you
turn on PrivateNetwork= all network interfaces besides lo
disappear. Long story short: depending on your use-case one, the other,
both or neither might be suitable for sand-boxing of your service. If
possible I'd always turn on both, for best security, and that's what
we do for all of systemd's own long-running services.
And that's all for now. Have fun with per-unit IP accounting and access lists!
TL;DR: you may now configure systemd to dynamically allocate a UNIX user ID for service processes when it starts them and release it when it stops them. It's pretty secure, mixes well with transient services, socket activated services and service templating.
Today we released systemd 235. Among other improvements this greatly extends the dynamic user logic of systemd. Dynamic users are a powerful but little known concept, supported in its basic form since systemd 232. With this blog story I hope to make it a bit better known.
The UNIX user concept is the most basic and well-understood security concept in POSIX operating systems. It is UNIX/POSIX' primary security concept, the one everybody can agree on, and most security concepts that came after it (such as process capabilities, SELinux and other MACs, user name-spaces, …) in some form or another build on it, extend it or at least interface with it. If you build a Linux kernel with all security features turned off, the user concept is pretty much the one you'll still retain.
Originally, the user concept was introduced to make multi-user systems
a reality, i.e. systems enabling multiple human users to share the
same system at the same time, cleanly separating their resources and
protecting them from each other. The majority of today's UNIX systems
don't really use the user concept like that anymore though. Most of
today's systems probably have only one actual human user (or even
less!), but their user databases (/etc/passwd) list a good number
more entries than that. Today, the majority of UNIX users in most
environments are system users, i.e. users that are not the technical
representation of a human sitting in front of a PC anymore, but the
security identity a system service — an executable program — runs
as. Even though traditional, simultaneous multi-user systems slowly
became less relevant, their ground-breaking basic concept became the
cornerstone of UNIX security. The OS is nowadays partitioned into
isolated services — and each service runs as its own system user, and
thus within its own, minimal security context.
The people behind the Android OS realized the relevance of the UNIX user concept as the primary security concept on UNIX, and took its use even further: on Android not only system services take benefit of the UNIX user concept, but each UI app gets its own, individual user identity too — thus neatly separating app resources from each other, and protecting app processes from each other, too.
Back in the more traditional Linux world things are a bit less advanced in this area. Even though users are the quintessential UNIX security concept, allocation and management of system users is still a pretty limited, raw and static affair. In most cases, RPM or DEB package installation scripts allocate a fixed number of (usually one) system users when you install the package of a service that wants to take benefit of the user concept, and from that point on the system user remains allocated on the system and is never deallocated again, even if the package is later removed again. Most Linux distributions limit the number of system users to 1000 (which isn't particularly a lot). Allocating a system user is hence expensive: the number of available users is limited, and there's no defined way to dispose of them after use. If you make use of system users too liberally, you are very likely to run out of them sooner rather than later.
You may wonder why system users are generally not deallocated when the package that registered them is uninstalled from a system (at least on most distributions). The reason for that is one relevant property of the user concept (you might even want to call this a design flaw): user IDs are sticky to files (and other objects such as IPC objects). If a service running as a specific system user creates a file at some location, and is then terminated and its package and user removed, then the created file still belongs to the numeric ID ("UID") the system user originally got assigned. When the next system user is allocated and — due to ID recycling — happens to get assigned the same numeric ID, then it will also gain access to the file, and that's generally considered a problem, given that the file belonged to a potentially very different service once upon a time, and likely should not be readable or changeable by anything coming after it. Distributions hence tend to avoid UID recycling which means system users remain registered forever on a system after they have been allocated once.
The above is a description of the status quo ante. Let's now focus on what systemd's dynamic user concept brings to the table, to improve the situation.
With systemd dynamic users we hope to make make it easier and cheaper to allocate system users on-the-fly, thus substantially increasing the possible uses of this core UNIX security concept.
If you write a systemd service unit file, you may enable the dynamic
user logic for it by setting the
DynamicUser=
option in its [Service] section to yes. If you do a system user is
dynamically allocated the instant the service binary is invoked, and
released again when the service terminates. The user is automatically
allocated from the UID range 61184–65519, by looking for a so far
unused UID.
Now you may wonder, how does this concept deal with the sticky user issue discussed above? In order to counter the problem, two strategies easily come to mind:
Prohibit the service from creating any files/directories or IPC objects
Automatically removing the files/directories or IPC objects the service created when it shuts down.
In systemd we implemented both strategies, but for different parts of the execution environment. Specifically:
Setting DynamicUser=yes implies
ProtectSystem=strict
and
ProtectHome=read-only. These
sand-boxing options turn off write access to pretty much the whole OS
directory tree, with a few relevant exceptions, such as the API file
systems /proc, /sys and so on, as well as /tmp and
/var/tmp. (BTW: setting these two options on your regular services
that do not use DynamicUser= is a good idea too, as it drastically
reduces the exposure of the system to exploited services.)
Setting DynamicUser=yes implies
PrivateTmp=yes. This
option sets up /tmp and /var/tmp for the service in a way that it
gets its own, disconnected version of these directories, that are not
shared by other services, and whose life-cycle is bound to the
service's own life-cycle. Thus if the service goes down, the user is
removed and all its temporary files and directories with it. (BTW: as
above, consider setting this option for your regular services that do
not use DynamicUser= too, it's a great way to lock things down
security-wise.)
Setting DynamicUser=yes implies
RemoveIPC=yes. This
option ensures that when the service goes down all SysV and POSIX IPC
objects (shared memory, message queues, semaphores) owned by the
service's user are removed. Thus, the life-cycle of the IPC objects is
bound to the life-cycle of the dynamic user and service, too. (BTW:
yes, here too, consider using this in your regular services, too!)
With these four settings in effect, services with dynamic users are
nicely sand-boxed. They cannot create files or directories, except in
/tmp and /var/tmp, where they will be removed automatically when
the service shuts down, as will any IPC objects created. Sticky
ownership of files/directories and IPC objects is hence dealt with
effectively.
The
RuntimeDirectory=
option may be used to open up a bit the sandbox to external
programs. If you set it to a directory name of your choice, it will be
created below /run when the service is started, and removed in its
entirety when it is terminated. The ownership of the directory is
assigned to the service's dynamic user. This way, a dynamic user
service can expose API interfaces (AF_UNIX sockets, …) to other
services at a well-defined place and again bind the life-cycle of it to
the service's own run-time. Example: set RuntimeDirectory=foobar in
your service, and watch how a directory /run/foobar appears at the
moment you start the service, and disappears the moment you stop
it again. (BTW: Much like the other settings discussed above,
RuntimeDirectory= may be used outside of the DynamicUser= context
too, and is a nice way to run any service with a properly owned,
life-cycle-managed run-time directory.)
Of course, a service running in such an environment (although already very useful for many cases!), has a major limitation: it cannot leave persistent data around it can reuse on a later run. As pretty much the whole OS directory tree is read-only to it, there's simply no place it could put the data that survives from one service invocation to the next.
With systemd 235 this limitation is removed: there are now three new
settings:
StateDirectory=,
LogsDirectory= and CacheDirectory=. In many ways they operate like
RuntimeDirectory=, but create sub-directories below /var/lib,
/var/log and /var/cache, respectively. There's one major
difference beyond that however: directories created that way are
persistent, they will survive the run-time cycle of a service, and
thus may be used to store data that is supposed to stay around between
invocations of the service.
Of course, the obvious question to ask now is: how do these three settings deal with the sticky file ownership problem?
For that we lifted a concept from container managers. Container
managers have a very similar problem: each container and the host
typically end up using a very similar set of numeric UIDs, and unless
user name-spacing is deployed this means that host users might be able
to access the data of specific containers that also have a user by the
same numeric UID assigned, even though it actually refers to a very
different identity in a different context. (Actually, it's even worse
than just getting access, due to the existence of setuid file bits,
access might translate to privilege elevation.) The way container
managers protect the container images from the host (and from each
other to some level) is by placing the container trees below a
boundary directory, with very restrictive access modes and ownership
(0700 and root:root or so). A host user hence cannot take advantage
of the files/directories of a container user of the same UID inside of
a local container tree, simply because the boundary directory makes it
impossible to even reference files in it. After all on UNIX, in order
to get access to a specific path you need access to every single
component of it.
How is that applied to dynamic user services? Let's say
StateDirectory=foobar is set for a service that has DynamicUser=
turned off. The instant the service is started, /var/lib/foobar is
created as state directory, owned by the service's user and remains in
existence when the service is stopped. If the same service now is run
with DynamicUser= turned on, the implementation is slightly
altered. Instead of a directory /var/lib/foobar a symbolic link by
the same path is created (owned by root), pointing to
/var/lib/private/foobar (the latter being owned by the service's
dynamic user). The /var/lib/private directory is created as boundary
directory: it's owned by root:root, and has a restrictive access
mode of 0700. Both the symlink and the service's state directory will
survive the service's life-cycle, but the state directory will remain,
and continues to be owned by the now disposed dynamic UID — however it
is protected from other host users (and other services which might get
the same dynamic UID assigned due to UID recycling) by the boundary
directory.
The obvious question to ask now is: but if the boundary directory
prohibits access to the directory from unprivileged processes, how can
the service itself which runs under its own dynamic UID access it
anyway? This is achieved by invoking the service process in a slightly
modified mount name-space: it will see most of the file hierarchy the
same way as everything else on the system (modulo /tmp and
/var/tmp as mentioned above), except for /var/lib/private, which
is over-mounted with a read-only tmpfs file system instance, with a
slightly more liberal access mode permitting the service read
access. Inside of this tmpfs file system instance another mount is
placed: a bind mount to the host's real /var/lib/private/foobar
directory, onto the same name. Putting this together these means that
superficially everything looks the same and is available at the same
place on the host and from inside the service, but two important
changes have been made: the /var/lib/private boundary directory lost
its restrictive character inside the service, and has been emptied of
the state directories of any other service, thus making the protection
complete. Note that the symlink /var/lib/foobar hides the fact that
the boundary directory is used (making it little more than an
implementation detail), as the directory is available this way under
the same name as it would be if DynamicUser= was not used. Long
story short: for the daemon and from the view from the host the
indirection through /var/lib/private is mostly transparent.
This logic of course raises another question: what happens to the state directory if a dynamic user service is started with a state directory configured, gets UID X assigned on this first invocation, then terminates and is restarted and now gets UID Y assigned on the second invocation, with X ≠ Y? On the second invocation the directory — and all the files and directories below it — will still be owned by the original UID X so how could the second instance running as Y access it? Our way out is simple: systemd will recursively change the ownership of the directory and everything contained within it to UID Y before invoking the service's executable.
Of course, such recursive ownership changing (chown()ing) of whole
directory trees can become expensive (though according to my
experiences, IRL and for most services it's much cheaper than you
might think), hence in order to optimize behavior in this regard, the
allocation of dynamic UIDs has been tweaked in two ways to avoid the
necessity to do this expensive operation in most cases: firstly, when
a dynamic UID is allocated for a service an allocation loop is
employed that starts out with a UID hashed from the service's
name. This means a service by the same name is likely to always use
the same numeric UID. That means that a stable service name translates
into a stable dynamic UID, and that means recursive file ownership
adjustments can be skipped (of course, after validation). Secondly, if
the configured state directory already exists, and is owned by a
suitable currently unused dynamic UID, it's preferably used above
everything else, thus maximizing the chance we can avoid the
chown()ing. (That all said, ultimately we have to face it, the
currently available UID space of 4K+ is very small still, and
conflicts are pretty likely sooner or later, thus a chown()ing has to
be expected every now and then when this feature is used extensively).
Note that CacheDirectory= and LogsDirectory= work very similar to
StateDirectory=. The only difference is that they manage directories
below the /var/cache and /var/logs directories, and their boundary
directory hence is /var/cache/private and /var/log/private,
respectively.
So, after all this introduction, let's have a look how this all can be put together. Here's a trivial example:
# cat > /etc/systemd/system/dynamic-user-test.service <<EOF
[Service]
ExecStart=/usr/bin/sleep 4711
DynamicUser=yes
EOF
# systemctl daemon-reload
# systemctl start dynamic-user-test
# systemctl status dynamic-user-test
● dynamic-user-test.service
Loaded: loaded (/etc/systemd/system/dynamic-user-test.service; static; vendor preset: disabled)
Active: active (running) since Fri 2017-10-06 13:12:25 CEST; 3s ago
Main PID: 2967 (sleep)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/dynamic-user-test.service
└─2967 /usr/bin/sleep 4711
Okt 06 13:12:25 sigma systemd[1]: Started dynamic-user-test.service.
# ps -e -o pid,comm,user | grep 2967
2967 sleep dynamic-user-test
# id dynamic-user-test
uid=64642(dynamic-user-test) gid=64642(dynamic-user-test) groups=64642(dynamic-user-test)
# systemctl stop dynamic-user-test
# id dynamic-user-test
id: ‘dynamic-user-test’: no such user
In this example, we create a unit file with DynamicUser= turned on,
start it, check if it's running correctly, have a look at the service
process' user (which is named like the service; systemd does this
automatically if the service name is suitable as user name, and you
didn't configure any user name to use explicitly), stop the service
and verify that the user ceased to exist too.
That's already pretty cool. Let's step it up a notch, by doing the
same in an interactive transient service (for those who don't know
systemd well: a transient service is a service that is defined and
started dynamically at run-time, for example via the
systemd-run
command from the shell. Think: run a service without having to write a
unit file first):
# systemd-run --pty --property=DynamicUser=yes --property=StateDirectory=wuff /bin/sh
Running as unit: run-u15750.service
Press ^] three times within 1s to disconnect TTY.
sh-4.4$ id
uid=63122(run-u15750) gid=63122(run-u15750) groups=63122(run-u15750) context=system_u:system_r:initrc_t:s0
sh-4.4$ ls -al /var/lib/private/
total 0
drwxr-xr-x. 3 root root 60 6. Okt 13:21 .
drwxr-xr-x. 1 root root 852 6. Okt 13:21 ..
drwxr-xr-x. 1 run-u15750 run-u15750 8 6. Okt 13:22 wuff
sh-4.4$ ls -ld /var/lib/wuff
lrwxrwxrwx. 1 root root 12 6. Okt 13:21 /var/lib/wuff -> private/wuff
sh-4.4$ ls -ld /var/lib/wuff/
drwxr-xr-x. 1 run-u15750 run-u15750 0 6. Okt 13:21 /var/lib/wuff/
sh-4.4$ echo hello > /var/lib/wuff/test
sh-4.4$ exit
exit
# id run-u15750
id: ‘run-u15750’: no such user
# ls -al /var/lib/private
total 0
drwx------. 1 root root 66 6. Okt 13:21 .
drwxr-xr-x. 1 root root 852 6. Okt 13:21 ..
drwxr-xr-x. 1 63122 63122 8 6. Okt 13:22 wuff
# ls -ld /var/lib/wuff
lrwxrwxrwx. 1 root root 12 6. Okt 13:21 /var/lib/wuff -> private/wuff
# ls -ld /var/lib/wuff/
drwxr-xr-x. 1 63122 63122 8 6. Okt 13:22 /var/lib/wuff/
# cat /var/lib/wuff/test
hello
The above invokes an interactive shell as transient service
run-u15750.service (systemd-run picked that name automatically,
since we didn't specify anything explicitly) with a dynamic user whose
name is derived automatically from the service name. Because
StateDirectory=wuff is used, a persistent state directory for the
service is made available as /var/lib/wuff. In the interactive shell
running inside the service, the ls commands show the
/var/lib/private boundary directory and its contents, as well as the
symlink that is placed for the service. Finally, before exiting the
shell, a file is created in the state directory. Back in the original
command shell we check if the user is still allocated: it is not, of
course, since the service ceased to exist when we exited the shell and
with it the dynamic user associated with it. From the host we check
the state directory of the service, with similar commands as we did
from inside of it. We see that things are set up pretty much the same
way in both cases, except for two things: first of all the user/group
of the files is now shown as raw numeric UIDs instead of the
user/group names derived from the unit name. That's because the user
ceased to exist at this point, and "ls" shows the raw UID for files
owned by users that don't exist. Secondly, the access mode of the
boundary directory is different: when we look at it from outside of
the service it is not readable by anyone but root, when we looked from
inside we saw it it being world readable.
Now, let's see how things look if we start another transient service, reusing the state directory from the first invocation:
# systemd-run --pty --property=DynamicUser=yes --property=StateDirectory=wuff /bin/sh
Running as unit: run-u16087.service
Press ^] three times within 1s to disconnect TTY.
sh-4.4$ cat /var/lib/wuff/test
hello
sh-4.4$ ls -al /var/lib/wuff/
total 4
drwxr-xr-x. 1 run-u16087 run-u16087 8 6. Okt 13:22 .
drwxr-xr-x. 3 root root 60 6. Okt 15:42 ..
-rw-r--r--. 1 run-u16087 run-u16087 6 6. Okt 13:22 test
sh-4.4$ id
uid=63122(run-u16087) gid=63122(run-u16087) groups=63122(run-u16087) context=system_u:system_r:initrc_t:s0
sh-4.4$ exit
exit
Here, systemd-run picked a different auto-generated unit name, but
the used dynamic UID is still the same, as it was read from the
pre-existing state directory, and was otherwise unused. As we can see
the test file we generated earlier is accessible and still contains
the data we left in there. Do note that the user name is different
this time (as it is derived from the unit name, which is different),
but the UID it is assigned to is the same one as on the first
invocation. We can thus see that the mentioned optimization of the UID
allocation logic (i.e. that we start the allocation loop from the UID
owner of any existing state directory) took effect, so that no
recursive chown()ing was required.
And that's the end of our example, which hopefully illustrated a bit how this concept and implementation works.
Now that we had a look at how to enable this logic for a unit and how it is implemented, let's discuss where this actually could be useful in real life.
One major benefit of dynamic user IDs is that running a privilege-separated service leaves no artifacts in the system. A system user is allocated and made use of, but it is discarded automatically in a safe and secure way after use, in a fashion that is safe for later recycling. Thus, quickly invoking a short-lived service for processing some job can be protected properly through a user ID without having to pre-allocate it and without this draining the available UID pool any longer than necessary.
In many cases, starting a service no longer requires
package-specific preparation. Or in other words, quite often
useradd/mkdir/chown/chmod invocations in "post-inst" package
scripts, as well as
sysusers.d
and
tmpfiles.d
drop-ins become unnecessary, as the DynamicUser= and
StateDirectory=/CacheDirectory=/LogsDirectory= logic can do the
necessary work automatically, on-demand and with a well-defined
life-cycle.
By combining dynamic user IDs with the transient unit concept, new
creative ways of sand-boxing are made available. For example, let's say
you don't trust the correct implementation of the sort command. You
can now lock it into a simple, robust, dynamic UID sandbox with a
simple systemd-run and still integrate it into a shell pipeline like
any other command. Here's an example, showcasing a shell pipeline
whose middle element runs as a dynamically on-the-fly allocated UID,
that is released when the pipelines ends.
# cat some-file.txt | systemd-run ---pipe --property=DynamicUser=1 sort -u | grep -i foobar > some-other-file.txt
By combining dynamic user IDs with the systemd templating logic it
is now possible to do much more fine-grained and fully automatic UID
management. For example, let's say you have a template unit file
/etc/systemd/system/foobard@.service:
[Service]
ExecStart=/usr/bin/myfoobarserviced
DynamicUser=1
StateDirectory=foobar/%i
Now, let's say you want to start one instance of this service for each of your customers. All you need to do now for that is:
# systemctl enable foobard@customerxyz.service --now
And you are done. (Invoke this as many times as you like, each time
replacing customerxyz by some customer identifier, you get the
idea.)
By combining dynamic user IDs with socket activation you may easily
implement a system where each incoming connection is served by a
process instance running as a different, fresh, newly allocated UID
within its own sandbox. Here's an example waldo.socket:
[Socket]
ListenStream=2048
Accept=yes
With a matching waldo@.service:
[Service]
ExecStart=-/usr/bin/myservicebinary
DynamicUser=yes
With the two unit files above, systemd will listen on TCP/IP port
2048, and for each incoming connection invoke a fresh instance of
waldo@.service, each time utilizing a different, new,
dynamically allocated UID, neatly isolated from any other
instance.
Dynamic user IDs combine very well with state-less systems,
i.e. systems that come up with an unpopulated /etc and /var. A
service using dynamic user IDs and the StateDirectory=,
CacheDirectory=, LogsDirectory= and RuntimeDirectory= concepts
will implicitly allocate the users and directories it needs for
running, right at the moment where it needs it.
Dynamic users are a very generic concept, hence a multitude of other uses are thinkable; the list above is just supposed to trigger your imagination.
I am pretty sure that a large number of services shipped with today's
distributions could benefit from using DynamicUser= and
StateDirectory= (and related settings). It often allows removal of
post-inst packaging scripts altogether, as well as any sysusers.d
and tmpfiles.d drop-ins by unifying the needed declarations in the
unit file itself. Hence, as a packager please consider switching your
unit files over. That said, there are a number of conditions where
DynamicUser= and StateDirectory= (and friends) cannot or should
not be used. To name a few:
Service that need to write to files outside of /run/<package>,
/var/lib/<package>, /var/cache/<package>, /var/log/<package>,
/var/tmp, /tmp, /dev/shm are generally incompatible with this
scheme. This rules out daemons that upgrade the system as one example,
as that involves writing to /usr.
Services that maintain a herd of processes with different user IDs. Some SMTP services are like this. If your service has such a super-server design, UID management needs to be done by the super-server itself, which rules out systemd doing its dynamic UID magic for it.
Services which run as root (obviously…) or are otherwise privileged.
Services that need to live in the same mount name-space as the host
system (for example, because they want to establish mount points
visible system-wide). As mentioned DynamicUser= implies
ProtectSystem=, PrivateTmp= and related options, which all require
the service to run in its own mount name-space.
Your focus is older distributions, i.e. distributions that do not
have systemd 232 (for DynamicUser=) or systemd 235 (for
StateDirectory= and friends) yet.
If your distribution's packaging guides don't allow it. Consult your packaging guides, and possibly start a discussion on your distribution's mailing list about this.
A couple of additional, random notes about the implementation and use of these features:
Do note that allocating or deallocating a dynamic user leaves
/etc/passwd untouched. A dynamic user is added into the user
database through the glibc NSS module
nss-systemd,
and this information never hits the disk.
On traditional UNIX systems it was the job of the daemon process
itself to drop privileges, while the DynamicUser= concept is
designed around the service manager (i.e. systemd) being responsible
for that. That said, since v235 there's a way to marry DynamicUser=
and such services which want to drop privileges on their own. For
that, turn on DynamicUser= and set
User=
to the user name the service wants to setuid() to. This has the
effect that systemd will allocate the dynamic user under the specified
name when the service is started. Then, prefix the command line you
specify in
ExecStart=
with a single ! character. If you do, the user is allocated for the
service, but the daemon binary is invoked as root instead of the
allocated user, under the assumption that the daemon changes its UID
on its own the right way. Note that after registration the user will
show up instantly in the user database, and is hence resolvable like
any other by the daemon process. Example:
ExecStart=!/usr/bin/mydaemond
You may wonder why systemd uses the UID range 61184–65519 for its
dynamic user allocations (side note: in hexadecimal this reads as
0xEF00–0xFFEF). That's because distributions (specifically Fedora)
tend to allocate regular users from below the 60000 range, and we
don't want to step into that. We also want to stay away from 65535 and
a bit around it, as some of these UIDs have special meanings (65535 is
often used as special value for "invalid" or "no" UID, as it is
identical to the 16bit value -1; 65534 is generally mapped to the
"nobody" user, and is where some kernel subsystems map unmappable
UIDs). Finally, we want to stay within the 16bit range. In a user
name-spacing world each container tends to have much less than the full
32bit UID range available that Linux kernels theoretically
provide. Everybody apparently can agree that a container should at
least cover the 16bit range though — already to include a nobody
user. (And quite frankly, I am pretty sure assigning 64K UIDs per
container is nicely systematic, as the the higher 16bit of the 32bit
UID values this way become a container ID, while the lower 16bit
become the logical UID within each container, if you still follow what
I am babbling here…). And before you ask: no this range cannot be
changed right now, it's compiled in. We might change that eventually
however.
You might wonder what happens if you already used UIDs from the 61184–65519 range on your system for other purposes. systemd should handle that mostly fine, as long as that usage is properly registered in the user database: when allocating a dynamic user we pick a UID, see if it is currently used somehow, and if yes pick a different one, until we find a free one. Whether a UID is used right now or not is checked through NSS calls. Moreover the IPC object lists are checked to see if there are any objects owned by the UID we are about to pick. This means systemd will avoid using UIDs you have assigned otherwise. Note however that this of course makes the pool of available UIDs smaller, and in the worst cases this means that allocating a dynamic user might fail because there simply are no unused UIDs in the range.
If not specified otherwise the name for a dynamically allocated
user is derived from the service name. Not everything that's valid in
a service name is valid in a user-name however, and in some cases a
randomized name is used instead to deal with this. Often it makes
sense to pick the user names to register explicitly. For that use
User= and choose whatever you like.
If you pick a user name with User= and combine it with
DynamicUser= and the user already exists statically it will be used
for the service and the dynamic user logic is automatically
disabled. This permits automatic up- and downgrades between static and
dynamic UIDs. For example, it provides a nice way to move a system
from static to dynamic UIDs in a compatible way: as long as you select
the same User= value before and after switching DynamicUser= on,
the service will continue to use the statically allocated user if it
exists, and only operates in the dynamic mode if it does not. This is
useful for other cases as well, for example to adapt a service that
normally would use a dynamic user to concepts that require statically
assigned UIDs, for example to marry classic UID-based file system
quota with such services.
systemd always allocates a pair of dynamic UID and GID at the same time, with the same numeric ID.
If the Linux kernel had a "shiftfs" or similar functionality,
i.e. a way to mount an existing directory to a second place, but map
the exposed UIDs/GIDs in some way configurable at mount time, this
would be excellent for the implementation of StateDirectory= in
conjunction with DynamicUser=. It would make the recursive
chown()ing step unnecessary, as the host version of the state
directory could simply be mounted into a the service's mount
name-space, with a shift applied that maps the directory's owner to the
services' UID/GID. But I don't have high hopes in this regard, as all
work being done in this area appears to be bound to user name-spacing
— which is a concept not used here (and I guess one could say user
name-spacing is probably more a source of problems than a solution to
one, but you are welcome to disagree on that).
And that's all for now. Enjoy your dynamic users!
I am happy to announce that we have published the All Systems Go! 2017 schedule! We are very happy with the large number and the quality of the submissions we got, and the resulting schedule is exceptionally strong.
Without further ado:
Here's the schedule for the first day (Saturday, 21st of October).
And here's the schedule for the second day (Sunday, 22nd of October).
Here are a couple of keywords from the topics of the talks: 1password, azure, bluetooth, build systems, casync, cgroups, cilium, cockpit, containers, ebpf, flatpak, habitat, IoT, kubernetes, landlock, meson, OCI, rkt, rust, secureboot, skydive, systemd, testing, tor, varlink, virtualization, wifi, and more.
Our speakers are from all across the industry: Chef CoreOS, Covalent, Facebook, Google, Intel, Kinvolk, Microsoft, Mozilla, Pantheon, Pengutronix, Red Hat, SUSE and more.
For further information about All Systems Go! visit our conference web site.
Make sure to buy your ticket for All Systems Go! 2017 now! A limited number of tickets are left at this point, so make sure you get yours before we are all sold out! Find all details here.
See you in Berlin!
Please make sure to get your presentation proprosals forAll Systems Go! 2017 in now! The CfP closes on sunday!
In case you haven't heard about All Systems Go! yet, here's a quick reminder what kind of conference it is, and why you should attend and speak there:
All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward. All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd. All Systems Go! is a 2-day event with 2-3 talks happening in parallel. Full presentation slots are 30-45 minutes in length and lightning talk slots are 5-10 minutes.
In particular, we are looking for sessions including, but not limited to, the following topics:
While our focus is definitely more on the user-space side of things, talks about kernel projects are welcome too, as long as they have a clear and direct relevance for user-space.
To submit your proposal now please visit our CFP submission web site.
For further information about All Systems Go! visit our conference web site.
systemd.conf will not take place this year in lieu of All Systems Go!. All Systems Go! welcomes all projects that contribute to Linux user space, which, of course, includes systemd. Thus, anything you think was appropriate for submission to systemd.conf is also fitting for All Systems Go!
Don't forget to send in your submissions to the All Systems Go! 2017 CfP! Proposals are accepted until September 3rd!
A couple of headline speakers have been announced now:
These folks will also review your submissions as part of the papers committee!
All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward.
All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd.
To submit your proposal now please visit our CFP submission web site.
For further information about All Systems Go! visit our conference web site.
The great folks at kinvolk have uploaded a video of my casync presentation at their offices last week.
The slides are available as well.
Enjoy!
After blogging about
casync
I realized I never blogged about the
mkosi tool that combines nicely
with it. mkosi has been around for a while already, and its time to
make it a bit better known. mkosi stands for Make Operating System
Image, and is a tool for precisely that: generating an OS tree or
image that can be booted.
Yes, there are many tools like mkosi, and a number of them are quite
well known and popular. But mkosi has a number of features that I
think make it interesting for a variety of use-cases that other tools
don't cover that well.
What are those use-cases, and what does mkosi precisely set apart?
mkosi is definitely a tool with a focus on developer's needs for
building OS images, for testing and debugging, but also for generating
production images with cryptographic protection. A typical use-case
would be to add a mkosi.default file to an existing project (for
example, one written in C or Python), and thus making it easy to
generate an OS image for it. mkosi will put together the image with
development headers and tools, compile your code in it, run your test
suite, then throw away the image again, and build a new one, this time
without development headers and tools, and install your build
artifacts in it. This final image is then "production-ready", and only
contains your built program and the minimal set of packages you
configured otherwise. Such an image could then be deployed with
casync (or any other tool of course) to be delivered to your set of
servers, or IoT devices or whatever you are building.
mkosi is supposed to be legacy-free: the focus is clearly on
today's technology, not yesteryear's. Specifically this means that
we'll generate GPT partition tables, not MBR/DOS ones. When you tell
mkosi to generate a bootable image for you, it will make it bootable
on EFI, not on legacy BIOS. The GPT images generated follow
specifications such as the Discoverable Partitions
Specification,
so that /etc/fstab can remain unpopulated and tools such as
systemd-nspawn can automatically dissect the image and boot from
them.
So, let's have a look on the specific images it can generate:
When any of the GPT choices above are selected, a couple of additional options are available:
/home and /srv may be added in/home and /srv partitions may be optionally encrypted with LUKSdm-verity, thus making offline attacks on the generated system harddm-verity root hash is automatically added to the kernel command line, and the kernel together with its initial RAM disk and the kernel command line is optionally cryptographically signed for UEFI SecureBootNote that mkosi is distribution-agnostic. It currently can build
images based on the following Linux distributions:
Note though that not all distributions are supported at the same
feature level currently. Also, as mkosi is based on dnf
--installroot, debootstrap, pacstrap and zypper, and those
packages are not packaged universally on all distributions, you might
not be able to build images for all those distributions on arbitrary
host distributions.
The GPT images are put together in a way that they aren't just
compatible with UEFI systems, but also with VM and container managers
(that is, at least the smart ones, i.e. VM managers that know UEFI,
and container managers that grok GPT disk images) to a large
degree. In fact, the idea is that you can use mkosi to build a
single GPT image that may be used to:
systemd-nspawn containerRootImage= unit file settingNote that in all four cases the dm-verity data is automatically used
if available to ensure the image is not tampered with (yes, you read
that right, systemd-nspawn and systemd's RootImage= setting
automatically do dm-verity these days if the image has it.)
The simplest usage of mkosi is by simply invoking it without
parameters (as root):
# mkosi
Without any configuration this will create a GPT disk image for you,
will call it image.raw and drop it in the current directory. The
distribution used will be the same one as your host runs.
Of course in most cases you want more control about how the image is
put together, i.e. select package sets, select the distribution, size
partitions and so on. Most of that you can actually specify on the
command line, but it is recommended to instead create a couple of
mkosi.$SOMETHING files and directories in some directory. Then,
simply change to that directory and run mkosi without any further
arguments. The tool will then look in the current working directory
for these files and directories and make use of them (similar to how
make looks for a Makefile…). Every single file/directory is
optional, but if they exist they are honored. Here's a list of the
files/directories mkosi currently looks for:
mkosi.default — This is the main configuration file, here you
can configure what kind of image you want, which distribution, which
packages and so on.
mkosi.extra/ — If this directory exists, then mkosi will copy
everything inside it into the images built. You can place arbitrary
directory hierarchies in here, and they'll be copied over whatever is
already in the image, after it was put together by the distribution's
package manager. This is the best way to drop additional static files
into the image, or override distribution-supplied ones.
mkosi.build — This executable file is supposed to be a build
script. When it exists, mkosi will build two images, one after the
other in the mode already mentioned above: the first version is the
build image, and may include various build-time dependencies such as
a compiler or development headers. The build script is also copied
into it, and then run inside it. The script should then build
whatever shall be built and place the result in $DESTDIR (don't
worry, popular build tools such as Automake or Meson all honor
$DESTDIR anyway, so there's not much to do here explicitly). It may
also run a test suite, or anything else you like. After the script
finished, the build image is removed again, and a second image (the
final image) is built. This time, no development packages are
included, and the build script is not copied into the image again —
however, the build artifacts from the first run (i.e. those placed in
$DESTDIR) are copied into the image.
mkosi.postinst — If this executable script exists, it is invoked
inside the image (inside a systemd-nspawn invocation) and can
adjust the image as it likes at a very late point in the image
preparation. If mkosi.build exists, i.e. the dual-phased
development build process used, then this script will be invoked
twice: once inside the build image and once inside the final
image. The first parameter passed to the script clarifies which phase
it is run in.
mkosi.nspawn — If this file exists, it should contain a
container configuration file for systemd-nspawn (see
systemd.nspawn(5)
for details), which shall be shipped along with the final image and
shall be included in the check-sum calculations (see below).
mkosi.cache/ — If this directory exists, it is used as package
cache directory for the builds. This directory is effectively bind
mounted into the image at build time, in order to speed up building
images. The package installers of the various distributions will
place their package files here, so that subsequent runs can reuse
them.
mkosi.passphrase — If this file exists, it should contain a
pass-phrase to use for the LUKS encryption (if that's enabled for the
image built). This file should not be readable to other users.
mkosi.secure-boot.crt and mkosi.secure-boot.key should be an
X.509 key pair to use for signing the kernel and initrd for UEFI
SecureBoot, if that's enabled.
So, let's come back to our most trivial example, without any of the
mkosi.$SOMETHING files around:
# mkosi
As mentioned, this will create a build file image.raw in the current
directory. How do we use it? Of course, we could dd it onto some USB
stick and boot it on a bare-metal device. However, it's much simpler
to first run it in a container for testing:
# systemd-nspawn -bi image.raw
And there you go: the image should boot up, and just work for you.
Now, let's make things more interesting. Let's still not use any of
the mkosi.$SOMETHING files around:
# mkosi -t raw_btrfs --bootable -o foobar.raw
# systemd-nspawn -bi foobar.raw
This is similar as the above, but we made three changes: it's no
longer GPT + ext4, but GPT + btrfs. Moreover, the system is made
bootable on UEFI systems, and finally, the output is now called
foobar.raw.
Because this system is bootable on UEFI systems, we can run it in KVM:
qemu-kvm -m 512 -smp 2 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -drive format=raw,file=foobar.raw
This will look very similar to the systemd-nspawn invocation, except
that this uses full VM virtualization rather than container
virtualization. (Note that the way to run a UEFI qemu/kvm instance
appears to change all the time and is different on the various
distributions. It's quite annoying, and I can't really tell you what
the right qemu command line is to make this work on your system.)
Of course, it's not all raw GPT disk images with mkosi. Let's try
a plain directory image:
# mkosi -d fedora -t directory -o quux
# systemd-nspawn -bD quux
Of course, if you generate the image as plain directory you can't boot it on bare-metal just like that, nor run it in a VM.
A more complex command line is the following:
# mkosi -d fedora -t raw_squashfs --checksum --xz --package=openssh-clients --package=emacs
In this mode we explicitly pick Fedora as the distribution to use, ask
mkosi to generate a compressed GPT image with a root squashfs,
compress the result with xz, and generate a SHA256SUMS file with
the hashes of the generated artifacts. The package will contain the
SSH client as well as everybody's favorite editor.
Now, let's make use of the various mkosi.$SOMETHING files. Let's
say we are working on some Automake-based project and want to make it
easy to generate a disk image off the development tree with the
version you are hacking on. Create a configuration file:
# cat > mkosi.default <<EOF
[Distribution]
Distribution=fedora
Release=24
[Output]
Format=raw_btrfs
Bootable=yes
[Packages]
# The packages to appear in both the build and the final image
Packages=openssh-clients httpd
# The packages to appear in the build image, but absent from the final image
BuildPackages=make gcc libcurl-devel
EOF
And let's add a build script:
# cat > mkosi.build <<EOF
#!/bin/sh
./autogen.sh
./configure --prefix=/usr
make -j `nproc`
make install
EOF
# chmod +x mkosi.build
And with all that in place we can now build our project into a disk image, simply by typing:
# mkosi
Let's try it out:
# systemd-nspawn -bi image.raw
Of course, if you do this you'll notice that building an image like this can be quite slow. And slow build times are actively hurtful to your productivity as a developer. Hence let's make things a bit faster. First, let's make use of a package cache shared between runs:
# mkdir mkosi.cache
Building images now should already be substantially faster (and
generate less network traffic) as the packages will now be downloaded
only once and reused. However, you'll notice that unpacking all those
packages and the rest of the work is still quite slow. But mkosi can
help you with that. Simply use mkosi's incremental build feature. In
this mode mkosi will make a copy of the build and final images
immediately before dropping in your build sources or artifacts, so
that building an image becomes a lot quicker: instead of always
starting totally from scratch a build will now reuse everything it can
reuse from a previous run, and immediately begin with building your
sources rather than the build image to build your sources in. To
enable the incremental build feature use -i:
# mkosi -i
Note that if you use this option, the package list is not updated anymore from your distribution's servers, as the cached copy is made after all packages are installed, and hence until you actually delete the cached copy the distribution's network servers aren't contacted again and no RPMs or DEBs are downloaded. This means the distribution you use becomes "frozen in time" this way. (Which might be a bad thing, but also a good thing, as it makes things kinda reproducible.)
Of course, if you run mkosi a couple of times you'll notice that it
won't overwrite the generated image when it already exists. You can
either delete the file yourself first (rm image.raw) or let mkosi
do it for you right before building a new image, with mkosi -f. You
can also tell mkosi to not only remove any such pre-existing images,
but also remove any cached copies of the incremental feature, by using
-f twice.
I wrote mkosi originally in order to test systemd, and quickly
generate a disk image of various distributions with the most current
systemd version from git, without all that affecting my host system. I
regularly use mkosi for that today, in incremental mode. The two
commands I use most in that context are:
# mkosi -if && systemd-nspawn -bi image.raw
And sometimes:
# mkosi -iff && systemd-nspawn -bi image.raw
The latter I use only if I want to regenerate everything based on the very newest set of RPMs provided by Fedora, instead of a cached snapshot of it.
BTW, the mkosi files for systemd are included in the systemd git
tree:
mkosi.default
and
mkosi.build. This
way, any developer who wants to quickly test something with current
systemd git, or wants to prepare a patch based on it and test it can
check out the systemd repository and simply run mkosi in it and a
few minutes later he has a bootable image he can test in
systemd-nspawn or KVM. casync has similar files:
mkosi.default,
mkosi.build.
As mentioned already, mkosi will generate dm-verity enabled
disk images if you ask for it. For that use the --verity switch on
the command line or Verity= setting in mkosi.default. Of course,
dm-verity implies that the root volume is read-only. In this mode
the top-level dm-verity hash will be placed along-side the output
disk image in a file named the same way, but with the .roothash
suffix. If the image is to be created bootable, the root hash is also
included on the kernel command line in the roothash= parameter,
which current systemd versions can use to both find and activate the
root partition in a dm-verity protected way. BTW: it's a good idea
to combine this dm-verity mode with the raw_squashfs image mode,
to generate a genuinely protected, compressed image suitable for
running in your IoT device.
As indicated above, mkosi can automatically create a check-sum
file SHA256SUMS for you (--checksum) covering all the files it
outputs (which could be the image file itself, a matching .nspawn
file using the mkosi.nspawn file mentioned above, as well as the
.roothash file for the dm-verity root hash.) It can then
optionally sign this with gpg (--sign). Note that systemd's
machinectl pull-tar and machinectl pull-raw command can download
these files and the SHA256SUMS file automatically and verify things
on download. With other words: what mkosi outputs is perfectly
ready for downloads using these two systemd commands.
As mentioned, mkosi is big on supporting UEFI SecureBoot. To
make use of that, place your X.509 key pair in two files
mkosi.secureboot.crt and mkosi.secureboot.key, and set
SecureBoot= or --secure-boot. If so, mkosi will sign the
kernel/initrd/kernel command line combination during the build. Of
course, if you use this mode, you should also use
Verity=/--verity=, otherwise the setup makes only partial
sense. Note that mkosi will not help you with actually enrolling
the keys you use in your UEFI BIOS.
mkosi has minimal support for GIT checkouts: when it recognizes
it is run in a git checkout and you use the mkosi.build script
stuff, the source tree will be copied into the build image, but will
all files excluded by .gitignore removed.
There's support for encryption in place. Use --encrypt= or
Encrypt=. Note that the UEFI ESP is never encrypted though, and the
root partition only if explicitly requested. The /home and /srv
partitions are unconditionally encrypted if that's enabled.
Images may be built with all documentation removed.
The password for the root user and additional kernel command line arguments may be configured for the image to generate.
Current mkosi requires Python 3.5, and has a number of dependencies,
listed in the
README. Most
notably you need a somewhat recent systemd version to make use of its
full feature set: systemd 233. Older versions are already packaged for
various distributions, but much of what I describe above is only
available in the most recent release mkosi 3.
The UEFI SecureBoot support requires sbsign which currently isn't
available in Fedora, but there's a
COPR.
It is my intention to continue turning mkosi into a tool suitable
for:
One of the biggest goals I have for the future is to teach mkosi and
systemd/sd-boot native support for A/B IoT style partition
setups. The idea is that the combination of systemd, casync and
mkosi provides generic building blocks for building secure,
auto-updating devices in a generic way from, even though all pieces
may be used individually, too.
Why are you reinventing the wheel again? This is exactly like
$SOMEOTHERPROJECT! — Well, to my knowledge there's no tool that
integrates this nicely with your project's development tree, and can
do dm-verity and UEFI SecureBoot and all that stuff for you. So
nope, I don't think this exactly like $SOMEOTHERPROJECT, thank you
very much.
What about creating MBR/DOS partition images? — That's really out of focus to me. This is an exercise in figuring out how generic OSes and devices in the future should be built and an attempt to commoditize OS image building. And no, the future doesn't speak MBR, sorry. That said, I'd be quite interested in adding support for booting on Raspberry Pi, possibly using a hybrid approach, i.e. using a GPT disk label, but arranging things in a way that the Raspberry Pi boot protocol (which is built around DOS partition tables), can still work.
Is this portable? — Well, depends what you mean by
portable. No, this tool runs on Linux only, and as it uses
systemd-nspawn during the build process it doesn't run on
non-systemd systems either. But then again, you should be able to
create images for any architecture you like with it, but of course if
you want the image bootable on bare-metal systems only systems doing
UEFI are supported (but systemd-nspawn should still work fine on
them).
Where can I get this stuff? — Try GitHub. And some distributions carry packaged versions, but I think none of them the current v3 yet.
Is this a systemd project? — Yes, it's hosted under the
systemd GitHub umbrella. And yes,
during run-time systemd-nspawn in a current version is required. But
no, the code-bases are separate otherwise, already because systemd
is a C project, and mkosi Python.
Requiring systemd 233 is a pretty steep requirement, no? —
Yes, but the feature we need kind of matters (systemd-nspawn's
--overlay= switch), and again, this isn't supposed to be a tool for
legacy systems.
Can I run the resulting images in LXC or Docker? — Humm, I am
not an LXC nor Docker guy. If you select directory or subvolume
as image type, LXC should be able to boot the generated images just
fine, but I didn't try. Last time I looked, Docker doesn't permit
running proper init systems as PID 1 inside the container, as they
define their own run-time without intention to emulate a proper
system. Hence, no I don't think it will work, at least not with an
unpatched Docker version. That said, again, don't ask me questions
about Docker, it's not precisely my area of expertise, and quite
frankly I am not a fan. To my knowledge neither LXC nor Docker are
able to run containers directly off GPT disk images, hence the
various raw_xyz image types are definitely not compatible with
either. That means if you want to generate a single raw disk image
that can be booted unmodified both in a container and on bare-metal,
then systemd-nspawn is the container manager to go for
(specifically, its -i/--image= switch).
Well, that's up to you really.
If you hack on some complex project and need a quick way to compile
and run your project on a specific current Linux distribution, then
mkosi is an excellent way to do that. Simply drop the mkosi.default
and mkosi.build files in your git tree and everything will be
easy. (And of course, as indicated above: if the project you are
hacking on happens to be called systemd or casync be aware that
those files are already part of the git tree — you can just use them.)
If you hack on some embedded or IoT device, then mkosi is a great
choice too, as it will make it reasonably easy to generate secure
images that are protected against offline modification, by using
dm-verity and UEFI SecureBoot.
If you are an administrator and need a nice way to build images for a
VM or systemd-nspawn container, or a portable service then mkosi
is an excellent choice too.
If you care about legacy computers, old distributions, non-systemd
init systems, old VM managers, Docker, … then no, mkosi is not for
you, but there are plenty of well-established alternatives around that
cover that nicely.
And never forget: mkosi is an Open Source project. We are happy to
accept your patches and other contributions.
Oh, and one unrelated last thing: don't forget to submit your talk
proposal
and/or buy a ticket for
All Systems Go! 2017 in Berlin — the
conference where things like systemd, casync and mkosi are
discussed, along with a variety of other Linux userspace projects used
for building systems.
We’d like to invite presentation proposals for All Systems Go! 2017!
All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward.
All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd.
All Systems Go! is a 2-day event with 2-3 talks happening in parallel. Full presentation slots are 30-45 minutes in length and lightning talk slots are 5-10 minutes.
We are now accepting submissions for presentation proposals. In particular, we are looking for sessions including, but not limited to, the following topics:
While our focus is definitely more on the user-space side of things, talks about kernel projects are welcome too, as long as they have a clear and direct relevance for user-space.
Please submit your proposals by September 3rd. Notification of acceptance will be sent out 1-2 weeks later.
To submit your proposal now please visit our CFP submission web site.
For further information about All Systems Go! visit our conference web site.
systemd.conf will not take place this year in lieu of All Systems Go!. All Systems Go! welcomes all projects that contribute to Linux user space, which, of course, includes systemd. Thus, anything you think was appropriate for submission to systemd.conf is also fitting for All Systems Go!