If you have already watched my presentation on systemd I gave at linux.conf.au 2011 then this video of my talk on the same topic which I have gave at FOSDEM 2011 in Brussels, Belgium will probably not be all new to you, but the questions from the audience (and hopefully my responses) might answer a question or two you might still have. So do watch it:

Hmm, seems p.g.o strips the video from the blog post. So either read the original blog story or watch it directly on YouTube.

Oh, and FOSDEM rocked, like every year!

I won't spare you the video of my talk about systemd at linux.conf.au 2011 in Brisbane, Australia last week:

Hmm, seems p.g.o strips the video from the blog post. So either read the original blog story or watch it directly on blip.tv.

LCA was fantastic and especially impressive given the circumstances of the recent floodings in Queensland. Really good conference, and congratulations to the organizers!

The FOSDEM organizers just published a brief interview with yours truly regarding the presentation about systemd I will be giving there on Sat. Feb. 5th, 3pm. If you come to Brussels make sure to drop by! And even if you don't have a look on the interview!

If you don't make it to Brussels, there are two more stops in my little systemd World Tour in the next weeks: today (Wed. Jan. 26th, 2:30pm) I will be speaking at linux.conf.au in Brisbane, Australia. And on Fri. Feb. 11th, 1:20pm I'll be speaking at the Red Hat Developer Conference in Brno, Czech Republic.

Here's the fourth installment of my ongoing series about systemd for administrators.

Killing Services

Killing a system daemon is easy, right? Or is it?

Sure, as long as your daemon persists only of a single process this might actually be somewhat true. You type killall rsyslogd and the syslog daemon is gone. However it is a bit dirty to do it like that given that this will kill all processes which happen to be called like this, including those an unlucky user might have named that way by accident. A slightly more correct version would be to read the .pid file, i.e. kill `cat /var/run/syslogd.pid`. That already gets us much further, but still, is this really what we want?

More often than not it actually isn't. Consider a service like Apache, or crond, or atd, which as part of their usual operation spawn child processes. Arbitrary, user configurable child processes, such as cron or at jobs, or CGI scripts, even full application servers. If you kill the main apache/crond/atd process this might or might not pull down the child processes too, and it's up to those processes whether they want to stay around or go down as well. Basically that means that terminating Apache might very well cause its CGI scripts to stay around, reassigned to be children of init, and difficult to track down.

systemd to the rescue: With systemctl kill you can easily send a signal to all processes of a service. Example:

# systemctl kill crond.service

This will ensure that SIGTERM is delivered to all processes of the crond service, not just the main process. Of course, you can also send a different signal if you wish. For example, if you are bad-ass you might want to go for SIGKILL right-away:

# systemctl kill -s SIGKILL crond.service

And there you go, the service will be brutally slaughtered in its entirety, regardless how many times it forked, whether it tried to escape supervision by double forking or fork bombing.

Sometimes all you need is to send a specific signal to the main process of a service, maybe because you want to trigger a reload via SIGHUP. Instead of going via the PID file, here's an easier way to do this:

# systemctl kill -s HUP --kill-who=main crond.service

So again, what is so new and fancy about killing services in systemd? Well, for the first time on Linux we can actually properly do that. Previous solutions were always depending on the daemons to actually cooperate to bring down everything they spawned if they themselves terminate. However, usually if you want to use SIGTERM or SIGKILL you are doing that because they actually do not cooperate properly with you.

How does this relate to systemctl stop? kill goes directly and sends a signal to every process in the group, however stop goes through the official configured way to shut down a service, i.e. invokes the stop command configured with ExecStop= in the service file. Usually stop should be sufficient. kill is the tougher version, for cases where you either don't want the official shutdown command of a service to run, or when the service is hosed and hung in other ways.

(It's up to you BTW to specify signal names with or without the SIG prefix on the -s switch. Both works.)

It's a bit surprising that we have come so far on Linux without even being able to properly kill services. systemd for the first time enables you to do this properly.

It has been a while since my last status update on systemd. Here's another short, incomprehensive status update on what we worked on for systemd since then.

• Fedora F15 (Rawhide) now includes a split up /etc/init.d/rc.sysinit (Bill Nottingham). This allows us to keep only a minimal compatibility set of shell scripts around, and boot otherwise a system without any shell scripts at all. In fact, shell scripts during early boot are only used in exceptional cases, i.e. when you enabled autoswapping (bad idea anyway), when a full SELinux relabel is necessary, during the first boot after initialization, if you have static kernel modules to load (which are not configured via the systemd-native way to do that), if you boot from a read-only NFS server, or when you rely on LVM/RAID/Multipath. If nothing of this applies to you can easily disable these parts of early boot and save several seconds on boot. How to do this I will describe in a later blog story.
• We have a fully C coded shutdown logic that kills all remaining processes, unmounts all remaining file systems, detaches all loop devices and DM volumes and does that in the right way to ensure that all these things are properly teared down even if they depend on each other in arbitrary ways. This is not only considerably faster then the traditional shell hackery for this, but also a lot safer, since we try to unmount/remount the remaining file systems with a little bit of brains. This feature is available via systemctl --force poweroff to the administrator. The --force controls whether the usual shutdown of all services is run or whether this is skipped and we immediately shall enter this final C shutdown logic. Using --force hence is a much safer replacement for the old /sbin/reboot -f and does not leave dirty file systems behind. (Thanks to Fabiano Fidencio has his colleagues from ProFUSION for this).
• systemd now includes a minmalistic readahead implementation, based on fanotify(), fadvise() and mincore(). It supports btrfs defragmentation and both SSD and HDD disks. While the effect on boots that are anyway fast (such as most stuff involving SSD) is minimal, slower and older machines benefit from this more substantially.
• We now control fsck and quota during early boot with a C tool that ensure maximum parallelization but properly implements the necessary high-level administration logic.
• Every service, every user and every user session now gets its own cgroup in the 'cpu' hierarchy thus creating better fairness between the logged in users and their sessions.
• We now provide /dev/log logging from early boot to late shutdown. If no syslog daemon is running the output is passed on to kmsg. As soon as a proper syslog daemon starts up the kmsg buffer is flushed to syslog, and hence we will have complete log coverage in syslog even for early boot.
• systemctl kill was introduced, an easy command to send a signal to all processes of a service. Expect a blog story with more details about this shortly.
• systemd gained the ability to load the SELinux policy if necessary, thus supporting non-initrd boots and initrd boots from the same binary with no duplicate work. This is in fact (and surprisingly) a first among Linux init systems.
• We now initialize and set the system locale inside PID 1 to be inherited by all services and users.
• systemd has native support for /etc/crypttab and can activate encrypted LUKS/dm-crypt disks both at boot-up and during runtime. A minimal password querying infrastructure is available, where multiple agents can be used to present the password to the user. During boot the password is queried either via Plymouth or directly on the console. If a system crypto disk is plugged in after boot you are queried for the password via a GNOME agent, or a wall(1) agent. Finally, while you run systemctl start (or a similar command) a minimal TTY password agent is available which asks you for passwords right-away if this is necessary. The password querying logic is very simple, additional agents can be implemented in a trivial amount of code (Yupp, KDE folks, you can add an agent for this, too). Note that the password querying logic in systemd is only for non-user passwords, i.e. passwords that have no relation to a specific user, but rather to specific hardware or system software. In future we hope to extend this so that this can be used to query the password of SSL certificates when Apache or other servers start.
• We offer a minimal interface that external projects can use to extend the dependency graph systemd manages. In fact, the cryptsetup logic mentioned above is implemented via this 'plugin'-like system. Since we did not want to add code that deals with cryptographic disks into the systemd process itself we introduced this interface (after all cryptographic volumes are not an essential feature of a minimal OS, and unncessary on most embedded uses; also the future might bring us STC which might make this at least partially obsolete). Simply by dropping a generator binary into /lib/systemd/system-generators which should write out systemd unit files into a temporary directory third-party packages may extend the systemd dependency tree dynamically. This could be useful for example to automatically create a systemd service for each KVM machine or LXC container. With that in place those containers/machines could be managed and supervised with the same tools as the usual system services.
• We integrated automatic clean-up of directories such as /tmp into the tmpfiles logic we already had in place that recreates files and directories on volatile file systems such as /var/run, /var/lock or /tmp.
• We now always measure and write to the log files the system startup time we measured, broken up into how many time was spent on the kernel, the initrd and the initialization of userspace.
• We now safely destroy all user session before going down. This is a feature long missing on Linux: since user processes were not killed until the very last moment the unhealthy situation that user code was running at a time where no other daemon was remaining was a normal part of shutdown.
• systemd now understands an 'extreme' form of disabling a service: if you symlink a service name in /etc/systemd/system to /dev/null then systemd will mark it as masked and completely refuse starting it, regardless if this is requested manually or automaticallly. Normally it should be sufficient to simply call systemctl disable to disable a service which still allows manual activation but no automatic activation. Masking a service goes one step further.
• There's now a simple condition syntax in places which allows skipping or enabling units depending on the existance of a file, whether a directory is empty or whether a kernel command line option is set.
• In addition to normal shutdowns for reboot, halt or poweroff we now similarly support a kexec reboot, that reboots the machine without going though the BIOS code again.
• We have bash completion support for systemctl. (Ran Benita)
• Andrew Edmunds contributed basic support to boot Ubuntu with systemd.
• Michael Biebl and Tollef Fog Heen have worked on the systemd integration into Debian to a level that it is now possible to boot a system without having the old initscripts packaged installed. For more details see the Debian Wiki. Michael even tested this integration on an Ubuntu Natty system and as it turns out this works almost equally well on Ubuntu already. If you are interesting in playing around with this, ping Michael.

And that's it for now. There's a lot of other stuff in the git commits, but most of it is smaller and I will it thus spare you.

We have come quite far in the last year. systemd is about a year old now, and we are now able to boot a system without legacy shell scripts remaining, something that appeared to be a task for the distant future.

All of this is available in systemd 13 and in F15/Rawhide as I type this. If you want to play around with this then consider installing Rawhide (it's fun!).

I really wonder why on earth the 27C3 accepted a nonsensical paper like this into their programme. So .. stupid. You read half the proposal and it's already kinda obvious that the presenter has no idea what he is talking of. Fundamental errors, obvious misinterpretations, outdated issues: this is just FUD.

And apparently this talk even is anonymous? Such a coward! FUDing around anonymously is acceptable at the CCC?

Last week LPC and GS 2010 took place in Cambridge, MA. Like the last years, LPC showed again that -- at least for me -- it is one of the most relevant Linux conferences in existence, if not the single most relevant one.

Here's a terse, incomprehensive report of the different discussions I took part in with various folks at the conference, in no particular order:

The Boot and Init track led by Kay Sievers (Suse) was a great success. We had exciting talks which I think helped quite a bit in clearing a few things up, and hopefully helps us in consolidating the full Linux boot process among all the components involved. We had talks covering everything from the BIOS boot, to initrds, graphical boot splashes and systemd. Kay Sievers and I spoke about systemd, also covering the state of it in the Fedora and openSUSE distributions. Gustavo Barbieri (ProFUSION, Gentoo) and Michael Biebl (Debian) gave interesting talks about systemd adoption in their respective distributions. I was particularly interested in the various statistics Michael showed about SysV/LSB init script usage in Debian, because this gives an idea how much work we have in front of us in the long run. A longer discussion about the future of initrds and the logic necessary to find the root file system on boot was quite enlightening. I think this track was helpful to increase the unification and consolidation of the way Linux systems boot up and are maintained during runtime.

Kay and I and some other folks sat down with Arjan van de Ven (Intel), to talk about the prospects of systemd in Meego. The discussions were very positive. In particular Arjan hat some great suggestions regarding use of the Simple Boot Flag in systemd (expect this in one of the next versions) and readahead. Before systemd can find adoption in Meego we'd have to add a short number of features to systemd first, most of them should be easy to add.

Similarly, I sat down with Martin Pitt and James Hunt (both Canonical) and discussed systemd in relation to Ubuntu. I think we managed to clear a lot of things up, and have a good chance to improve cooperation between Ubuntu and systemd in relation to APIs and maybe even more.

We talked to Thomas Gleixner regarding userspace notifications when the wallclock time jumps relative to the monotonic clock. This is important to systemd so that we can schedule calendar jobs similar to cron, but without having to wake up periodically to check whether the wallclock time changed relatively to the monotonic clock so that we can recalculate the next point in time a calendar event is triggered. There has been previous work in this area in the kernel world, but nothing got merged. Thomas' suggestion how to add this facility should be much easier than anything proposed so far.

I also tried to talk Andreas Grünbacher into supporting file system user extended attributes in various virtual file systems such as procfs, cgroupfs, sysfs and tmpfs. I hope I convinced him that this would be a good idea, since this would allow setting externally accessible attributes to all kinds of kernel objects, such as processes and devices. This would not only have uses in systemd (where we could easily store all meta information systemd needs to know about a service in the cgroupfs via xattrs, so that systemd could even crash or go away at any time and we still can read all runtime information necessary beyond mere cgrouping from the file system when systemd comes to live again) but also in the desktop environments, so that we could for example attach the human readable application name, an icon or a desktop file to the processes currently running, in a simple way where the data we attach follows the lifecycle of the process itself.

The Audio track went really well, too. I was particularly excited about Pierre-Louis Bossart's (Intel) plans regarding AC3 (and other codecs) support in PulseAudio, and the simplicity of his approach. Also great was hearing about Laurent Pinchart's project to expose audio and video device routing to userspace. Finally, I really enjoyed David Henningsson's and Luke Yelavich's (both Canonical) talk regarding tracking down audio bugs on Ubuntu. I was really impressed by the elaborate tools they created to test audio drivers on users machines. Pretty cool stuff. Maybe this can be extended into a test suite for driver writers, because the current approach for driver writers (i.e. "If PulseAudio works correctly, your driver is correct") doesn't really scale (although I like the idea and take it as a compliment...). I also liked the timechart profiling results Pierre showed me that he generated for PulseAudio. Seems PulseAudio is behaving quite nicely these days.

Together with Harald Hoyer I got a demo of David Zeuthen's disk assembly daemon (stc), which makes RAID/MD/LVM assembly more dynamic. Great stuff, and I think we convinced him to leave actual mounting of file systems to systemd instead of doing it himself.

Harald and I also hashed out a few things to make integration between dracut and systemd nicer (i.e. passing along profiling information between the two, and information regarding the root fsck).

I also hope I convinced Ray Strode to make Plymouth actively listen to udev for notifications about DRM devices, so that further synchronization between udev and plymouth won't be necessary, which both makes things more robust and a little bit faster.

Kay and I talked to Greg Kroah-Hartman regarding the brokeness of VT_WAITEVENT in kernel TTY layer, and discussed what to do about this. After returning from the US Kay now did the necessary hacking work to provide a minimal sysfs based solution that allows userspace query to which TTYs /dev/console and /dev/tty0 currently point, and get notifications when this changes. This should allow us to greatly simplify ConsoleKit and make it possible to add console-triggered activation to systemd (think: getty gets started the moment you switch to its virtual terminal, not already at boot).

I also spent some time discussing the upcoming deadline scheduling kernel logic with Dario, Dhaval and Tommaso regarding its possible use in PulseAudio. I believe deadline schedule is a useful tool to hand out real-time scheduling to applications securely. As an easy path to supporting deadline scheduling in PulseAudio I suggested patching RealtimeKit to optionally use deadline scheduling for its clients. This would magically teach PA (and other clients) to use deadline scheduling without further patching in the clients.

At GNOME Summit I sat down with Ryan Lortie and Will Thompson to discuss the the future of the D-Bus session bus and how we can move to a machine/user bus instead in a nice way. We managed to come to a nice agreement here, and this should enable us to introduce systemd for session management soonishly. Now we only need to convince the other folks having stakes in D-Bus that what we discussed is actually a good idea, expect more about this soon on dbus-devel. Ryan and I also hashed out our remaining differences regarding the exact semantics of XDG_RUNTIME_DIR, the result of which you can already see on the XDG mailing list. Ryan already did the GLib work to introduce XDG_RUNTIME_DIR and systemd already supports this inofficially since a few versions.

I quite appreciate how Michael Meeks quoted me in his final keynote. ;-)

There was a lot of other stuff going on at the conference, and what I wrote above is in no way complete. And of course, besides all the technical stuff, it was great meeting all the good Linux folks again, especially my colleagues from Red Hat.

I am still amazed how systemd is received so positively and with open arms all across the board. It's particularly amazing that systemd at this point in time has already been adopted by various companies in the automotive and aviation industry.

Later this week the Linux Plumbers Conference 2010 will take place at the Hyatt Regency in Cambridge.

Together with Mark Brown I'll be running the conference track about Audio, and I believe we managed to put together quite a nice schedule with various interesting talks covering many areas of what Audio on Linux is about.

I'll also be around at the Boot and Init Systems track which Kay Sievers is running. Together with Kay I'll do a session about systemd, everybody's favourite system and session manager. We also managed to convince a number of distribution maintainers of systemd to do short presentations about the state of systemd adoption in their respective distributions: Michael Biebl from Debian, Gustavo Barbieri from Gentoo, Kay for openSUSE and yours truly for Fedora.

Because there never can be enough systemd coverage at a conference I'll do another talk about systemd, in Vincent Untz' Desktop track, this time focussing less on how to boot and maintain a system, but more on doing the same for desktop sessions, in particular GNOME.

I'll also stick around for the the first two days of the GNOME Boston Summit.

See you in Cambridge!

I just submitted my paper^[1] for FOSS.in 2010 in Bangalore/India. Don't forget to submit yours! The CFP closes on 10th of October. That's this Sunday! Hurry up, before it is too late!

FOSS.in is one of the most amazing Free Software conferences this world has to offer (hey, and I think I can say that because I have presented at quite a few). A dedicated audience, flawless organization, magic hospitality, and all this in incredible India! Both the technical programme and everything around it are impressive. Which other conference can offer you a concert of one of India's greatest acts as part of the schedule? Which other international conference host city can be such a positive attack on your senses as Bangalore (see that endless sea of flowers below)? And where else do they serve pure silver as part of the conference catering?

Read the CFP! Or, go straight to submitting a paper.

Footnotes

^[1] About systemd.

Here's the third installment of my ongoing series about systemd for administrators.

How Do I Convert A SysV Init Script Into A systemd Service File?

Traditionally, Unix and Linux services (daemons) are started via SysV init scripts. These are Bourne Shell scripts, usually residing in a directory such as /etc/rc.d/init.d/ which when called with one of a few standardized arguments (verbs) such as start, stop or restart controls, i.e. starts, stops or restarts the service in question. For starts this usually involves invoking the daemon binary, which then forks a background process (more precisely daemonizes). Shell scripts tend to be slow, needlessly hard to read, very verbose and fragile. Although they are immensly flexible (after all, they are just code) some things are very hard to do properly with shell scripts, such as ordering parallized execution, correctly supervising processes or just configuring execution contexts in all detail. systemd provides compatibility with these shell scripts, but due to the shortcomings pointed out it is recommended to install native systemd service files for all daemons installed. Also, in contrast to SysV init scripts which have to be adjusted to the distribution systemd service files are compatible with any kind of distribution running systemd (which become more and more these days...). What follows is a terse guide how to take a SysV init script and translate it into a native systemd service file. Ideally, upstream projects should ship and install systemd service files in their tarballs. If you have successfully converted a SysV script according to the guidelines it might hence be a good idea to submit the file as patch to upstream. How to prepare a patch like that will be discussed in a later installment, suffice to say at this point that the daemon(7) manual page shipping with systemd contains a lot of useful information regarding this.

So, let's jump right in. As an example we'll convert the init script of the ABRT daemon into a systemd service file. ABRT is a standard component of every Fedora install, and is an acronym for Automatic Bug Reporting Tool, which pretty much describes what it does, i.e. it is a service for collecting crash dumps. Its SysV script I have uploaded here.

The first step when converting such a script is to read it (surprise surprise!) and distill the useful information from the usually pretty long script. In almost all cases the script consists of mostly boilerplate code that is identical or at least very similar in all init scripts, and usually copied and pasted from one to the other. So, let's extract the interesting information from the script linked above:

• A description string for the service is "Daemon to detect crashing apps". As it turns out, the header comments include a redundant number of description strings, some of them describing less the actual service but the init script to start it. systemd services include a description too, and it should describe the service and not the service file.
• The LSB header^[1] contains dependency information. systemd due to its design around socket-based activation usually needs no (or very little) manually configured dependencies. (For details regarding socket activation see the original announcement blog post.) In this case the dependency on $syslog (which encodes that abrtd requires a syslog daemon), is the only valuable information. While the header lists another dependency ($local_fs) this one is redundant with systemd as normal system services are always started with all local file systems available.
• The LSB header suggests that this service should be started in runlevels 3 (multi-user) and 5 (graphical).
• The daemon binary is /usr/sbin/abrtd

And that's already it. The entire remaining content of this 115-line shell script is simply boilerplate or otherwise redundant code: code that deals with synchronizing and serializing startup (i.e. the code regarding lock files) or that outputs status messages (i.e. the code calling echo), or simply parsing of the verbs (i.e. the big case block).

From the information extracted above we can now write our systemd service file:

[Unit]
Description=Daemon to detect crashing apps
After=syslog.target

[Service]
ExecStart=/usr/sbin/abrtd
Type=forking

[Install]
WantedBy=multi-user.target

A little explanation of the contents of this file: The [Unit] section contains generic information about the service. systemd not only manages system services, but also devices, mount points, timer, and other components of the system. The generic term for all these objects in systemd is a unit, and the [Unit] section encodes information about it that might be applicable not only to services but also in to the other unit types systemd maintains. In this case we set the following unit settings: we set the description string and configure that the daemon shall be started after Syslog^[2], similar to what is encoded in the LSB header of the original init script. For this Syslog dependency we create a dependency of type After= on a systemd unit syslog.target. The latter is a special target unit in systemd and is the standardized name to pull in a syslog implementation. For more information about these standardized names see the systemd.special(7). Note that a dependency of type After= only encodes the suggested ordering, but does not actually cause syslog to be started when abrtd is -- and this is exactly what we want, since abrtd actually works fine even without syslog being around. However, if both are started (and usually they are) then the order in which they are is controlled with this dependency.

The next section is [Service] which encodes information about the service itself. It contains all those settings that apply only to services, and not the other kinds of units systemd maintains (mount points, devices, timers, ...). Two settings are used here: ExecStart= takes the path to the binary to execute when the service shall be started up. And with Type= we configure how the service notifies the init system that it finished starting up. Since traditional Unix daemons do this by returning to the parent process after having forked off and initialized the background daemon we set the type to forking here. That tells systemd to wait until the start-up binary returns and then consider the processes still running afterwards the daemon processes.

The final section is [Install]. It encodes information about how the suggested installation should look like, i.e. under which circumstances and by which triggers the service shall be started. In this case we simply say that this service shall be started when the multi-user.target unit is activated. This is a special unit (see above) that basically takes the role of the classic SysV Runlevel 3^[3]. The setting WantedBy= has little effect on the daemon during runtime. It is only read by the systemctl enable command, which is the recommended way to enable a service in systemd. This command will simply ensure that our little service gets automatically activated as soon as multi-user.target is requested, which it is on all normal boots^[4].

And that's it. Now we already have a minimal working systemd service file. To test it we copy it to /etc/systemd/system/abrtd.service and invoke systemctl daemon-reload. This will make systemd take notice of it, and now we can start the service with it: systemctl start abrtd.service. We can verify the status via systemctl status abrtd.service. And we can stop it again via systemctl stop abrtd.service. Finally, we can enable it, so that it is activated by default on future boots with systemctl enable abrtd.service.

The service file above, while sufficient and basically a 1:1 translation (feature- and otherwise) of the SysV init script still has room for improvement. Here it is a little bit updated:

[Unit]
Description=ABRT Automated Bug Reporting Tool
After=syslog.target

[Service]
Type=dbus
BusName=com.redhat.abrt
ExecStart=/usr/sbin/abrtd -d -s

[Install]
WantedBy=multi-user.target

So, what did we change? Two things: we improved the description string a bit. More importantly however, we changed the type of the service to dbus and configured the D-Bus bus name of the service. Why did we do this? As mentioned classic SysV services daemonize after startup, which usually involves double forking and detaching from any terminal. While this is useful and necessary when daemons are invoked via a script, this is unnecessary (and slow) as well as counterproductive when a proper process babysitter such as systemd is used. The reason for that is that the forked off daemon process usually has little relation to the original process started by systemd (after all the daemonizing scheme's whole idea is to remove this relation), and hence it is difficult for systemd to figure out after the fork is finished which process belonging to the service is actually the main process and which processes might just be auxiliary. But that information is crucial to implement advanced babysitting, i.e. supervising the process, automatic respawning on abnormal termination, collectig crash and exit code information and suchlike. In order to make it easier for systemd to figure out the main process of the daemon we changed the service type to dbus. The semantics of this service type are appropriate for all services that take a name on the D-Bus system bus as last step of their initialization^[5]. ABRT is one of those. With this setting systemd will spawn the ABRT process, which will no longer fork (this is configured via the -d -s switches to the daemon), and systemd will consider the service fully started up as soon as com.redhat.abrt appears on the bus. This way the process spawned by systemd is the main process of the daemon, systemd has a reliable way to figure out when the daemon is fully started up and systemd can easily supervise it.

And that's all there is to it. We have a simple systemd service file now that encodes in 10 lines more information than the original SysV init script encoded in 115. And even now there's a lot of room left for further improvement utilizing more features systemd offers. For example, we could set Restart=restart-always to tell systemd to automatically restart this service when it dies. Or, we could use OOMScoreAdjust=-500 to ask the kernel to please leave this process around when the OOM killer wreaks havoc. Or, we could use CPUSchedulingPolicy=idle to ensure that abrtd processes crash dumps in background only, always allowing the kernel to give preference to whatever else might be running and needing CPU time.

For more information about the configuration options mentioned here, see the respective man pages systemd.unit(5), systemd.service(5), systemd.exec(5). Or, browse all of systemd's man pages.

Of course, not all SysV scripts are as easy to convert as this one. But gladly, as it turns out the vast majority actually are.

That's it for today, come back soon for the next installment in our series.

Footnotes

[1] The LSB header of init scripts is a convention of including meta data about the service in comment blocks at the top of SysV init scripts and is defined by the Linux Standard Base. This was intended to standardize init scripts between distributions. While most distributions have adopted this scheme, the handling of the headers varies greatly between the distributions, and in fact still makes it necessary to adjust init scripts for every distribution. As such the LSB spec never kept the promise it made.

[2] Strictly speaking, this dependency does not even have to be encoded here, as it is redundant in a system where the Syslog daemon is socket activatable. Modern syslog systems (for example rsyslog v5) have been patched upstream to be socket-activatable. If such a init system is used configuration of the After=syslog.target dependency is redundant and implicit. However, to maintain compatibility with syslog services that have not been updated we include this dependency here.

[3] At least how it used to be defined on Fedora.

[4] Note that in systemd the graphical bootup (graphical.target, taking the role of SysV runlevel 5) is an implicit superset of the console-only bootup (multi-user.target, i.e. like runlevel 3). That means hooking a service into the latter will also hook it into the former.

[5] Actually the majority of services of the default Fedora install now take a name on the bus after startup.