Category: projects
Here's the second installment of my ongoing series about systemd for administrators.
Which Service Owns Which Processes?
On most Linux systems the number of processes that are running by default is substantial. Knowing which process does what and where it belongs to becomes increasingly difficult. Some services even maintain a couple of worker processes which clutter the "ps" output with many additional processes that are often not easy to recognize. This is further complicated if daemons spawn arbitrary 3rd-party processes, as Apache does with CGI processes, or cron does with user jobs.
A slight remedy for this is often the process inheritance tree, as shown by "ps xaf". However this is usually not reliable, as processes whose parents die get reparented to PID 1, and hence all information about inheritance gets lost. If a process "double forks" it hence loses its relationships to the processes that started it. (This actually is supposed to be a feature and is relied on for the traditional Unix daemonizing logic.) Furthermore processes can freely change their names with PR_SETNAME or by patching argv[0], thus making it harder to recognize them. In fact they can play hide-and-seek with the administrator pretty nicely this way.
In systemd we place every process that is spawned in a control group named after its service. Control groups (or cgroups) at their most basic are simply groups of processes that can be arranged in a hierarchy and labelled individually. When processes spawn other processes these children are automatically made members of the parents cgroup. Leaving a cgroup is not possible for unprivileged processes. Thus, cgroups can be used as an effective way to label processes after the service they belong to and be sure that the service cannot escape from the label, regardless how often it forks or renames itself. Furthermore this can be used to safely kill a service and all processes it created, again with no chance of escaping.
In today's installment I want to introduce you to two commands you may use to relate systemd services and processes. The first one, is the well known ps command which has been updated to show cgroup information along the other process details. And this is how it looks:
$ ps xawf -eo pid,user,cgroup,args
PID USER CGROUP COMMAND
2 root - [kthreadd]
3 root - \_ [ksoftirqd/0]
[...]
4281 root - \_ [flush-8:0]
1 root name=systemd:/systemd-1 /sbin/init
455 root name=systemd:/systemd-1/sysinit.service /sbin/udevd -d
28188 root name=systemd:/systemd-1/sysinit.service \_ /sbin/udevd -d
28191 root name=systemd:/systemd-1/sysinit.service \_ /sbin/udevd -d
1096 dbus name=systemd:/systemd-1/dbus.service /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
1131 root name=systemd:/systemd-1/auditd.service auditd
1133 root name=systemd:/systemd-1/auditd.service \_ /sbin/audispd
1135 root name=systemd:/systemd-1/auditd.service \_ /usr/sbin/sedispatch
1171 root name=systemd:/systemd-1/NetworkManager.service /usr/sbin/NetworkManager --no-daemon
4028 root name=systemd:/systemd-1/NetworkManager.service \_ /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-wlan0.pid -lf /var/lib/dhclient/dhclient-7d32a784-ede9-4cf6-9ee3-60edc0bce5ff-wlan0.lease -
1175 avahi name=systemd:/systemd-1/avahi-daemon.service avahi-daemon: running [epsilon.local]
1194 avahi name=systemd:/systemd-1/avahi-daemon.service \_ avahi-daemon: chroot helper
1193 root name=systemd:/systemd-1/rsyslog.service /sbin/rsyslogd -c 4
1195 root name=systemd:/systemd-1/cups.service cupsd -C /etc/cups/cupsd.conf
1207 root name=systemd:/systemd-1/mdmonitor.service mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
1210 root name=systemd:/systemd-1/irqbalance.service irqbalance
1216 root name=systemd:/systemd-1/dbus.service /usr/sbin/modem-manager
1219 root name=systemd:/systemd-1/dbus.service /usr/libexec/polkit-1/polkitd
1242 root name=systemd:/systemd-1/dbus.service /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
1249 68 name=systemd:/systemd-1/haldaemon.service hald
1250 root name=systemd:/systemd-1/haldaemon.service \_ hald-runner
1273 root name=systemd:/systemd-1/haldaemon.service \_ hald-addon-input: Listening on /dev/input/event3 /dev/input/event9 /dev/input/event1 /dev/input/event7 /dev/input/event2 /dev/input/event0 /dev/input/event8
1275 root name=systemd:/systemd-1/haldaemon.service \_ /usr/libexec/hald-addon-rfkill-killswitch
1284 root name=systemd:/systemd-1/haldaemon.service \_ /usr/libexec/hald-addon-leds
1285 root name=systemd:/systemd-1/haldaemon.service \_ /usr/libexec/hald-addon-generic-backlight
1287 68 name=systemd:/systemd-1/haldaemon.service \_ /usr/libexec/hald-addon-acpi
1317 root name=systemd:/systemd-1/abrtd.service /usr/sbin/abrtd -d -s
1332 root name=systemd:/systemd-1/getty@.service/tty2 /sbin/mingetty tty2
1339 root name=systemd:/systemd-1/getty@.service/tty3 /sbin/mingetty tty3
1342 root name=systemd:/systemd-1/getty@.service/tty5 /sbin/mingetty tty5
1343 root name=systemd:/systemd-1/getty@.service/tty4 /sbin/mingetty tty4
1344 root name=systemd:/systemd-1/crond.service crond
1346 root name=systemd:/systemd-1/getty@.service/tty6 /sbin/mingetty tty6
1362 root name=systemd:/systemd-1/sshd.service /usr/sbin/sshd
1376 root name=systemd:/systemd-1/prefdm.service /usr/sbin/gdm-binary -nodaemon
1391 root name=systemd:/systemd-1/prefdm.service \_ /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1 --force-active-vt
1394 root name=systemd:/systemd-1/prefdm.service \_ /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-f2KUOh/database -nolisten tcp vt1
1495 root name=systemd:/user/lennart/1 \_ pam: gdm-password
1521 lennart name=systemd:/user/lennart/1 \_ gnome-session
1621 lennart name=systemd:/user/lennart/1 \_ metacity
1635 lennart name=systemd:/user/lennart/1 \_ gnome-panel
1638 lennart name=systemd:/user/lennart/1 \_ nautilus
1640 lennart name=systemd:/user/lennart/1 \_ /usr/libexec/polkit-gnome-authentication-agent-1
1641 lennart name=systemd:/user/lennart/1 \_ /usr/bin/seapplet
1644 lennart name=systemd:/user/lennart/1 \_ gnome-volume-control-applet
1646 lennart name=systemd:/user/lennart/1 \_ /usr/sbin/restorecond -u
1652 lennart name=systemd:/user/lennart/1 \_ /usr/bin/devilspie
1662 lennart name=systemd:/user/lennart/1 \_ nm-applet --sm-disable
1664 lennart name=systemd:/user/lennart/1 \_ gnome-power-manager
1665 lennart name=systemd:/user/lennart/1 \_ /usr/libexec/gdu-notification-daemon
1670 lennart name=systemd:/user/lennart/1 \_ /usr/libexec/evolution/2.32/evolution-alarm-notify
1672 lennart name=systemd:/user/lennart/1 \_ /usr/bin/python /usr/share/system-config-printer/applet.py
1674 lennart name=systemd:/user/lennart/1 \_ /usr/lib64/deja-dup/deja-dup-monitor
1675 lennart name=systemd:/user/lennart/1 \_ abrt-applet
1677 lennart name=systemd:/user/lennart/1 \_ bluetooth-applet
1678 lennart name=systemd:/user/lennart/1 \_ gpk-update-icon
1408 root name=systemd:/systemd-1/console-kit-daemon.service /usr/sbin/console-kit-daemon --no-daemon
1419 gdm name=systemd:/systemd-1/prefdm.service /usr/bin/dbus-launch --exit-with-session
1453 root name=systemd:/systemd-1/dbus.service /usr/libexec/upowerd
1473 rtkit name=systemd:/systemd-1/rtkit-daemon.service /usr/libexec/rtkit-daemon
1496 root name=systemd:/systemd-1/accounts-daemon.service /usr/libexec/accounts-daemon
1499 root name=systemd:/systemd-1/systemd-logger.service /lib/systemd/systemd-logger
1511 lennart name=systemd:/systemd-1/prefdm.service /usr/bin/gnome-keyring-daemon --daemonize --login
1534 lennart name=systemd:/user/lennart/1 dbus-launch --sh-syntax --exit-with-session
1535 lennart name=systemd:/user/lennart/1 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
1603 lennart name=systemd:/user/lennart/1 /usr/libexec/gconfd-2
1612 lennart name=systemd:/user/lennart/1 /usr/libexec/gnome-settings-daemon
1615 lennart name=systemd:/user/lennart/1 /usr/libexec/gvfsd
1626 lennart name=systemd:/user/lennart/1 /usr/libexec//gvfs-fuse-daemon /home/lennart/.gvfs
1634 lennart name=systemd:/user/lennart/1 /usr/bin/pulseaudio --start --log-target=syslog
1649 lennart name=systemd:/user/lennart/1 \_ /usr/libexec/pulse/gconf-helper
1645 lennart name=systemd:/user/lennart/1 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=24
1668 lennart name=systemd:/user/lennart/1 /usr/libexec/im-settings-daemon
1701 lennart name=systemd:/user/lennart/1 /usr/libexec/gvfs-gdu-volume-monitor
1707 lennart name=systemd:/user/lennart/1 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
1725 lennart name=systemd:/user/lennart/1 /usr/libexec/clock-applet
1727 lennart name=systemd:/user/lennart/1 /usr/libexec/wnck-applet
1729 lennart name=systemd:/user/lennart/1 /usr/libexec/notification-area-applet
1733 root name=systemd:/systemd-1/dbus.service /usr/libexec/udisks-daemon
1747 root name=systemd:/systemd-1/dbus.service \_ udisks-daemon: polling /dev/sr0
1759 lennart name=systemd:/user/lennart/1 gnome-screensaver
1780 lennart name=systemd:/user/lennart/1 /usr/libexec/gvfsd-trash --spawner :1.9 /org/gtk/gvfs/exec_spaw/0
1864 lennart name=systemd:/user/lennart/1 /usr/libexec/gvfs-afc-volume-monitor
1874 lennart name=systemd:/user/lennart/1 /usr/libexec/gconf-im-settings-daemon
1903 lennart name=systemd:/user/lennart/1 /usr/libexec/gvfsd-burn --spawner :1.9 /org/gtk/gvfs/exec_spaw/1
1909 lennart name=systemd:/user/lennart/1 gnome-terminal
1913 lennart name=systemd:/user/lennart/1 \_ gnome-pty-helper
1914 lennart name=systemd:/user/lennart/1 \_ bash
29231 lennart name=systemd:/user/lennart/1 | \_ ssh tango
2221 lennart name=systemd:/user/lennart/1 \_ bash
4193 lennart name=systemd:/user/lennart/1 | \_ ssh tango
2461 lennart name=systemd:/user/lennart/1 \_ bash
29219 lennart name=systemd:/user/lennart/1 | \_ emacs systemd-for-admins-1.txt
15113 lennart name=systemd:/user/lennart/1 \_ bash
27251 lennart name=systemd:/user/lennart/1 \_ empathy
29504 lennart name=systemd:/user/lennart/1 \_ ps xawf -eo pid,user,cgroup,args
1968 lennart name=systemd:/user/lennart/1 ssh-agent
1994 lennart name=systemd:/user/lennart/1 gpg-agent --daemon --write-env-file
18679 lennart name=systemd:/user/lennart/1 /bin/sh /usr/lib64/firefox-3.6/run-mozilla.sh /usr/lib64/firefox-3.6/firefox
18741 lennart name=systemd:/user/lennart/1 \_ /usr/lib64/firefox-3.6/firefox
28900 lennart name=systemd:/user/lennart/1 \_ /usr/lib64/nspluginwrapper/npviewer.bin --plugin /usr/lib64/mozilla/plugins/libflashplayer.so --connection /org/wrapper/NSPlugins/libflashplayer.so/18741-6
4016 root name=systemd:/systemd-1/sysinit.service /usr/sbin/bluetoothd --udev
4094 smmsp name=systemd:/systemd-1/sendmail.service sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
4096 root name=systemd:/systemd-1/sendmail.service sendmail: accepting connections
4112 ntp name=systemd:/systemd-1/ntpd.service /usr/sbin/ntpd -n -u ntp:ntp -g
27262 lennart name=systemd:/user/lennart/1 /usr/libexec/mission-control-5
27265 lennart name=systemd:/user/lennart/1 /usr/libexec/telepathy-haze
27268 lennart name=systemd:/user/lennart/1 /usr/libexec/telepathy-logger
27270 lennart name=systemd:/user/lennart/1 /usr/libexec/dconf-service
27280 lennart name=systemd:/user/lennart/1 /usr/libexec/notification-daemon
27284 lennart name=systemd:/user/lennart/1 /usr/libexec/telepathy-gabble
27285 lennart name=systemd:/user/lennart/1 /usr/libexec/telepathy-salut
27297 lennart name=systemd:/user/lennart/1 /usr/libexec/geoclue-yahoo
(Note that this output is shortened, I have removed most of the kernel threads here, since they are not relevant in the context of this blog story)
In the third column you see the cgroup systemd assigned to each process. You'll find that the udev processes are in the name=systemd:/systemd-1/sysinit.service cgroup, which is where systemd places all processes started by the sysinit.service service, which covers early boot.
My personal recommendation is to set the shell alias psc to the ps command line shown above:
alias psc='ps xawf -eo pid,user,cgroup,args'
With this service information of processes is just four keypresses away!
A different way to present the same information is the systemd-cgls tool we ship with systemd. It shows the cgroup hierarchy in a pretty tree. Its output looks like this:
$ systemd-cgls
+ 2 [kthreadd]
[...]
+ 4281 [flush-8:0]
+ user
| \ lennart
| \ 1
| + 1495 pam: gdm-password
| + 1521 gnome-session
| + 1534 dbus-launch --sh-syntax --exit-with-session
| + 1535 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
| + 1603 /usr/libexec/gconfd-2
| + 1612 /usr/libexec/gnome-settings-daemon
| + 1615 /ushr/libexec/gvfsd
| + 1621 metacity
| + 1626 /usr/libexec//gvfs-fuse-daemon /home/lennart/.gvfs
| + 1634 /usr/bin/pulseaudio --start --log-target=syslog
| + 1635 gnome-panel
| + 1638 nautilus
| + 1640 /usr/libexec/polkit-gnome-authentication-agent-1
| + 1641 /usr/bin/seapplet
| + 1644 gnome-volume-control-applet
| + 1645 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=24
| + 1646 /usr/sbin/restorecond -u
| + 1649 /usr/libexec/pulse/gconf-helper
| + 1652 /usr/bin/devilspie
| + 1662 nm-applet --sm-disable
| + 1664 gnome-power-manager
| + 1665 /usr/libexec/gdu-notification-daemon
| + 1668 /usr/libexec/im-settings-daemon
| + 1670 /usr/libexec/evolution/2.32/evolution-alarm-notify
| + 1672 /usr/bin/python /usr/share/system-config-printer/applet.py
| + 1674 /usr/lib64/deja-dup/deja-dup-monitor
| + 1675 abrt-applet
| + 1677 bluetooth-applet
| + 1678 gpk-update-icon
| + 1701 /usr/libexec/gvfs-gdu-volume-monitor
| + 1707 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
| + 1725 /usr/libexec/clock-applet
| + 1727 /usr/libexec/wnck-applet
| + 1729 /usr/libexec/notification-area-applet
| + 1759 gnome-screensaver
| + 1780 /usr/libexec/gvfsd-trash --spawner :1.9 /org/gtk/gvfs/exec_spaw/0
| + 1864 /usr/libexec/gvfs-afc-volume-monitor
| + 1874 /usr/libexec/gconf-im-settings-daemon
| + 1882 /usr/libexec/gvfs-gphoto2-volume-monitor
| + 1903 /usr/libexec/gvfsd-burn --spawner :1.9 /org/gtk/gvfs/exec_spaw/1
| + 1909 gnome-terminal
| + 1913 gnome-pty-helper
| + 1914 bash
| + 1968 ssh-agent
| + 1994 gpg-agent --daemon --write-env-file
| + 2221 bash
| + 2461 bash
| + 4193 ssh tango
| + 15113 bash
| + 18679 /bin/sh /usr/lib64/firefox-3.6/run-mozilla.sh /usr/lib64/firefox-3.6/firefox
| + 18741 /usr/lib64/firefox-3.6/firefox
| + 27251 empathy
| + 27262 /usr/libexec/mission-control-5
| + 27265 /usr/libexec/telepathy-haze
| + 27268 /usr/libexec/telepathy-logger
| + 27270 /usr/libexec/dconf-service
| + 27280 /usr/libexec/notification-daemon
| + 27284 /usr/libexec/telepathy-gabble
| + 27285 /usr/libexec/telepathy-salut
| + 27297 /usr/libexec/geoclue-yahoo
| + 28900 /usr/lib64/nspluginwrapper/npviewer.bin --plugin /usr/lib64/mozilla/plugins/libflashplayer.so --connection /org/wrapper/NSPlugins/libflashplayer.so/18741-6
| + 29219 emacs systemd-for-admins-1.txt
| + 29231 ssh tango
| \ 29519 systemd-cgls
\ systemd-1
+ 1 /sbin/init
+ ntpd.service
| \ 4112 /usr/sbin/ntpd -n -u ntp:ntp -g
+ systemd-logger.service
| \ 1499 /lib/systemd/systemd-logger
+ accounts-daemon.service
| \ 1496 /usr/libexec/accounts-daemon
+ rtkit-daemon.service
| \ 1473 /usr/libexec/rtkit-daemon
+ console-kit-daemon.service
| \ 1408 /usr/sbin/console-kit-daemon --no-daemon
+ prefdm.service
| + 1376 /usr/sbin/gdm-binary -nodaemon
| + 1391 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1 --force-active-vt
| + 1394 /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-f2KUOh/database -nolisten tcp vt1
| + 1419 /usr/bin/dbus-launch --exit-with-session
| \ 1511 /usr/bin/gnome-keyring-daemon --daemonize --login
+ getty@.service
| + tty6
| | \ 1346 /sbin/mingetty tty6
| + tty4
| | \ 1343 /sbin/mingetty tty4
| + tty5
| | \ 1342 /sbin/mingetty tty5
| + tty3
| | \ 1339 /sbin/mingetty tty3
| \ tty2
| \ 1332 /sbin/mingetty tty2
+ abrtd.service
| \ 1317 /usr/sbin/abrtd -d -s
+ crond.service
| \ 1344 crond
+ sshd.service
| \ 1362 /usr/sbin/sshd
+ sendmail.service
| + 4094 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
| \ 4096 sendmail: accepting connections
+ haldaemon.service
| + 1249 hald
| + 1250 hald-runner
| + 1273 hald-addon-input: Listening on /dev/input/event3 /dev/input/event9 /dev/input/event1 /dev/input/event7 /dev/input/event2 /dev/input/event0 /dev/input/event8
| + 1275 /usr/libexec/hald-addon-rfkill-killswitch
| + 1284 /usr/libexec/hald-addon-leds
| + 1285 /usr/libexec/hald-addon-generic-backlight
| \ 1287 /usr/libexec/hald-addon-acpi
+ irqbalance.service
| \ 1210 irqbalance
+ avahi-daemon.service
| + 1175 avahi-daemon: running [epsilon.local]
+ NetworkManager.service
| + 1171 /usr/sbin/NetworkManager --no-daemon
| \ 4028 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-wlan0.pid -lf /var/lib/dhclient/dhclient-7d32a784-ede9-4cf6-9ee3-60edc0bce5ff-wlan0.lease -cf /var/run/nm-dhclient-wlan0.conf wlan0
+ rsyslog.service
| \ 1193 /sbin/rsyslogd -c 4
+ mdmonitor.service
| \ 1207 mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
+ cups.service
| \ 1195 cupsd -C /etc/cups/cupsd.conf
+ auditd.service
| + 1131 auditd
| + 1133 /sbin/audispd
| \ 1135 /usr/sbin/sedispatch
+ dbus.service
| + 1096 /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
| + 1216 /usr/sbin/modem-manager
| + 1219 /usr/libexec/polkit-1/polkitd
| + 1242 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
| + 1453 /usr/libexec/upowerd
| + 1733 /usr/libexec/udisks-daemon
| + 1747 udisks-daemon: polling /dev/sr0
| \ 29509 /usr/libexec/packagekitd
+ dev-mqueue.mount
+ dev-hugepages.mount
\ sysinit.service
+ 455 /sbin/udevd -d
+ 4016 /usr/sbin/bluetoothd --udev
+ 28188 /sbin/udevd -d
\ 28191 /sbin/udevd -d
(This too is shortened, the same way)
As you can see, this command shows the processes by their cgroup and hence service, as systemd labels the cgroups after the services. For example, you can easily see that the auditing service auditd.service spawns three individual processes, auditd, audisp and sedispatch.
If you look closely you will notice that a number of processes have been assigned to the cgroup /user/1. At this point let's simply leave it at that systemd not only maintains services in cgroups, but user session processes as well. In a later installment we'll discuss in more detail what this about.
So much for now, come back soon for the next installment!
Google just published a video interview with yours truly. Watch it! Oh, and Vincent, I even put on a red shirt for you!
It has been a while since my original announcement of systemd. Here's a little status update, on what happened since then. For simplicity's sake I'll just list here what we worked on in a bulleted list, with no particular order and without trying to cover this comprehensively:
- systemd has been accepted as Feature for Fedora 14, and as it looks right now everything worked out nicely and we'll ship F14 with systemd as init system.
- We added a number of additional unit types: .timer for cron-style timer-based activation of services, .swap exposes swap files and partitions the same way we handle mount points, and .path can be used to activate units dependending on the existance/creation of files or fill status of spool directories.
- We hooked systemd up to SELinux: systemd is now capabale of properly labelling directories, sockets and FIFOs it creates according to the SELinux policy for the services we maintain.
- We hooked systemd up to the Linux auditing subsystem: as first init system at all systemd now generates auditing records for all services it starts/stops, including their failure status.
- We hooked systemd up to TCP wrappers, for all socket connections it accepts.
- We hooked systemd up to PAM, so that optionally, when systemd runs a service as a different user it initializes the usual PAM session setup and teardown hooks.
- We hooked systemd up to D-Bus, so that D-Bus passes activation requests to systemd and systemd becomes the central point for all kinds of activation, thus greatly extending the control of the execution environment of bus activated services, and making them accessible through the same utilities as SysV services. Also, this enables us to do race-free parallelized start-up for D-Bus services and their clients, thus speeding up things even further.
- systemd is now able to handle various Debian and OpenSUSE-specific extensions to the classic SysV init script formats natively, on top of the Fedora extensions we already parse.
- The D-Bus coverage of the systemd interface is now complete, allowing both introspection of runtime data and of parsed configuration data. It's fun now to introspect systemd with gdbus or d-feet.
- We added a systemd PAM module, which assigns the processes of each user session to its own cgroup in the systemd cgroup tree. This also enables reliable killing of all processes associated with a session when the user logs out. This also manages a secure per-user /var/run-style directory which is supposed to be used for sockets and similar files that shall be cleaned up when the user logs out.
- There's a new tool systemd-cgls, which plots a pretty process tree based on the systemd cgroup hierarchy. It's really pretty. Try it!
- We now have our own cgroup hierarchy beneath /cgroup/systemd (though is will move to /sys/fs/ before the F14 release).
- We have pretty code that automatically spawns a getty on a serial port when the kernel console is redirected to a serial TTY.
- systemctl got beefed up substantially (it can even draw dependency graphs now, via dot!), and the SysV compatiblity tools were extended to more completely and correctly support what was historically provided by SysV. For example, we'll now warn the user when systemd service files have changed but systemd was not asked to reload its configuration. Also, you can now use systemd's native client tools to reboot or shut-down an Upstart or sysvinit system, to facilitate upgrades.
- We provide a reference implementation for the socket activation and other APIs for nicer interaction with systemd.
- We have a pretty complete set of documentation now, some of it even extending to areas not directly related to systemd itself.
- Quite a number of upstream packages now ship with systemd service files out-of-the-box now, that work across all distributions that have adopted systemd. It is our intention to unify the boot and service management between distributions with systemd, and this shows fruits already. Furthermore a number of upstream packages now ship our patches for socket-based activation.
- Even more options that control the process execution environment or the sockets we create are now supported.
- Earlier today I began my series of blog stories on systemd for administrators.
- We reimplemented almost all boot-up and shutdown scripts of the standard Fedora install in much smaller, simpler and faster C utilities, or in systemd itself. Most of this will not be enabled in F14 however, even though it is shipped with systemd upstream. With this enabled the entire Linux system gains a completely new feeling as the number of shells we spawn approaches zero, and the PID of the first user terminal is way < 500 now, and the early boot-up is fully parallelized. We looked at the boot scripts of Fedora, OpenSUSE and Debian and distilled from this a list of functionality that makes up the early boot process and reimplemented this in C, if possible following the bahaviour of one of the existing implementations from these three distributions. This turned out to be much less effort than anticipated, and we are actually quite excited about this. Look forward to the fruits of this work in F15, when we might be able to present you a shell-less boot at least for standard desktop/laptop systems.
- We spent some time reinvestigating the current syslog logic, and came up with an elegant and simple scheme to provide /dev/log compatible logging right from the time systemd is first initialized right until the time the kernel halts the machine. Through the wonders of socket based activation we first connect the /dev/log socket with a minimal bridge to the kernel log buffer (kmsg) and then, as soon as the real syslog is started up as part of the later bootup phase, we dynamically replace this minimal bridge by the real syslog daemon -- without losing a single log message. Since one of the first things the real syslog daemon does is flushing the kernel log buffer into log files, all logged messages will sooner or later be stored on disk, regardless whether they have been generated during early boot, late boot or system runtime. On top of that if the syslog daemon terminates or is shut down during runtime, the bridge becomes active again and log output is written to kmsg again. The same applies when the system goes down. This provides a simple an robust way how we can ensure that no logs will ever be lost again, and logging is available from the beginning of boot-up to the end of shut-down. Plymouth will most likely adopt a similar scheme for initrd logging, thus ensuring that everything ever logged on the system will properly end up in the log files, whether it comes from the kernel, from the initrd, from early-boot, from runtime or shutdown. And if syslogd is not around, dmesg will provide you with access to the log messages. While this bridge is part of systemd upstream, we'll most likely enable this bridge in Fedora only starting with F15. Also note that embedded systems that have no interest in shipping a full syslogd solution can simply use this syslog bridge during the entire runtime, and thus making the kernel log buffer the centralized log storage, with all the advantages this offers: zero disk IO at runtime, access to serial and netconsole logging, and remote debug access to the kernel log buffer.
- We now install autofs units for many "API" kernel virtual file systems by default, such as binfmt_misc or hugetlbfs. That means that the file system access is readily available, client code no longer has to manually load the respective kernel modules, as they are autoloaded on first access of the file system. This has many advantages: it is not only faster to set up during boot, but also simpler for applications, as they can just assume the functionality is available. On top of that permission problems for the initialization go away, since manual module loading requires root privileges.
- Many smaller fixes and enhancements, all across the board, which if mentioned here would make this blog story another blog novel. Suffice to say, we did a lot of polishing to ready systemd for F14.
All in all, systemd is progressing nicely, and the features we have been working on in the last months are without exception features not existing in any other of the init systems available on Linux and our feature set already was far ahead of what the older init implementations provide. And we have quite a bit planned for the future. So, stay tuned!
Also note that I'll speak about systemd at LinuxKongress 2010 in Nuremberg, Germany. Later this year I'll also be speaking at the Linux Plumbers Conference in Boston, MA. Make sure to drop by if you want to learn about systemd or discuss exiciting new ideas or features with us.
As many of you know, systemd is the new Fedora init system, starting with F14, and it is also on its way to being adopted in a number of other distributions as well (for example, OpenSUSE). For administrators systemd provides a variety of new features and changes and enhances the administrative process substantially. This blog story is the first part of a series of articles I plan to post roughly every week for the next months. In every post I will try to explain one new feature of systemd. Many of these features are small and simple, so these stories should be interesting to a broader audience. However, from time to time we'll dive a little bit deeper into the great new features systemd provides you with.
Verifying Bootup
Traditionally, when booting up a Linux system, you see a lot of little messages passing by on your screen. As we work on speeding up and parallelizing the boot process these messages are becoming visible for a shorter and shorter time only and be less and less readable -- if they are shown at all, given we use graphical boot splash technology like Plymouth these days. Nonetheless the information of the boot screens was and still is very relevant, because it shows you for each service that is being started as part of bootup, wether it managed to start up successfully or failed (with those green or red [ OK ] or [ FAILED ] indicators). To improve the situation for machines that boot up fast and parallelized and to make this information more nicely available during runtime, we added a feature to systemd that tracks and remembers for each service whether it started up successfully, whether it exited with a non-zero exit code, whether it timed out, or whether it terminated abnormally (by segfaulting or similar), both during start-up and runtime. By simply typing systemctl in your shell you can query the state of all services, both systemd native and SysV/LSB services:
[root@lambda] ~# systemctl
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
dev-hugepages.automount loaded active running Huge Pages File System Automount Point
dev-mqueue.automount loaded active running POSIX Message Queue File System Automount Point
proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable File Formats File System Automount Point
sys-kernel-debug.automount loaded active waiting Debug File System Automount Point
sys-kernel-security.automount loaded active waiting Security File System Automount Point
sys-devices-pc...0000:02:00.0-net-eth0.device loaded active plugged 82573L Gigabit Ethernet Controller
[...]
sys-devices-virtual-tty-tty9.device loaded active plugged /sys/devices/virtual/tty/tty9
-.mount loaded active mounted /
boot.mount loaded active mounted /boot
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
home.mount loaded active mounted /home
proc-sys-fs-binfmt_misc.mount loaded active mounted Arbitrary Executable File Formats File System
abrtd.service loaded active running ABRT Automated Bug Reporting Tool
accounts-daemon.service loaded active running Accounts Service
acpid.service loaded active running ACPI Event Daemon
atd.service loaded active running Execution Queue Daemon
auditd.service loaded active running Security Auditing Service
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
bluetooth.service loaded active running Bluetooth Manager
console-kit-daemon.service loaded active running Console Manager
cpuspeed.service loaded active exited LSB: processor frequency scaling support
crond.service loaded active running Command Scheduler
cups.service loaded active running CUPS Printing Service
dbus.service loaded active running D-Bus System Message Bus
getty@tty2.service loaded active running Getty on tty2
getty@tty3.service loaded active running Getty on tty3
getty@tty4.service loaded active running Getty on tty4
getty@tty5.service loaded active running Getty on tty5
getty@tty6.service loaded active running Getty on tty6
haldaemon.service loaded active running Hardware Manager
hdapsd@sda.service loaded active running sda shock protection daemon
irqbalance.service loaded active running LSB: start and stop irqbalance daemon
iscsi.service loaded active exited LSB: Starts and stops login and scanning of iSCSI devices.
iscsid.service loaded active exited LSB: Starts and stops login iSCSI daemon.
livesys-late.service loaded active exited LSB: Late init script for live image.
livesys.service loaded active exited LSB: Init script for live image.
lvm2-monitor.service loaded active exited LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
mdmonitor.service loaded active running LSB: Start and stop the MD software RAID monitor
modem-manager.service loaded active running Modem Manager
netfs.service loaded active exited LSB: Mount and unmount network filesystems.
NetworkManager.service loaded active running Network Manager
ntpd.service loaded maintenance maintenance Network Time Service
polkitd.service loaded active running Policy Manager
prefdm.service loaded active running Display Manager
rc-local.service loaded active exited /etc/rc.local Compatibility
rpcbind.service loaded active running RPC Portmapper Service
rsyslog.service loaded active running System Logging Service
rtkit-daemon.service loaded active running RealtimeKit Scheduling Policy Service
sendmail.service loaded active running LSB: start and stop sendmail
sshd@172.31.0.53:22-172.31.0.4:36368.service loaded active running SSH Per-Connection Server
sysinit.service loaded active running System Initialization
systemd-logger.service loaded active running systemd Logging Daemon
udev-post.service loaded active exited LSB: Moves the generated persistent udev rules to /etc/udev/rules.d
udisks.service loaded active running Disk Manager
upowerd.service loaded active running Power Manager
wpa_supplicant.service loaded active running Wi-Fi Security Service
avahi-daemon.socket loaded active listening Avahi mDNS/DNS-SD Stack Activation Socket
cups.socket loaded active listening CUPS Printing Service Sockets
dbus.socket loaded active running dbus.socket
rpcbind.socket loaded active listening RPC Portmapper Socket
sshd.socket loaded active listening sshd.socket
systemd-initctl.socket loaded active listening systemd /dev/initctl Compatibility Socket
systemd-logger.socket loaded active running systemd Logging Socket
systemd-shutdownd.socket loaded active listening systemd Delayed Shutdown Socket
dev-disk-by\x1...x1db22a\x1d870f1adf2732.swap loaded active active /dev/disk/by-uuid/fd626ef7-34a4-4958-b22a-870f1adf2732
basic.target loaded active active Basic System
bluetooth.target loaded active active Bluetooth
dbus.target loaded active active D-Bus
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User
network.target loaded active active Network
remote-fs.target loaded active active Remote File Systems
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
JOB = Pending job for the unit.
221 units listed. Pass --all to see inactive units, too.
[root@lambda] ~#
(I have shortened the output above a little, and removed a few lines not relevant for this blog post.)
Look at the ACTIVE column, which shows you the high-level state of a service (or in fact of any kind of unit systemd maintains, which can be more than just services, but we'll have a look on this in a later blog posting), whether it is active (i.e. running), inactive (i.e. not running) or in any other state. If you look closely you'll see one item in the list that is marked maintenance and highlighted in red. This informs you about a service that failed to run or otherwise encountered a problem. In this case this is ntpd. Now, let's find out what actually happened to ntpd, with the systemctl status command:
[root@lambda] ~# systemctl status ntpd.service
ntpd.service - Network Time Service
Loaded: loaded (/etc/systemd/system/ntpd.service)
Active: maintenance
Main: 953 (code=exited, status=255)
CGroup: name=systemd:/systemd-1/ntpd.service
[root@lambda] ~#
This shows us that NTP terminated during runtime (when it ran as PID 953), and tells us exactly the error condition: the process exited with an exit status of 255.
In a later systemd version, we plan to hook this up to ABRT, as soon as this enhancement request is fixed. Then, if systemctl status shows you information about a service that crashed it will direct you right-away to the appropriate crash dump in ABRT.
Summary: use systemctl and systemctl status as modern, more complete replacements for the traditional boot-up status messages of SysV services. systemctl status not only captures in more detail the error condition but also shows runtime errors in addition to start-up errors.
That's it for this week, make sure to come back next week, for the next posting about systemd for administrators!
I too forgot to mention that my accommodation at GUADEC was sponsored by the GNOME Foundation. Thanks guys!
Today I came across this blog post of your design team. In context of the recent criticism you had to endure regarding upstream contributions I am disappointed that you have not bothered to ping anybody from the upstream freedesktop sound theme (for example yours truly) about this in advance. No, you went to cook your own soup. What really disappoints me is that we have asked multiple times for help and support and contributions for the sound theme, to only very little success, and I even asked some of the Canonical engineers about this topic and in particular regarding some clarifications of the licensing of the old Ubuntu sound theme. I am sorry, but if you had listened, or looked, or asked you would have been aware that we were looking for somebody to maintain this actively, upstream -- and because we didn't have the time to maintain this we only did the absolute minimum work necessary and we only maintain this ourselves because noone else wanted to.
It should be upstream first, downstream second.
I am sorry if I sound like an always complaining prick to you. But believe me, I am not saying this because I wouldn't like you or anything like that. I am just saying this because I believe you could do things oh so much better.
Please fix this. We want your contributions. Upstream.
I guess it's a bit beating a dead horse, but I had a good laugh today when I learned that I alone contributed more to GNOME than the entirety of Canonical, and only 800 additional commits seperating me from being more awesome than Nokia.
/me is amused
Here's a podcast interview with yours truly where I speak a little about PulseAudio and systemd. Seek to 64:43 for my lovely impetuous voice. There's also an interview with Owen just before mine.
The Call for Papers for the Linux Plumbers Conference (LPC) in November in Cambridge, Massachusetts is ending soon, on July 19th 2010 (That's the upcoming monday!). It's a conference about the core infrastructure of Linux systems: the part of the system where userspace and the kernel interface. It's the only conference where the focus is specifically on getting together the kernel people who work on the userspace interfaces and the userspace people who have to deal with kernel interfaces. It's supposed to be a place where all the people doing infrastructure work sit down and talk, so that both parties understand better what the requirements and needs of the other are, and where we can work towards fixing the major problems we currently have with our lower-level infrastructure and APIs.
The two previous LPCs were hugely successful (as reported on LWN on various occasions), and this time we hope to repeat that.
Like the previous years, I will be running the Audio conference track of LPC, this time together with Mark Brown. Audio infrastructure on Linux has been steadily improving the last years all over the place, but there's still a lot to do. Join us at the LPC to discuss the next steps and help improving Linux audio further! If you are doing audio infrastructure work on Linux, make sure to attend and submit a paper!
Sign up soon! Send in your paper quickly! Only three days left to the end of the CFP!
(I am also planning to do a presentation there about systemd, together with Kay. Make sure to attend if you are interested in that topic.)
See you in Boston!