1 files changed, 28 insertions, 15 deletions

diff --git a/Software/systemd/ContainerInterface.mdwn b/Software/systemd/ContainerInterface.mdwn
index b392a544..d371b74d 100644
--- a/Software/systemd/ContainerInterface.mdwn
+++ b/Software/systemd/ContainerInterface.mdwn
@@ -4,21 +4,34 @@
 
 Also consult [[Writing Virtual Machine or Container Managers|http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers]].
 
-systemd has a number of interfaces for interaction with container managers when systemd is used within an OS container. If you write a container solution, please consider supporting the following interfaces: 
-
-* To allow systemd (and other code) to identify that it is executed within a container, please set the $container= environment variable for PID 1 in the container to a short lowercase string identifying your implementation. With this in place the ConditionVirtualization= setting in unit files will work properly. Example: "container=lxc-libvirt" 
-* systemd has special support for allowing container managers to initialize the UUID for /etc/machine-id to some manager supplied value. This is only enabled if /etc/machine-id is empty (i.e. not yet set) at boot time of the container. The container manager should set $container_uuid= as environment variable for the container's PID 1 to the container UUID it wants to set. (This is similar to the effect of qemu's -uuid switch). Note that you should pass only a UUID here that is actually unique (i.e. only one running container should have a specific UUID), and gets changed when a container gets duplicated. Also note that systemd will try to persistently store the UUID in /etc/machine-id (if writable) when this option is used, hence you should always pass the same UUID here. Keeping the externally used UUID for a container and the internal one in sync is hopefully useful to minimize surprise for the administrator.
-* systemd can automatically spawn login gettys on additional ptys. A container manager can set the $container_ttys= environment variable for the container's PID 1 to tell it on which ptys to spawn gettys. The variable should take a space seperated list of pty names, without the leading "/dev" prefix, but with the "pts/" prefix included. Note that despite the variable's name you may only specify ptys, and not other types of ttys. Also you need to specify the pty itself, a symlink will not suffice. This is implemented in systemd-getty-generator(8). Note that this variable should not include the pty that /dev/console maps to if it maps to one (see below). Example: if the container receives "container_ttys=pts/7 pts/8 pts/14" it will spawn three additionally login gettys on ptys 7, 8 and 14.
-* If the container manager wants to influence the hostname for a machine it should just set it before invoking systemd in the container, and systemd will leave it unmodified (that is unless there's an explicit hostname configured in /etc/hostname which overrides whatever is pre-initialized by the container manager) 
-* Make sure to pre-mount /sys, and /proc, /sys/fs/selinux before invoking systemd, and mount /proc/sys and the entirety of /sys and /sys/fs/selinux read-only in order to avoid that the container can alter the host kernel's configuration settings. systemd and various tools (such as the selinux) have been modified to detect whether these file systems are read-only, and will behave accordingly. 
-* Consider syncing /etc/localtime from the host file system into the container. Make it a relative symlink to the containers's zoneinfo dir, as usual. Tools rely on being able to determine the timezone setting from the symlink value, and by making it relative it looks nice even if people list the containers' /etc from the host. 
-* Pre-mount /dev as (container private) tmpfs for the container and bind mount some suitable TTY to /dev/console. Also, make sure to create device nodes for /dev/null, /dev/zero, /dev/full, /dev/random, /dev/urandom, /dev/tty, /dev/ptmx in /dev. It is not necessary to create /dev/fd or /dev/stdout, as systemd will do that on its own. Make sure to set up the "devices" cgroup controller so that no other devices but these may be created in the container. Note that many systemd services these days use PrivateDevices=, which means that systemd will set up a private /dev for them for which it needs to be able to create these device nodes. Dropping CAP_MKNOD for containers is hence generally not OK.
-* udev is not available in containers (and refuses to start), and hence device dependencies are unavailable. The udev unit files will check for /sys being read-only, as an indication whether device management can work. Hence make sure to mount /sys read-only in the container (see above).
-* If systemd detects it is run in a container it will spawn a single shell on /dev/console, and not care about VTs or multiple gettys on VTs. (But see $container_ttys above.)
-* Make the container journal available in the host, by automatically symlinking the container journal directory into the host journal directory. More precisely, link /var/log/journal/<container-machine-id> of the container into the same dir of the host. Administrators can then automatically browse all container journals (correctly interleaved) by issuing "journalctl -m". The container machine ID you can determine from /etc/machine-id in the container. 
-* If the container manager wants to cleanly shutdown the container, it might be a good idea to send SIGRTMIN+3 to its init process. systemd will then do a clean shutdown. Note however, that only systemd understands SIGRTMIN+3 like this, this might confuse other init systems. 
-* To support [[Socket Activated Containers|http://0pointer.de/blog/projects/socket-activated-containers.html]] the container manager should be capable of being run as a systemd service. It will then receive the sockets starting with FD 3, the number of passed FDs in $LISTEN_FDS and its PID as $LISTEN_PID. It should take these and pass them on to the container's init process, also setting $LISTEN_FDS and $LISTEN_PID (basically, it can just leave the FDs and $LISTEN_FDS untouched, but it needs to set $LISTEN_PID to for the container init process). That's all that's necessary to make socket activation work. The protocol to hand sockets from systemd to services is hence the same as from a container manager to a container systemd. For further details see the [[explanations of sd_listen_fds(1)|http://0pointer.de/public/systemd-man/sd_listen_fds.html]] and [[the blog story for service developers|http://0pointer.de/blog/projects/socket-activation.html]]. 
-* Container managers should stay away from the "name=systemd" cgroup hierarchy. That's private property of systemd, and no other code should interfere with it. 
+systemd has a number of interfaces for interaction with container managers when systemd is used within an OS container. If you write a container solution, please consider supporting the following interfaces.
+
+## Execution Environment
+1. If the container manager wants to influence the hostname for a machine it should just set it before invoking systemd in the container, and systemd will leave it unmodified (that is unless there's an explicit hostname configured in /etc/hostname which overrides whatever is pre-initialized by the container manager) 
+2. Make sure to pre-mount /sys, and /proc, /sys/fs/selinux before invoking systemd, and mount /proc/sys and the entirety of /sys and /sys/fs/selinux read-only in order to avoid that the container can alter the host kernel's configuration settings. systemd and various tools (such as the selinux) have been modified to detect whether these file systems are read-only, and will behave accordingly. 
+3. Pre-mount /dev as (container private) tmpfs for the container and bind mount some suitable TTY to /dev/console. Also, make sure to create device nodes for /dev/null, /dev/zero, /dev/full, /dev/random, /dev/urandom, /dev/tty, /dev/ptmx in /dev. It is not necessary to create /dev/fd or /dev/stdout, as systemd will do that on its own. Make sure to set up the "devices" cgroup controller so that no other devices but these may be created in the container. Note that many systemd services these days use PrivateDevices=, which means that systemd will set up a private /dev for them for which it needs to be able to create these device nodes. Dropping CAP_MKNOD for containers is hence generally not OK.
+4. udev is not available in containers (and refuses to start), and hence device dependencies are unavailable. The udev unit files will check for /sys being read-only, as an indication whether device management can work. Hence make sure to mount /sys read-only in the container (see above).
+5. If systemd detects it is run in a container it will spawn a single shell on /dev/console, and not care about VTs or multiple gettys on VTs. (But see $container_ttys above.)
+
+## Environment Variables
+1. To allow systemd (and other code) to identify that it is executed within a container, please set the $container= environment variable for PID 1 in the container to a short lowercase string identifying your implementation. With this in place the ConditionVirtualization= setting in unit files will work properly. Example: "container=lxc-libvirt" 
+2. systemd has special support for allowing container managers to initialize the UUID for /etc/machine-id to some manager supplied value. This is only enabled if /etc/machine-id is empty (i.e. not yet set) at boot time of the container. The container manager should set $container_uuid= as environment variable for the container's PID 1 to the container UUID it wants to set. (This is similar to the effect of qemu's -uuid switch). Note that you should pass only a UUID here that is actually unique (i.e. only one running container should have a specific UUID), and gets changed when a container gets duplicated. Also note that systemd will try to persistently store the UUID in /etc/machine-id (if writable) when this option is used, hence you should always pass the same UUID here. Keeping the externally used UUID for a container and the internal one in sync is hopefully useful to minimize surprise for the administrator.
+3. systemd can automatically spawn login gettys on additional ptys. A container manager can set the $container_ttys= environment variable for the container's PID 1 to tell it on which ptys to spawn gettys. The variable should take a space seperated list of pty names, without the leading "/dev" prefix, but with the "pts/" prefix included. Note that despite the variable's name you may only specify ptys, and not other types of ttys. Also you need to specify the pty itself, a symlink will not suffice. This is implemented in systemd-getty-generator(8). Note that this variable should not include the pty that /dev/console maps to if it maps to one (see below). Example: if the container receives "container_ttys=pts/7 pts/8 pts/14" it will spawn three additionally login gettys on ptys 7, 8 and 14.
+
+## Advanced Integration
+1. Consider syncing /etc/localtime from the host file system into the container. Make it a relative symlink to the containers's zoneinfo dir, as usual. Tools rely on being able to determine the timezone setting from the symlink value, and by making it relative it looks nice even if people list the containers' /etc from the host. 
+2. Make the container journal available in the host, by automatically symlinking the container journal directory into the host journal directory. More precisely, link /var/log/journal/<container-machine-id> of the container into the same dir of the host. Administrators can then automatically browse all container journals (correctly interleaved) by issuing "journalctl -m". The container machine ID you can determine from /etc/machine-id in the container. 
+3. If the container manager wants to cleanly shutdown the container, it might be a good idea to send SIGRTMIN+3 to its init process. systemd will then do a clean shutdown. Note however, that only systemd understands SIGRTMIN+3 like this, this might confuse other init systems. 
+4. To support [[Socket Activated Containers|http://0pointer.de/blog/projects/socket-activated-containers.html]] the container manager should be capable of being run as a systemd service. It will then receive the sockets starting with FD 3, the number of passed FDs in $LISTEN_FDS and its PID as $LISTEN_PID. It should take these and pass them on to the container's init process, also setting $LISTEN_FDS and $LISTEN_PID (basically, it can just leave the FDs and $LISTEN_FDS untouched, but it needs to set $LISTEN_PID to for the container init process). That's all that's necessary to make socket activation work. The protocol to hand sockets from systemd to services is hence the same as from a container manager to a container systemd. For further details see the [[explanations of sd_listen_fds(1)|http://0pointer.de/public/systemd-man/sd_listen_fds.html]] and [[the blog story for service developers|http://0pointer.de/blog/projects/socket-activation.html]]. 
+5. Container managers should stay away from the "name=systemd" cgroup hierarchy. That's private property of systemd, and no other code should interfere with it. 
+
+## Networking
+
+1. Inside of a container, if a veth link is named "host0", systemd-networkd running inside of the container will by default do DHCPv4 client, DHCPv6 client and IPv4LL on it. It is thus recommended that container managers that add a veth link to a container name it "host0", to get automatically configured networked, with no manual interference from outside.
+2. Outside of a container, if a veth link is prefixed "ve-" will by default do DHCPv4 server and DHCPv6 serer on it, as well as IPv4LL. It is thus recommended that container managers that add a veth link to a container name the external side "ve-" followed by the container name.
+3. It is recommended to configure stable MAC addresses to container veth devices, for example hashed out of the container names. That way it is more likely that DHCP and Ipv4LL will acquire stable addresses.
+
+## Final Words
 If you write software that wants to detect whether it is run in a container, please check /proc/1/environ and look for the container= environment variable. Do not assume the environment variable is inherited down the process tree. It generally is not. Hence check the environment block of PID 1, not your own. 
 
 Note that it is our intention to make systemd systems work flawlessly and out-of-the-box in containers. In fact we are interested to ensure that the same OS image can be booted on a bare system, in a VM and in a container, and behave correctly each time. If you notice that some component in systemd does not work in a container as it should please contact us.