[systemd-devel] DynamicUsers and read-only /var

[Antoine Pietri]

Hi,

Our organization uses a diskless setup to boot hundreds of machines
using a read-only NFS export of their common rootfs.

To be able to run services that need to write in /var, we can't just
have /var as a tmpfs, because it contains files installed by packages
that are required by some services to run. Our current solution was to
have /var in read-only, but have a list of directories where some
services actually write (/var/log, /var/spool/mail, etc) and mount
them as tmpfs.

This year, some services like systemd-timesyncd are shipped with
DynamicUser=yes by default in our distribution (Archlinux), which
means the above solution no longer works. My understanding is that
systemd requires a writable /var to be able to symlink the state
directory the first time it is launched.

Our only option here, if we don't want to manually disable dynamic
users in all the services, seems to be to mount /var in a
copy-on-write overlayfs. We could do that, but it's a bit cutting edge
and dangerous for us. Two years ago, overlayfs didn't even support nfs
as its lower directory, that's why we avoided it so far.

As I know you don't like to add requirements to have a writable /var,
I'd love to have your input on this issue! Is there anything we missed
that would allow us to keep using dynamic user services with a
read-only /var, or do we have to use the overlay solution?

Thanks,

[Jérémy Rosen]

hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

Post by Antoine Pietri
Hi,
Our organization uses a diskless setup to boot hundreds of machines
using a read-only NFS export of their common rootfs.
To be able to run services that need to write in /var, we can't just
have /var as a tmpfs, because it contains files installed by packages
that are required by some services to run. Our current solution was to
have /var in read-only, but have a list of directories where some
services actually write (/var/log, /var/spool/mail, etc) and mount
them as tmpfs.
This year, some services like systemd-timesyncd are shipped with
DynamicUser=yes by default in our distribution (Archlinux), which
means the above solution no longer works. My understanding is that
systemd requires a writable /var to be able to symlink the state
directory the first time it is launched.
Our only option here, if we don't want to manually disable dynamic
users in all the services, seems to be to mount /var in a
copy-on-write overlayfs. We could do that, but it's a bit cutting edge
and dangerous for us. Two years ago, overlayfs didn't even support nfs
as its lower directory, that's why we avoided it so far.
As I know you don't like to add requirements to have a writable /var,
I'd love to have your input on this issue! Is there anything we missed
that would allow us to keep using dynamic user services with a
read-only /var, or do we have to use the overlay solution?
Thanks,

--
SMILE <http://www.smile.eu/>

20 rue des Jardins
92600 AsniÚres-sur-Seine


*Jérémy ROSEN*
Architecte technique
Responsable de l'expertise Smile-ECS

email ***@smile.fr <mailto:***@smile.fr>
phone [phone]
url http://www.smile.eu

Twitter <https://twitter.com/GroupeSmile> Facebook
<https://www.facebook.com/smileopensource> LinkedIn
<https://www.linkedin.com/company/smile> Github
<https://github.com/Smile-SA>


Découvrez l’univers Smile, rendez-vous sur smile.eu
<http://smile.eu/?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÚte, n'imprimez ce mail que si c'est nécessaire

[Antoine Pietri]

Hi Jérémy,

Post by Jérémy Rosen
hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

That's another interesting workaround, but ideally we'd like to let
all the packages install stuff in /var/lib like they would normally,
and only put some tmpfs in /var after that. I'll still keep that
solution in my mind if we have to use overlayfs and it doesn't work
for some reason.

Thanks,

[Simon McVittie]

Post by Antoine Pietri

Post by Jérémy Rosen
hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

That's another interesting workaround, but ideally we'd like to let
all the packages install stuff in /var/lib like they would normally,
and only put some tmpfs in /var after that.

The purpose of /var is that it contains variable data, so a read-only
/var seems like a rather contradictory goal?

I think you'd really be better off redirecting the packaged
or package-manager-produced contents of /var to /usr/var or
/usr/share/factory/var or something (perhaps using your package manager's
equivalent of dpkg-divert if it has one), and using systemd-tmpfiles to
populate a tmpfs with copies or symbolic links (or possibly bind-mounting
selected directories from the read-only copy, if entire subtrees like
/var/lib/dpkg are read-only except during package manager operations).

Projects like libostree and rpm-ostree might have some useful concepts
or code for managing immutable, read-only rootfs or /usr deployments,
since that's what they do: in an ostree-based OS, /usr is an
atomically-updated immutable tree, directories like /var and /home are
locally-maintained, and /etc is a three-way merge between the old
/usr/etc, the new /usr/etc and the local /etc.

smcv

[Lennart Poettering]

Post by Simon McVittie

Post by Antoine Pietri

Post by Jérémy Rosen
hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

That's another interesting workaround, but ideally we'd like to let
all the packages install stuff in /var/lib like they would normally,
and only put some tmpfs in /var after that.

The purpose of /var is that it contains variable data, so a read-only
/var seems like a rather contradictory goal?
I think you'd really be better off redirecting the packaged
or package-manager-produced contents of /var to /usr/var or
/usr/share/factory/var or something (perhaps using your package manager's

Yes, /usr/share/factory is the recommended place for this, if you
follow tmpfiles.d logic. See the "C" specififier documentation for
tmpfiles.d, it suggests using that directory for the purpose of early
population of /var and /etc with non-empty files or dirs.

Lennart

[Colin Walters]

Post by Simon McVittie
Projects like libostree and rpm-ostree might have some useful concepts
or code for managing immutable, read-only rootfs or /usr deployments,
since that's what they do: in an ostree-based OS, /usr is an
atomically-updated immutable tree, directories like /var and /home are
locally-maintained, and /etc is a three-way merge between the old
/usr/etc, the new /usr/etc and the local /etc.

Yep, the libostree manual suggests converting contents in /var to systemd
tmpfiles snippets:

https://ostree.readthedocs.io/en/latest/manual/adapting-existing/

And that's in fact what rpm-ostree does:

https://github.com/projectatomic/rpm-ostree/blob/bff3a54e1ee67398bd7a2cfdc359326689479954/src/libpriv/rpmostree-postprocess.c#L556

[Lennart Poettering]

Post by Antoine Pietri
Hi Jérémy,

Post by Jérémy Rosen
hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

That's another interesting workaround, but ideally we'd like to let

It's not a "workaround". It's the recommended mode of operation to
support stateless systems and "factory reset" concepts, i.e. systems
that start with an empty, uninitialized /var.

Lennart

[Lennart Poettering]

Post by Antoine Pietri
Hi,
Our organization uses a diskless setup to boot hundreds of machines
using a read-only NFS export of their common rootfs.
To be able to run services that need to write in /var, we can't just
have /var as a tmpfs, because it contains files installed by packages
that are required by some services to run. Our current solution was to
have /var in read-only, but have a list of directories where some
services actually write (/var/log, /var/spool/mail, etc) and mount
them as tmpfs.

Uh, if /var is read-only you kinda void your warranty. I mean, it's
already in the name that /var is "variable", no?

You are fine to make pretty much everything else read-only, but /tmp,
/var and /dev/shm really need to be writable for a normal system to
operate. They don't have to be persistant, but writable they really
should be. If you are this rule, then that's of course OK, but you get
to keep the pieces. This is even documented, very explicitly in
file-hierarchy(7).

I am not sure I grok your reason for making /var read-only
though. Note that today's systems should be mostly fine with /var
being empty at boot, as they ship enough configuration with tmpfiles.d
drop-ins so that what is missing in /var is created as needed early on
boot, and hence /var is sufficiently populated for everything to
work. If you are encounter a package that does not ship the right
tmpfiles.d/ snippets to make empty /var bootups just work, then please
consider working with the respective upstreams to fix that. It's as
easy as shipping one more drop-in file in the rpm/deb, and everybody's
life is easier.

In the unlikely case that some package needs files in /var that are
more than empty dirs/files with the right perms, then you can even
make use of tmpfiles' "factory logic" where it can automatically copy
a set of files from /usr early on, to ensure /var is properly
populated.

Post by Antoine Pietri
This year, some services like systemd-timesyncd are shipped with
DynamicUser=yes by default in our distribution (Archlinux), which
means the above solution no longer works. My understanding is that
systemd requires a writable /var to be able to symlink the state
directory the first time it is launched.

Correct.

Lennart