Discussion:
DynamicUsers and read-only /var
(too old to reply)
Antoine Pietri
2018-05-16 11:29:17 UTC
Permalink
Raw Message
Hi,

Our organization uses a diskless setup to boot hundreds of machines
using a read-only NFS export of their common rootfs.

To be able to run services that need to write in /var, we can't just
have /var as a tmpfs, because it contains files installed by packages
that are required by some services to run. Our current solution was to
have /var in read-only, but have a list of directories where some
services actually write (/var/log, /var/spool/mail, etc) and mount
them as tmpfs.

This year, some services like systemd-timesyncd are shipped with
DynamicUser=yes by default in our distribution (Archlinux), which
means the above solution no longer works. My understanding is that
systemd requires a writable /var to be able to symlink the state
directory the first time it is launched.

Our only option here, if we don't want to manually disable dynamic
users in all the services, seems to be to mount /var in a
copy-on-write overlayfs. We could do that, but it's a bit cutting edge
and dangerous for us. Two years ago, overlayfs didn't even support nfs
as its lower directory, that's why we avoided it so far.

As I know you don't like to add requirements to have a writable /var,
I'd love to have your input on this issue! Is there anything we missed
that would allow us to keep using dynamic user services with a
read-only /var, or do we have to use the overlay solution?

Thanks,

--
Antoine Pietri
Jérémy Rosen
2018-05-16 13:05:23 UTC
Permalink
Raw Message
hmm, I think you could have the whole /var as a tmpfs and use
systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
copying some template directory from a read-only location (typicalli in
/usr)

On 16/05/2018 13:29, Antoine Pietri wrote:
> Hi,
>
> Our organization uses a diskless setup to boot hundreds of machines
> using a read-only NFS export of their common rootfs.
>
> To be able to run services that need to write in /var, we can't just
> have /var as a tmpfs, because it contains files installed by packages
> that are required by some services to run. Our current solution was to
> have /var in read-only, but have a list of directories where some
> services actually write (/var/log, /var/spool/mail, etc) and mount
> them as tmpfs.
>
> This year, some services like systemd-timesyncd are shipped with
> DynamicUser=yes by default in our distribution (Archlinux), which
> means the above solution no longer works. My understanding is that
> systemd requires a writable /var to be able to symlink the state
> directory the first time it is launched.
>
> Our only option here, if we don't want to manually disable dynamic
> users in all the services, seems to be to mount /var in a
> copy-on-write overlayfs. We could do that, but it's a bit cutting edge
> and dangerous for us. Two years ago, overlayfs didn't even support nfs
> as its lower directory, that's why we avoided it so far.
>
> As I know you don't like to add requirements to have a writable /var,
> I'd love to have your input on this issue! Is there anything we missed
> that would allow us to keep using dynamic user services with a
> read-only /var, or do we have to use the overlay solution?
>
> Thanks,
>

--
SMILE <http://www.smile.eu/>

20 rue des Jardins
92600 AsniÚres-sur-Seine


*Jérémy ROSEN*
Architecte technique
Responsable de l'expertise Smile-ECS

email ***@smile.fr <mailto:***@smile.fr>
phone +33141402967
url http://www.smile.eu

Twitter <https://twitter.com/GroupeSmile> Facebook
<https://www.facebook.com/smileopensource> LinkedIn
<https://www.linkedin.com/company/smile> Github
<https://github.com/Smile-SA>


Découvrez l’univers Smile, rendez-vous sur smile.eu
<http://smile.eu/?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÚte, n'imprimez ce mail que si c'est nécessaire
Antoine Pietri
2018-05-16 14:33:08 UTC
Permalink
Raw Message
Hi Jérémy,

On Wed, May 16 at 13:05 PM, Jérémy Rosen <***@smile.fr> wrote:
> hmm, I think you could have the whole /var as a tmpfs and use
> systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
> copying some template directory from a read-only location (typicalli in
> /usr)

That's another interesting workaround, but ideally we'd like to let
all the packages install stuff in /var/lib like they would normally,
and only put some tmpfs in /var after that. I'll still keep that
solution in my mind if we have to use overlayfs and it doesn't work
for some reason.

Thanks,

--
Antoine Pietri
Simon McVittie
2018-05-16 14:56:06 UTC
Permalink
Raw Message
On Wed, 16 May 2018 at 16:33:08 +0200, Antoine Pietri wrote:
> On Wed, May 16 at 13:05 PM, Jérémy Rosen <***@smile.fr> wrote:
> > hmm, I think you could have the whole /var as a tmpfs and use
> > systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
> > copying some template directory from a read-only location (typicalli in
> > /usr)
>
> That's another interesting workaround, but ideally we'd like to let
> all the packages install stuff in /var/lib like they would normally,
> and only put some tmpfs in /var after that.

The purpose of /var is that it contains variable data, so a read-only
/var seems like a rather contradictory goal?

I think you'd really be better off redirecting the packaged
or package-manager-produced contents of /var to /usr/var or
/usr/share/factory/var or something (perhaps using your package manager's
equivalent of dpkg-divert if it has one), and using systemd-tmpfiles to
populate a tmpfs with copies or symbolic links (or possibly bind-mounting
selected directories from the read-only copy, if entire subtrees like
/var/lib/dpkg are read-only except during package manager operations).

Projects like libostree and rpm-ostree might have some useful concepts
or code for managing immutable, read-only rootfs or /usr deployments,
since that's what they do: in an ostree-based OS, /usr is an
atomically-updated immutable tree, directories like /var and /home are
locally-maintained, and /etc is a three-way merge between the old
/usr/etc, the new /usr/etc and the local /etc.

smcv
Lennart Poettering
2018-05-16 20:59:57 UTC
Permalink
Raw Message
On Mi, 16.05.18 15:56, Simon McVittie (***@collabora.com) wrote:

> On Wed, 16 May 2018 at 16:33:08 +0200, Antoine Pietri wrote:
> > On Wed, May 16 at 13:05 PM, Jérémy Rosen <***@smile.fr> wrote:
> > > hmm, I think you could have the whole /var as a tmpfs and use
> > > systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
> > > copying some template directory from a read-only location (typicalli in
> > > /usr)
> >
> > That's another interesting workaround, but ideally we'd like to let
> > all the packages install stuff in /var/lib like they would normally,
> > and only put some tmpfs in /var after that.
>
> The purpose of /var is that it contains variable data, so a read-only
> /var seems like a rather contradictory goal?
>
> I think you'd really be better off redirecting the packaged
> or package-manager-produced contents of /var to /usr/var or
> /usr/share/factory/var or something (perhaps using your package
> manager's

Yes, /usr/share/factory is the recommended place for this, if you
follow tmpfiles.d logic. See the "C" specififier documentation for
tmpfiles.d, it suggests using that directory for the purpose of early
population of /var and /etc with non-empty files or dirs.

Lennart

--
Lennart Poettering, Red Hat
Colin Walters
2018-05-24 14:51:02 UTC
Permalink
Raw Message
On Wed, May 16, 2018, at 10:56 AM, Simon McVittie wrote:
>
> Projects like libostree and rpm-ostree might have some useful concepts
> or code for managing immutable, read-only rootfs or /usr deployments,
> since that's what they do: in an ostree-based OS, /usr is an
> atomically-updated immutable tree, directories like /var and /home are
> locally-maintained, and /etc is a three-way merge between the old
> /usr/etc, the new /usr/etc and the local /etc.

Yep, the libostree manual suggests converting contents in /var to systemd
tmpfiles snippets:

https://ostree.readthedocs.io/en/latest/manual/adapting-existing/

And that's in fact what rpm-ostree does:

https://github.com/projectatomic/rpm-ostree/blob/bff3a54e1ee67398bd7a2cfdc359326689479954/src/libpriv/rpmostree-postprocess.c#L556
Lennart Poettering
2018-05-16 20:41:14 UTC
Permalink
Raw Message
On Mi, 16.05.18 16:33, Antoine Pietri (***@gmail.com) wrote:

> Hi Jérémy,
>
> On Wed, May 16 at 13:05 PM, Jérémy Rosen <***@smile.fr> wrote:
> > hmm, I think you could have the whole /var as a tmpfs and use
> > systemd-tmpfiles (man:tmpfiles.d) to initialize /var at startup by
> > copying some template directory from a read-only location (typicalli in
> > /usr)
>
> That's another interesting workaround, but ideally we'd like to let

It's not a "workaround". It's the recommended mode of operation to
support stateless systems and "factory reset" concepts, i.e. systems
that start with an empty, uninitialized /var.

Lennart

--
Lennart Poettering, Red Hat
Lennart Poettering
2018-05-16 20:38:39 UTC
Permalink
Raw Message
On Mi, 16.05.18 13:29, Antoine Pietri (***@gmail.com) wrote:

> Hi,
>
> Our organization uses a diskless setup to boot hundreds of machines
> using a read-only NFS export of their common rootfs.
>
> To be able to run services that need to write in /var, we can't just
> have /var as a tmpfs, because it contains files installed by packages
> that are required by some services to run. Our current solution was to
> have /var in read-only, but have a list of directories where some
> services actually write (/var/log, /var/spool/mail, etc) and mount
> them as tmpfs.

Uh, if /var is read-only you kinda void your warranty. I mean, it's
already in the name that /var is "variable", no?

You are fine to make pretty much everything else read-only, but /tmp,
/var and /dev/shm really need to be writable for a normal system to
operate. They don't have to be persistant, but writable they really
should be. If you are this rule, then that's of course OK, but you get
to keep the pieces. This is even documented, very explicitly in
file-hierarchy(7).

I am not sure I grok your reason for making /var read-only
though. Note that today's systems should be mostly fine with /var
being empty at boot, as they ship enough configuration with tmpfiles.d
drop-ins so that what is missing in /var is created as needed early on
boot, and hence /var is sufficiently populated for everything to
work. If you are encounter a package that does not ship the right
tmpfiles.d/ snippets to make empty /var bootups just work, then please
consider working with the respective upstreams to fix that. It's as
easy as shipping one more drop-in file in the rpm/deb, and everybody's
life is easier.

In the unlikely case that some package needs files in /var that are
more than empty dirs/files with the right perms, then you can even
make use of tmpfiles' "factory logic" where it can automatically copy
a set of files from /usr early on, to ensure /var is properly
populated.

> This year, some services like systemd-timesyncd are shipped with
> DynamicUser=yes by default in our distribution (Archlinux), which
> means the above solution no longer works. My understanding is that
> systemd requires a writable /var to be able to symlink the state
> directory the first time it is launched.

Correct.

Lennart

--
Lennart Poettering, Red Hat
Loading...