[systemd-devel] early mounts in systemd

[Rick Winscot]

We have an embedded product that uses a minimal Linux distribution
generated via Buildroot.

Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations might
cause problems on the eMMC. At the same time, making logging (e.g. /var)
persistent for debugging was added to requirements. Persistent storage
would be achieved by mounting /var to a separate partition that is
read-write.

Several-hundred hours later... with many systemd-analyze reports and
various configurations tested, we have determined that managing the /var
mount with stadard services is not going to work due to tightly coupled and
precisely timed dependencies. Attempts with /etc/fstab and the systemd
generator are also unstable.

Getting /var mounted in proximity to the initialization of
systemd-journald.service seemed illusive.

Several days ago I found a post on Stackoverflow that tied into udev
triggers that seemed promising; resulting in the method outlined below.
Initial testing shows proper timing with all dependencies satisfied.
However, the solution feels... hackey.

My question for anyone on the list, is the method outlined below a
reasonable solution to mounting /var early in the start-up cycle?

Or... is there a better way? Some trimming


Step One: Create a systemd service that mounts /var to the specified
partition
Service: /etc/systemd/system/var.service

[Unit]
Description=service for mounting /var
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p6


Step Two: Add a nofail mount
fstab: /etc/fstab

/dev/root / auto rw 0 1
/dev/mmcblk0p6 /var ext4 rw,nofail 0 0


Step Three: Add a wants dependency on the mount in udev triggers (some
lines deleted for brevity)
Service: /usr/lib/systemd/system/systemd-udev-trigger.service

[Unit]
...
Wants=systemd-udevd.service var.service

[Service]
...
ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm
trigger


Finally, systemd-analyze plot shows that the mount works as desired.

systemd-udev-trigger.service
var.service
dev-mmcblk0p6.device
var.mount
....
systemd-remount-fs.service
systemd-journal-flush.service
local-fs-pre.target
....

[Michael Biebl]

What is the actual problem you have with a separate /var and systemd-journald?
For completeness sake, which systemd version do you have?

Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot

We have an embedded product that uses a minimal Linux distribution generated via Buildroot.
Early in the project it was decided to make the rootfs read-only... in an effort to improve durability in environments where power fluctuations might cause problems on the eMMC. At the same time, making logging (e.g. /var) persistent for debugging was added to requirements. Persistent storage would be achieved by mounting /var to a separate partition that is read-write.
Several-hundred hours later... with many systemd-analyze reports and various configurations tested, we have determined that managing the /var mount with stadard services is not going to work due to tightly coupled and precisely timed dependencies. Attempts with /etc/fstab and the systemd generator are also unstable.
Getting /var mounted in proximity to the initialization of systemd-journald.service seemed illusive.
Several days ago I found a post on Stackoverflow that tied into udev triggers that seemed promising; resulting in the method outlined below. Initial testing shows proper timing with all dependencies satisfied. However, the solution feels... hackey.
My question for anyone on the list, is the method outlined below a reasonable solution to mounting /var early in the start-up cycle?
Or... is there a better way? Some trimming
Step One: Create a systemd service that mounts /var to the specified partition
Service: /etc/systemd/system/var.service
[Unit]
Description=service for mounting /var
DefaultDependencies=no
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p6
Step Two: Add a nofail mount
fstab: /etc/fstab
/dev/root / auto rw 0 1
/dev/mmcblk0p6 /var ext4 rw,nofail 0 0
Step Three: Add a wants dependency on the mount in udev triggers (some lines deleted for brevity)
Service: /usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
...
Wants=systemd-udevd.service var.service
[Service]
...
ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger
Finally, systemd-analyze plot shows that the mount works as desired.
systemd-udev-trigger.service
var.service
dev-mmcblk0p6.device
var.mount
....
systemd-remount-fs.service
systemd-journal-flush.service
local-fs-pre.target
....
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[Rick Winscot]

systemd 247

/etc/systemd/journald.conf storage is persistent,
systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.

Mounting /var on a separate read-write partition handles the persistent log
requirement as well as offloading other read-write operations that can no
longer live on the rootfs... being read-only.

Post by Michael Biebl
What is the actual problem you have with a separate /var and
systemd-journald?
For completeness sake, which systemd version do you have?
Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot

Post by Rick Winscot
We have an embedded product that uses a minimal Linux distribution

generated via Buildroot.

Post by Rick Winscot
Early in the project it was decided to make the rootfs read-only... in

an effort to improve durability in environments where power fluctuations
might cause problems on the eMMC. At the same time, making logging (e.g.
/var) persistent for debugging was added to requirements. Persistent
storage would be achieved by mounting /var to a separate partition that is
read-write.

Post by Rick Winscot
Several-hundred hours later... with many systemd-analyze reports and

various configurations tested, we have determined that managing the /var
mount with stadard services is not going to work due to tightly coupled and
precisely timed dependencies. Attempts with /etc/fstab and the systemd
generator are also unstable.

Post by Rick Winscot
Getting /var mounted in proximity to the initialization of

systemd-journald.service seemed illusive.

Post by Rick Winscot
Several days ago I found a post on Stackoverflow that tied into udev

triggers that seemed promising; resulting in the method outlined below.
Initial testing shows proper timing with all dependencies satisfied.
However, the solution feels... hackey.

Post by Rick Winscot
My question for anyone on the list, is the method outlined below a

reasonable solution to mounting /var early in the start-up cycle?

Post by Rick Winscot
Or... is there a better way? Some trimming
Step One: Create a systemd service that mounts /var to the specified

partition

Post by Rick Winscot
Service: /etc/systemd/system/var.service
[Unit]
Description=service for mounting /var
DefaultDependencies=no
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p6
Step Two: Add a nofail mount
fstab: /etc/fstab
/dev/root / auto rw 0 1
/dev/mmcblk0p6 /var ext4 rw,nofail 0 0
Step Three: Add a wants dependency on the mount in udev triggers (some

lines deleted for brevity)

Post by Rick Winscot
Service: /usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
...
Wants=systemd-udevd.service var.service
[Service]
...
ExecStart=udevadm trigger --type=subsystems --action=add ;

/usr/bin/udevadm trigger

Post by Rick Winscot
Finally, systemd-analyze plot shows that the mount works as desired.
systemd-udev-trigger.service
var.service
dev-mmcblk0p6.device
var.mount
....
systemd-remount-fs.service
systemd-journal-flush.service
local-fs-pre.target
....
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[Michael Biebl]

Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot

Post by Rick Winscot
systemd 247

Ok, thanks

Post by Rick Winscot
/etc/systemd/journald.conf storage is persistent, systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.
Mounting /var on a separate read-write partition handles the persistent log requirement as well as offloading other read-write operations that can no longer live on the rootfs... being read-only.

Sure, but what is the actual problem? I do have systems with a
separate /var and don't remember any ordering issues / race
conditions.
Do you have any error messages / journal logs which show the issue?

[Rick Winscot]

15:31:18 localhost systemd[1]: Dependency failed for Flush Journal to
Persistent Storage.
░░ Subject: A start job for unit systemd-journal-flush.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit systemd-journal-flush.service has finished with a
failure.
░░
░░ The job identifier is 38 and the job result is dependency.

At this point, flush is attempting to re-route /run/log/journal to
/var/log/journal ... and the /var partition is not yet mounted. Units
generated for fstab in /run/systemd/generator that manage the mount have an
After=local-fs-pre.target which is too late.

Other dependency errors appear in the log; all with the same root cause. By
the time [ a specified service ] that uses /var is ready, the partition has
not yet been mounted. My solution resolves the /var mount as soon as the
block device is seen by udev - which made all the dependency errors go away.

Post by Michael Biebl
Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot

Post by Rick Winscot
systemd 247

Ok, thanks

Post by Rick Winscot
/etc/systemd/journald.conf storage is persistent,

systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.

Post by Rick Winscot
Mounting /var on a separate read-write partition handles the persistent

log requirement as well as offloading other read-write operations that can
no longer live on the rootfs... being read-only.
Sure, but what is the actual problem? I do have systems with a
separate /var and don't remember any ordering issues / race
conditions.
Do you have any error messages / journal logs which show the issue?

[Michael Biebl]

Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot

At this point, flush is attempting to re-route /run/log/journal to /var/log/journal ... and the /var partition is not yet mounted. Units generated for fstab in /run/systemd/generator that manage the mount have an After=local-fs-pre.target which is too late.
Other dependency errors appear in the log; all with the same root cause. By the time [ a specified service ] that uses /var is ready, the partition has not yet been mounted. My solution resolves the /var mount as soon as the block device is seen by udev - which made all the dependency errors go away.

Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
correctly started after /var has been mounted.
In case you are interested, I attached a journalctl dump, /etc/fstab
and systemd-analyze dump as well.
As you can see, systemd-journal-flush.service has a proper
After=var.mount ordering.

I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.

[Luca Boccassi]

Post by Michael Biebl
Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot

Post by Rick Winscot
At this point, flush is attempting to re-route /run/log/journal to

/var/log/journal ... and the /var partition is not yet mounted. Units
generated for fstab in /run/systemd/generator that manage the mount have an
After=local-fs-pre.target which is too late.

Post by Michael Biebl

Post by Rick Winscot
Other dependency errors appear in the log; all with the same root

cause. By the time [ a specified service ] that uses /var is ready, the
partition has not yet been mounted. My solution resolves the /var mount as
soon as the block device is seen by udev - which made all the dependency
errors go away.

Post by Michael Biebl
Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
correctly started after /var has been mounted.
In case you are interested, I attached a journalctl dump, /etc/fstab
and systemd-analyze dump as well.
As you can see, systemd-journal-flush.service has a proper
After=var.mount ordering.
I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.

As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service - I
had the same issue, because my /var/log partition is also not mounted in
the initramfs for $reasons, so it appears late at boot.
I'd suggest again to try and override it.

[Michael Biebl]

Am Sa., 1. Mai 2021 um 08:44 Uhr schrieb Luca Boccassi

Post by Michael Biebl
I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.

As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service - I had the same issue, because my /var/log partition is also not mounted in the initramfs for $reasons, so it appears late at boot.
I'd suggest again to try and override it.

I guess my point is, that before trying to suggest solutions, we
should understand what the underlying problem is first.

[Rick Winscot]

Luca, I have a similar setup where /var/log is not mounted in initramfs for
$reasons. I can’t believe I forgot to mention this!

Overriding with the tips you all have mentioned have things on the right
track!

Many thanks to the gurus of systemd!

Post by Rick Winscot

Post by Michael Biebl
Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot

Post by Rick Winscot
At this point, flush is attempting to re-route /run/log/journal to

/var/log/journal ... and the /var partition is not yet mounted. Units
generated for fstab in /run/systemd/generator that manage the mount have an
After=local-fs-pre.target which is too late.

Post by Michael Biebl

Post by Rick Winscot
Other dependency errors appear in the log; all with the same root

cause. By the time [ a specified service ] that uses /var is ready, the
partition has not yet been mounted. My solution resolves the /var mount as
soon as the block device is seen by udev - which made all the dependency
errors go away.

Post by Michael Biebl
Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
correctly started after /var has been mounted.
In case you are interested, I attached a journalctl dump, /etc/fstab
and systemd-analyze dump as well.
As you can see, systemd-journal-flush.service has a proper
After=var.mount ordering.
I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.

As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service -
I had the same issue, because my /var/log partition is also not mounted in
the initramfs for $reasons, so it appears late at boot.
I'd suggest again to try and override it.

[Norbert Lange]

I believe you can have a hard time if you dont do (the last step
atleast) with a var.mount file.
The mount files are special as systemd will pull those up implicitly
if they are needed.

Maybe you could rework your solution, so that one service mounts your
first or second choice in
a separate directory (/run/mymount) then you add a var.mount unit that
depends ("BindsTo") on said service
and bind-mounts /run/mymount to var.
That way you will hook into the existing systemd machinery.

I did something similar with an overlayfs once. Tested to work correctly,
even if you have further mounts like /var/tmp (systemd should
automatically add dependency to var.mount).

Norbert

# file rootfs-bindmount-var.service
[Unit]
Description=Bind-mount variable storage (/var)
Documentation=man:file-hierarchy(7)
ConditionPathIsSymbolicLink=!/var
# ConditionPathIsReadWrite=!/var
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=local-fs-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=-/bin/mkdir /run/varoverlay
ExecStartPre=/bin/mount --make-private -n -t tmpfs tmpfs_root_ovl
/run/varoverlay
ExecStartPre=/bin/mkdir /run/varoverlay/lower /run/varoverlay/upper
/run/varoverlay/work
ExecStart=/bin/mount --make-private -n --bind /var /run/varoverlay/lower
ExecStop=/bin/umount -n /run/varoverlay/lower
ExecStopPost=/bin/umount -n /run/varoverlay
ExecStopPost=/bin/rmdir /run/varoverlay


# file var.mount
[Unit]
Description=variable storage (/var)
Documentation=man:file-hierarchy(7)
ConditionPathIsSymbolicLink=!/var
ConditionPathIsReadWrite=!/var
After=rootfs-bindmount-var.service
BindsTo=rootfs-bindmount-var.service

[Mount]
What=overlay_var
Where=/var
Type=overlay
Options=lowerdir=/run/varoverlay/lower,upperdir=/run/varoverlay/upper,workdir=/run/varoverlay/work,redirect_dir=on,index=on,xino=on

[Dave Howorth]

On Fri, 30 Apr 2021 22:08:17 +0200

Post by Michael Biebl
Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot

Post by Rick Winscot
At this point, flush is attempting to re-route /run/log/journal
to /var/log/journal ... and the /var partition is not yet mounted.
Units generated for fstab in /run/systemd/generator that manage the
mount have an After=local-fs-pre.target which is too late.
Other dependency errors appear in the log; all with the same root
cause. By the time [ a specified service ] that uses /var is ready,
the partition has not yet been mounted. My solution resolves
the /var mount as soon as the block device is seen by udev - which
made all the dependency errors go away.

Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
correctly started after /var has been mounted.
In case you are interested, I attached a journalctl dump, /etc/fstab
and systemd-analyze dump as well.
As you can see, systemd-journal-flush.service has a proper
After=var.mount ordering.
I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.

If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
polite and log that fact somewhere. No?

[Michael Biebl]

Post by Dave Howorth
If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
polite and log that fact somewhere. No?

It should, yes.

[Michael Biebl]

Post by Michael Biebl

Post by Dave Howorth
If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
polite and log that fact somewhere. No?

It should, yes.

To be precise, it should log a message like here
https://sources.debian.org/src/systemd/247.3-5/src/core/transaction.c/?hl=414#L414

[Dave Howorth]

On Sat, 1 May 2021 18:08:55 +0200

Am Sa., 1. Mai 2021 um 18:07 Uhr schrieb Michael Biebl

Am Sa., 1. Mai 2021 um 17:46 Uhr schrieb Dave Howorth

Post by Dave Howorth
If systemd removes (i.e. doesn't obey) a directive, I'd expect it
to be polite and log that fact somewhere. No?

It should, yes.

To be precise, it should log a message like here
https://sources.debian.org/src/systemd/247.3-5/src/core/transaction.c/?hl=414#L414

Many thanks :) Rick and/or Luca should be able to find that if it is the
source of their problems!

PS however you sent your mail, it meant that my MUA didn't
automatically send a copy to the list when I hit reply. I'd much prefer
to keep everything on list.

[Luca Boccassi]

Post by Rick Winscot
We have an embedded product that uses a minimal Linux distribution

generated via Buildroot.

Post by Rick Winscot
Early in the project it was decided to make the rootfs read-only... in an

effort to improve durability in environments where power fluctuations might
cause problems on the eMMC. At the same time, making logging (e.g. /var)
persistent for debugging was added to requirements. Persistent storage
would be achieved by mounting /var to a separate partition that is
read-write.

Post by Rick Winscot
Several-hundred hours later... with many systemd-analyze reports and

various configurations tested, we have determined that managing the /var
mount with stadard services is not going to work due to tightly coupled and
precisely timed dependencies. Attempts with /etc/fstab and the systemd
generator are also unstable.

Post by Rick Winscot
Getting /var mounted in proximity to the initialization of

systemd-journald.service seemed illusive.

Post by Rick Winscot
Several days ago I found a post on Stackoverflow that tied into udev

triggers that seemed promising; resulting in the method outlined below.
Initial testing shows proper timing with all dependencies satisfied.
However, the solution feels... hackey.

Post by Rick Winscot
My question for anyone on the list, is the method outlined below a

reasonable solution to mounting /var early in the start-up cycle?

Post by Rick Winscot
Or... is there a better way? Some trimming
Step One: Create a systemd service that mounts /var to the specified

partition

Post by Rick Winscot
Service: /etc/systemd/system/var.service
[Unit]
Description=service for mounting /var
DefaultDependencies=no
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p6
Step Two: Add a nofail mount
fstab: /etc/fstab
/dev/root / auto rw 0 1
/dev/mmcblk0p6 /var ext4 rw,nofail 0 0
Step Three: Add a wants dependency on the mount in udev triggers (some

lines deleted for brevity)

Post by Rick Winscot
Service: /usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
...
Wants=systemd-udevd.service var.service
[Service]
...
ExecStart=udevadm trigger --type=subsystems --action=add ;

/usr/bin/udevadm trigger

Post by Rick Winscot
Finally, systemd-analyze plot shows that the mount works as desired.
systemd-udev-trigger.service
var.service
dev-mmcblk0p6.device
var.mount
....
systemd-remount-fs.service
systemd-journal-flush.service
local-fs-pre.target
....
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

You don't need to synchronize with the actual journald unit - you need to
synchronize with systemd-journal-flush.service which is what moves the
journal from /run/log to /var/log.
The default flush unit is ordered with systemd-tmpfiles-setup which is also
somewhat early-boot - what I do is mask systemd-journal-flush.service, and
provide my own without that ordering constraint.

If you mount your /var via mount units/fstab generator, then
RequiresMountsFor=/var/log/journal will take care of the ordering
automagically.

[Silvio Knizek]

Post by Rick Winscot
My question for anyone on the list, is the method outlined below a
reasonable solution to mounting /var early in the start-up cycle?
Or... is there a better way? Some trimming

Hi Rick,

by definition if you need to mount /var (or /usr for this argument),
you need an initrd [1] which actually set up everything as you
requires. Anything else has a tendency to break in unpleasant ways due
to race conditions and such. You don't need much, just enough to set up
everything required for the root and API file systems.

BR
Silvio

[1] https://systemd.io/INITRD_INTERFACE/

[Lennart Poettering]

Post by Silvio Knizek

Post by Rick Winscot
My question for anyone on the list, is the method outlined below a
reasonable solution to mounting /var early in the start-up cycle?
Or... is there a better way? Some trimming

Hi Rick,
by definition if you need to mount /var (or /usr for this argument),
you need an initrd [1] which actually set up everything as you
requires. Anything else has a tendency to break in unpleasant ways due
to race conditions and such. You don't need much, just enough to set up
everything required for the root and API file systems.

From systemd's side we actually explicitly support environments where
/var is mounted after the initrd transition. From our side everything
should just work, we should have all the necessary deps in place to
make /var being mounted post-initrd but still during early boot just
work. I'd consider a bug in systemd if something of systemd#s own
components can't deal with /var/ being mounted after the transition.

(I mean, there have been discussions on whether we shouldn't require
/var to be mounted from initrd, but so far we didn't decide that this
was necessary, given the political effort this would take to require)

Lennart

--
Lennart Poettering, Berlin

[Kenneth Porter]

--On Friday, April 30, 2021 11:39 AM -0400 Rick Winscot

Post by Rick Winscot
Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations
might cause problems on the eMMC. At the same time, making logging (e.g.
/var) persistent for debugging was added to requirements. Persistent
storage would be achieved by mounting /var to a separate partition that is
read-write.

Does /etc need to be read-only? On my last server I decided to make /usr
read-only but root is writable and /var is part of that. I put /home on its
own partition.

[Lennart Poettering]

Post by Kenneth Porter
--On Friday, April 30, 2021 11:39 AM -0400 Rick Winscot

Post by Rick Winscot
Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations
might cause problems on the eMMC. At the same time, making logging (e.g.
/var) persistent for debugging was added to requirements. Persistent
storage would be achieved by mounting /var to a separate partition that is
read-write.

Does /etc need to be read-only? On my last server I decided to make /usr
read-only but root is writable and /var is part of that. I put /home on its
own partition.

I think making /usr read-only makes a ton of sense.

The way I see it, besides the traditional Linux scheme where the whole
fs is writable the following two scenarios make the most sense, and
are what I personally intend to support in systemd very well:

1. root fs writable, /var/ part of it, but /usr/ separate and
read-only/immutable.

2. rootfs read-only/immutable, /usr/ part of it, but /var/ separate
and writable.

The main difference I that in the second case the configuration is
immutable too, while the firt case allows it to be changed locally.

Lennart

--
Lennart Poettering, Berlin