Discussion:
systemd.volatile=yes
(too old to reply)
Tobias Hunger
2017-09-03 18:23:59 UTC
Permalink
Raw Message
Hi,

I have been running a system based on a tmpfs as '/' and with a
read-only /usr for a while now and am rather happy with that setup. I
added "mount.usr" and similar flags to systemd ages ago, so that I
could configure that setup via kernel parameters. That has worked
great so far.

Recently I saw "systemd.volatile" in the documentation (e.g. here:
https://www.freedesktop.org/software/systemd/man/kernel-command-line.html)
and that "mount.usr*" is no longer documented. So I thought I'd move
over to the new way of doing things. The change was pretty simple to
do, I moved from "rootfs=tmpfs root=tmpfs rootflags=default
mount.usr=/device/path mount.usrflags=ro mount.usrfs=somefs" over to
"systemd.volatile=yes root=/device/path rootflags=ro rootfs=somefs".
Much simpler, nice:-)

The one pitfall I ran into is that I had to add a "usr" folder into
the usr partition for systemd-volatile-root.service to work. The
system boots well and seems to work nicely with this change.

But then I discovered one strange problem: I can not ssh into the root
account anymore!

ssh -v shows that a connection is established, then ssh is checking
for key files in /root/.ssh and does not find anything in there. Doing
"ls -alF /root/.ssh" as root does list keys there.

Mounting the same usr partition via "mount.usr*" kernel command line
parameters fixes the ssh login again.

The sshd.service files has no hardening options applied that might
explain the behavior.

Calling systemctl daemon-reload does not change anything (even when
making sure to stop sshd.socket and all SSH processes before doing
so).

My usr-partition does not contain anything but a /usr folder (with all
the necessary data) now that the typical folders found in /usr were
pushed down into that folder. There is no /root folder on it.

Any ideas what might be going wrong here?

Best Regards,
Tobias
Lennart Poettering
2017-09-04 09:06:45 UTC
Permalink
Raw Message
Post by Tobias Hunger
Hi,
I have been running a system based on a tmpfs as '/' and with a
read-only /usr for a while now and am rather happy with that setup. I
added "mount.usr" and similar flags to systemd ages ago, so that I
could configure that setup via kernel parameters. That has worked
great so far.
https://www.freedesktop.org/software/systemd/man/kernel-command-line.html)
and that "mount.usr*" is no longer documented. So I thought I'd move
over to the new way of doing things. The change was pretty simple to
do, I moved from "rootfs=tmpfs root=tmpfs rootflags=default
mount.usr=/device/path mount.usrflags=ro mount.usrfs=somefs" over to
"systemd.volatile=yes root=/device/path rootflags=ro rootfs=somefs".
Much simpler, nice:-)
Hmm, mount.usr= should continue to be supported. It's documented in
the systemd-fstab-generator man page however, not in the
kernel-command-line one. We should fix that however, can you file a
bug?
Post by Tobias Hunger
The one pitfall I ran into is that I had to add a "usr" folder into
the usr partition for systemd-volatile-root.service to work. The
system boots well and seems to work nicely with this change.
Uh, this shouldn't be necessary. Can you file a bug? I am really
surprised by this I must say... In my testing it didn't do that
either...
Post by Tobias Hunger
But then I discovered one strange problem: I can not ssh into the root
account anymore!
ssh -v shows that a connection is established, then ssh is checking
for key files in /root/.ssh and does not find anything in there. Doing
"ls -alF /root/.ssh" as root does list keys there.
This is very strange... Did you check that the perms of eahc component
of the path to /root/.ssh/[keys] actually are the same in both cases?

Lennart
--
Lennart Poettering, Red Hat
Tobias Hunger
2017-09-04 11:48:04 UTC
Permalink
Raw Message
Hi Lennart,

On Mon, Sep 4, 2017 at 11:06 AM, Lennart Poettering
Post by Lennart Poettering
Hmm, mount.usr= should continue to be supported. It's documented in
the systemd-fstab-generator man page however, not in the
kernel-command-line one. We should fix that however, can you file a
bug?
I'll file a merge request for that this week. I guess this is not that urgent;-)
Post by Lennart Poettering
Post by Tobias Hunger
The one pitfall I ran into is that I had to add a "usr" folder into
the usr partition for systemd-volatile-root.service to work. The
system boots well and seems to work nicely with this change.
Uh, this shouldn't be necessary. Can you file a bug? I am really
surprised by this I must say... In my testing it didn't do that
either...
src/volatile-root/volatile-root.c line 53: return log_error_errno(r,
"/usr not available in old root: %m");

Rereading the documentation on systemd.volatile, that is also pretty
much exactly what it says there: "[...] only /usr is mounted from the
file system configured as root device, in read-only mode.". My
assumption was that I can take a usr-partition as is (the one I used
to use with mount.usr*) is wrong, I need to move things down one
level.

But I do understand why you implemented this as is: Your way allows to
use any existing rootfs in a stateless setup without any special
preparation (provided /usr is not in a separate partition:-)

Once I get my setup rolling again, I plan to add dm-verity support to
my setup. I am curious how that will like your "remount the usr folder
from the already mounted root partition" approach.
Post by Lennart Poettering
Post by Tobias Hunger
But then I discovered one strange problem: I can not ssh into the root
account anymore!
ssh -v shows that a connection is established, then ssh is checking
for key files in /root/.ssh and does not find anything in there. Doing
"ls -alF /root/.ssh" as root does list keys there.
This is very strange... Did you check that the perms of eahc component
of the path to /root/.ssh/[keys] actually are the same in both cases?
Nope, since I have no idea how to move into the mount namespace that
sshd is running in.

The journal just lists the attempts to access /root/.ssh/idrsa (and
others), each followed by a line that the file is not found.

These files are actually created on the tmpfs by a custom
systemd-service in the initrd that just takes a file from the usr
partition and extracts it onto /. This service is run before the root
is moved over from the initrd to the real one.

The whole setup works nicely when using mount.usr* instead of
systemd.volatile, so I do not expect the files or their permissions to
be wrong themselves. They do also have the expected permissions when
checking them in the shell.

Best Regards,
Tobias
Tobias Hunger
2017-09-04 18:32:37 UTC
Permalink
Raw Message
Hi Lennart,

I probed a bit deeper: Apparently the openssh package is currently
borked in arch linux:-/ I ended up with a slightly different version
in the non systemd.volatile case which does work:-/

Sorry for the false alarm and wasting your time.

Best Regards,
Tobias

PS: I did send the update to the kernel-command-line man page via github.
Lennart Poettering
2017-09-05 07:16:12 UTC
Permalink
Raw Message
Post by Tobias Hunger
Hi Lennart,
On Mon, Sep 4, 2017 at 11:06 AM, Lennart Poettering
Post by Lennart Poettering
Hmm, mount.usr= should continue to be supported. It's documented in
the systemd-fstab-generator man page however, not in the
kernel-command-line one. We should fix that however, can you file a
bug?
I'll file a merge request for that this week. I guess this is not that urgent;-)
Post by Lennart Poettering
Post by Tobias Hunger
The one pitfall I ran into is that I had to add a "usr" folder into
the usr partition for systemd-volatile-root.service to work. The
system boots well and seems to work nicely with this change.
Uh, this shouldn't be necessary. Can you file a bug? I am really
surprised by this I must say... In my testing it didn't do that
either...
src/volatile-root/volatile-root.c line 53: return log_error_errno(r,
"/usr not available in old root: %m");
Rereading the documentation on systemd.volatile, that is also pretty
much exactly what it says there: "[...] only /usr is mounted from the
file system configured as root device, in read-only mode.". My
assumption was that I can take a usr-partition as is (the one I used
to use with mount.usr*) is wrong, I need to move things down one
level.
Oh right of course, we make this requirement so that you can take a
normal root fs, and boot it with systemd.volatile= sometimes but not
always, i.e. that you can choose on every boot whether you want to
boot in stateful or stateless mode. Or to say this differently: that a
root fs may contain /etc and /var and all that garbage, but if you
boot in volatile mode these are ignored, and only /usr is used.

Lennart
--
Lennart Poettering, Red Hat
Loading...