Discussion:
User configs, BTRFS subvolume for /home and automatic service start during boot?
(too old to reply)
Thorsten Schöning
2018-05-03 19:07:49 UTC
Permalink
Raw Message
Hi all,

I'm using Ubuntu 16.04 LTS server with BTRFS, which created one
subvolume for / and another one for /home. While I do like this setup,
it makes me trouble with how I deploy our own software developed for
various customers. For many reasons that software is stored under
/home[/tenant1|tenant2|...] currently, including systemd service files
nowadays. The good thing is that enabling those files is as easy as

> systemctl enable ...

with the absolute path. The bad thing is that because /home is another
file system, during boot systemd can't find my service files and
therefore can't start my services[1].

I'm looking for alternatives[2] now and thought of user configs. The
good thing is that my service files seem to be properly available to
systemd if I login e.g. using SSH, the status of the service is loaded
and I'm able to start it manually. But I have trouble getting systemd
to automatically start the service during boot and even during SSH
login.

I'm pretty sure to already have use the suggested "loginctl enable-linger ...",
but might have some error somewhere of course. But it might as well be
that the subvolume /home itself is making trouble here again. In the
end, there's the following sentence in the docs which doesn't
distinguish between system wide and user configs:

> The file system where the linked unit files are located must be
> accessible when systemd is started (e.g. anything underneath /home
> or /var is not allowed, unless those directories are located on the
> root file system).

So, are user configs for systemd supported to be placed on a non-root
file system AND be started during boot time? I don't have any users
logging in in the end.

Thanks!

[1]: https://serverfault.com/questions/910173/why-cant-systemd-find-service-files-within-a-btrfs-subvolume-within-a-btrfs-roo
[2]: https://serverfault.com/questions/910274/how-to-make-initrd-of-ubuntu-16-04-mount-home-automatically-in-addition-to

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: ***@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Andrei Borzenkov
2018-05-04 04:57:28 UTC
Permalink
Raw Message
03.05.2018 22:07, Thorsten Schöning пишет:

> I'm looking for alternatives[2] now and thought of user configs. The
> good thing is that my service files seem to be properly available to
> systemd if I login e.g. using SSH, the status of the service is loaded
> and I'm able to start it manually. But I have trouble getting systemd
> to automatically start the service during boot and even during SSH
> login.
>
> I'm pretty sure to already have use the suggested "loginctl enable-linger ...",
> but might have some error somewhere of course. But it might as well be
> that the subvolume /home itself is making trouble here again. In the
> end, there's the following sentence in the docs which doesn't
> distinguish between system wide and user configs:>
> So, are user configs for systemd supported to be placed on a non-root
> file system

I would say "yes", at least I do not see anything that prevents it.

> AND be started during boot time?

User services may be started by user instance which is started normally
by the first user session (which is not necessarily user login) and
stopped when the last user session ends. Or you can enable linger in
which case user instance will be started on boot.

Which services will actually be started depends on this user's
configuration, which services are enabled. It sounds like you may not
have enabled them or at least not enabled for the correct top-level target.

> I don't have any users
> logging in in the end.
>
Thorsten Schöning
2018-05-04 06:14:18 UTC
Permalink
Raw Message
Guten Tag Andrei Borzenkov,
am Freitag, 4. Mai 2018 um 06:57 schrieben Sie:

> Or you can enable linger in
> which case user instance will be started on boot.

That's the part not working in my setup for some reason.

> Which services will actually be started depends on this user's
> configuration, which services are enabled. It sounds like you may not
> have enabled them or at least not enabled for the correct top-level target.

I'm pretty sure they are enabled, all the documented directories,
files and links are available AND the services are started if I do it
manually using SSH and "systemctl start ...". The service itself is
pretty standard as well:

> [Unit]
> Description=Test /home being a subvolume in BTRFS and containing a service.
>
> [Install]
> WantedBy=multi-user.target
>
> [Service]
> Type=oneshot
> ExecStart=/bin/bash -c "echo Works: `date` >> /home/tschoening/btrfs_home_test/ping.txt"

Do you see any error? It is enabled using "systemctl --user enable ..."
using the absolut path and can be diabled the same way and started
manually. It's only not started automaticalyl, neither if I login
using SSH, nor at boot time.

So am I missing something in the service config or regarding
"enable-linger"?

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: ***@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Andrei Borzenkov
2018-05-04 10:07:11 UTC
Permalink
Raw Message
On Fri, May 4, 2018 at 9:14 AM, Thorsten Schöning <***@am-soft.de> wrote:
> Guten Tag Andrei Borzenkov,
> am Freitag, 4. Mai 2018 um 06:57 schrieben Sie:
>
>> Or you can enable linger in
>> which case user instance will be started on boot.
>
> That's the part not working in my setup for some reason.
>
>> Which services will actually be started depends on this user's
>> configuration, which services are enabled. It sounds like you may not
>> have enabled them or at least not enabled for the correct top-level target.
>
> I'm pretty sure they are enabled, all the documented directories,
> files and links are available AND the services are started if I do it
> manually

How does it confirm that units are enabled?

> using SSH and "systemctl start ...". The service itself is
> pretty standard as well:
>
>> [Unit]
>> Description=Test /home being a subvolume in BTRFS and containing a service.
>>
>> [Install]
>> WantedBy=multi-user.target
>>
>> [Service]
>> Type=oneshot
>> ExecStart=/bin/bash -c "echo Works: `date` >> /home/tschoening/btrfs_home_test/ping.txt"
>
> Do you see any error?

There is no multi-user.target in user instance (unless you added it
yourself), user instance starts default.target. Which is incidentally
true also for system instance unless explicitly overridden, usually on
kernel command line.

> It is enabled using "systemctl --user enable ..."
> using the absolut path

That just makes it depedency of multi-user.target which itself does not exist.

> and can be diabled the same way and started
> manually. It's only not started automaticalyl, neither if I login
> using SSH, nor at boot time.
>
> So am I missing something in the service config or regarding
> "enable-linger"?
>
> Mit freundlichen Grüßen,
>
> Thorsten Schöning
>
> --
> Thorsten Schöning E-Mail: ***@AM-SoFT.de
> AM-SoFT IT-Systeme http://www.AM-SoFT.de/
>
> Telefon...........05151- 9468- 55
> Fax...............05151- 9468- 88
> Mobil..............0178-8 9468- 04
>
> AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
> AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
>
> _______________________________________________
> systemd-devel mailing list
> systemd-***@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Thorsten Schöning
2018-05-04 14:13:35 UTC
Permalink
Raw Message
Guten Tag Andrei Borzenkov,
am Freitag, 4. Mai 2018 um 12:07 schrieben Sie:

> There is no multi-user.target in user instance (unless you added it
> yourself), user instance starts default.target.

Thanks, that was the problem, after changing the target everything
seems to work as expected. I didn't spot that difference to the docs I
was using.

Is there some counterpart for "systemctl --user ..." allowing me to
provide a username to work on instead of the "current user"? The only
thing in this direction I see is "--root" and providing a path to some
user's home dir.

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: ***@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Lennart Poettering
2018-05-04 16:09:44 UTC
Permalink
Raw Message
On Fr, 04.05.18 16:13, Thorsten Schöning (***@am-soft.de) wrote:

> Guten Tag Andrei Borzenkov,
> am Freitag, 4. Mai 2018 um 12:07 schrieben Sie:
>
> > There is no multi-user.target in user instance (unless you added it
> > yourself), user instance starts default.target.
>
> Thanks, that was the problem, after changing the target everything
> seems to work as expected. I didn't spot that difference to the docs I
> was using.
>
> Is there some counterpart for "systemctl --user ..." allowing me to
> provide a username to work on instead of the "current user"? The only
> thing in this direction I see is "--root" and providing a path to some
> user's home dir.

No, that's not going to work. You might be able to use "sudo" or "su"
for that however.

Lennart

--
Lennart Poettering, Red Hat
Lennart Poettering
2018-05-04 16:07:42 UTC
Permalink
Raw Message
On Do, 03.05.18 21:07, Thorsten Schöning (***@am-soft.de) wrote:

> Hi all,
>
> I'm using Ubuntu 16.04 LTS server with BTRFS, which created one
> subvolume for / and another one for /home. While I do like this setup,
> it makes me trouble with how I deploy our own software developed for
> various customers. For many reasons that software is stored under
> /home[/tenant1|tenant2|...] currently, including systemd service files
> nowadays. The good thing is that enabling those files is as easy as
>
> > systemctl enable ...
>
> with the absolute path. The bad thing is that because /home is another
> file system, during boot systemd can't find my service files and
> therefore can't start my services[1].

Hmm, you said /home was a subvolume? That suggests they are on the
same fs? Not following here?

> I'm looking for alternatives[2] now and thought of user configs. The
> good thing is that my service files seem to be properly available to
> systemd if I login e.g. using SSH, the status of the service is loaded
> and I'm able to start it manually. But I have trouble getting systemd
> to automatically start the service during boot and even during SSH
> login.
>
> I'm pretty sure to already have use the suggested "loginctl enable-linger ...",
> but might have some error somewhere of course. But it might as well be
> that the subvolume /home itself is making trouble here again. In the
> end, there's the following sentence in the docs which doesn't
> distinguish between system wide and user configs:
>
> > The file system where the linked unit files are located must be
> > accessible when systemd is started (e.g. anything underneath /home
> > or /var is not allowed, unless those directories are located on the
> > root file system).
>
> So, are user configs for systemd supported to be placed on a non-root
> file system AND be started during boot time? I don't have any users
> logging in in the end.

The comment above only applies to the systemd system instance,
i.e. PID 1. It does not apply to systemd user instances. I figure we
should clarify that in the docs...

Lennart

--
Lennart Poettering, Red Hat
Thorsten Schöning
2018-05-04 16:59:57 UTC
Permalink
Raw Message
Guten Tag Lennart Poettering,
am Freitag, 4. Mai 2018 um 18:07 schrieben Sie:

> Hmm, you said /home was a subvolume? That suggests they are on the
> same fs? Not following here?

I guess "/etc/fstab" is clearer than my explanation:

> # <file system> <mount point> <type> <options> <dump> <pass>
> # / was on /dev/sda1 during installation
> UUID=0841ef72-e9d4-45ca-af22-a403783859c6 / btrfs noatime,nodiratime,subvol=@ 0 1
>
> # /home was on /dev/sda1 during installation
> UUID=0841ef72-e9d4-45ca-af22-a403783859c6 /home btrfs noatime,nodiratime,subvol=@home 0 2

So the same BTRFS file system containing two sub volumes. That seems
to work for user configs, my problem was a wrong "WantedBy".

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: ***@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Lennart Poettering
2018-05-04 17:03:59 UTC
Permalink
Raw Message
On Fr, 04.05.18 18:59, Thorsten Schöning (***@am-soft.de) wrote:

> Guten Tag Lennart Poettering,
> am Freitag, 4. Mai 2018 um 18:07 schrieben Sie:
>
> > Hmm, you said /home was a subvolume? That suggests they are on the
> > same fs? Not following here?
>
> I guess "/etc/fstab" is clearer than my explanation:
>
> > # <file system> <mount point> <type> <options> <dump> <pass>
> > # / was on /dev/sda1 during installation
> > UUID=0841ef72-e9d4-45ca-af22-a403783859c6 / btrfs noatime,nodiratime,subvol=@ 0 1
> >
> > # /home was on /dev/sda1 during installation
> > UUID=0841ef72-e9d4-45ca-af22-a403783859c6 /home btrfs noatime,nodiratime,subvol=@home 0 2
>
> So the same BTRFS file system containing two sub volumes. That seems
> to work for user configs, my problem was a wrong "WantedBy".

Note that btrfs subvolumes are just special directories, hence you can
also do without an explicit entry for the home subvol, and simply
create it below the root subvol.

Lennart

--
Lennart Poettering, Red Hat
Thorsten Schöning
2018-05-04 17:35:24 UTC
Permalink
Raw Message
Guten Tag Lennart Poettering,
am Freitag, 4. Mai 2018 um 19:03 schrieben Sie:

> Note that btrfs subvolumes are just special directories, hence you can
> also do without an explicit entry for the home subvol, and simply
> create it below the root subvol.

Thanks for the hint, I'll think about that. The provided fstab was
from an Ubuntu default installation and I would like to keep that as
much as possible.

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: ***@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Julian Andres Klode
2018-05-04 18:55:48 UTC
Permalink
Raw Message
On Fri, May 04, 2018 at 07:03:59PM +0200, Lennart Poettering wrote:
> On Fr, 04.05.18 18:59, Thorsten Schöning (***@am-soft.de) wrote:
>
> > Guten Tag Lennart Poettering,
> > am Freitag, 4. Mai 2018 um 18:07 schrieben Sie:
> >
> > > Hmm, you said /home was a subvolume? That suggests they are on the
> > > same fs? Not following here?
> >
> > I guess "/etc/fstab" is clearer than my explanation:
> >
> > > # <file system> <mount point> <type> <options> <dump> <pass>
> > > # / was on /dev/sda1 during installation
> > > UUID=0841ef72-e9d4-45ca-af22-a403783859c6 / btrfs noatime,nodiratime,subvol=@ 0 1
> > >
> > > # /home was on /dev/sda1 during installation
> > > UUID=0841ef72-e9d4-45ca-af22-a403783859c6 /home btrfs noatime,nodiratime,subvol=@home 0 2
> >
> > So the same BTRFS file system containing two sub volumes. That seems
> > to work for user configs, my problem was a wrong "WantedBy".
>
> Note that btrfs subvolumes are just special directories, hence you can
> also do without an explicit entry for the home subvol, and simply
> create it below the root subvol.

That's not the best advise IMO. Subvolumes inside subvolumes have
weird behavior that sometimes bites you very hard. For example:
You can't restore an older / snapshot because it won't contain
your /home because it's a subvolume of the current /, but not of the
snapshot. It's just going to be a pain if you forget that.

I had this problem with lxd - it keeps containers in a btrfs subvolume
below /, and then I restored / to an older snapshot and all my containers
were gone / broken - I had to recreate them, as I only noticed later and
already had deleted my broken old "/"; so I switched to a mount for
them:

/dev/mapper/ubuntu--vg-root /var/snap/lxd/common/lxd/storage-pools/default btrfs defaults,subvol=@lxd 0 2

For Ubuntu, it's worth noting that the @ subvolume that's mounted to / is automatically
snapshotted when upgrading distributions; or even on normal upgrades with apt-btrfs-snapshot.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Loading...