Discussion:
systemd and NFS "bg" mounts.
Add Reply
NeilBrown
2017-05-26 02:46:43 UTC
Reply
Permalink
Raw Message
Hi all,
it appears that systemd doesn't play well with NFS "bg" mounts.
I can see a few options for how to address this and wonder if anyone
has any opinions.

"bg" mounts will try to mount the filesystem just like normal, but
if the server cannot be contacted before a "major timeout" (4.5 minutes
by default for TCP), mount.nfs will fork and continue in the
background. Meanwhile the original mount process reports success (even
though the filesystem wasn't mounted). This allows the boot process to
continue and succeed.

Currently if you specify the "bg" option in /etc/fstab and are using
systemd, the "bg" has no useful effect.
systemd imposes its own timeout of 90 seconds (which is less than 4.5
minutes).
After 90 seconds, systemd will kill the mount process and decide that
the mount failed. This will lead to remote-fs.target not being reached,
and boot not completing.

If you set TimeoutSec=0 (aka "infinity") for the mount unit, either by
hacking fstab-generator or adding "x-systemd.mount-timeout=infinity"
if you have systemd 233 or later, then systemd won't kill the mount
process, and after 4.5 minutes it will exit.
(to quote a comment in systemd/src/core/mount.c
" /bin/mount lies to us and is broken"
:-)

This is better, but the background mount.nfs can persist for a long
time. I don't think it persists forever, but at least half an hour I
think.

When the foo.mount unit is stopped, the mount.nfs process isn't killed.
I don't think this is a major problem, but it is unfortunate and could
be confusing. During testing I've had multiple mount.nfs background
processes all attached to the one .mount unit.

What should we do about bg NFS mounts with systemd?
Some options:
- declare "bg" to be not-supported. If you don't need the filesystem
to be present for boot, then use x-systemd.automount, or some other
automount scheme.
If we did this, we probably need to make it very obvious that "bg"
mounts aren't supported - maybe a log message that appears when
you do "systemctl status ..." ??

- decide that "bg" is really just a lame attempt at automounting, and
that now we have real automounting, "bg" can be translated to that.
So systemd-fstab-generator would treat "bg" like
"x-systemd.automount" and otherwise strip it from the list of
options.

- do our best to really support "bg". That means, at least, applying
a much larger timeout to "bg" mounts, and preferably killing any
background processes when a mount unit is stopped. Below is a
little patch which does this last bit, but I'm not sure it is generally
safe.

A side question is: should this knowledge about NFS be encoded in
systemd, or should nfs-utils add the necessary knowledge?

i.e. we could add an nfs-fstab-generator to nfs-utils which creates
drop-ins to modify the behaviour of the drop-ins provided by
systemd-fstab-generator.
Adding the TimeoutSec= would be easy. Stripping the "bg" would be
possible.
Changing the remote-fs.target.requires/foo.mount symlink to be
remote-fs.target.requires/foo.automount would be problematic though.

Could we teach systemd-fstab-generator to ignore $TYPE filesystems
if TYPE-fstab-generator existed?

Or should we just build all this filesystem-specific knowledge into
systemd?

Thanks for your thoughts,
NeilBrown


hackish patch to kill backgrounded mount.nfs processes:

diff --git a/src/core/mount.c b/src/core/mount.c
index ca0c4b0d5eed..91939b48d11a 100644
--- a/src/core/mount.c
+++ b/src/core/mount.c
@@ -883,6 +883,18 @@ static void mount_enter_unmounting(Mount *m) {
MOUNT_UNMOUNTING_SIGKILL))
m->n_retry_umount = 0;

+ if (m->result == MOUNT_SUCCESS &&
+ !m->from_proc_self_mountinfo) {
+ /* There is no mountpoint, but mount seemed to succeed.
+ * Could be a bg mount.nfs.
+ * In any case, kill any processes that might be hanging
+ * around, they cannot be doing anything useful.
+ */
+ sd_bus_error error = SD_BUS_ERROR_NULL;
+ unit_kill_common(UNIT(m), KILL_ALL, SIGTERM, -1, -1, &error);
+ }
+
+
m->control_command_id = MOUNT_EXEC_UNMOUNT;
m->control_command = m->exec_command + MOUNT_EXEC_UNMOUNT;
Lennart Poettering
2017-05-29 13:38:14 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Hi all,
it appears that systemd doesn't play well with NFS "bg" mounts.
I can see a few options for how to address this and wonder if anyone
has any opinions.
Yeah, this has come up before. Long story short: "bg" is simply not
compatible with systemd. We assume that the /bin/mount's effect is
visible in /proc/self/mountinfo, and everything else is considered a
bug, i.e. /bin/mount lying to us. And I think that's a pretty rational
assumption and requirement to make.

I am not particularly convinced the "bg" usecase is really such a
great idea, since it is necessarily racy: you never know whether mount
is actually in effect or not, even though /bin/mount claims so. I am
pretty sure other options (such as autofs mounts, which are dead-easy
to use in system: just replace the "bg" mount option in fstab by
"x-systemd.automount") are much better approaches to the problem at
hand: they also make your local system less dependent on remote
network access, but they do give proper guarantees about their
runtime: when the autofs mount is established the path is available.

Hence I am tempted to treat the issue as a documentation and warning
issue: accept that "bg" is not supported, but document this better. In
addition, we should probably log about "bg" being broken in the
fstab-generator. I file a bug about that now:

https://github.com/systemd/systemd/issues/6046
Post by NeilBrown
This is better, but the background mount.nfs can persist for a long
time. I don't think it persists forever, but at least half an hour I
think.
When the foo.mount unit is stopped, the mount.nfs process isn't killed.
Hmm, if you see this, then this would be a bug: mount units that are
stopped should not leave processes around.
Post by NeilBrown
I don't think this is a major problem, but it is unfortunate and could
be confusing. During testing I've had multiple mount.nfs background
processes all attached to the one .mount unit.
Humpf, could you file a bug?

While I think the "bg" concept is broken, as discussed above, having
FUSE mounts with processes in the background is actually supported,
and we should clean them up properly when the mount unit is stopped.

Hmm, maybe mount.nfs isn't properly killable? i.e. systemd tries to
kill it, but it refuses to be killed?
Post by NeilBrown
What should we do about bg NFS mounts with systemd?
- declare "bg" to be not-supported. If you don't need the filesystem
to be present for boot, then use x-systemd.automount, or some other
automount scheme.
If we did this, we probably need to make it very obvious that "bg"
mounts aren't supported - maybe a log message that appears when
you do "systemctl status ..." ??
I am all for this, as suggested above. I'd only log from
fstab-generator though. (That said, if we want something stronger, we
could also add the fact that we stripped "bg" from the mount optoins
to the DEscription= of the generated mount unit.)
Post by NeilBrown
- decide that "bg" is really just a lame attempt at automounting, and
that now we have real automounting, "bg" can be translated to that.
So systemd-fstab-generator would treat "bg" like
"x-systemd.automount" and otherwise strip it from the list of
options.
I am a bit afraid of this I must say. The semantics are different
enough to likely cause more problems then we'd solve with this. Not
supporting this at all sounds like the much better approach here:
let's strip "bg" when specified.
Post by NeilBrown
- do our best to really support "bg". That means, at least, applying
a much larger timeout to "bg" mounts, and preferably killing any
background processes when a mount unit is stopped. Below is a
little patch which does this last bit, but I'm not sure it is generally
safe.
As mentioned I think this would just trade one race for a couple of
new ones, and that appears to be a bad idea to me.
Post by NeilBrown
A side question is: should this knowledge about NFS be encoded in
systemd, or should nfs-utils add the necessary knowledge?
I am pretty sure we should keep special understanding of NFS at a
minimum in PID 1, but I think we can be less strict in
fstab-generator, as its primary job is compat with UNIX anyway.
Post by NeilBrown
i.e. we could add an nfs-fstab-generator to nfs-utils which creates
drop-ins to modify the behaviour of the drop-ins provided by
systemd-fstab-generator.
Adding the TimeoutSec= would be easy. Stripping the "bg" would be
possible.
Changing the remote-fs.target.requires/foo.mount symlink to be
remote-fs.target.requires/foo.automount would be problematic
though.
Well, I'd be fine with letting NFS do its own handling of the NFS
/etc/fstab entries, but I think the special casing of "bg" is fine to
simply add to the existing generator in systemd.
Post by NeilBrown
Could we teach systemd-fstab-generator to ignore $TYPE filesystems
if TYPE-fstab-generator existed?
Well, generators can override each other in very limited ways (as
there are three different output directories a gnerator can write to,
which are inserted at different places in the unit file search path),
we could build on that. That said, I think adding this to
fstab-generator in systemd is OK.
Post by NeilBrown
Or should we just build all this filesystem-specific knowledge into
systemd?
For now, I think adding this to systemd's fstab-generator would be fine.

Hope this makes sense,

Lennart
--
Lennart Poettering, Red Hat
NeilBrown
2017-05-29 22:05:57 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by NeilBrown
Hi all,
it appears that systemd doesn't play well with NFS "bg" mounts.
I can see a few options for how to address this and wonder if anyone
has any opinions.
Yeah, this has come up before. Long story short: "bg" is simply not
compatible with systemd. We assume that the /bin/mount's effect is
visible in /proc/self/mountinfo, and everything else is considered a
bug, i.e. /bin/mount lying to us. And I think that's a pretty rational
assumption and requirement to make.
I am not particularly convinced the "bg" usecase is really such a
great idea, since it is necessarily racy: you never know whether mount
is actually in effect or not, even though /bin/mount claims so. I am
pretty sure other options (such as autofs mounts, which are dead-easy
to use in system: just replace the "bg" mount option in fstab by
"x-systemd.automount") are much better approaches to the problem at
hand: they also make your local system less dependent on remote
network access, but they do give proper guarantees about their
runtime: when the autofs mount is established the path is available.
Hence I am tempted to treat the issue as a documentation and warning
issue: accept that "bg" is not supported, but document this better. In
addition, we should probably log about "bg" being broken in the
https://github.com/systemd/systemd/issues/6046
There is a weird distorted sort of justice here. When NFS first
appeared, it broke various long-standing Unix practices, such as
O_EXCL|O_CREAT for lock files. Now systemd appears and breaks a
long-standing NFS practice: bg mounts.

I hoped we could find a way to make them work, but I won't cry over
their demise. I much prefer automount .... I think all NFS mounts
should be automounts.

I see this is already documented in systemd.mount:

The NFS mount option bg for NFS background mounts as documented in nfs(5) is
not supported in /etc/fstab entries.

I wonder how many people actually read that.
We should probably add symmetric documentation to nfs(5)
Both should give clear pointers to x-systemd.automount.
Post by Lennart Poettering
Post by NeilBrown
This is better, but the background mount.nfs can persist for a long
time. I don't think it persists forever, but at least half an hour I
think.
When the foo.mount unit is stopped, the mount.nfs process isn't killed.
Hmm, if you see this, then this would be a bug: mount units that are
stopped should not leave processes around.
Post by NeilBrown
I don't think this is a major problem, but it is unfortunate and could
be confusing. During testing I've had multiple mount.nfs background
processes all attached to the one .mount unit.
Humpf, could you file a bug?
https://github.com/systemd/systemd/issues/6048
Post by Lennart Poettering
While I think the "bg" concept is broken, as discussed above, having
FUSE mounts with processes in the background is actually supported,
and we should clean them up properly when the mount unit is stopped.
Hmm, maybe mount.nfs isn't properly killable? i.e. systemd tries to
kill it, but it refuses to be killed?
mount.nfs responds cleanly to SIGTERM.
Post by Lennart Poettering
Post by NeilBrown
What should we do about bg NFS mounts with systemd?
- declare "bg" to be not-supported. If you don't need the filesystem
to be present for boot, then use x-systemd.automount, or some other
automount scheme.
If we did this, we probably need to make it very obvious that "bg"
mounts aren't supported - maybe a log message that appears when
you do "systemctl status ..." ??
I am all for this, as suggested above. I'd only log from
fstab-generator though. (That said, if we want something stronger, we
could also add the fact that we stripped "bg" from the mount optoins
to the DEscription= of the generated mount unit.)
That last bit sounds like a very good idea. Stripping "bg" could be seen
as a "surprising" thing for fstab-generator to do. Making it as obvious
as possible to the sys-admin would be a good thing (and would probably
make support personnel happy too).
Post by Lennart Poettering
Post by NeilBrown
- decide that "bg" is really just a lame attempt at automounting, and
that now we have real automounting, "bg" can be translated to that.
So systemd-fstab-generator would treat "bg" like
"x-systemd.automount" and otherwise strip it from the list of
options.
I am a bit afraid of this I must say. The semantics are different
enough to likely cause more problems then we'd solve with this. Not
let's strip "bg" when specified.
Post by NeilBrown
- do our best to really support "bg". That means, at least, applying
a much larger timeout to "bg" mounts, and preferably killing any
background processes when a mount unit is stopped. Below is a
little patch which does this last bit, but I'm not sure it is generally
safe.
As mentioned I think this would just trade one race for a couple of
new ones, and that appears to be a bad idea to me.
Post by NeilBrown
A side question is: should this knowledge about NFS be encoded in
systemd, or should nfs-utils add the necessary knowledge?
I am pretty sure we should keep special understanding of NFS at a
minimum in PID 1, but I think we can be less strict in
fstab-generator, as its primary job is compat with UNIX anyway.
I was thinking about which source package the knowledge would be in, and
hence which set of maintainers would over-see it. I don't expect
systemd maintainers to be fully in-touch with the details of NFS, but
then NFS developers cannot be fully in-touch with how systemd works.

Apart from some documentation changes, we probably don't need to put
anything new in nfs-utils at the moment.
Post by Lennart Poettering
Post by NeilBrown
i.e. we could add an nfs-fstab-generator to nfs-utils which creates
drop-ins to modify the behaviour of the drop-ins provided by
systemd-fstab-generator.
Adding the TimeoutSec= would be easy. Stripping the "bg" would be
possible.
Changing the remote-fs.target.requires/foo.mount symlink to be
remote-fs.target.requires/foo.automount would be problematic
though.
Well, I'd be fine with letting NFS do its own handling of the NFS
/etc/fstab entries, but I think the special casing of "bg" is fine to
simply add to the existing generator in systemd.
Post by NeilBrown
Could we teach systemd-fstab-generator to ignore $TYPE filesystems
if TYPE-fstab-generator existed?
Well, generators can override each other in very limited ways (as
there are three different output directories a gnerator can write to,
which are inserted at different places in the unit file search path),
we could build on that. That said, I think adding this to
fstab-generator in systemd is OK.
Ahh, of course. That is what normal-dir / early-dir / late-dir is for.
I'll keep that in mind in case I do ever need it.
Post by Lennart Poettering
Post by NeilBrown
Or should we just build all this filesystem-specific knowledge into
systemd?
For now, I think adding this to systemd's fstab-generator would be fine.
Hope this makes sense,
Yes it does. It wasn't the outcome I was hoping for, but it is hard to
argue against it.

Thanks a lot,
NeilBrown
NeilBrown
2017-05-29 22:19:16 UTC
Reply
Permalink
Raw Message
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.

Explain this.

See also https://github.com/systemd/systemd/issues/6046

Signed-off-by: NeilBrown <***@suse.com>
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
--
2.12.2
Niels de Vos
2017-05-30 04:47:54 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
Explain this.
See also https://github.com/systemd/systemd/issues/6046
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
--
2.12.2
I like this, it makes it so much easier for users to find the better
alternative.

FWIW,
Reviewed-by: Niels de Vos <***@redha.com>
Michael Biebl
2017-05-30 07:40:07 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
+.B bg
+option is not supported, and may be stripped from the option list.
Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.

Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
NeilBrown
2017-05-30 08:55:15 UTC
Reply
Permalink
Raw Message
Post by Michael Biebl
Post by NeilBrown
+.B bg
+option is not supported, and may be stripped from the option list.
Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.
It depends on which version of systemd is in use.
Even if/when systemd is updated to strip the 'bg', it would not
be correct to say "and will be stripped..." as that isn't true for all
versions.
We could spell it out "and may be stripped from the option list,
depending on which version of systemd is installed", but I'm not sure
that really helps...
"... and some versions of systemd will strip 'bg' from the option list"
??

Thanks,
NeilBrown
Michael Biebl
2017-05-30 09:15:10 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Post by Michael Biebl
Post by NeilBrown
+.B bg
+option is not supported, and may be stripped from the option list.
Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.
It depends on which version of systemd is in use.
Even if/when systemd is updated to strip the 'bg', it would not
be correct to say "and will be stripped..." as that isn't true for all
versions.
We could spell it out "and may be stripped from the option list,
depending on which version of systemd is installed", but I'm not sure
that really helps...
"... and some versions of systemd will strip 'bg' from the option list"
??
So far, no version of systemd strips bg from the option list.
If in version X systemd actually starts stripping bg, we should
explicitly mention that version in the man page imho.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2017-05-30 12:45:58 UTC
Reply
Permalink
Raw Message
Post by Michael Biebl
Post by NeilBrown
Post by Michael Biebl
Post by NeilBrown
+.B bg
+option is not supported, and may be stripped from the option list.
Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.
It depends on which version of systemd is in use.
Even if/when systemd is updated to strip the 'bg', it would not
be correct to say "and will be stripped..." as that isn't true for all
versions.
We could spell it out "and may be stripped from the option list,
depending on which version of systemd is installed", but I'm not sure
that really helps...
"... and some versions of systemd will strip 'bg' from the option list"
??
So far, no version of systemd strips bg from the option list.
If in version X systemd actually starts stripping bg, we should
explicitly mention that version in the man page imho.
I figure it's up to the NFS folks to decide on this, but if I were
them I would not mention any version numbers. That just gets out of
date. Instead, document the new behaviour and indicate in the README
file that the documentation assumes systemd with a certain version is
used, and then let the packers figure out the rest, i.e. if they want
to place some kind of weak package version dependency or not.

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-05-30 12:43:12 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
Explain this.
See also https://github.com/systemd/systemd/issues/6046
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
The option is called "x-systemd.automount", i.e. one more "d".
Post by NeilBrown
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
Looks good to me otherwise!

Lennart
--
Lennart Poettering, Red Hat
Steve Dickson
2017-06-06 18:07:57 UTC
Reply
Permalink
Raw Message
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...

It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???

So I'm a bit hesitant to commit this since not accurate, yet.

Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...

Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.

Note, the 'bg' is used by clients that do want their
booting to hang by servers that are down so if the
option is rip out, boots will start hang. This
will make it very difficult to debug since the bg
will still exist in fstab.

Again, the whole concept of systemd messing with mounts options
is just not a good one... IMHO..

steved.
Post by NeilBrown
Explain this.
See also https://github.com/systemd/systemd/issues/6046
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
Michael Biebl
2017-06-06 19:57:32 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Personally, I would prefer if systemd simply logged a warning/error
message but would *not* strip the bg option.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2017-06-07 08:13:13 UTC
Reply
Permalink
Raw Message
Post by Michael Biebl
Post by Steve Dickson
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Personally, I would prefer if systemd simply logged a warning/error
message but would *not* strip the bg option.
What good does that do if "bg" doesn't actually work properly?

Lennart
--
Lennart Poettering, Red Hat
Steve Dickson
2017-06-07 09:42:39 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by Michael Biebl
Post by Steve Dickson
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Personally, I would prefer if systemd simply logged a warning/error
message but would *not* strip the bg option.
What good does that do if "bg" doesn't actually work properly?
At the moment it works just fine...

steved.
Post by Lennart Poettering
Lennart
NeilBrown
2017-06-06 21:49:24 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.

A working "bg" mount would successfully mount the filesystem is the
server came back after 5 minutes. If you use current systemd in the
default configuration, it won't.

bg mounts do work sometimes, but I don't think they are reliable, and
there seems to be no interest in changing this.
Maybe the text should say "Systemd does not, and will not, reliably
support "bg" mounts...".
Post by Steve Dickson
It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???
Might be related to my patch
[PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()

sent 25th May.
Post by Steve Dickson
So I'm a bit hesitant to commit this since not accurate, yet.
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
It isn't exactly systemd, it is systemd-fstab-generator.
The options lists in /etc/fstab are not all equal. Some
are relevant to /bin/mount, some to mount.nfs, some to the kernel.
I think /bin/mount processes the option lists before passing it
to mount.nfs. There is no intrinsic reason that systemd-fstab-generator
shouldn't do the same thing.
Post by Steve Dickson
Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.
I could suggest that automount is uniformly better than bg. Give how
long automount has been around, and how easy it is to use with systemd,
it might be time to start encouraging people to stop using the inferior
technology.

I could say that, but I'm not 100% sure. The difference between
automount and bg is that with bg it is easy to see if the mount has
succeeded yet - just look for an empty directory. With automount,
you'll typically get a delay at that point. We could possibly wind down
that delay...
Post by Steve Dickson
Note, the 'bg' is used by clients that do want their
booting to hang by servers that are down so if the
option is rip out, boots will start hang. This
will make it very difficult to debug since the bg
will still exist in fstab.
Not exactly.
Current default behaviour is that systemd will wait 90 seconds, then
kill the mount program and fail the boot. If we strip out "bg", the
same thing will happen.

I'm OK with the patch not being applied just yet. I think we need to
resolve this issue, but it isn't 100% clear to me what the best
resolution would be. So I'm happy to see a conversation happening and
opinions being shared.
I'd be particularly pleased if you could double check how "bg" is
currently handled on some systemd-enabled system that you use.
Does the mount program get killed like I see? Does boot succeed if
there is a bg mount from an unresponsive server?

Thanks,
NeilBrown
Post by Steve Dickson
Again, the whole concept of systemd messing with mounts options
is just not a good one... IMHO..
steved.
Post by NeilBrown
Explain this.
See also https://github.com/systemd/systemd/issues/6046
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
Steve Dickson
2017-06-07 10:08:18 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me

# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"

# ps ax | grep mount.nfs
2270 ? Ss 0:00 /sbin/mount.nfs rhel7srv:/home/tmp /mnt/tmp -v -o rw,bg
Post by NeilBrown
A working "bg" mount would successfully mount the filesystem is the
server came back after 5 minutes. If you use current systemd in the
default configuration, it won't.
When I add a bg mount to fstab again... it works just fine. This
is with the latest upstream nfs-utils.
Post by NeilBrown
bg mounts do work sometimes, but I don't think they are reliable, and
there seems to be no interest in changing this.
Maybe the text should say "Systemd does not, and will not, reliably
support "bg" mounts...".
not reliable maybe... I'm just doing very simple testing...
Post by NeilBrown
Post by Steve Dickson
It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???
Might be related to my patch
[PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()
sent 25th May.
I'll take a look.. thanks!
Post by NeilBrown
Post by Steve Dickson
So I'm a bit hesitant to commit this since not accurate, yet.
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
It isn't exactly systemd, it is systemd-fstab-generator.
The options lists in /etc/fstab are not all equal. Some
are relevant to /bin/mount, some to mount.nfs, some to the kernel.
I think /bin/mount processes the option lists before passing it
to mount.nfs. There is no intrinsic reason that systemd-fstab-generator
shouldn't do the same thing.
OK.
Post by NeilBrown
Post by Steve Dickson
Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.
I could suggest that automount is uniformly better than bg. Give how
long automount has been around, and how easy it is to use with systemd,
it might be time to start encouraging people to stop using the inferior
technology.
Yes... bg mounts are a poor man's automount...
Post by NeilBrown
I could say that, but I'm not 100% sure. The difference between
automount and bg is that with bg it is easy to see if the mount has
succeeded yet - just look for an empty directory. With automount,
you'll typically get a delay at that point. We could possibly wind down
that delay...
Post by Steve Dickson
Note, the 'bg' is used by clients that do want their
booting to hang by servers that are down so if the
option is rip out, boots will start hang. This
will make it very difficult to debug since the bg
will still exist in fstab.
Not exactly.
Current default behaviour is that systemd will wait 90 seconds, then
kill the mount program and fail the boot. If we strip out "bg", the
same thing will happen.
Again.. I'm not seeing this 90 sec delay when I add a bg mount
to /etc/fstab.
Post by NeilBrown
I'm OK with the patch not being applied just yet. I think we need to
resolve this issue, but it isn't 100% clear to me what the best
resolution would be. So I'm happy to see a conversation happening and
opinions being shared.
I'd be particularly pleased if you could double check how "bg" is
currently handled on some systemd-enabled system that you use.
Does the mount program get killed like I see?
No. after adding a bg mount to fstab and rebooting (with the server
down) I see the following mount in backgroup
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
Post by NeilBrown
Does boot succeed if there is a bg mount from an unresponsive server?
Yes. Then when I bring up the server the mount succeeds

steved.

P.S. I'm taking some PTO today so I will not be back in the office
until later today or tomorrow...

steved.
Post by NeilBrown
Thanks,
NeilBrown
Post by Steve Dickson
Again, the whole concept of systemd messing with mounts options
is just not a good one... IMHO..
steved.
Post by NeilBrown
Explain this.
See also https://github.com/systemd/systemd/issues/6046
---
utils/mount/nfs.man | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@ Alternatively these issues can be addressed
using an automounter (refer to
.BR automount (8)
for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option. This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot. The mount still happens in
+the "background", though in a different way.
.TP 1.5i
.BR rdirplus " / " nordirplus
Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
.BR rpc.idmapd (8),
.BR rpc.gssd (8),
.BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
.sp
RFC 768 for the UDP specification.
.br
Lennart Poettering
2017-06-07 12:02:21 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me
# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"
We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.

Lennart
--
Lennart Poettering, Red Hat
Steve Dickson
2017-06-07 19:48:04 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by Steve Dickson
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me
# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"
We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.
I guess I was not clear... After adding a bg mount to fstab and
reboot, mounting a server that is not up, there is a mount in
background that looks like

# ps ax | grep mount
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg

Looking at the remote-fs.target status:

# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)

Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.

It appears to be successful

steved.
NeilBrown
2017-06-08 05:16:52 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
Post by Lennart Poettering
Post by Steve Dickson
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me
# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"
We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.
I guess I was not clear... After adding a bg mount to fstab and
reboot, mounting a server that is not up, there is a mount in
background that looks like
# ps ax | grep mount
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)
Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
It appears to be successful
Strange ... not for me.

I have a recent compiled-from-source upstream systemd and a very recent
upstream kernel.

I add a line to /etc/fstab

192.168.1.111:/nowhere /mnt nfs bg 0 0

and reboot (192.168.1.111 is on a different subnet to the VM I am
testing in, but no host responds to the address).

There is a 1m30s wait while trying to mount /mnt, then boot completes
(I was wrong when I said that it didn't).

● mnt.mount - /mnt
Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
Where: /mnt
What: 192.168.1.111:/nowhere
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)

Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.


The /bin/mount process has been killed (SIGTERM) after the 90 second
timeout

● remote-fs.target - Remote File Systems
Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/remote-fs.target.d
└─50-insserv.conf.conf
Active: inactive (dead)
Docs: man:systemd.special(7)

Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.


remote-fs.target has not been reached.

Because remote-fs.target is WantedBy multi-user.target, but need
RequiredBy it, boot completes.
But if anything did Require remote-fs.target, it would fail if "bg"
mounts were not mounted within 90 seconds.
Post by Steve Dickson
Post by Lennart Poettering
Post by Steve Dickson
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
background the process immediately. Systemd sees that as success
(despite the fact that the filesystem isn't actually mounted) and
doesn't clean up the background processes (which is arguably a bug).

If you try to mount from a server which doesn't even return ECONNREFUSED or
EHOSTUNREACH, such that the first mount attempt takes longer than 90
seconds, then you should get the same results as me.

Your results go some way to explaining why Lennart hasn't received many
complaints, but I'm convinced that the current situation is not ideal
(because SUSE has received some complaints).

I've been pondering the possibility of making "bg" work properly with
systemd and I think I've found a promising approach. It involves having
systemd take responsibility for the "run in the background" part.

If we get systemd-fstab-generator to translate "bg" to "retry=10000",
then "mount.nfs" will behave like the background version of
"mount.nfs -o bg". i.e. it will retry for one month (nearly). If there is
already a 'retry=' option, we just strip the "bg".

For this to work, we would need to add
TimeoutSec=infinity
to the .mount unit file so that systemd doesn't kill the mount.
We would also need to add (the effect of) "nofail", so that systemd
doesn't wait for the mount to complete...
Except that the current implementation of "nofail" is faulty.
It removes the default "Before=remote-fs.mount", which has the unwanted
consequence of unmounting the filesystem too early at shutdown.

I have a solution for that too (which I'll submit a push request for
shortly).
My solution to "nofail" is to treat it much like "automount", but
instead of using an automount unit to trigger the mount, we use
a timer unit (with OnActiveSec=0).
By triggering the mount unit with a timer instead of Wanting it
directly, it gets run in a separate transaction. This means that the
"Before=remote-fs.target" doesn't have the effect of delaying
remote-fs.target. Before/After only order jobs within the one
transaction.
When it comes time to shutdown, remote-fs.target and the foo.mount will
be in the same transaction, so the Before= will ensure foo.mount
isn't unmounted until after remote-fs.target has been allows to finish,
which is after any services that might be using the filesystem.

So I think I've found a solution for systemd to handle "bg" nfs mounts
correctly. I'll submit some pull requests for consideration.

Thanks,
NeilBrown
Mantas Mikulėnas
2017-06-08 05:56:41 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
background the process immediately. Systemd sees that as success
(despite the fact that the filesystem isn't actually mounted) and
doesn't clean up the background processes (which is arguably a bug).
Cleaning up background processes for successful mounts would break every
FUSE filesystem.
--
Mantas Mikulėnas <***@gmail.com>
Steve Dickson
2017-06-08 20:24:06 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Post by Steve Dickson
Post by Lennart Poettering
Post by Steve Dickson
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me
# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"
We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.
I guess I was not clear... After adding a bg mount to fstab and
reboot, mounting a server that is not up, there is a mount in
background that looks like
# ps ax | grep mount
1104 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)
Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
It appears to be successful
Strange ... not for me.
I have a recent compiled-from-source upstream systemd and a very recent
upstream kernel.
I add a line to /etc/fstab
192.168.1.111:/nowhere /mnt nfs bg 0 0
and reboot (192.168.1.111 is on a different subnet to the VM I am
testing in, but no host responds to the address).
There is a 1m30s wait while trying to mount /mnt, then boot completes
(I was wrong when I said that it didn't).
● mnt.mount - /mnt
Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
Where: /mnt
What: 192.168.1.111:/nowhere
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)
Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.
The /bin/mount process has been killed (SIGTERM) after the 90 second
timeout
● remote-fs.target - Remote File Systems
Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Drop-In: /run/systemd/generator/remote-fs.target.d
└─50-insserv.conf.conf
Active: inactive (dead)
Docs: man:systemd.special(7)
Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
remote-fs.target has not been reached.
I'm seeing this now... the server has to be up to cause this hang.
Post by NeilBrown
Because remote-fs.target is WantedBy multi-user.target, but need
RequiredBy it, boot completes.
But if anything did Require remote-fs.target, it would fail if "bg"
mounts were not mounted within 90 seconds.
Post by Steve Dickson
Post by Lennart Poettering
Post by Steve Dickson
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
background the process immediately. Systemd sees that as success
(despite the fact that the filesystem isn't actually mounted) and
doesn't clean up the background processes (which is arguably a bug).
No... Systemd is doing the right thing in this case... Letting
bg mounts work...
Post by NeilBrown
If you try to mount from a server which doesn't even return ECONNREFUSED or
EHOSTUNREACH, such that the first mount attempt takes longer than 90
seconds, then you should get the same results as me.
This is clearly a bug in systemd! The 'bg' mounts work with the server
down but hangs when the server is up. How can't this be a bug in systemd??

Why is systemd even looking at or interpreting the mount options???
This clear an overreach of systemd, IMHO. Then just to blindly rip
them out... WOW!

I'm only assuming... but it appears, do to this overreach, that systemd
actually think it know how to mount a file system better than the
actual mount command tailor for particular that file system and
if it doesn't like an mount option it just going to rip it
out! Again, I hope this is not the case because if its...
that's just crazy!!!
Post by NeilBrown
Your results go some way to explaining why Lennart hasn't received many
complaints, but I'm convinced that the current situation is not ideal
(because SUSE has received some complaints).
It just proves bg mounts are being used.
Post by NeilBrown
I've been pondering the possibility of making "bg" work properly with
systemd and I think I've found a promising approach. It involves having
systemd take responsibility for the "run in the background" part.
If we get systemd-fstab-generator to translate "bg" to "retry=10000",
then "mount.nfs" will behave like the background version of
"mount.nfs -o bg". i.e. it will retry for one month (nearly). If there is
already a 'retry=' option, we just strip the "bg".
For this to work, we would need to add
TimeoutSec=infinity
to the .mount unit file so that systemd doesn't kill the mount.
We would also need to add (the effect of) "nofail", so that systemd
doesn't wait for the mount to complete...
Except that the current implementation of "nofail" is faulty.
It removes the default "Before=remote-fs.mount", which has the unwanted
consequence of unmounting the filesystem too early at shutdown.
I have a solution for that too (which I'll submit a push request for
shortly).
My solution to "nofail" is to treat it much like "automount", but
instead of using an automount unit to trigger the mount, we use
a timer unit (with OnActiveSec=0).
By triggering the mount unit with a timer instead of Wanting it
directly, it gets run in a separate transaction. This means that the
"Before=remote-fs.target" doesn't have the effect of delaying
remote-fs.target. Before/After only order jobs within the one
transaction.
When it comes time to shutdown, remote-fs.target and the foo.mount will
be in the same transaction, so the Before= will ensure foo.mount
isn't unmounted until after remote-fs.target has been allows to finish,
which is after any services that might be using the filesystem.
So I think I've found a solution for systemd to handle "bg" nfs mounts
correctly. I'll submit some pull requests for consideration.
Neil, your an excellent engineer so I'm sure you will craft
something very cool... but I truly feel you are breaking one
of the NFS communities long standing golden rule... We never add
fix to client that fixes a bug in the server... Fix the server...

That is the case here, IMHO... There is a bug in systemd that is not
letting a mount command succeed... It's clearly their bug,
let them fix it... Again, why systemd looking at mount options
is just mind boggling...

steved.
NeilBrown
2017-06-08 21:54:09 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
So I think I've found a solution for systemd to handle "bg" nfs mounts
correctly. I'll submit some pull requests for consideration.
Out of curiosity, after that change is there still any reason you'd
recommend any new user actually use "bg" (as opposed to an automount)?
Me? Recommend? Who would listen? Who would even hear?
For the last few years I've been recommending that automount should
be used for *all* NFS mounts at every opportunity. I think I've had two
opportunities.

But no, I would not recommend "bg". I would recommend automount and
then when they reported problems, I would help fix them.

I would be much happier recommending automount if it were easier.
Setting up /etc/auto.direct with automountd is fairly easy, but you need
to actually enable it but modifying auto.master or auto.master.d, which
is slightly annoying.

systemd does make it easier to do direct mounts, but it is ugly.
You need to include "comment=systemd.automount" or "x-systemd.automount"
in /etc/fstab instead of just "automount" or "ondemand". I understand
exactly why they did that and I cannot fault the logic. But it still
looks clumsy.

With systemd, you cannot divorce the timeout that an application has to
wait when accessing the mountpoint while the server is down, from the
timeout imposed on the mount program. i.e., mount cannot keep trying
in the background. - that could be useful if you want really-short
timeouts... at least it seems to me that they should be separate.

The timeout is configured differently if mounting from a device, or
mounting from anything else such as NFS. The first uses
x-systemd.device-timeout. The other needs x-systemd.mount-timeout.


But I'm ranting... I should probably shut up and send patches.
A generator for /etc/fstab.auto??
I appreciate the effort to keep existing systems working, I'm just
curious.
Compatibility with existing practice is certainly the main driver.

Thanks,
NeilBrown
Andrei Borzenkov
2017-06-11 06:07:11 UTC
Reply
Permalink
Raw Message
Post by NeilBrown
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.
A working "bg" mount would successfully mount the filesystem is the
server came back after 5 minutes. If you use current systemd in the
default configuration, it won't.
What about bumping timeout to the (default) value of retry= option in
this case? It sounds better than silently ignoring it and will mostly be
compatible with how it worked previously?

Lennart Poettering
2017-06-07 08:12:57 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???
So I'm a bit hesitant to commit this since not accurate, yet.
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.
Well "bg" doesn't really work on systemd systems, and this was never
different, hence I think it's only fair to document that it is
incompatible with systemd. In addition, I have the suspicion it is not
used very widely, since I never actually got complaints about it.

Lennart
--
Lennart Poettering, Red Hat
Steve Dickson
2017-06-07 10:04:12 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by Steve Dickson
Hello,
Post by NeilBrown
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel...
It appears there is a problem with a 4.12 kernel. The mount no
longer errors out with ECONNREFUSED it just hangs in the
kernel trying forever... It sounds like a bug to me but
maybe that change was intentional.. Anna?? Trond???
So I'm a bit hesitant to commit this since not accurate, yet.
Finally, the whole idea of systemd randomly/silently
strip off mount options is crazy... IMHO...
Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.
Well "bg" doesn't really work on systemd systems, and this was never
different, hence I think it's only fair to document that it is
incompatible with systemd.
I'm seeing it work just fine...

/etc/fstab: nfssrv:/home/tmp /mnt/tmp nfs bg 0 0
nfssrv is down
reboot client
login to client
# ps ax | grep mount
980 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
Post by Lennart Poettering
In addition, I have the suspicion it is not
used very widely, since I never actually got complaints about it.
Since it seems to be still working we probably would not hear
any complaints about it...

steved.
Steve Dickson
2017-06-08 20:34:05 UTC
Reply
Permalink
Raw Message
Post by Steve Dickson
# ps ax | grep mount
980 ? Ss 0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
Right, but I think we also need to see a "systemctl status
remote-fs.target", or something, to verify whether that's the forked
background process or just the foreground process that's still hanging
up some part of the boot process (even though it's gotten far enough
along that you can log in--unless logins aren't permitted till remote
fs's are mounted, I don't know.)
It succeeds...

# systemctl status remote-fs.target
* remote-fs.target - Remote File Systems
Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
Docs: man:systemd.special(7)

Jun 06 12:36:51 f26 systemd[1]: Reached target Remote File Systems.

The reason being, as Neil pointed out, the mount.nfs gets the
ECONNREFUSED right away because the server is down. So a
child is quickly forked that continues to try the mount...
Basically sneaking around systemd back... Which is hard
to do... these day 8-)

steved.
Loading...