Discussion:
Synchronization Between Services at Shutdown
(too old to reply)
Kurt von Laven
2015-04-02 01:22:50 UTC
Permalink
Hello folks,

I am trying to ensure that my system DBus service has time to handle a DBus
method call before it shuts down. My DBus service is implemented as a
system systemd service; let's call it service A. It runs a single-threaded,
synchronous event loop that listens for DBus method calls and handles them
appropriately. This event loop also listens for SIGTERM. When SIGTERM is
received, service A currently terminates the event loop, performs any
necessary writes to disk, and releases its resources. I am assuming that
when the system is shut down cleanly, systemd initially sends SIGTERM to
systemd services and only sends SIGKILL to abort it abruptly if it takes
too long to shut down when asked nicely.

I have another system systemd service, call it service B, that runs an
event loop listening for SIGTERM. When service B receives a SIGTERM, it
makes the aforementioned DBus method call and then flushes the system bus.
Service B specifies Requires=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=>Service
A and After=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=>Service
A. Service A makes no mention of service B in its unit configuration file.
Does systemd make any sort of guarantee that service A will get a chance to
handle the DBus method call before receiving SIGTERM, or does my code as
written have a race condition?

Be well,
Kurt
--
Kurt von Laven | Endless Mobile | 732.784.7356 | EndlessM.com
<http://endlessm.com/>
Andrei Borzenkov
2015-04-02 03:26:48 UTC
Permalink
В Wed, 1 Apr 2015 18:22:50 -0700
Post by Kurt von Laven
Hello folks,
I am trying to ensure that my system DBus service has time to handle a DBus
method call before it shuts down. My DBus service is implemented as a
system systemd service; let's call it service A. It runs a single-threaded,
synchronous event loop that listens for DBus method calls and handles them
appropriately. This event loop also listens for SIGTERM. When SIGTERM is
received, service A currently terminates the event loop, performs any
necessary writes to disk, and releases its resources. I am assuming that
when the system is shut down cleanly, systemd initially sends SIGTERM to
systemd services and only sends SIGKILL to abort it abruptly if it takes
too long to shut down when asked nicely.
I have another system systemd service, call it service B, that runs an
event loop listening for SIGTERM. When service B receives a SIGTERM, it
makes the aforementioned DBus method call and then flushes the system bus.
Service B specifies Requires=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=>Service
A and After=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=>Service
A. Service A makes no mention of service B in its unit configuration file.
Does systemd make any sort of guarantee that service A will get a chance to
handle the DBus method call before receiving SIGTERM, or does my code as
written have a race condition?
systemd will not initiate stopping of A before B has stopped. If it
does, this should be considered a bug. So it is up to service B to
handle termination correctly. Note that systemd will send SIGKILL after
timeout if service has not terminated.
Kurt von Laven
2015-04-02 06:44:16 UTC
Permalink
Thanks for the quick reply, Andrei. It sounds like there is indeed a race
condition in my code then since there is nothing explicit in my
implementation of service A that makes it wait until it has processed the
method call from service B before it stops listening for method calls. Did
I get that right?
В Wed, 1 Apr 2015 18:22:50 -0700
Post by Kurt von Laven
Hello folks,
I am trying to ensure that my system DBus service has time to handle a
DBus
Post by Kurt von Laven
method call before it shuts down. My DBus service is implemented as a
system systemd service; let's call it service A. It runs a
single-threaded,
Post by Kurt von Laven
synchronous event loop that listens for DBus method calls and handles
them
Post by Kurt von Laven
appropriately. This event loop also listens for SIGTERM. When SIGTERM is
received, service A currently terminates the event loop, performs any
necessary writes to disk, and releases its resources. I am assuming that
when the system is shut down cleanly, systemd initially sends SIGTERM to
systemd services and only sends SIGKILL to abort it abruptly if it takes
too long to shut down when asked nicely.
I have another system systemd service, call it service B, that runs an
event loop listening for SIGTERM. When service B receives a SIGTERM, it
makes the aforementioned DBus method call and then flushes the system
bus.
Post by Kurt von Laven
Service B specifies Requires=
<
http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=
Post by Kurt von Laven
Service
A and After=
<
http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=
Post by Kurt von Laven
Service
A. Service A makes no mention of service B in its unit configuration
file.
Post by Kurt von Laven
Does systemd make any sort of guarantee that service A will get a chance
to
Post by Kurt von Laven
handle the DBus method call before receiving SIGTERM, or does my code as
written have a race condition?
systemd will not initiate stopping of A before B has stopped. If it
does, this should be considered a bug. So it is up to service B to
handle termination correctly. Note that systemd will send SIGKILL after
timeout if service has not terminated.
--
Kurt von Laven | Endless Mobile | EndlessM.com <http://endlessm.com/>
Lennart Poettering
2015-04-02 09:06:50 UTC
Permalink
Post by Kurt von Laven
Hello folks,
I am trying to ensure that my system DBus service has time to handle a DBus
method call before it shuts down. My DBus service is implemented as a
system systemd service; let's call it service A. It runs a single-threaded,
synchronous event loop that listens for DBus method calls and handles them
appropriately. This event loop also listens for SIGTERM. When SIGTERM is
received, service A currently terminates the event loop, performs any
necessary writes to disk, and releases its resources. I am assuming that
when the system is shut down cleanly, systemd initially sends SIGTERM to
systemd services and only sends SIGKILL to abort it abruptly if it takes
too long to shut down when asked nicely.
I have another system systemd service, call it service B, that runs an
event loop listening for SIGTERM. When service B receives a SIGTERM, it
makes the aforementioned DBus method call and then flushes the system bus.
Service B specifies Requires=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Requires=>Service
A and After=
<http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=>Service
A. Service A makes no mention of service B in its unit configuration file.
Does systemd make any sort of guarantee that service A will get a chance to
handle the DBus method call before receiving SIGTERM, or does my code as
written have a race condition?
The general rule in systemd is that the shutdown order is the reverse
startup order. If you declare After=foo.service in a service
bar.service, then this will hence have two effects: 1) when both are
started foo.service is started before bar.service. And 2) when both
are stopped bar.service is stopped before foo.service.

If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.

Lennart
--
Lennart Poettering, Red Hat
Michael Biebl
2015-04-02 11:00:19 UTC
Permalink
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).

Incidentally, pitti had to deal with bug reports the last couple of
days, where dbus.service was stopped too early during shutdown, so
e.g. wpa_supplicant.service and other D-Bus using services failed to
stop properly on shutdown.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2015-04-02 11:03:00 UTC
Permalink
Post by Michael Biebl
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...

Lennart
--
Lennart Poettering, Red Hat
Michael Biebl
2015-04-02 11:07:31 UTC
Permalink
Post by Lennart Poettering
Post by Michael Biebl
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
Apparently they don't. There were all sorts of failures caused by dbus
being shut down too early.

https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
the related bug reports afair.

Since Martin has been working on this, I'll let him comment on this.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2015-04-02 13:31:22 UTC
Permalink
Post by Michael Biebl
Post by Lennart Poettering
Post by Michael Biebl
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down before
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
Apparently they don't. There were all sorts of failures caused by dbus
being shut down too early.
https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
the related bug reports afair.
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...

You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that. See systemd.special(7) for details.

I really don't see what dbus has to do with all this...

Lennart
--
Lennart Poettering, Red Hat
Simon McVittie
2015-04-02 14:04:54 UTC
Permalink
Post by Lennart Poettering
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
It does inasmuch as it requires networking to be up, which *might*
require dbus (e.g. for NetworkManager).
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target.
AIUI services with the default dependencies depend on remote-fs.target?
Distributions have historically supported NFS /usr and /var if
networking is done via ifupdown or something, and used (the LSB
equivalent of) remote-fs.target to represent that. NFS /usr or /var with
NetworkManager or similar already didn't work, because that needs D-Bus,
which needs /usr and /var.

systemd mandates an initramfs that mounts /usr (which has thankfully
made it into Debian 8, although for a while it didn't look as though
that was going to happen), but AIUI /var is still something that comes
up later?

It's very easy to get into circular dependencies here; there's the
remote filesystem issue, and also the fact that dbus-daemon wants to
resolve users/groups, hence might need NIS or LDAP to be up. I
personally consider some sort of automatically-synched local cache like
sssd to be a far better answer than NIS or LDAP, and NFS root better
than NFS /usr and/or /var, but I am not the only one whose opinion
matters when considering whether a feature regression in distributions
is acceptable.

I think the best solution might be a way for "dbus-daemon --system" to
not be required to be Before early targets like sysinit.target (to avoid
circular dependencies), but for it to survive until the final killing
spree anyway. Does systemd have a way to say that, or are
startup/shutdown dependencies always symmetric?

Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
on this; one possible hack is to have dbus.service's ExecStop not
actually stop dbus-daemon, so that it stays around until the final
killing spree just before /usr and /var are unmounted.

I'm happy to modify dbus.service if that's (part or all of) the
solution, but before I can do that we need to come up with a solution
that doesn't cause new dependency cycles.

S
--
Simon McVittie
Collabora Ltd. <http://www.collabora.com/>
Dan Williams
2015-04-02 14:49:12 UTC
Permalink
Post by Simon McVittie
Post by Lennart Poettering
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
It does inasmuch as it requires networking to be up, which *might*
require dbus (e.g. for NetworkManager).
NM requires wpa_supplicant running for WiFi because WiFi requires more
active management; the AP might go away, you might need to re-key, roam,
etc at any moment. The supplicant can't really stop until you no longer
need the interface. But in reality, (a) anyone mounting a rootfs over
WiFi is asking for trouble, and (b) why the F does NFS still not handle
network dropouts reliably? But I digress.
Post by Simon McVittie
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target.
AIUI services with the default dependencies depend on remote-fs.target?
Distributions have historically supported NFS /usr and /var if
networking is done via ifupdown or something, and used (the LSB
equivalent of) remote-fs.target to represent that. NFS /usr or /var with
NetworkManager or similar already didn't work, because that needs D-Bus,
which needs /usr and /var.
NM hasn't needed dbus-daemon running for a long time in early boot, or
for any root-level operation. So there's no requirement for /usr
or /var or anything else mounted, because root communication happens on
a private socket in /run.

wpa_supplicant does require dbus, since it has no private socket code.

Dan
Post by Simon McVittie
systemd mandates an initramfs that mounts /usr (which has thankfully
made it into Debian 8, although for a while it didn't look as though
that was going to happen), but AIUI /var is still something that comes
up later?
It's very easy to get into circular dependencies here; there's the
remote filesystem issue, and also the fact that dbus-daemon wants to
resolve users/groups, hence might need NIS or LDAP to be up. I
personally consider some sort of automatically-synched local cache like
sssd to be a far better answer than NIS or LDAP, and NFS root better
than NFS /usr and/or /var, but I am not the only one whose opinion
matters when considering whether a feature regression in distributions
is acceptable.
I think the best solution might be a way for "dbus-daemon --system" to
not be required to be Before early targets like sysinit.target (to avoid
circular dependencies), but for it to survive until the final killing
spree anyway. Does systemd have a way to say that, or are
startup/shutdown dependencies always symmetric?
Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
on this; one possible hack is to have dbus.service's ExecStop not
actually stop dbus-daemon, so that it stays around until the final
killing spree just before /usr and /var are unmounted.
I'm happy to modify dbus.service if that's (part or all of) the
solution, but before I can do that we need to come up with a solution
that doesn't cause new dependency cycles.
S
Lennart Poettering
2015-04-03 12:01:01 UTC
Permalink
Post by Simon McVittie
Post by Lennart Poettering
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
It does inasmuch as it requires networking to be up, which *might*
require dbus (e.g. for NetworkManager).
In this case you should make sure that NM stays up until the NFS
shares are unmounted, and if NM wants to have dbus around until its
very end it should use After=dbus.service. However NFS mounts and dbus
have little to do with each other.
Post by Simon McVittie
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target.
AIUI services with the default dependencies depend on
remote-fs.target?
No. local-fs.target is part of the default dependencies, but
remote-fs.target is not.
Post by Simon McVittie
Distributions have historically supported NFS /usr and /var if
networking is done via ifupdown or something, and used (the LSB
equivalent of) remote-fs.target to represent that. NFS /usr or /var with
NetworkManager or similar already didn't work, because that needs D-Bus,
which needs /usr and /var.
Split-out /usr is not supported by systemd unless already mounted in
the initrd.

Split-out /var is supported and does not have to be done in the
initrd. It's incompatible of course with NM because NM needs dbus, and
dbus needs /var.
Post by Simon McVittie
systemd mandates an initramfs that mounts /usr (which has thankfully
made it into Debian 8, although for a while it didn't look as though
that was going to happen), but AIUI /var is still something that comes
up later?
Yes.
Post by Simon McVittie
It's very easy to get into circular dependencies here; there's the
remote filesystem issue, and also the fact that dbus-daemon wants to
resolve users/groups, hence might need NIS or LDAP to be up.
Well, no, this is not supported.

dbus is a late boot service, and it's difficult to change that. The
major reason is bus activation: as long as dbus-daemon is running all
its activatable names are activatable, and this easily results in
deadlocks, since actually activating many of them would mean
activating late boot services, which results in deadlocks all over the
place. (simply because there is so much code that tries to contact
some service if it can, but skips if it doesn't, that is run from udev
rules or weird other early-boot stuff).

In kdbus this problem goes away since bus name are not always
actviatable, but can be made activatable only during later boot. With
kdbus we can schedule when names become activtable, which dbus-daemon
simply doesn't support.
Post by Simon McVittie
I personally consider some sort of automatically-synched local cache
like sssd to be a far better answer than NIS or LDAP, and NFS root
better than NFS /usr and/or /var, but I am not the only one whose
opinion matters when considering whether a feature regression in
distributions is acceptable.
Well, the way this is supposed to work with systemd is that NIS/LDAP
becomes available with the nss-user-lookup.target target, and user
logins wait for that (as well as remote-fs.target). NIS/LDAP only adds
real (i.e. "human") users to the user database, and system users
*must* exist all the time before that.

Both remote-fs.target and nss-user-lookup.target are *not* included in
basic.target but are pulled in later.
Post by Simon McVittie
I think the best solution might be a way for "dbus-daemon --system" to
not be required to be Before early targets like sysinit.target (to avoid
circular dependencies), but for it to survive until the final killing
spree anyway. Does systemd have a way to say that, or are
startup/shutdown dependencies always symmetric?
Yes they are always symmetric.

dbus-daemon cannot really be started before basic.target
Post by Simon McVittie
Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
on this; one possible hack is to have dbus.service's ExecStop not
actually stop dbus-daemon, so that it stays around until the final
killing spree just before /usr and /var are unmounted.
Urks! Don't do that.
Post by Simon McVittie
I'm happy to modify dbus.service if that's (part or all of) the
solution, but before I can do that we need to come up with a solution
that doesn't cause new dependency cycles.
Honestly, services should not require dbus to be around when shutting
down. And if they do anyway, they should simply specifiy the explicit
After=dbus.service dependency, and all is good. Don't fuck up the dbus
unit file, with lines like the above!

Lennart
--
Lennart Poettering, Red Hat
Dan Williams
2015-04-06 17:28:25 UTC
Permalink
Post by Lennart Poettering
Post by Simon McVittie
Post by Lennart Poettering
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
It does inasmuch as it requires networking to be up, which *might*
require dbus (e.g. for NetworkManager).
In this case you should make sure that NM stays up until the NFS
shares are unmounted, and if NM wants to have dbus around until its
very end it should use After=dbus.service. However NFS mounts and dbus
have little to do with each other.
NM does not need D-Bus to be up and around, and it doesn't care if D-Bus
goes away.

The only issue with D-Bus going away is for interfaces that require
external management daemons, like WiFi or Bluetooth, where if those
daemons exit the interface must be shut down because they have much more
hands-on control that is essentially required for the interface to
operate. Those daemons either need to not exit when D-Bus isn't
available, or they need to get ported to kdbus or something.

Dan
Post by Lennart Poettering
Post by Simon McVittie
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target.
AIUI services with the default dependencies depend on
remote-fs.target?
No. local-fs.target is part of the default dependencies, but
remote-fs.target is not.
Post by Simon McVittie
Distributions have historically supported NFS /usr and /var if
networking is done via ifupdown or something, and used (the LSB
equivalent of) remote-fs.target to represent that. NFS /usr or /var with
NetworkManager or similar already didn't work, because that needs D-Bus,
which needs /usr and /var.
Split-out /usr is not supported by systemd unless already mounted in
the initrd.
Split-out /var is supported and does not have to be done in the
initrd. It's incompatible of course with NM because NM needs dbus, and
dbus needs /var.
Post by Simon McVittie
systemd mandates an initramfs that mounts /usr (which has thankfully
made it into Debian 8, although for a while it didn't look as though
that was going to happen), but AIUI /var is still something that comes
up later?
Yes.
Post by Simon McVittie
It's very easy to get into circular dependencies here; there's the
remote filesystem issue, and also the fact that dbus-daemon wants to
resolve users/groups, hence might need NIS or LDAP to be up.
Well, no, this is not supported.
dbus is a late boot service, and it's difficult to change that. The
major reason is bus activation: as long as dbus-daemon is running all
its activatable names are activatable, and this easily results in
deadlocks, since actually activating many of them would mean
activating late boot services, which results in deadlocks all over the
place. (simply because there is so much code that tries to contact
some service if it can, but skips if it doesn't, that is run from udev
rules or weird other early-boot stuff).
In kdbus this problem goes away since bus name are not always
actviatable, but can be made activatable only during later boot. With
kdbus we can schedule when names become activtable, which dbus-daemon
simply doesn't support.
Post by Simon McVittie
I personally consider some sort of automatically-synched local cache
like sssd to be a far better answer than NIS or LDAP, and NFS root
better than NFS /usr and/or /var, but I am not the only one whose
opinion matters when considering whether a feature regression in
distributions is acceptable.
Well, the way this is supposed to work with systemd is that NIS/LDAP
becomes available with the nss-user-lookup.target target, and user
logins wait for that (as well as remote-fs.target). NIS/LDAP only adds
real (i.e. "human") users to the user database, and system users
*must* exist all the time before that.
Both remote-fs.target and nss-user-lookup.target are *not* included in
basic.target but are pulled in later.
Post by Simon McVittie
I think the best solution might be a way for "dbus-daemon --system" to
not be required to be Before early targets like sysinit.target (to avoid
circular dependencies), but for it to survive until the final killing
spree anyway. Does systemd have a way to say that, or are
startup/shutdown dependencies always symmetric?
Yes they are always symmetric.
dbus-daemon cannot really be started before basic.target
Post by Simon McVittie
Please see https://bugs.freedesktop.org/show_bug.cgi?id=89847 for more
on this; one possible hack is to have dbus.service's ExecStop not
actually stop dbus-daemon, so that it stays around until the final
killing spree just before /usr and /var are unmounted.
Urks! Don't do that.
Post by Simon McVittie
I'm happy to modify dbus.service if that's (part or all of) the
solution, but before I can do that we need to come up with a solution
that doesn't cause new dependency cycles.
Honestly, services should not require dbus to be around when shutting
down. And if they do anyway, they should simply specifiy the explicit
After=dbus.service dependency, and all is good. Don't fuck up the dbus
unit file, with lines like the above!
Lennart
Martin Pitt
2015-04-02 14:31:07 UTC
Permalink
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that.
wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Dan Williams
2015-04-02 14:51:44 UTC
Permalink
Post by Martin Pitt
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that.
wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.
At least on the NM side, NM shouldn't be doing anything when dbus-daemon
exits, except logging something and trying to reconnect. It shouldn't
touch network interfaces except those that require a management daemon,
like wpa_supplicant or Bluetooth or teamd, provided you use the internal
DHCP client. But ethernet, bridge/bond/vlan/infiniband/etc should be
fine. Obviously if the supplicant goes down, WiFi goes down, but
running something like NFS over WiFi is kinda asking for trouble, given
that NFS doesn't handle network dropouts well in the first place.

Dan
Martin Pitt
2015-04-02 20:28:39 UTC
Permalink
Hello,
Post by Dan Williams
Post by Martin Pitt
wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.
At least on the NM side, NM shouldn't be doing anything when dbus-daemon
exits, except logging something and trying to reconnect.
Ah, that's correct. It complains a lot, but indeed our bug reporters
say that everything works fine with ethernet on shutdown. The remote
fs specific problems are due to wpasupplicant going down early with
dbus instead of at the (much later) time when we intend it to go down.
Avahi, libvirt etc. are more "typical" dbus services which go down
when the system dbus does, and do cause trouble.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Lennart Poettering
2015-04-03 12:08:27 UTC
Permalink
Post by Martin Pitt
Hello,
Post by Dan Williams
Post by Martin Pitt
wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.
At least on the NM side, NM shouldn't be doing anything when dbus-daemon
exits, except logging something and trying to reconnect.
Ah, that's correct. It complains a lot, but indeed our bug reporters
say that everything works fine with ethernet on shutdown. The remote
fs specific problems are due to wpasupplicant going down early with
dbus instead of at the (much later) time when we intend it to go down.
Avahi, libvirt etc. are more "typical" dbus services which go down
when the system dbus does, and do cause trouble.
Avahi reconnects when dbus goes away, it does not exit.

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-04-03 12:06:19 UTC
Permalink
Post by Martin Pitt
Post by Lennart Poettering
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that.
wpa_supplicant and NM already have Before=network.target which is even
stronger. But that doesn't help -- it's D-Bus which we need to leave
running longer, as dbus going down early tears down all dbus clients as well
prematurely, thus rendering the above dependencies useless.
Well, it sounds really wrong btw, if NM shuts down the ifaces just
because it got kicked off the bus... That's massively wrong.

Lennart
--
Lennart Poettering, Red Hat
Martin Pitt
2015-04-04 07:10:57 UTC
Permalink
Post by Lennart Poettering
Well, it sounds really wrong btw, if NM shuts down the ifaces just
because it got kicked off the bus... That's massively wrong.
It doesn't -- wpa_supplicant does.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Kurt von Laven
2015-04-02 16:49:25 UTC
Permalink
Both services A and B specify Requires=dbus.service and After=dbus.service.
If I am interpreting everything that's being said correctly, there is a
race condition in my code since service A's implementation doesn't require
it to wait until it has processed the method call from service B before it
stops listening for method calls. Does that sound right?
Post by Kurt von Laven
Post by Michael Biebl
Post by Lennart Poettering
Post by Michael Biebl
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down
before
Post by Michael Biebl
Post by Lennart Poettering
Post by Michael Biebl
Post by Lennart Poettering
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
Apparently they don't. There were all sorts of failures caused by dbus
being shut down too early.
https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
the related bug reports afair.
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that. See systemd.special(7) for details.
I really don't see what dbus has to do with all this...
Lennart
--
Lennart Poettering, Red Hat
--
Kurt von Laven | Endless Mobile | EndlessM.com <http://endlessm.com/>
Kurt von Laven
2015-04-03 03:55:16 UTC
Permalink
Actually I think I just answered my own question and in the process
realized that I didn't provide enough context in my original question.
There is a race condition in my code because when service B receives
SIGTERM, it calls g_dbus_connection_flush_sync
<https://developer.gnome.org/gio/stable/GDBusConnection.html#g-dbus-connection-flush-sync>
after
a g_dbus_proxy_call
<https://developer.gnome.org/gio/stable/GDBusProxy.html#g-dbus-proxy-call>
to service A in an incorrect attempt to ensure that service A has handled
the DBus proxy call. The problem with this is that
g_dbus_connection_flush_sync ensures all outgoing messages are sent over
the DBus connection but doesn't wait around for replies to those messages.
Service B should instead call g_dbus_proxy_call_sync
<https://developer.gnome.org/gio/stable/GDBusProxy.html#g-dbus-proxy-call-sync>
and
call neither g_dbus_connection_flush_sync nor g_dbus_proxy_call. Please
correct me if I am missing something.

On a separate note, how long does systemd wait after sending a SIGTERM to a
service before sending it a SIGKILL? Is it reasonable to assume that a few
g_dbus_proxy_call_sync calls would complete in that time if the slowest
part of those calls is just the standard overhead of a DBus call? I ask
because currently I acquire a shutdown inhibitor
<http://www.freedesktop.org/wiki/Software/systemd/inhibit/> in service B,
and I now suspect this is unnecessary.

Be well,
Kurt
Post by Kurt von Laven
Both services A and B specify Requires=dbus.service and
After=dbus.service. If I am interpreting everything that's being said
correctly, there is a race condition in my code since service A's
implementation doesn't require it to wait until it has processed the method
call from service B before it stops listening for method calls. Does that
sound right?
Post by Kurt von Laven
Post by Michael Biebl
Post by Lennart Poettering
2015-04-02 11:06 GMT+02:00 Lennart Poettering <
Post by Lennart Poettering
If you want to ensure that bus communication still works in your
shutdown code, you hence need to make sure you place
After=dbus.service in your services, so that you are shut down
before
Post by Michael Biebl
Post by Lennart Poettering
Post by Lennart Poettering
dbus is.
Type=dbus service currently only get a dependency on dbus.socket (via
After=basic.target).
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
Apparently they don't. There were all sorts of failures caused by dbus
being shut down too early.
https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/1438612 is one of
the related bug reports afair.
Hmm? I really don't see how the NFS vs wpa_supplicant issue has
anything to do with dbus? NFS doesn't care about dbus at all...
You need to order wpa_supplicant and NM Before=remote-fs-pre.target
and pull it it via Wants=remote-fs-pre.target. With that in place
during shutdown the mounts will be unmounted first, and NM/wpa only
shut down after that. See systemd.special(7) for details.
I really don't see what dbus has to do with all this...
Lennart
--
Lennart Poettering, Red Hat
--
Kurt von Laven | Endless Mobile | EndlessM.com <http://endlessm.com/>
--
Kurt von Laven | Endless Mobile | EndlessM.com <http://endlessm.com/>
Andrei Borzenkov
2015-04-03 05:51:32 UTC
Permalink
Post by Kurt von Laven
On a separate note, how long does systemd wait after sending a SIGTERM to a
service before sending it a SIGKILL?
http://www.freedesktop.org/software/systemd/man/systemd.kill.html
Lennart Poettering
2015-04-03 12:12:48 UTC
Permalink
Post by Kurt von Laven
On a separate note, how long does systemd wait after sending a SIGTERM to a
service before sending it a SIGKILL?
90s. This may be configured per-service in TimeoutStopSec= and
globally with DefaultTimeoutStopSec= in system.conf.

he default of 90s appears to be not documented anyway right
now. That's a bug. It used to be documented, but somehow was
lost... Will fix.

Lennart
--
Lennart Poettering, Red Hat
Martin Pitt
2015-04-02 14:28:32 UTC
Permalink
Post by Lennart Poettering
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
They do, but that's irrelevant here. The problem is that dbus.service
shuts down very early as nothing prevents it from that. All
dependencies just refer to dbus.socket, but TBH socket (re-)activation
totally doesn't work for dbus. On restarts it forgets all of its
previous state/connections, so a service which still needs to talk to
dbus during shutdown will just fail if dbus.service already stopped.

So what happens is that dbus.service gets stopped early, takes
NetworkManager, wpasupplicant etc. with it, and then the network
mounts can't be unmounted any more as the network is already down
(which isn't intended at this point yet, as network.target is still
active).

Synthesizing After=dbus.service dependencies instead of dbus.socket
ones for Type=dbus would help, as then you'd have the ordering which
prevents dbus.service from going down early. But this is prone to
dependency loops.

This has also been discussed with upstream on
https://bugs.freedesktop.org/show_bug.cgi?id=89847

I don't really have a good solution, the "don't really stop dbus" is a
relatively safe hack that's appropriate for the "we need to release in
two weeks" freeze. Obviously the really good solution is to get kdbus,
then dbus will always be available :-)

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Lennart Poettering
2015-04-03 12:04:10 UTC
Permalink
Post by Martin Pitt
Post by Lennart Poettering
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
They do, but that's irrelevant here. The problem is that dbus.service
shuts down very early as nothing prevents it from that. All
dependencies just refer to dbus.socket, but TBH socket (re-)activation
totally doesn't work for dbus.
systemd will not reactivate services when shutting down, we refuse to
go in circles...

And even if it would reactivate dbus, it wouldn't help you, since all
services disappeared from the bus then...
Post by Martin Pitt
On restarts it forgets all of its
previous state/connections, so a service which still needs to talk to
dbus during shutdown will just fail if dbus.service already stopped.
Well, if that service is really this broken that it needs dbus when
all it should do is exit, then it should declare After=dbus.service,
and all is good.

Lennart
--
Lennart Poettering, Red Hat
Andrei Borzenkov
2015-04-03 16:37:34 UTC
Permalink
В Fri, 3 Apr 2015 14:04:10 +0200
Post by Lennart Poettering
Post by Martin Pitt
Post by Lennart Poettering
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
They do, but that's irrelevant here. The problem is that dbus.service
shuts down very early as nothing prevents it from that. All
dependencies just refer to dbus.socket, but TBH socket (re-)activation
totally doesn't work for dbus.
systemd will not reactivate services when shutting down, we refuse to
go in circles...
Could you elaborate? systemd will obviously start services when
starting shutdown.target. How does systemd differentiate between
services that can and can not be started?
Lennart Poettering
2015-04-07 16:41:58 UTC
Permalink
Post by Andrei Borzenkov
В Fri, 3 Apr 2015 14:04:10 +0200
Post by Lennart Poettering
Post by Martin Pitt
Post by Lennart Poettering
Yeah, and rightfully so. I mean, a service really should be able to
shutdown if dbus is dead. In fact, it should be able to shutdown in
pretty much any situation...
They do, but that's irrelevant here. The problem is that dbus.service
shuts down very early as nothing prevents it from that. All
dependencies just refer to dbus.socket, but TBH socket (re-)activation
totally doesn't work for dbus.
systemd will not reactivate services when shutting down, we refuse to
go in circles...
Could you elaborate? systemd will obviously start services when
starting shutdown.target. How does systemd differentiate between
services that can and can not be started?
Socket units for which a stop job is queued will suppress
activations until the stop job is cancelled (ot it is stopped).

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...