[systemd-devel] network/openvswitch dependency loop/deadlock

Post by Ian Pilcher
Recently, Fedora shipped an update which starts the Open vSwitch service
on demand -- whenever an Open vSwitch bridge or port is "ifup'ed". In
theory, I should now be able to simply write traditional ifcfg-* files
for all of my interfaces and use the "network" service to start them.
Unfortunately, my interfaces are not starting properly. I believe that
* systemd starts the network service (/etc/rc.d/init.d/network)
* The network tries to start an interface (ifup eth0)
* ifup reads "TYPE=ovs" from ifcfg-eth0 and executes ifup-ovs
* ifup-ovs sees that the Open vSwitch daemon is not running
(/var/lock/subsys/openvswitch does not exist) and executes "service
openvswitch start"
* /usr/sbin/service executes "systemctl start openvswitch.service"
* systemd sees "Before=... network.target" in openvswitch.service and
waits for the network service to complete -- which will never happen,
because the network service is waiting for the openvswitch servicr to
start.
* DEADLOCK!

This last step shouldn't (in theory) be a problem as far as I understand
it. Before=network.target doesn't imply it that has to wait for
network.service to complete - it should only imply that both
network.service and openvswitch.service are both have to start before
network.target is considered reached. If it said After=network.target
then I would see an obvious deadlock, but with both saying Before= they
should be able to work fine.

I'm guessing that the job simply isn't scheduled because a current
transaction is running and the scheduler does not fiddle with it while
it's running. It just queues another job for later after it's done
(although this also doesn't seem correct... as if that were the case why
would --ignore-dependencies work).

Post by Ian Pilcher
At this point, everything grinds to a halt. The "systemctl start
openvswitch service" process hangs, and the network service sits there
waiting for it to complete.
After 5 minutes, systemd kills the network service. Since the network
service is no longer running, systemd considers that the network.target
has been reached and starts the Open vSwitch daemon.
https://bugzilla.redhat.com/show_bug.cgi?id=818754#c21
All of this is background to my question -- is this dependency loop/
deadlock the expected behavior in this case?
Assuming that the answer is yes, what is the best way to work around
this?
* Removing network.target from the Before=... line in
openvswitch.service is not an option. See comment #1 of that bug.
* Changing the network startup script (ifup-ovs) to use "systemctl
--ignore-dependencies start openvswitch.service" appears to work, but
the man page discourages its use for anything but debugging.

Depending on how the daemon is used, it might make more sense to use
--no-block. This will return control to the command line straight away,
but obviously the daemon may not be "ready" for communications yet and
the script may fail.

I'm not familiar with the daemon or what it does and how any IPC may
work (i.e. how you talk to the daemon).

To me, it would make more sense to make it socket or dbus activatable if
that's what it uses, but that by itself will be unlikely to
fundamentally solve the transactional problems here (especially if two
way comms are required during the ifup-ovs execution)

Post by Ian Pilcher
It would be very nice to not have to create another unit file just to
ignore this single dependency in this single circumstance.

I get the feeling I'm perhaps misinterpreting something. I think the
real reason for the deadlock would be good to track down. It could be
that it is being artificially held back from completing or some other
dep is causing the problem.

Perhaps a "systemctl list-jobs" when it's stuck would help? Also the
systemctl show from network.service, openvswitch.service and
network.target are likely useful too to track down which dep may be
causing problems.

It's just odd that removing "Before=network.target" solves the
problem... seems odd.

Also re the initscripts tweaks and the if statement proposed in the bug,
there is a SYSTEMCTL_IGNORE_DEPENDENCIES=1 env var you can export that
will make "service openvswitch start" pass the --ignore-dependencies
argument if it redirects to systemctl. That's likely cleaner than the if
[ -x /usr/bin/systemctl ] check. Obviously as this is arguably not the
right fix anyway, it's perhaps a moot point.

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

Ian Pilcher

2013-02-07 15:55:01 UTC

Post by Ian Pilcher
* systemd sees "Before=... network.target" in openvswitch.service and
waits for the network service to complete -- which will never happen,
because the network service is waiting for the openvswitch servicr to
start.
* DEADLOCK!

Well it would help if my brain worked a bit better. openvswitch.service
does, in fact, contain:

After=syslog.target network.target

So there's the "obvious deadlock".

Post by Ian Pilcher
Assuming that the answer is yes, what is the best way to work around
this?
* Removing network.target from the Before=... line in
openvswitch.service is not an option. See comment #1 of that bug.
* Changing the network startup script (ifup-ovs) to use "systemctl
--ignore-dependencies start openvswitch.service" appears to work, but
the man page discourages its use for anything but debugging.

--no-block is not going to be an option. The script is trying to bring
up an Open vSwitch bridge, and it can't do so without a running daemon.

Post by Colin Guthrie
I'm not familiar with the daemon or what it does and how any IPC may
work (i.e. how you talk to the daemon).

The script uses ovs-vsctl to talk to ovs-vswitchd.

Post by Colin Guthrie
I get the feeling I'm perhaps misinterpreting something. I think the
real reason for the deadlock would be good to track down. It could be
that it is being artificially held back from completing or some other
dep is causing the problem.

You made the mistake of believing what I wrote.

Post by Colin Guthrie
Also re the initscripts tweaks and the if statement proposed in the bug,
there is a SYSTEMCTL_IGNORE_DEPENDENCIES=1 env var you can export that
will make "service openvswitch start" pass the --ignore-dependencies
argument if it redirects to systemctl. That's likely cleaner than the if
[ -x /usr/bin/systemctl ] check. Obviously as this is arguably not the
right fix anyway, it's perhaps a moot point.

Cool.

So given that it is in fact "After=... network.target" (in
openvswitch.service) that is causing the problem, do you see a better
solution than using SYSTEMCTL_IGNORE_DEPENDENCIES?

Is there any chance that this would work?

After=syslog.target
Requires=network.target

(While not breaking the use case in comment #34 of the bug.)

I'm very unclear on what Requires=network.target would actually mean.

Thanks for your help, and sorry about the brain-cramp!

Frederic Crozat

2013-02-07 16:09:34 UTC

Well it would help if my brain worked a bit better. openvswitch.service
After=syslog.target network.target
So there's the "obvious deadlock".

--no-block is not going to be an option. The script is trying to bring
up an Open vSwitch bridge, and it can't do so without a running daemon.

Post by Colin Guthrie
I'm not familiar with the daemon or what it does and how any IPC may
work (i.e. how you talk to the daemon).

The script uses ovs-vsctl to talk to ovs-vswitchd.

You made the mistake of believing what I wrote.

Cool.
So given that it is in fact "After=... network.target" (in
openvswitch.service) that is causing the problem, do you see a better
solution than using SYSTEMCTL_IGNORE_DEPENDENCIES?

We have exactly the same issue on openSUSE with some network services,
and we are using --ignore-dependencies in those case (because --no-block
is not always the "right" solution).

--
Frederic Crozat <***@suse.com>
SUSE

Ian Pilcher

2013-02-08 21:05:18 UTC

Post by Frederic Crozat
We have exactly the same issue on openSUSE with some network services,
and we are using --ignore-dependencies in those case (because --no-block
is not always the "right" solution).

Yeah. I thrashed around for a couple of days until I finally your note
from last month.

So thanks for the cluebat!

Thomas Graf

2013-02-12 13:07:45 UTC

Cool.
So given that it is in fact "After=... network.target" (in
openvswitch.service) that is causing the problem, do you see a better
solution than using SYSTEMCTL_IGNORE_DEPENDENCIES?
Is there any chance that this would work?
After=syslog.target
Requires=network.target
(While not breaking the use case in comment #34 of the bug.)
I'm very unclear on what Requires=network.target would actually mean.

The case of Open vSwitch is a bit special because there is a chicken
egg problem.

OVS bridges and ports are configured using the ovs-vsctl utility. It
requires the OVS daemons to be running. So naturally we want to start
the openvswitch unit (if it is not running yet) before we ifup any
OVS type interfaces.

We also want the openvswitch unit to be started after network on boot
because an OVS bridge may be using a controller on the network that
depends on network connectivity.

So what we did is introduce a new ifcfg key that can be used to define
the interface dependencies of a bridge. Let's say ovsbr0 is configured
to use a controller that is behind interface em1. So before bringing
up ovsbr0 we want to bring up em1 and after that we want to start the
openvswitch unit and then configure the bridge.

So it seems what we need is a way to tell systemd to disregard the
network dependency if we are starting the openvswitch unit from within
ifup. Something like --ignore-dependency=network.target because we
still want it to depend on syslog.target obviously.

I guess we could also be providing a separate units files:
openvswitch-no-network.service
openvswitch.service

What is cleaner from a systemd perspective?

Lennart Poettering

2013-02-13 03:06:34 UTC

Post by Thomas Graf

Cool.
So given that it is in fact "After=... network.target" (in
openvswitch.service) that is causing the problem, do you see a better
solution than using SYSTEMCTL_IGNORE_DEPENDENCIES?
Is there any chance that this would work?
After=syslog.target
Requires=network.target
(While not breaking the use case in comment #34 of the bug.)
I'm very unclear on what Requires=network.target would actually mean.

The case of Open vSwitch is a bit special because there is a chicken
egg problem.
OVS bridges and ports are configured using the ovs-vsctl utility. It
requires the OVS daemons to be running. So naturally we want to start
the openvswitch unit (if it is not running yet) before we ifup any
OVS type interfaces.
We also want the openvswitch unit to be started after network on boot
because an OVS bridge may be using a controller on the network that
depends on network connectivity.
So what we did is introduce a new ifcfg key that can be used to define
the interface dependencies of a bridge. Let's say ovsbr0 is configured
to use a controller that is behind interface em1. So before bringing
up ovsbr0 we want to bring up em1 and after that we want to start the
openvswitch unit and then configure the bridge.
So it seems what we need is a way to tell systemd to disregard the
network dependency if we are starting the openvswitch unit from within
ifup. Something like --ignore-dependency=network.target because we
still want it to depend on syslog.target obviously.
openvswitch-no-network.service
openvswitch.service
What is cleaner from a systemd perspective?

Well, there still is a dep loop, that is unfixed, right?

I mean, you want your service to run as part of bringing up some
networks, but also after the network is up. So what do you want now?
Start it after or at the same time, that's contradicting.

a) if you want to start it only after then, you should drop those
callouts from ifup.

b) if you want to start it as part of the network config, then you
should drop any reference to network.target.

To me it appears that you aren't entirely clear which way you want it,
and try to use --ignore-dependencies as a work-around for that... But I
really think the first step needs to be: what it's gonna be?

Lennart

--
Lennart Poettering - Red Hat, Inc.

Thomas Graf

2013-02-13 07:49:34 UTC

Post by Lennart Poettering
Well, there still is a dep loop, that is unfixed, right?
I mean, you want your service to run as part of bringing up some
networks, but also after the network is up. So what do you want now?
Start it after or at the same time, that's contradicting.
a) if you want to start it only after then, you should drop those
callouts from ifup.
b) if you want to start it as part of the network config, then you
should drop any reference to network.target.
To me it appears that you aren't entirely clear which way you want it,
and try to use --ignore-dependencies as a work-around for that... But I
really think the first step needs to be: what it's gonna be?

Let's put it in another way, there is two ways to run OVS:

a) _not_ part of the network config. OVS will load the config from
its database when the unit is started, configures itself and
then runs. This is a no brainer and requires After=network.target
syslog.service

b) As part of the network config scripts. Each virtual bridge, port,
tunnel, mirror, and bond has its ifcfg file. This is to allow to
ifdown/ifup virtual bridges and ports using a well known interface
and persistent storage method.

Dependencies:
1. interfaces listed in OVSREQUIRES= in each OVS ifcfg must be
brought up first to bring up the uplink interfaces for bonds,
tunnels, or native uplinks. We do this within ifup.
2. openvswitch unit must be started with After=syslog. The
dependency on network.target is already resolved in the first
step.
3. OVS gets configured
4. Any remaining interfaces get brought up

So essentially we need both.

Lennart Poettering

2013-02-13 19:03:57 UTC

Post by Thomas Graf

a) _not_ part of the network config. OVS will load the config from
its database when the unit is started, configures itself and
then runs. This is a no brainer and requires After=network.target
syslog.service

Well, so first of all, no service needs syslog.service anymore. The
socket of syslog is established during early boot, and you should not
add deps for it anymore.

Why is After=network.target actually needed? Shouldn't OVS be capable of
picking up devices as they appear? Device probing is ansynchronous, so
we kinda try to get people away from assuming that there was a point in
time where all devices are finished probing, and we prefer if people
just pick the devices up as they appear... YOu are the libnl guy, I
figure you know that anyway...

Post by Thomas Graf
b) As part of the network config scripts. Each virtual bridge, port,
tunnel, mirror, and bond has its ifcfg file. This is to allow to
ifdown/ifup virtual bridges and ports using a well known interface
and persistent storage method.
1. interfaces listed in OVSREQUIRES= in each OVS ifcfg must be
brought up first to bring up the uplink interfaces for bonds,
tunnels, or native uplinks. We do this within ifup.
2. openvswitch unit must be started with After=syslog. The
dependency on network.target is already resolved in the first
step.
3. OVS gets configured
4. Any remaining interfaces get brought up
So essentially we need both.

But under different configuration, right?

So, you could either enable that target I mentioned before as a barrier
that is enabled only sometimes, or you could just use two different unit
files for this, and pull in one for the configuration a) and the other
for b)?

Lennart

--
Lennart Poettering - Red Hat, Inc.

Thomas Graf

2013-02-13 22:55:23 UTC

Post by Lennart Poettering
Well, so first of all, no service needs syslog.service anymore. The
socket of syslog is established during early boot, and you should not
add deps for it anymore.

OK, I'll remove that dependency. We inherited that from the upstream
unit file.

Post by Lennart Poettering
Why is After=network.target actually needed? Shouldn't OVS be capable of
picking up devices as they appear? Device probing is ansynchronous, so
we kinda try to get people away from assuming that there was a point in
time where all devices are finished probing, and we prefer if people
just pick the devices up as they appear... YOu are the libnl guy, I
figure you know that anyway...

A virtual switch may depend on a OpenFlow controller doing the switching
decisions for the switch. An OpenFlow controller typically runs
logically centralized but physically distributed on the network.

The virtual switch may also have sFlow and NetFlow agents configured to
send packet samples access the network and the receivers don't want to
miss out on samples just because the network is not ready yet.

Therefore the generic network.target dependency. I agree that it's not
ideal which is why we added a new ifcfg key=value allowing to define
dependencies on specific interfaces but we have a lot of existing setups
that are not using that new key yet.

Post by Lennart Poettering
But under different configuration, right?
So, you could either enable that target I mentioned before as a barrier
that is enabled only sometimes, or you could just use two different unit
files for this, and pull in one for the configuration a) and the other
for b)?

Alright, two separate unit files sounds good to me. The point of
this discussion was to figure out if there was another well known best
practice.

Thanks

Colin Guthrie

2013-02-13 11:10:13 UTC

Post by Lennart Poettering

Post by Thomas Graf

Cool.
So given that it is in fact "After=... network.target" (in
openvswitch.service) that is causing the problem, do you see a better
solution than using SYSTEMCTL_IGNORE_DEPENDENCIES?
Is there any chance that this would work?
After=syslog.target
Requires=network.target
(While not breaking the use case in comment #34 of the bug.)
I'm very unclear on what Requires=network.target would actually mean.

The case of Open vSwitch is a bit special because there is a chicken
egg problem.
OVS bridges and ports are configured using the ovs-vsctl utility. It
requires the OVS daemons to be running. So naturally we want to start
the openvswitch unit (if it is not running yet) before we ifup any
OVS type interfaces.
We also want the openvswitch unit to be started after network on boot
because an OVS bridge may be using a controller on the network that
depends on network connectivity.
So what we did is introduce a new ifcfg key that can be used to define
the interface dependencies of a bridge. Let's say ovsbr0 is configured
to use a controller that is behind interface em1. So before bringing
up ovsbr0 we want to bring up em1 and after that we want to start the
openvswitch unit and then configure the bridge.
So it seems what we need is a way to tell systemd to disregard the
network dependency if we are starting the openvswitch unit from within
ifup. Something like --ignore-dependency=network.target because we
still want it to depend on syslog.target obviously.
openvswitch-no-network.service
openvswitch.service
What is cleaner from a systemd perspective?

Well, there still is a dep loop, that is unfixed, right?
I mean, you want your service to run as part of bringing up some
networks, but also after the network is up. So what do you want now?
Start it after or at the same time, that's contradicting.
a) if you want to start it only after then, you should drop those
callouts from ifup.
b) if you want to start it as part of the network config, then you
should drop any reference to network.target.
To me it appears that you aren't entirely clear which way you want it,
and try to use --ignore-dependencies as a work-around for that... But I
really think the first step needs to be: what it's gonna be?

Is what is really needed here some kind of netdev-generator which parses
the individual ifcfg-* files and writes out ***@eth0.service
instances (or maybe not a single ***@.service but several for the
different kinds of interfaces?)

That way the deps can be properly parsed and ordered in the units.
Actually due to the interplay of deps it might not be possible to use
templated units here, but just generate specific, individual units (just
like mount units).

Either way, is a generator the way to go here longer term, or is there
some other plan afoot to modernise the static network configs?

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

Kay Sievers

2013-02-13 13:01:32 UTC

Post by Colin Guthrie
Either way, is a generator the way to go here longer term, or is there
some other plan afoot to modernise the static network configs?

The static network config files, file format and parsing will some day
be part of the core systemd.

Looking at it from the cross-distribution/upstream perspective, it's a
complete mess, and it's probably the weakest and most "broken "part of
the core system at the moment.

But we are kind of busy with other topics at the moment, so it might
take a while ... :)

Kay

Ian Pilcher

2013-02-08 21:00:11 UTC

FYI, I just tried this on Fedora 18, and it didn't work. It looks like
the version of systemctl in F18 may not recognize that variable:

$ strings /usr/bin/systemctl | grep SYSTEMCTL_IGNORE_DEPENDENCIES
$

Oh well.

Colin Guthrie

2013-02-10 13:14:25 UTC

FYI, I just tried this on Fedora 18, and it didn't work. It looks like
$ strings /usr/bin/systemctl | grep SYSTEMCTL_IGNORE_DEPENDENCIES
$
Oh well.

This isn't something built into systemctl itself, but rather in the
"/usr/bin/service" wrapper and in initscripts functions generally:

e.g. in the "initscripts" git: git://git.fedorahosted.org/initscripts.git

[***@jimmy initscripts (master)]$ git grep SYSTEMCTL_IGNORE_DEPENDENCIES
initscripts.spec:- /sbin/service: honor SYSTEMCTL_IGNORE_DEPENDENCIES
(<***@poettering.net>)
rc.d/init.d/functions: if [ -n "$SYSTEMCTL_IGNORE_DEPENDENCIES" ]
; then
service: export SYSTEMCTL_IGNORE_DEPENDENCIES=1
service: env -i PATH="$PATH" TERM="$TERM"
SYSTEMCTL_IGNORE_DEPENDENCIES=${SYSTEMCTL_IGNORE_DEPENDENCIES}
SYSTEMCTL_SKIP_REDIRECT=${SYSTEMCTL_SKIP_REDIRECT} "${SERVICEDIR}/${SERVICE
service: env -i PATH="$PATH" TERM="$TERM"
SYSTEMCTL_IGNORE_DEPENDENCIES=${SYSTEMCTL_IGNORE_DEPENDENCIES}
SYSTEMCTL_SKIP_REDIRECT=${SYSTEMCTL_SKIP_REDIRECT} "${ACTIONDIR}/${SERVICE}

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

Lennart Poettering

2013-02-13 03:11:40 UTC