Discussion:
`Found ordering cycle on _SERVICE_` loops due to `local-fs.target` with `Requires` instead of `Wants` for mounts generated from `/etc/fstab`
Add Reply
Alexey Matyushin
2018-04-24 23:26:21 UTC
Reply
Permalink
Raw Message
Hi there,

I initially created a bug report for this topic (apologize, Lennart), but would like to discuss this topic in mode detail here (as suggested by Lennart), and really appreciate your opinion here,

I've seen that once you define network (NFS) mount together with local (referring to that network (NFS) mount) mount in `/etc/fstab` then it will always produce dependency loops (due to `local-fs.target`),

Details and how to reproduce it,

# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

# cat /etc/*release
CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.4.1708 (Core)

### Expected behaviour you didn't see

All local mounts (dependencies) are defined for `local-fs.target` with `Wants` parameter.

### Unexpected behaviour you saw

All local mounts are defined for `local-fs.target` with `Requires` parameter instead of `Wants` parameter.

### Steps to reproduce the problem

1. Add NFS mount into `/etc/fstab`, example
```server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0```
2. Add local mount with dependency on NFS mount, example,
```/mnt/some_other_mount/ /mnt/test/media/ none bind,rw 0 0```

Example of `/etc/fstab`,
```
#
# /etc/fstab
# Created by anaconda on Mon Nov 27 22:04:35 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=2f15c101-f516-4ee8-a4b7-89ad880610db / xfs defaults 0 0

server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0
/mnt/some_other_mount/ /mnt/test/medi none bind,rw 0 0
```

Question,
I know that this issue with dependency loop on `local-fs.target` can be corrected by `nofail`/`_netdev` options in `/etc/fstab` file for all local shares (as then all local mounts will be only wanted (`Wants` parameter), and not required (`Requires` parameter) by `local-fs.target), but my question is about behavior without these options specified, do you think that this (explained) behavior is correct?

This looks for me as quite dangerous behavior (without `nofail`/`_netdev` options) since in this scenario we may delete start jobs for up to 10 different services/targets (including network stack (`network.service`)), here is the full list of possible services/targets which may never start due to this issue (ordering cycle (loop)),

systemd-journal-catalog-update.service
systemd-machine-id-commit.service
systemd-tmpfiles-setup.service
rhel-import-state.service
systemd-update-done.service
auditd.service
network.service
nfs-config.service
local-fs.target

Thanks a lot in advance!

Kind regards,
Alexey
Lennart Poettering
2018-04-25 08:53:33 UTC
Reply
Permalink
Raw Message
Post by Alexey Matyushin
server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0
/mnt/some_other_mount/ /mnt/test/medi none bind,rw 0 0
```
Question, I know that this issue with dependency loop on
`local-fs.target` can be corrected by `nofail`/`_netdev` options in
`/etc/fstab` file for all local shares (as then all local mounts
will be only wanted (`Wants` parameter), and not required
(`Requires` parameter) by `local-fs.target), but my question is
about behavior without these options specified, do you think that
this (explained) behavior is correct?
This looks for me as quite dangerous behavior (without
`nofail`/`_netdev` options) since in this scenario we may delete
start jobs for up to 10 different services/targets (including
network stack (`network.service`)), here is the full list of
possible services/targets which may never start due to this issue
(ordering cycle (loop)),
Yeah, when dependency cycles are detected systemd will try to fix
them, by dropping jobs from the reaction. This algorithm might not do
what people would like it to do though, as there's no metric for
knowing which one is the least important job, and hence the one to
drop.

There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.

Lennart
--
Lennart Poettering, Red Hat
John Florian
2018-04-26 20:04:01 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.
Having been the author of numerous custom services that are intended to
wedge into those provided by systemd/Fedora I've faced resolving these
several times and I've never felt all that competent at the task. 
Lennart, you were immensely helpful on one occasion by pointing me to
`systemctl show FOO` but even then hunting the problem down was far from
simple (for me).  I've done the dot/graphviz thing and found it just as
useless to me as when I've tried applying it to Puppet's
ordering/dependency looping.  I'm not blaming those tools (or systemd)
because I'm well aware much of the problem is my inability to use them
effectively.  The graphs always seem to be overly simple and revealing
no problem or overly detailed and obscuring the problem.  Compound that
with "an arrow pointing this way means what exactly?"

Is there anything else that could be done to make hunting these loops
down easier?  Is there an example of any product that has a similar
situation where they excel at helping the developer?  Or are we already
there and I just need more practice?  I'm sure part of my struggle is
just not encountering these very regularly and integrating into much
that I'm only partly familiar with, but the result is the same,
trepidation and loathing.
John Florian
2018-04-26 20:20:15 UTC
Reply
Permalink
Raw Message
Post by John Florian
Post by Lennart Poettering
There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.
Having been the author of numerous custom services that are intended
to wedge into those provided by systemd/Fedora I've faced resolving
these several times and I've never felt all that competent at the
task.  Lennart, you were immensely helpful on one occasion by pointing
me to `systemctl show FOO` but even then hunting the problem down was
far from simple (for me).  I've done the dot/graphviz thing and found
it just as useless to me as when I've tried applying it to Puppet's
ordering/dependency looping. I'm not blaming those tools (or systemd)
because I'm well aware much of the problem is my inability to use them
effectively.  The graphs always seem to be overly simple and revealing
no problem or overly detailed and obscuring the problem.  Compound
that with "an arrow pointing this way means what exactly?"
Is there anything else that could be done to make hunting these loops
down easier?  Is there an example of any product that has a similar
situation where they excel at helping the developer?  Or are we
already there and I just need more practice?  I'm sure part of my
struggle is just not encountering these very regularly and integrating
into much that I'm only partly familiar with, but the result is the
same, trepidation and loathing.
Since I hate to grumble w/o so much as offering any possible
suggestions... It seems like it might be helpful to have something like
`systemctl show` but only dumping the properties used for ordering and
dependencies.  Ideally, only ordering OR dependencies as the case
warrants and for all units involved, but only those units.
Andrei Borzenkov
2018-04-27 04:01:25 UTC
Reply
Permalink
Raw Message
Post by John Florian
Post by John Florian
Post by Lennart Poettering
There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.
Having been the author of numerous custom services that are intended
to wedge into those provided by systemd/Fedora I've faced resolving
these several times and I've never felt all that competent at the
task.  Lennart, you were immensely helpful on one occasion by pointing
me to `systemctl show FOO` but even then hunting the problem down was
far from simple (for me).  I've done the dot/graphviz thing and found
it just as useless to me as when I've tried applying it to Puppet's
ordering/dependency looping. I'm not blaming those tools (or systemd)
because I'm well aware much of the problem is my inability to use them
effectively.  The graphs always seem to be overly simple and revealing
no problem or overly detailed and obscuring the problem.  Compound
that with "an arrow pointing this way means what exactly?"
Is there anything else that could be done to make hunting these loops
down easier?  Is there an example of any product that has a similar
situation where they excel at helping the developer?  Or are we
already there and I just need more practice?  I'm sure part of my
struggle is just not encountering these very regularly and integrating
into much that I'm only partly familiar with, but the result is the
same, trepidation and loathing.
Since I hate to grumble w/o so much as offering any possible
suggestions... It seems like it might be helpful to have something like
`systemctl show` but only dumping the properties used for ordering and
dependencies.  Ideally, only ordering OR dependencies as the case
warrants and for all units involved, but only those units.
Well, you could use "systemctl show -p Id -p After -p Before" but the
problem is to find the correct unit set. You really need something that
simulates transaction and evaluates dependencies like systemd does it.

Note that systemd should print offending cycle when it hits it:

апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start


It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.
Jérémy Rosen
2018-04-27 07:56:09 UTC
Reply
Permalink
Raw Message
Ok, there are a couple of tools that have not be mentionned..

"systemd-analyze verify <service>" will check all sorts of stuff on a
service, including checking recursively the deps and finding ordering cycles


systemctl list-dependencies can show ordering/req dependencies, up or
down, recursively or not

to be completely honest, I never use dot for cycles.... I sometime
read/grep the .dot file itself, but I never generate the image...

systemd-analyze is my favorite tool for that sort of problems
Post by Andrei Borzenkov
Post by John Florian
Post by John Florian
Post by Lennart Poettering
There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.
Having been the author of numerous custom services that are intended
to wedge into those provided by systemd/Fedora I've faced resolving
these several times and I've never felt all that competent at the
task.  Lennart, you were immensely helpful on one occasion by pointing
me to `systemctl show FOO` but even then hunting the problem down was
far from simple (for me).  I've done the dot/graphviz thing and found
it just as useless to me as when I've tried applying it to Puppet's
ordering/dependency looping. I'm not blaming those tools (or systemd)
because I'm well aware much of the problem is my inability to use them
effectively.  The graphs always seem to be overly simple and revealing
no problem or overly detailed and obscuring the problem.  Compound
that with "an arrow pointing this way means what exactly?"
Is there anything else that could be done to make hunting these loops
down easier?  Is there an example of any product that has a similar
situation where they excel at helping the developer?  Or are we
already there and I just need more practice?  I'm sure part of my
struggle is just not encountering these very regularly and integrating
into much that I'm only partly familiar with, but the result is the
same, trepidation and loathing.
Since I hate to grumble w/o so much as offering any possible
suggestions... It seems like it might be helpful to have something like
`systemctl show` but only dumping the properties used for ordering and
dependencies.  Ideally, only ordering OR dependencies as the case
warrants and for all units involved, but only those units.
Well, you could use "systemctl show -p Id -p After -p Before" but the
problem is to find the correct unit set. You really need something that
simulates transaction and evaluates dependencies like systemd does it.
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start
It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
SMILE <http://www.smile.eu/>

20 rue des Jardins
92600 AsniÚres-sur-Seine


*Jérémy ROSEN*
Architecte technique
Responsable de l'expertise Smile-ECS

email ***@smile.fr <mailto:***@smile.fr>
phone +33141402967
url http://www.smile.eu

Twitter <https://twitter.com/GroupeSmile> Facebook
<https://www.facebook.com/smileopensource> LinkedIn
<https://www.linkedin.com/company/smile> Github
<https://github.com/Smile-SA>


Découvrez l’univers Smile, rendez-vous sur smile.eu
<http://smile.eu/?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÚte, n'imprimez ce mail que si c'est nécessaire
John Florian
2018-04-27 13:13:30 UTC
Reply
Permalink
Raw Message
Post by Jérémy Rosen
Ok, there are a couple of tools that have not be mentionned..
"systemd-analyze verify <service>" will check all sorts of stuff on a
service, including checking recursively the deps and finding ordering cycles
systemctl list-dependencies can show ordering/req dependencies, up or
down, recursively or not
to be completely honest, I never use dot for cycles.... I sometime
read/grep the .dot file itself, but I never generate the image...
systemd-analyze is my favorite tool for that sort of problems
Post by Andrei Borzenkov
Well, you could use "systemctl show -p Id -p After -p Before" but the
problem is to find the correct unit set. You really need something that
simulates transaction and evaluates dependencies like systemd does it.
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start
Well these are good things to know.  I can't recall if I ever tried to
use them when faced with a loop.  I'd guess that I tried
`systemd-analyze verify` as it sort of jumps off the man page whereas
`systemctl list-dependencies` is kind of buried in the details.
Post by Jérémy Rosen
Post by Andrei Borzenkov
It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.
Maybe one really helpful immediate change would be to have systemd log
some hints when it breaks a loop to suggest a few commands that might be
run to help understand the situation more clearly.
Michal Sekletar
2018-05-09 15:39:26 UTC
Reply
Permalink
Raw Message
Post by Andrei Borzenkov
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start
It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.
I was thinking that in addition to better log messages we could
generate dot graph of the cycle and dump it to journal as a custom
field. So people can then turn it into the picture and get a better
understanding of the cycle. Do you think it would be helpful?

Michal
Michal Sekletar
2018-05-10 07:51:11 UTC
Reply
Permalink
Raw Message
What information would the graph contain? The basic structure of a
cycle is always just a simple ring, and I don't see what benefit making
a graph of that would give over just listing the nodes in order.
The simple transformation of the current text output to graph form is
not very helpful, but I think we could provide more than that. For
example, highlighting a job that systemd removed to break the cycle,
annotating edges with dependency origin (e.g. implicit dependencies
colour-coded differently), highlighting anchor job of the transaction
if it is present in the cycle.

Basically, I am trying to figure out what relevant pieces of the
runtime state could be captured in such graph and help admins (or
maintainers) to understand and fix the dependency cycle in their
environment.

Michal

Loading...