Discussion:
`Found ordering cycle on _SERVICE_` loops due to `local-fs.target` with `Requires` instead of `Wants` for mounts generated from `/etc/fstab`
(too old to reply)
Alexey Matyushin
2018-04-24 23:26:21 UTC
Permalink
Raw Message
Hi there,

I initially created a bug report for this topic (apologize, Lennart), but would like to discuss this topic in mode detail here (as suggested by Lennart), and really appreciate your opinion here,

I've seen that once you define network (NFS) mount together with local (referring to that network (NFS) mount) mount in `/etc/fstab` then it will always produce dependency loops (due to `local-fs.target`),

Details and how to reproduce it,

# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

# cat /etc/*release
CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.4.1708 (Core)

### Expected behaviour you didn't see

All local mounts (dependencies) are defined for `local-fs.target` with `Wants` parameter.

### Unexpected behaviour you saw

All local mounts are defined for `local-fs.target` with `Requires` parameter instead of `Wants` parameter.

### Steps to reproduce the problem

1. Add NFS mount into `/etc/fstab`, example
```server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0```
2. Add local mount with dependency on NFS mount, example,
```/mnt/some_other_mount/ /mnt/test/media/ none bind,rw 0 0```

Example of `/etc/fstab`,
```
#
# /etc/fstab
# Created by anaconda on Mon Nov 27 22:04:35 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=2f15c101-f516-4ee8-a4b7-89ad880610db / xfs defaults 0 0

server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0
/mnt/some_other_mount/ /mnt/test/medi none bind,rw 0 0
```

Question,
I know that this issue with dependency loop on `local-fs.target` can be corrected by `nofail`/`_netdev` options in `/etc/fstab` file for all local shares (as then all local mounts will be only wanted (`Wants` parameter), and not required (`Requires` parameter) by `local-fs.target), but my question is about behavior without these options specified, do you think that this (explained) behavior is correct?

This looks for me as quite dangerous behavior (without `nofail`/`_netdev` options) since in this scenario we may delete start jobs for up to 10 different services/targets (including network stack (`network.service`)), here is the full list of possible services/targets which may never start due to this issue (ordering cycle (loop)),

systemd-journal-catalog-update.service
systemd-machine-id-commit.service
systemd-tmpfiles-setup.service
rhel-import-state.service
systemd-update-done.service
auditd.service
network.service
nfs-config.service
local-fs.target

Thanks a lot in advance!

Kind regards,
Alexey
Lennart Poettering
2018-04-25 08:53:33 UTC
Permalink
Raw Message
On Di, 24.04.18 19:26, Alexey Matyushin (***@protonmail.ch) wrote:

>
> server.domain:/ /mnt/test nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0
> /mnt/some_other_mount/ /mnt/test/medi none bind,rw 0 0
> ```
>
> Question, I know that this issue with dependency loop on
> `local-fs.target` can be corrected by `nofail`/`_netdev` options in
> `/etc/fstab` file for all local shares (as then all local mounts
> will be only wanted (`Wants` parameter), and not required
> (`Requires` parameter) by `local-fs.target), but my question is
> about behavior without these options specified, do you think that
> this (explained) behavior is correct?
>
> This looks for me as quite dangerous behavior (without
> `nofail`/`_netdev` options) since in this scenario we may delete
> start jobs for up to 10 different services/targets (including
> network stack (`network.service`)), here is the full list of
> possible services/targets which may never start due to this issue
> (ordering cycle (loop)),

Yeah, when dependency cycles are detected systemd will try to fix
them, by dropping jobs from the reaction. This algorithm might not do
what people would like it to do though, as there's no metric for
knowing which one is the least important job, and hence the one to
drop.

There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.

Lennart

--
Lennart Poettering, Red Hat
John Florian
2018-04-26 20:04:01 UTC
Permalink
Raw Message
On 2018-04-25 04:53, Lennart Poettering wrote:
> There have been requests in improving the cycle breaking algorithm,
> but not much has been done in this area, since it's not clear what can
> be done. Ultimately it's just polishing a broken situation, and
> the better way out is to fix things properly, i.e. just correct the
> cycle in the dependencies in the first place.

Having been the author of numerous custom services that are intended to
wedge into those provided by systemd/Fedora I've faced resolving these
several times and I've never felt all that competent at the task. 
Lennart, you were immensely helpful on one occasion by pointing me to
`systemctl show FOO` but even then hunting the problem down was far from
simple (for me).  I've done the dot/graphviz thing and found it just as
useless to me as when I've tried applying it to Puppet's
ordering/dependency looping.  I'm not blaming those tools (or systemd)
because I'm well aware much of the problem is my inability to use them
effectively.  The graphs always seem to be overly simple and revealing
no problem or overly detailed and obscuring the problem.  Compound that
with "an arrow pointing this way means what exactly?"

Is there anything else that could be done to make hunting these loops
down easier?  Is there an example of any product that has a similar
situation where they excel at helping the developer?  Or are we already
there and I just need more practice?  I'm sure part of my struggle is
just not encountering these very regularly and integrating into much
that I'm only partly familiar with, but the result is the same,
trepidation and loathing.
John Florian
2018-04-26 20:20:15 UTC
Permalink
Raw Message
On 2018-04-26 16:04, John Florian wrote:
> On 2018-04-25 04:53, Lennart Poettering wrote:
>> There have been requests in improving the cycle breaking algorithm,
>> but not much has been done in this area, since it's not clear what can
>> be done. Ultimately it's just polishing a broken situation, and
>> the better way out is to fix things properly, i.e. just correct the
>> cycle in the dependencies in the first place.
>
> Having been the author of numerous custom services that are intended
> to wedge into those provided by systemd/Fedora I've faced resolving
> these several times and I've never felt all that competent at the
> task.  Lennart, you were immensely helpful on one occasion by pointing
> me to `systemctl show FOO` but even then hunting the problem down was
> far from simple (for me).  I've done the dot/graphviz thing and found
> it just as useless to me as when I've tried applying it to Puppet's
> ordering/dependency looping. I'm not blaming those tools (or systemd)
> because I'm well aware much of the problem is my inability to use them
> effectively.  The graphs always seem to be overly simple and revealing
> no problem or overly detailed and obscuring the problem.  Compound
> that with "an arrow pointing this way means what exactly?"
>
> Is there anything else that could be done to make hunting these loops
> down easier?  Is there an example of any product that has a similar
> situation where they excel at helping the developer?  Or are we
> already there and I just need more practice?  I'm sure part of my
> struggle is just not encountering these very regularly and integrating
> into much that I'm only partly familiar with, but the result is the
> same, trepidation and loathing.

Since I hate to grumble w/o so much as offering any possible
suggestions... It seems like it might be helpful to have something like
`systemctl show` but only dumping the properties used for ordering and
dependencies.  Ideally, only ordering OR dependencies as the case
warrants and for all units involved, but only those units.
Andrei Borzenkov
2018-04-27 04:01:25 UTC
Permalink
Raw Message
26.04.2018 23:20, John Florian пишет:
> On 2018-04-26 16:04, John Florian wrote:
>> On 2018-04-25 04:53, Lennart Poettering wrote:
>>> There have been requests in improving the cycle breaking algorithm,
>>> but not much has been done in this area, since it's not clear what can
>>> be done. Ultimately it's just polishing a broken situation, and
>>> the better way out is to fix things properly, i.e. just correct the
>>> cycle in the dependencies in the first place.
>>
>> Having been the author of numerous custom services that are intended
>> to wedge into those provided by systemd/Fedora I've faced resolving
>> these several times and I've never felt all that competent at the
>> task.  Lennart, you were immensely helpful on one occasion by pointing
>> me to `systemctl show FOO` but even then hunting the problem down was
>> far from simple (for me).  I've done the dot/graphviz thing and found
>> it just as useless to me as when I've tried applying it to Puppet's
>> ordering/dependency looping. I'm not blaming those tools (or systemd)
>> because I'm well aware much of the problem is my inability to use them
>> effectively.  The graphs always seem to be overly simple and revealing
>> no problem or overly detailed and obscuring the problem.  Compound
>> that with "an arrow pointing this way means what exactly?"
>>
>> Is there anything else that could be done to make hunting these loops
>> down easier?  Is there an example of any product that has a similar
>> situation where they excel at helping the developer?  Or are we
>> already there and I just need more practice?  I'm sure part of my
>> struggle is just not encountering these very regularly and integrating
>> into much that I'm only partly familiar with, but the result is the
>> same, trepidation and loathing.
>
> Since I hate to grumble w/o so much as offering any possible
> suggestions... It seems like it might be helpful to have something like
> `systemctl show` but only dumping the properties used for ordering and
> dependencies.  Ideally, only ordering OR dependencies as the case
> warrants and for all units involved, but only those units.

Well, you could use "systemctl show -p Id -p After -p Before" but the
problem is to find the correct unit set. You really need something that
simulates transaction and evaluates dependencies like systemd does it.

Note that systemd should print offending cycle when it hits it:

апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start


It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.
Jérémy Rosen
2018-04-27 07:56:09 UTC
Permalink
Raw Message
Ok, there are a couple of tools that have not be mentionned..

"systemd-analyze verify <service>" will check all sorts of stuff on a
service, including checking recursively the deps and finding ordering cycles


systemctl list-dependencies can show ordering/req dependencies, up or
down, recursively or not

to be completely honest, I never use dot for cycles.... I sometime
read/grep the .dot file itself, but I never generate the image...

systemd-analyze is my favorite tool for that sort of problems

On 27/04/2018 06:01, Andrei Borzenkov wrote:
> 26.04.2018 23:20, John Florian пОшет:
>> On 2018-04-26 16:04, John Florian wrote:
>>> On 2018-04-25 04:53, Lennart Poettering wrote:
>>>> There have been requests in improving the cycle breaking algorithm,
>>>> but not much has been done in this area, since it's not clear what can
>>>> be done. Ultimately it's just polishing a broken situation, and
>>>> the better way out is to fix things properly, i.e. just correct the
>>>> cycle in the dependencies in the first place.
>>> Having been the author of numerous custom services that are intended
>>> to wedge into those provided by systemd/Fedora I've faced resolving
>>> these several times and I've never felt all that competent at the
>>> task.  Lennart, you were immensely helpful on one occasion by pointing
>>> me to `systemctl show FOO` but even then hunting the problem down was
>>> far from simple (for me).  I've done the dot/graphviz thing and found
>>> it just as useless to me as when I've tried applying it to Puppet's
>>> ordering/dependency looping. I'm not blaming those tools (or systemd)
>>> because I'm well aware much of the problem is my inability to use them
>>> effectively.  The graphs always seem to be overly simple and revealing
>>> no problem or overly detailed and obscuring the problem.  Compound
>>> that with "an arrow pointing this way means what exactly?"
>>>
>>> Is there anything else that could be done to make hunting these loops
>>> down easier?  Is there an example of any product that has a similar
>>> situation where they excel at helping the developer?  Or are we
>>> already there and I just need more practice?  I'm sure part of my
>>> struggle is just not encountering these very regularly and integrating
>>> into much that I'm only partly familiar with, but the result is the
>>> same, trepidation and loathing.
>> Since I hate to grumble w/o so much as offering any possible
>> suggestions... It seems like it might be helpful to have something like
>> `systemctl show` but only dumping the properties used for ordering and
>> dependencies.  Ideally, only ordering OR dependencies as the case
>> warrants and for all units involved, but only those units.
> Well, you could use "systemctl show -p Id -p After -p Before" but the
> problem is to find the correct unit set. You really need something that
> simulates transaction and evaluates dependencies like systemd does it.
>
> Note that systemd should print offending cycle when it hits it:
>
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> ordering cycle on foo.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on bar.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on baz.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on foo.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
> ordering cycle by deleting job baz.service/start
>
>
> It would be helpful it if additionally printed kind of dependency (like
> foo.service is After bar.service) because this is by far not obvious for
> average user.
> _______________________________________________
> systemd-devel mailing list
> systemd-***@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel

--
SMILE <http://www.smile.eu/>

20 rue des Jardins
92600 AsniÚres-sur-Seine


*Jérémy ROSEN*
Architecte technique
Responsable de l'expertise Smile-ECS

email ***@smile.fr <mailto:***@smile.fr>
phone +33141402967
url http://www.smile.eu

Twitter <https://twitter.com/GroupeSmile> Facebook
<https://www.facebook.com/smileopensource> LinkedIn
<https://www.linkedin.com/company/smile> Github
<https://github.com/Smile-SA>


Découvrez l’univers Smile, rendez-vous sur smile.eu
<http://smile.eu/?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÚte, n'imprimez ce mail que si c'est nécessaire
John Florian
2018-04-27 13:13:30 UTC
Permalink
Raw Message
On 2018-04-27 03:56, Jérémy Rosen wrote:
> Ok, there are a couple of tools that have not be mentionned..
>
> "systemd-analyze verify <service>" will check all sorts of stuff on a
> service, including checking recursively the deps and finding ordering
> cycles
>
>
> systemctl list-dependencies can show ordering/req dependencies, up or
> down, recursively or not
>
> to be completely honest, I never use dot for cycles.... I sometime
> read/grep the .dot file itself, but I never generate the image...
>
> systemd-analyze is my favorite tool for that sort of problems
>
> On 27/04/2018 06:01, Andrei Borzenkov wrote:
>> Well, you could use "systemctl show -p Id -p After -p Before" but the
>> problem is to find the correct unit set. You really need something that
>> simulates transaction and evaluates dependencies like systemd does it.
>>
>> Note that systemd should print offending cycle when it hits it:
>>
>> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
>> ordering cycle on foo.service/start
>> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
>> dependency on bar.service/start
>> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
>> dependency on baz.service/start
>> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
>> dependency on foo.service/start
>> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
>> ordering cycle by deleting job baz.service/start

Well these are good things to know.  I can't recall if I ever tried to
use them when faced with a loop.  I'd guess that I tried
`systemd-analyze verify` as it sort of jumps off the man page whereas
`systemctl list-dependencies` is kind of buried in the details.

>> It would be helpful it if additionally printed kind of dependency (like
>> foo.service is After bar.service) because this is by far not obvious for
>> average user.

Maybe one really helpful immediate change would be to have systemd log
some hints when it breaks a loop to suggest a few commands that might be
run to help understand the situation more clearly.
Michal Sekletar
2018-05-09 15:39:26 UTC
Permalink
Raw Message
On Fri, Apr 27, 2018 at 6:01 AM, Andrei Borzenkov <***@gmail.com> wrote:

> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> ordering cycle on foo.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on bar.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on baz.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
> dependency on foo.service/start
> апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
> ordering cycle by deleting job baz.service/start
>
>
> It would be helpful it if additionally printed kind of dependency (like
> foo.service is After bar.service) because this is by far not obvious for
> average user.

I was thinking that in addition to better log messages we could
generate dot graph of the cycle and dump it to journal as a custom
field. So people can then turn it into the picture and get a better
understanding of the cycle. Do you think it would be helpful?

Michal
Michal Sekletar
2018-05-10 07:51:11 UTC
Permalink
Raw Message
On Wed, May 9, 2018 at 9:42 PM, Uoti Urpala <***@pp1.inet.fi> wrote:

> What information would the graph contain? The basic structure of a
> cycle is always just a simple ring, and I don't see what benefit making
> a graph of that would give over just listing the nodes in order.

The simple transformation of the current text output to graph form is
not very helpful, but I think we could provide more than that. For
example, highlighting a job that systemd removed to break the cycle,
annotating edges with dependency origin (e.g. implicit dependencies
colour-coded differently), highlighting anchor job of the transaction
if it is present in the cycle.

Basically, I am trying to figure out what relevant pieces of the
runtime state could be captured in such graph and help admins (or
maintainers) to understand and fix the dependency cycle in their
environment.

Michal
Loading...