Discussion:
Question about service dependency handling in systemd-228
(too old to reply)
Bao Nguyen
2017-11-25 05:08:17 UTC
Permalink
Hello everyone,

I would like to have a question regarding to the building dependency and
cycle dependency handling on systemd-228. In my system, I have some socket
and service files, it has a cycle on socket target, when I run on
system-228, systemd-228 throws

[ 40.358582] systemd[1]: Set hostname to <MY-2>.
[ 41.154231] systemd[1]: nss-lookup.target: Dependency
Before=nss-lookup.target dropped
[ 41.297229] systemd[1]: sockets.target: Found ordering cycle on
sockets.target/start
[ 41.297236] systemd[1]: sockets.target: Found dependency on
asi-My-5101.socket/start
[ 41.297239] systemd[1]: sockets.target: Found dependency on
My-sshd.target/start
[ 41.297241] systemd[1]: sockets.target: Found dependency on
My-syncd.service/start
[ 41.297244] systemd[1]: sockets.target: Found dependency on
My-nfs-client.service/start
[ 41.297246] systemd[1]: sockets.target: Found dependency on
My-handling.service/start
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5101
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5010
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5111
[ SKIP ] Ordering cycle found, skipping asi-vsftpd-MyIO_2.socket
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5110
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5002
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5100
[ SKIP ] Ordering cycle found, skipping Remo...ell Facilities Activation
Socket
[ SKIP ] Ordering cycle found, skipping My Telnet Server Socket on port 5011
[ SKIP ] Ordering cycle found, skipping My sshd target


It said that there is an ordering on the sockets.target, then break the
cycle and SKIP randomly starting other service => the system cannot start.
However I did not meet the same issue on systemd-210 with the same my
services and sockets. Systemd-210 does not break and skip, and my system
can start well.

My question is if there are any significant different about building tree
dependency and handling cycle dependency between systemd-210 and
systemd-228 that can lead to my current situation? I have checked the
change log, source code but not found any useful info

And what does the message "nss-lookup.target: Dependency
Before=nss-lookup.target dropped" mean? I do not see it in systemd-210.

Many thanks for your support,
Best regards,
Naru
Uoti Urpala
2017-11-25 14:44:36 UTC
Permalink
[ 41.154231] systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped
[ 41.297229] systemd[1]: sockets.target: Found ordering cycle on sockets.target/start
[ 41.297236] systemd[1]: sockets.target: Found dependency on asi-My-5101.socket/start
[ 41.297239] systemd[1]: sockets.target: Found dependency on My-sshd.target/start
[ 41.297241] systemd[1]: sockets.target: Found dependency on My-syncd.service/start
[ 41.297244] systemd[1]: sockets.target: Found dependency on My-nfs-client.service/start
[ 41.297246] systemd[1]: sockets.target: Found dependency on My-handling.service/start
My question is if there are any significant different about building tree dependency and handling cycle dependency between systemd-210 and systemd-228 that can lead to my current situation? I have checked the change log, source code but not found any useful info
Rather than start by trying to find differences between systemd
versions, I suggest you first find out exactly what goes wrong under
the newer systemd version. Exactly which dependency is wrong and
shouldn't be there? Where does that dependency come from? A system
where ordering dependencies form a cycle is not valid, so some
dependency explicitly listed in your unit files or implicitly added by
systemd must be wrong. After finding that out, you can then try to find
out what differs under the older systemd if it's still relevant.

In the above log, the most suspicious part is that it seems to say
"asi-My-5101.socket" depends on "My-sshd.target". A socket unit almost
certainly shouldn't have such dependencies, as normally a listening
socket can be opened regardless of the state of the rest of the system
(the main exception I can think of would be a UNIX socket at a
filesystem path that requires mounting something, but normally you
wouldn't do that...).
And what does the message "nss-lookup.target: Dependency Before=nss-lookup.target dropped" mean? I do not see it in systemd-210.
Apparently the target had a dependency saying that it should be started
before itself, and such a blatantly impossible dependency was ignored.
Bao Nguyen
2017-11-26 09:47:41 UTC
Permalink
Hi Uoti,

Thanks a lot for your answer, I have checked the cycle. It is created
by sockets.target
-> asi-My-5101.socket -> My-sshd.target -> My-syncd.service ->
My-nfs-client.service
-> My-handling.service -> basic.target -> sockets.target. I do not see the
same cycle in systemd-210 so I said that there is maybe a change in
systemd-210 and sytemd-228 like building dependency tree and handling
cycle. I can confirm there is no change in my scripts.

Regard to your question, "asi-My-5101.socket" depends on "My-sshd.target",
I think that in my case it is expected as my socket listens on a specific
address IP:port so it should start after a network service to configure and
assign IP address before my socket runs.

Could you please help me if it is due to the fault in sytemd-228 or I have
to adapt my script to overcome this issue? I tried to
add DefaultDependencies=no in my asi-My-5101.socket, the problem go away
(because sometimes I see it said the cycle created in basic.target, the
behavior is really strange in systemd-228).

Thanks,
Brs,
Bao
Post by Bao Nguyen
Post by Bao Nguyen
[ 41.154231] systemd[1]: nss-lookup.target: Dependency
Before=nss-lookup.target dropped
Post by Bao Nguyen
[ 41.297229] systemd[1]: sockets.target: Found ordering cycle on
sockets.target/start
Post by Bao Nguyen
[ 41.297236] systemd[1]: sockets.target: Found dependency on
asi-My-5101.socket/start
Post by Bao Nguyen
[ 41.297239] systemd[1]: sockets.target: Found dependency on
My-sshd.target/start
Post by Bao Nguyen
[ 41.297241] systemd[1]: sockets.target: Found dependency on
My-syncd.service/start
Post by Bao Nguyen
[ 41.297244] systemd[1]: sockets.target: Found dependency on
My-nfs-client.service/start
Post by Bao Nguyen
[ 41.297246] systemd[1]: sockets.target: Found dependency on
My-handling.service/start
Post by Bao Nguyen
My question is if there are any significant different about building
tree dependency and handling cycle dependency between systemd-210 and
systemd-228 that can lead to my current situation? I have checked the
change log, source code but not found any useful info
Rather than start by trying to find differences between systemd
versions, I suggest you first find out exactly what goes wrong under
the newer systemd version. Exactly which dependency is wrong and
shouldn't be there? Where does that dependency come from? A system
where ordering dependencies form a cycle is not valid, so some
dependency explicitly listed in your unit files or implicitly added by
systemd must be wrong. After finding that out, you can then try to find
out what differs under the older systemd if it's still relevant.
In the above log, the most suspicious part is that it seems to say
"asi-My-5101.socket" depends on "My-sshd.target". A socket unit almost
certainly shouldn't have such dependencies, as normally a listening
socket can be opened regardless of the state of the rest of the system
(the main exception I can think of would be a UNIX socket at a
filesystem path that requires mounting something, but normally you
wouldn't do that...).
Post by Bao Nguyen
And what does the message "nss-lookup.target: Dependency
Before=nss-lookup.target dropped" mean? I do not see it in systemd-210.
Apparently the target had a dependency saying that it should be started
before itself, and such a blatantly impossible dependency was ignored.
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Reindl Harald
2017-11-26 09:53:59 UTC
Permalink
Post by Bao Nguyen
Regard to your question, "asi-My-5101.socket" depends on
"My-sshd.target", I think that in my case it is expected as my socket
listens on a specific address IP:port so it should start after a network
service to configure and assign IP address before my socket runs
nonsense - the whole point of socket activation is to have sockets
listening before other stuff is up and running

https://www.freedesktop.org/software/systemd/man/systemd.socket.html
If an IP address is used here, it is often desirable to listen on it
before the interface it is configured on is up and running, and even
regardless of whether it will be up and running at any point. To deal
with this, it is recommended to set the FreeBind= option described below

FreeBind=
Takes a boolean value. Controls whether the socket can be bound to
non-local IP addresses. This is useful to configure sockets listening on
specific IP addresses before those IP addresses are successfully
configured on a network interface. This sets the IP_FREEBIND socket
option. For robustness reasons it is recommended to use this option
whenever you bind a socket to a specific IP address. Defaults to false.
Bao Nguyen
2017-11-27 04:23:05 UTC
Permalink
Hi,

Thanks all for your comments. I will try to use option FreeBind. However
could anyone explain for me that I did not use FreeBind option in
systems-210 but all my services start well? I am still inclined to the
different of systemd-228 and systemd-210 causes the current issue.

Thanks again,
Brs,
Bao
Post by Reindl Harald
Post by Bao Nguyen
Regard to your question, "asi-My-5101.socket" depends on
"My-sshd.target", I think that in my case it is expected as my socket
listens on a specific address IP:port so it should start after a network
service to configure and assign IP address before my socket runs
nonsense - the whole point of socket activation is to have sockets
listening before other stuff is up and running
https://www.freedesktop.org/software/systemd/man/systemd.socket.html
If an IP address is used here, it is often desirable to listen on it
before the interface it is configured on is up and running, and even
regardless of whether it will be up and running at any point. To deal with
this, it is recommended to set the FreeBind= option described below
FreeBind=
Takes a boolean value. Controls whether the socket can be bound to
non-local IP addresses. This is useful to configure sockets listening on
specific IP addresses before those IP addresses are successfully configured
on a network interface. This sets the IP_FREEBIND socket option. For
robustness reasons it is recommended to use this option whenever you bind a
socket to a specific IP address. Defaults to false.
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Reindl Harald
2017-11-27 09:11:44 UTC
Permalink
Post by Bao Nguyen
Thanks all for your comments. I will try to use option FreeBind. However
could anyone explain for me that I did not use FreeBind option in
systems-210 but all my services start well? I am still inclined to the
different of systemd-228 and systemd-210 causes the current issue.
beause your configuration was undefined behavior and never made any
sense when there are dependency loops and similar problems - systemd
does and did the best not throw you to the mergency console and boot the
system somehow, pointed out errors and now it's time to fi them

IMHO it would be justified not to boot at all if there is as example a
unit which has itself in After/Before/Requires as example when someone
don't read his systemlogs after change units and "systemctl
daemon-reload" :-)
Post by Bao Nguyen
Regard to your question, "asi-My-5101.socket" depends on
"My-sshd.target", I think that in my case it is expected as my
socket listens on a specific address IP:port so it should start
after a network service to configure and assign IP address
before my socket runs
nonsense - the whole point of socket activation is to have sockets
listening before other stuff is up and running
https://www.freedesktop.org/software/systemd/man/systemd.socket.html
<https://www.freedesktop.org/software/systemd/man/systemd.socket.html>
If an IP address is used here, it is often desirable to listen on it
before the interface it is configured on is up and running, and even
regardless of whether it will be up and running at any point. To
deal with this, it is recommended to set the FreeBind= option
described below
FreeBind=
Takes a boolean value. Controls whether the socket can be bound to
non-local IP addresses. This is useful to configure sockets
listening on specific IP addresses before those IP addresses are
successfully configured on a network interface. This sets the
IP_FREEBIND socket option. For robustness reasons it is recommended
to use this option whenever you bind a socket to a specific IP
address. Defaults to false.
Bao Nguyen
2017-11-30 05:23:46 UTC
Permalink
Hi all,

Thank you very much for your support.

I will try to fix the cycle.

Brs,
Post by Bao Nguyen
Thanks all for your comments. I will try to use option FreeBind. However
could anyone explain for me that I did not use FreeBind option in
systems-210 but all my services start well? I am still inclined to the
different of systemd-228 and systemd-210 causes the current issue.
beause your configuration was undefined behavior and never made any sense
when there are dependency loops and similar problems - systemd does and did
the best not throw you to the mergency console and boot the system somehow,
pointed out errors and now it's time to fi them
IMHO it would be justified not to boot at all if there is as example a
unit which has itself in After/Before/Requires as example when someone
don't read his systemlogs after change units and "systemctl daemon-reload"
:-)
Post by Bao Nguyen
Regard to your question, "asi-My-5101.socket" depends on
"My-sshd.target", I think that in my case it is expected as my
socket listens on a specific address IP:port so it should start
after a network service to configure and assign IP address
before my socket runs
nonsense - the whole point of socket activation is to have sockets
listening before other stuff is up and running
https://www.freedesktop.org/software/systemd/man/systemd.socket.html
<https://www.freedesktop.org/software/systemd/man/systemd.socket.html
If an IP address is used here, it is often desirable to listen on it
before the interface it is configured on is up and running, and even
regardless of whether it will be up and running at any point. To
deal with this, it is recommended to set the FreeBind= option
described below
FreeBind=
Takes a boolean value. Controls whether the socket can be bound to
non-local IP addresses. This is useful to configure sockets
listening on specific IP addresses before those IP addresses are
successfully configured on a network interface. This sets the
IP_FREEBIND socket option. For robustness reasons it is recommended
to use this option whenever you bind a socket to a specific IP
address. Defaults to false.
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Mantas Mikulėnas
2017-11-26 09:59:03 UTC
Permalink
Post by Bao Nguyen
Hi Uoti,
Thanks a lot for your answer, I have checked the cycle. It is created by sockets.target
-> asi-My-5101.socket -> My-sshd.target -> My-syncd.service -> My-nfs-client.service
-> My-handling.service -> basic.target -> sockets.target. I do not see
the same cycle in systemd-210 so I said that there is maybe a change in
systemd-210 and sytemd-228 like building dependency tree and handling
cycle. I can confirm there is no change in my scripts.
Could you please show the full (not paraphrased) dependencies that you
have? that is, [Unit] sections, *.wants/ contents, and such
Post by Bao Nguyen
Regard to your question, "asi-My-5101.socket" depends on
"My-sshd.target", I think that in my case it is expected as my socket
listens on a specific address IP:port so it should start after a network
service to configure and assign IP address before my socket runs.
"My-sshd.target" does not sound like a network service.

In any case, just use FreeBind=true instead.
--
Mantas Mikulėnas <***@gmail.com>
Loading...