Discussion:
loginctl hangs when run at boot time
Add Reply
Jeff Solomon
2018-04-05 06:19:57 UTC
Reply
Permalink
Raw Message
Hi,

On the CentOS7 AWS with the following systemd installed:

systemd-219-42.el7_4.4.x86_64

Any loginctl command that we try to run at boot time (during the AWS "user
data" section of cloud-init), will hang and then timeout.

The same loginctl command run after you ssh into the box and become root,
works fine.

I can currently get around this problem by running:

systemd-run loginctl ....

which implies that something about the boot time execution environment
doesn't allow loginctl to run.

When I strace the hanging command, I can see that it's waiting on
/run/dbus/system_bus_socket.

I don't think I have a unit ordering problem because systemd-run is able to
run the command. It must be the environment.

Sorry for what must be a dumb question, but what is different about the
execution environment at boot time that is preventing loginctl from running?

Thanks,

Jeff
Lennart Poettering
2018-04-05 08:42:21 UTC
Reply
Permalink
Raw Message
Post by Jeff Solomon
Hi,
systemd-219-42.el7_4.4.x86_64
Any loginctl command that we try to run at boot time (during the AWS "user
data" section of cloud-init), will hang and then timeout.
The same loginctl command run after you ssh into the box and become root,
works fine.
systemd-run loginctl ....
which implies that something about the boot time execution environment
doesn't allow loginctl to run.
When I strace the hanging command, I can see that it's waiting on
/run/dbus/system_bus_socket.
I don't think I have a unit ordering problem because systemd-run is able to
run the command. It must be the environment.
Sorry for what must be a dumb question, but what is different about the
execution environment at boot time that is preventing loginctl from running?
loginctl is a client tool for systemd-logind, the communicatoin
between both happens through dbus, hence both are client to
dbus-daemon. Both systemd-logind and dbus-daemon are run during late
boot (i.e. are ordered after basic.target). systemd-logind is bus
activated, and dbus-daemon is socket activated, which means that as
soon as dbus-daemon's socket unit is up (i.e. dbus.socket is started)
any client (such as loginctl) connecting to it will implicitly wait
first for dbus-daemon to be fully started, and then until logind to be
fully started.

Now, if you invoke loginctl from something that itself delays the boot
process, then you might create a deadlock: you are waiting for
dbus.service/systemd-logind.service to start-up but are at the same
time blocking it from being started.

Lennart
--
Lennart Poettering, Red Hat
Jeff Solomon
2018-04-05 20:02:01 UTC
Reply
Permalink
Raw Message
Thanks Lennart, I hear what you're saying but I'm still confused.

I can run any commands in the "User Data" section. And when I run
"systemctl" immediately prior to attempting to run loginctl (which hangs),
I see the following (snipped):

dbus.service loaded
active running D-Bus System Message Bus
dbus.socket loaded
active running D-Bus System Message Bus Socket
systemd-logind.service loaded
active running Login Service

which makes me think that even though cloud-init and these two services all
run after basic.target, the two needed to loginctl to run successfully are
already running.

Additionally, the following command works:

systemd-run loginctl ...

I understand what you're saying about dead-lock, but when I run
systemd-run, all I'm really doing is changing the execution environment,
not the flow of my entire script, and loginctl works in that case.

I guess I still don't understand how your explanation makes sense given the
behavior I observe.
Post by Jeff Solomon
Post by Jeff Solomon
Hi,
systemd-219-42.el7_4.4.x86_64
Any loginctl command that we try to run at boot time (during the AWS
"user
Post by Jeff Solomon
data" section of cloud-init), will hang and then timeout.
The same loginctl command run after you ssh into the box and become root,
works fine.
systemd-run loginctl ....
which implies that something about the boot time execution environment
doesn't allow loginctl to run.
When I strace the hanging command, I can see that it's waiting on
/run/dbus/system_bus_socket.
I don't think I have a unit ordering problem because systemd-run is able
to
Post by Jeff Solomon
run the command. It must be the environment.
Sorry for what must be a dumb question, but what is different about the
execution environment at boot time that is preventing loginctl from
running?
loginctl is a client tool for systemd-logind, the communicatoin
between both happens through dbus, hence both are client to
dbus-daemon. Both systemd-logind and dbus-daemon are run during late
boot (i.e. are ordered after basic.target). systemd-logind is bus
activated, and dbus-daemon is socket activated, which means that as
soon as dbus-daemon's socket unit is up (i.e. dbus.socket is started)
any client (such as loginctl) connecting to it will implicitly wait
first for dbus-daemon to be fully started, and then until logind to be
fully started.
Now, if you invoke loginctl from something that itself delays the boot
process, then you might create a deadlock: you are waiting for
dbus.service/systemd-logind.service to start-up but are at the same
time blocking it from being started.
Lennart
--
Lennart Poettering, Red Hat
Loading...