Discussion:
how to put max time limit on reboot time
Add Reply
prashantkumar dhotre
2018-04-05 06:26:57 UTC
Reply
Permalink
Raw Message
Hi
Currently it appears that systemd does not provide a way to put max time
limit for system reboot.

So there is no way to get system reboot within specified time limit even if
hardware watchdog is present in system.


Using RunTimeWatchDogSec,ShutDownTimeWatchDogSec and
JobTimeoutSec,JobTimeoutAction= in shutdown.target will not solve this
issue.

RunTimeWatchDogSec configures hardware watchdog (hw wd )with the given
timeout value and system manager will pat the wd at least once in half of
that time.
This wd patting will be done in 1st phase of shutdown also.
example: set this to 2 min and hw wd gets configured with 2 min timeout and
system manager keeps patting every 1min.

ShutDownTimeWatchDogSec will configured HW WD with given value and pats the
WD during 2nd phase of shutdown.
2nd phase of shutdown contains loop of 'try to clean up remaining things'.

So for ex, if we set ShutDownTimeWatchDogSec to 10 min, then during 2nd
phase of shutdown , if any task of 'try to clean up remaining things' in
the loop takes more than 10 min, then only HW WD get fired and system
reboots.

So this puts max time out on each iteration of loop and not on entire loop

So this setting does not help in putting max time out for 2nd phase of
shutdown.



But intention of ShutDownTimeWatchDogSec from man page is : “It works as a
safety net to ensure that the reboot takes place even if a clean reboot
attempt times out.”

Clearly this intention is not served by current implementation of
ShutDownTimeWatchDogSec.



Do we need to :

redefine ShutDownTimeWatchDogSec and use it to put max time out on entire
2nd part of shutdown.

i.e set the user given value of ShutDownTimeWatchDogSec to HW WD at
beginning of 2nd phase of shutdown but do not pat HW WD each time in in
each iteration of loop.

this way, if 2nd phase takes more than 10 min, then hw wd gets fired and
system reboots.



please let me know if we need to open a bug for this


1) related write up

https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdNFSMountShutdown
https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdShutdownWatchdog



2) my previous queries related to this are

https://lists.freedesktop.org/archives/systemd-devel/2018-March/040566.html
https://github.com/systemd/systemd/issues/8484



Thanks
Lennart Poettering
2018-04-06 10:09:23 UTC
Reply
Permalink
Raw Message
Post by prashantkumar dhotre
Hi
Currently it appears that systemd does not provide a way to put max time
limit for system reboot.
So there is no way to get system reboot within specified time limit even if
hardware watchdog is present in system.
Using RunTimeWatchDogSec,ShutDownTimeWatchDogSec and
JobTimeoutSec,JobTimeoutAction= in shutdown.target will not solve this
issue.
RunTimeWatchDogSec configures hardware watchdog (hw wd )with the given
timeout value and system manager will pat the wd at least once in half of
that time.
This wd patting will be done in 1st phase of shutdown also.
example: set this to 2 min and hw wd gets configured with 2 min timeout and
system manager keeps patting every 1min.
ShutDownTimeWatchDogSec will configured HW WD with given value and pats the
WD during 2nd phase of shutdown.
2nd phase of shutdown contains loop of 'try to clean up remaining things'.
So for ex, if we set ShutDownTimeWatchDogSec to 10 min, then during 2nd
phase of shutdown , if any task of 'try to clean up remaining things' in
the loop takes more than 10 min, then only HW WD get fired and system
reboots.
So this puts max time out on each iteration of loop and not on entire loop
So this setting does not help in putting max time out for 2nd phase of
shutdown.
But intention of ShutDownTimeWatchDogSec from man page is : “It works as a
safety net to ensure that the reboot takes place even if a clean reboot
attempt times out.”
Clearly this intention is not served by current implementation of
ShutDownTimeWatchDogSec.
redefine ShutDownTimeWatchDogSec and use it to put max time out on entire
2nd part of shutdown.
That's not watch a watchdog is supposed to do. The "watchdog" concept
is generally used to refer to "hang" checks, i.e. to detect cases
where some program is "stuck", i.e. doesn't respond regularly
enough. It's not supposed to put a limit on the overall runtime of a
program, as long as the program keeps pinging the watchdog.

What precisely are you trying to do? Do you want to set a hard time
limit over the entire shutdown logic as a whole? We currently have no
concept for that, and adding that would make sense but would be
completely new code. We do have an over-all timeout on the first phase
of the shutdown, and we have watchdog concepts for all parts of
shutdown, but neither an over-all timeout on the second phase of the
boot, nor an over-all timeout on the combination of both phases.

Quite frankly though I am not entirely sure how even to implement such
an over-all timeout in a fully robust way, even though I am
sympathetic to adding that.

One option might be fork off a "timebomb" subprocess from logind or
PID1 (probably the latter) when we start shutting down. This timebomb
process would just wait for the timeout to elapse, and when it does
would issue reboot(), and little else. This sounds simple but is
quite complex IRL afaics, for at least two reasons:

1. If the timebomb process is supposed to survive from the first into
the second phase of the reboot, then it would need to exclude
itself from the process killing spree the second phase does
initially. There are concepts for that (specifically, the
argv[0][0] = '@' hack), but it needs special care to be taken.

2. it would have to release references to any kind of resource that
might interfere with the shutdown logic (i.e. to the umount spree
for example). One such resource are file systems appearing in its
mount table or being pinned by the mmaped executable its running
from. This is particularly nasty, as this means the binary
implementing the timebomb would have to copy itself to some tmpfs
or so, reexec itself there, then change its fs namespace and ensure
the only remaining mount in it is that tmpfs and everything else is
released. That's doable, but jeeez, far from trivial to
implement...

Not doing #2 would mean some fs might stay mounted until the last
phase of boot, i.e. would cause it to stay possibly dirty, and if you
care about reliable, clean boots, then that's the worst option.

So yeah, if this is what you are asking for, then please file an RFE
about it. But given the complexity this is not going to be an easy
addition, so don't hold your breath. Would love to review/merge patchs
adding this though…

In theory there's also the option of repurposing the hw watchdog for
this: i.e. instead of pinging it regularly, just set it to the maximum
shutdown timeout early, and then leave it unpinged for the rest of the
shutdown. However, given the limited configurable time range of most
hw watchdogs this doesn't appear realistic to me, and very hw
specific.

Lennart
--
Lennart Poettering, Red Hat
Loading...