Discussion:
How do I remove dead process managed by Systemd?
(too old to reply)
Yun-Chih Chen
2017-12-04 15:15:33 UTC
Permalink
Raw Message
Hi, fellows:

How do I remove a dead process (D-state process) wrapped as a Systemd
service unit? The process's parent is systemd (pid=1) and I still did not
find a way to remove such process without rebooting. I have tried the
following:

$ systemctl restart <my_service> # timeout
$ systemctl daemon-reexec
$ kill -9 <pid of my process> # won't work of course

Or anyone knows of hacks that do the trick?

Thanks in advance!
Lennart Poettering
2017-12-04 18:14:51 UTC
Permalink
Raw Message
On Mo, 04.12.17 23:15, Yun-Chih Chen (***@gmail.com) wrote:

> Hi, fellows:
>
> How do I remove a dead process (D-state process) wrapped as a Systemd
> service unit? The process's parent is systemd (pid=1) and I still did not
> find a way to remove such process without rebooting. I have tried the
> following:
>
> $ systemctl restart <my_service> # timeout
> $ systemctl daemon-reexec
> $ kill -9 <pid of my process> # won't work of course
>
> Or anyone knows of hacks that do the trick?

You cannot kill "D" state processes on Linux. That's what "D" state
means ultimately: *uninterruptible* sleep. The kill command can't kill
such processes, and neither can systemd. If you see processes like
this, then either you have very slow I/O, or this points to some form
of kernel/driver problem, as sleeping in this state should usually be
for short periods only.

Lennart

--
Lennart Poettering, Red Hat
v***@pengaru.com
2017-12-04 18:41:53 UTC
Permalink
Raw Message
On Mon, Dec 04, 2017 at 07:14:51PM +0100, Lennart Poettering wrote:
> On Mo, 04.12.17 23:15, Yun-Chih Chen (***@gmail.com) wrote:
>
> > Hi, fellows:
> >
> > How do I remove a dead process (D-state process) wrapped as a Systemd
> > service unit? The process's parent is systemd (pid=1) and I still did not
> > find a way to remove such process without rebooting. I have tried the
> > following:
> >
> > $ systemctl restart <my_service> # timeout
> > $ systemctl daemon-reexec
> > $ kill -9 <pid of my process> # won't work of course
> >
> > Or anyone knows of hacks that do the trick?
>
> You cannot kill "D" state processes on Linux. That's what "D" state
> means ultimately: *uninterruptible* sleep. The kill command can't kill
> such processes, and neither can systemd. If you see processes like
> this, then either you have very slow I/O, or this points to some form
> of kernel/driver problem, as sleeping in this state should usually be
> for short periods only.
>

That's not strictly accurate AFAIK.

Not all D state processes are unkillable. D state contributes to load
average, so it's not just to represent unkillable tasks. IIRC it's
ambiguous from userspace by just glancing at D state if the process is
killable, but it can happen that the process is plain uninterruptible.

I don't have time at the moment to do much digging to confirm. If you
glance at wait.h from the kernel headers, see wait_event_killable().
TASK_KILLABLE includes TASK_UNINTERRUPTIBLE, and I believe that shows as a
D state in userspace, but it also includes TASK_WAKEKILL.

What Lennart describes still holds true for Yun-Chih; if the process is in
an unkillable state then there's nothing systemd can do.

Regards,
Vito Caputo
Loading...