Discussion:
Hints for upgrading systemd on a running system
(too old to reply)
Reindl Harald
2018-02-20 19:06:05 UTC
Permalink
Raw Message
Am 20.02.2018 um 20:00 schrieb Paul Menzel:
> Dear systemd folks,
>
> We finally are going to upgrade from a very old systemd version 27 from
> 2011 to the current systemd v237. (Historical reasons.)

hopefully you have a working backup

> Anyway, I already was told about `systemctl daemon-reexec`, and we got
> it working.

the "reexec" is misleading because it's not possible to terminate PID1
and start it again on a running system with a new binary

> After that, looking at the output of `systemctl`, there are many units
> from the old version, which were removed in the meantime.

i doubt that anybody has done such a version jump at all and even if the
environment won't match anyways - you need to reboot and hope it works
as well as be prepared for the case it won't boot - it's that easy

make at least sure that initrd has been updated with the new systemd!

> $ systemctl --state=not-found
>   UNIT                                 LOAD      ACTIVE   SUB DESCRIPTION
> ● dev-hugepages.automount              not-found active   waiting
> dev-hugepages.automount
> ● dev-mqueue.automount                 not-found active   waiting
> dev-mqueue.automount
> ● sys-kernel-debug.automount           not-found active   waiting
> sys-kernel-debug.automount
> ● sys-kernel-security.automount        not-found active   waiting
> sys-kernel-security.automount
> ● auditd.service                       not-found inactive dead
> auditd.service
> ● console-kit-log-system-start.service not-found active   exited
> console-kit-log-system-start.service
> ● display-manager.service              not-found inactive dead
> display-manager.service
> ● hwclock-load.service                 not-found active   exited
> hwclock-load.service
> ● plymouth-quit-wait.service           not-found inactive dead
> plymouth-quit-wait.service
> ● plymouth-start.service               not-found inactive dead
> plymouth-start.service
> ● remount-rootfs.service               not-found active   exited
> remount-rootfs.service
> ● syslog.service                       not-found inactive dead
> syslog.service
> ● systemd-kmsg-syslogd.service         not-found active   running
> systemd-kmsg-syslogd.service
> ● systemd-remount-api-vfs.service      not-found active   exited
> systemd-remount-api-vfs.service
> ● systemd-sysusers.service             not-found inactive dead
> systemd-sysusers.service
> ● udev-retry.service                   not-found active   exited
> udev-retry.service
> ● udev-settle.service                  not-found active   exited
> udev-settle.service
> ● systemd-logger.socket                not-found active   listening
> systemd-logger.socket
> ● systemd-shutdownd.socket             not-found active   listening
> systemd-shutdownd.socket
> ● cryptsetup.target                    not-found active   active
> cryptsetup.target
> ● syslog.target                        not-found active   active
> syslog.target
>
> LOAD   = Reflects whether the unit definition was properly loaded.
> ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
> SUB    = The low-level unit activation state, values depend on unit type.
>
> 21 loaded units listed. Pass --all to see loaded but inactive units, too.
> To show all installed unit files use 'systemctl list-unit-files'.
> ```
>
> Do I need to stop those manually beforehand, or is there another way to
> clean up?
>
> Is the recommended update procedure documented somewhere?
Michael Biebl
2018-02-20 19:09:37 UTC
Permalink
Raw Message
2018-02-20 20:00 GMT+01:00 Paul Menzel <pmenzel+systemd-***@molgen.mpg.de>:
>
> Do I need to stop those manually beforehand, or is there another way to
> clean up?

reboot.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Mantas Mikulėnas
2018-02-20 21:04:33 UTC
Permalink
Raw Message
On Tue, Feb 20, 2018, 21:06 Reindl Harald <***@thelounge.net> wrote:

>
>
> Am 20.02.2018 um 20:00 schrieb Paul Menzel:
> > Dear systemd folks,
> >
> > We finally are going to upgrade from a very old systemd version 27 from
> > 2011 to the current systemd v237. (Historical reasons.)
>
> hopefully you have a working backup
>
> > Anyway, I already was told about `systemctl daemon-reexec`, and we got
> > it working.
>
> the "reexec" is misleading because it's not possible to terminate PID1
> and start it again on a running system with a new binary
>

But you don't *have to* terminate pid1 to change the binary – you can just
exec the new one. Hence the "reexec".

It's possible. That's how initramfs works.

> --

Mantas Mikulėnas <***@gmail.com>
Sent from my phone
Reindl Harald
2018-02-20 21:12:38 UTC
Permalink
Raw Message
Am 20.02.2018 um 22:04 schrieb Mantas Mikulėnas:
> On Tue, Feb 20, 2018, 21:06 Reindl Harald <***@thelounge.net
> <mailto:***@thelounge.net>> wrote:
>
>
> Am 20.02.2018 um 20:00 schrieb Paul Menzel:
> > Dear systemd folks,
> >
> > We finally are going to upgrade from a very old systemd version
> 27 from
> > 2011 to the current systemd v237. (Historical reasons.)
>
> hopefully you have a working backup
>
> > Anyway, I already was told about `systemctl daemon-reexec`, and
> we got
> > it working.
>
> the "reexec" is misleading because it's not possible to terminate PID1
> and start it again on a running system with a new binary
>
>
> But you don't *have to* terminate pid1 to change the binary – you can
> just exec the new one. Hence the "reexec".
>
> It's possible. That's how initramfs works

a lot of other threads here made it clear that there is not much
difference between "daemon-reload" and "daemon-reexec" at all in the
past and i strongly doubt that it's technically possible to seamless
switch from version 27 (the first Fedora GA AFAIK hat version 44) to 237
and keep the system happily running as like nothing has changed without
reboot
Lennart Poettering
2018-02-20 22:12:37 UTC
Permalink
Raw Message
On Di, 20.02.18 20:00, Paul Menzel (pmenzel+systemd-***@molgen.mpg.de) wrote:

> Dear systemd folks,
>
>
> We finally are going to upgrade from a very old systemd version 27 from 2011
> to the current systemd v237. (Historical reasons.)
>
> Anyway, I already was told about `systemctl daemon-reexec`, and we got it
> working.

While we try to ensure that live upgrades of PID 1 like that work
quite well, this is generally tested only for small steps. Jumping 6
years ahead in one go is not something people typically test.

> After that, looking at the output of `systemctl`, there are many units from
> the old version, which were removed in the meantime.
>
> ```
> $ systemctl --state=not-found
> UNIT LOAD ACTIVE SUB DESCRIPTION
> ● dev-hugepages.automount not-found active waiting
> dev-hugepages.automount
> ● dev-mqueue.automount not-found active waiting
> dev-mqueue.automount
> ● sys-kernel-debug.automount not-found active waiting
> sys-kernel-debug.automount
> ● sys-kernel-security.automount not-found active waiting
> sys-kernel-security.automount
> ● auditd.service not-found inactive dead
> auditd.service
> ● console-kit-log-system-start.service not-found active exited
> console-kit-log-system-start.service
> ● display-manager.service not-found inactive dead
> display-manager.service
> ● hwclock-load.service not-found active exited
> hwclock-load.service
> ● plymouth-quit-wait.service not-found inactive dead
> plymouth-quit-wait.service
> ● plymouth-start.service not-found inactive dead
> plymouth-start.service
> ● remount-rootfs.service not-found active exited
> remount-rootfs.service
> ● syslog.service not-found inactive dead
> syslog.service
> ● systemd-kmsg-syslogd.service not-found active running
> systemd-kmsg-syslogd.service
> ● systemd-remount-api-vfs.service not-found active exited
> systemd-remount-api-vfs.service
> ● systemd-sysusers.service not-found inactive dead
> systemd-sysusers.service
> ● udev-retry.service not-found active exited
> udev-retry.service
> ● udev-settle.service not-found active exited
> udev-settle.service
> ● systemd-logger.socket not-found active listening
> systemd-logger.socket
> ● systemd-shutdownd.socket not-found active listening
> systemd-shutdownd.socket
> ● cryptsetup.target not-found active active
> cryptsetup.target
> ● syslog.target not-found active active
> syslog.target

My recommendation: simply reboot. That should clean up everything
properly.

Note that PID 1 itself is probably pretty Ok with such a massive
update in one step, but the unit files have been rearranged quite a
bit since then. Downstream distributions generally expect you to
reboot even between single-step distro updates, but this becomes much
more of a necessity if you jump even further.

Note that systemd upstream currently requires kernel 3.13 at least
which was released in 2014. Hence, if you update from a 2011 system
you have to reboot anyway, already to update the kernel...

> Do I need to stop those manually beforehand, or is there another way to
> clean up?
>
> Is the recommended update procedure documented somewhere?

Usually distributions document that invididually as systemd is just
one component of many that make up the distribution.

Lennart

--
Lennart Poettering, Red Hat
Lennart Poettering
2018-02-22 09:55:25 UTC
Permalink
Raw Message
On Mi, 21.02.18 08:10, Paul Menzel (pmenzel+systemd-***@molgen.mpg.de) wrote:

> > Note that PID 1 itself is probably pretty Ok with such a massive
> > update in one step, but the unit files have been rearranged quite a
> > bit since then. Downstream distributions generally expect you to
> > reboot even between single-step distro updates, but this becomes much
> > more of a necessity if you jump even further.
>
> But if reboot wouldn’t be an option, is there a way to get rid of not-found
> services?

stop them.

How did you upgrade your kernel without rebooting?

Note that there's also stuff such as dbus-daemon which cannot be
updated without rebooting.

Lennart

--
Lennart Poettering, Red Hat
Lennart Poettering
2018-02-20 22:30:50 UTC
Permalink
Raw Message
On Di, 20.02.18 20:06, Reindl Harald (***@thelounge.net) wrote:

>
>
> Am 20.02.2018 um 20:00 schrieb Paul Menzel:
> > Dear systemd folks,
> >
> > We finally are going to upgrade from a very old systemd version 27 from
> > 2011 to the current systemd v237. (Historical reasons.)
>
> hopefully you have a working backup
>
> > Anyway, I already was told about `systemctl daemon-reexec`, and we got
> > it working.
>
> the "reexec" is misleading because it's not possible to terminate PID1 and
> start it again on a running system with a new binary

Hmm? "systemctl daemon-reexec" is supposed to support upgrades between
systemd versions. Doing massive jumps v27 → v237 in one go isn't
precisely well tested, but smaller ones should work quite well.

Lennart

--
Lennart Poettering, Red Hat
Loading...