[systemd-devel] journal on btrfs subvolume, failed unmounting /var/log/journal

[Chris Murphy]

It looks like var-log-journal.mount is called before systemd-journald is finished using it. Is the failure message merely cosmetic? The file system comes up clean on reboot so I'm pretty sure it's being cleanly unmounted despite this message.



fstab
UUID=------------------ /var/log/journal btrfs subvol=journald,compress=lzo,ssd 0 0



[ 72.218513] f20v.localdomain systemd[1]: Stopping Local File Systems.
[ 72.218793] f20v.localdomain systemd[1]: Stopped target Local File Systems.
[ 72.219677] f20v.localdomain systemd[1]: Unmounting /var/log/journal...
[ 72.219967] f20v.localdomain systemd[1]: Unmounting /home...
[ 72.221971] f20v.localdomain systemd[1]: Unmounting /boot/efi...
[ 72.222683] f20v.localdomain systemd[1]: Unmounting Temporary Directory...
[ 72.222954] f20v.localdomain systemd[1]: Stopping Configure read-only root support...
[ 72.223327] f20v.localdomain systemd[1]: Stopped Configure read-only root support.
[ 72.223595] f20v.localdomain systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[ 72.221876] f20v.localdomain kernel: audit: type=1131 audit(1392176350.145:395): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u
[ 72.228742] f20v.localdomain umount[1080]: umount: /var/log/journal: target is busy
[ 72.229119] f20v.localdomain umount[1080]: (In some cases useful info about processes that
[ 72.229361] f20v.localdomain umount[1080]: use the device is found by lsof(8) or fuser(1).)
[ 72.233293] f20v.localdomain lvm[1084]: WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!
[ 72.233662] f20v.localdomain systemd[1]: var-log-journal.mount mount process exited, code=exited status=32
[ 72.234214] f20v.localdomain systemd[1]: Failed unmounting /var/log/journal.
[ 72.241336] f20v.localdomain systemd[1]: Unmounted /home.
[ 72.242288] f20v.localdomain systemd[1]: Unmounted Temporary Directory.
[ 72.242581] f20v.localdomain systemd[1]: Unmounted /boot/efi.
[ 72.242827] f20v.localdomain systemd[1]: Unmounting /boot...
[ 72.246561] f20v.localdomain lvm[1084]: No volume groups found
[ 72.249517] f20v.localdomain systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[ 72.249888] f20v.localdomain systemd[1]: Stopping LVM2 metadata daemon...
[ 72.248968] f20v.localdomain kernel: audit: type=1131 audit(1392176350.172:396): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u
[ 72.251329] f20v.localdomain systemd[1]: Stopped LVM2 metadata daemon.
[ 72.250831] f20v.localdomain kernel: audit: type=1131 audit(1392176350.174:397): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u
[ 72.251886] f20v.localdomain systemd[1]: Unmounted /boot.
[ 72.252350] f20v.localdomain systemd[1]: Starting Unmount All Filesystems.
[ 72.252805] f20v.localdomain systemd[1]: Reached target Unmount All Filesystems.
[ 72.253142] f20v.localdomain systemd[1]: Stopping Local File Systems (Pre).
[ 72.253388] f20v.localdomain systemd[1]: Stopped target Local File Systems (Pre).
[ 72.253633] f20v.localdomain systemd[1]: Stopping Remount Root and Kernel File Systems...
[ 72.253880] f20v.localdomain systemd[1]: Stopped Remount Root and Kernel File Systems.
[ 72.254160] f20v.localdomain systemd[1]: Starting Shutdown.
[ 72.254895] f20v.localdomain systemd[1]: Reached target Shutdown.
[ 72.255266] f20v.localdomain systemd[1]: Starting Final Step.
[ 72.255716] f20v.localdomain systemd[1]: Reached target Final Step.
[ 72.255966] f20v.localdomain systemd[1]: Starting Reboot...
[ 72.252368] f20v.localdomain kernel: audit: type=1131 audit(1392176350.176:398): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u
[ 72.259828] f20v.localdomain systemd[1]: Shutting down.
[ 72.283437] f20v.localdomain systemd-journal[361]: Journal stopped

[Lennart Poettering]

Post by Chris Murphy
It looks like var-log-journal.mount is called before systemd-journald
is finished using it. Is the failure message merely cosmetic? The file
system comes up clean on reboot so I'm pretty sure it's being cleanly
unmounted despite this message.

Yeah, there's a bit of an ugliness here: journald stays running until
the final killing spree which keeps the fs busy. The main reason for
that is simply that we have no nice way to tell journald synchronously
that it should stop access to /var and continue logging into
/run. Normally we'd use dbus for requesting this, but since dbus-daemon
is a client of journald and we really can't have the cyclic dep betwen
journald and dbus-daemon we cannot make journald be on the bus. With the
advent of kdbus this will change however, as then we have dbus around
all the time, and the cyclic dep is gone.

The issue is mostly cosmetic as we still will cleanly unmount all
remaining file systems right before shutting down. We will also cleanly
terminate journald then.

Lennart

[Chris Murphy]

Post by Lennart Poettering

Post by Chris Murphy
It looks like var-log-journal.mount is called before systemd-journald
is finished using it. Is the failure message merely cosmetic? The file
system comes up clean on reboot so I'm pretty sure it's being cleanly
unmounted despite this message.

Yeah, there's a bit of an ugliness here: journald stays running until
the final killing spree which keeps the fs busy. The main reason for
that is simply that we have no nice way to tell journald synchronously
that it should stop access to /var and continue logging into
/run. Normally we'd use dbus for requesting this, but since dbus-daemon
is a client of journald and we really can't have the cyclic dep betwen
journald and dbus-daemon we cannot make journald be on the bus. With the
advent of kdbus this will change however, as then we have dbus around
all the time, and the cyclic dep is gone.
The issue is mostly cosmetic as we still will cleanly unmount all
remaining file systems right before shutting down. We will also cleanly
terminate journald then.

OK no problem.

I expect the same issue, but I'm thinking it's better to put /var/log on its own subvolume instead of /var/log/journal. Then it enabled immunity from rollbacks for other logs, in addition to the journal. This way all logs can be kept persistent regardless of what subvolume the user boots from. And this is also applicable for LVM Thin Provisioning layouts.

Chris Murphy