Discussion:
magically disappearing filesystems
(too old to reply)
Andre Maasikas
2017-06-14 13:58:24 UTC
Permalink
Raw Message
Hi,

Having done this numerous times over the years I proceeded to move our data
to new storage array, first time on a new os version though.
Goes always like this,
* attach array, create LUNS, create multipath conf etc...
* umount old filesystem, mount new filesystems, mount old filesys to temp
location, copy data over, * update fstab, unmount temporary/old filesystem.
DONE
A day later ... Now to cleanly remove the old array/devices I did
# multipath -f olddev
# echo 1 > /sys/block/sdx/device/delete
# echo 1 > /sys/block/sdy/device/delete
etc..

After double checking I see that none of the new filesystems are mounted !
OOPS moment. I estimate I have about 10 minutes now to figure this out
before the transaction logs fill up and lots of revenue goes down. Probably
doesn't look good for me as I discovered the issue after i executed the
removal on most production systems and all clustered nodes

OK, let's mount manually,

mount /dev/mapper/newdev /mountpoint
no errors, seems ok
still df, mount and /proc/mounts show nothing...
WTF moment

mount -v tells me smth like :
/dev/newdev does not contain SELinux labels.
You just mounted an file system that supports labels which does not
contain labels, onto an SELinux box. It is likely that confined
applications will generate AVC messages and not be allowed access to
this file system. For more details see restorecon(8) and mount(8).

Searching google for 3 minutes if I may be confined to a space where I am
not allowed to see the filesystem anymore proved futile.

dmesg is filled with messages about how the filesystem is busy for umount
and for the mount attempt:
kernel: XFS (dm-22): Mounting V4 Filesystem
kernel: XFS (dm-22): Ending clean mount
systemd: Unit xx.mount is bound to inactive unit dev-mapper-xx.device.
Stopping, too.
systemd: Unmounting /mountpoint...
kernel: XFS (dm-22): Unmounting Filesystem
systemd: Unmounted /mountpoint.
systemd: Unit xxx.mount entered failed state.

(dev-mapper-xx being the old/removed device-mapper device)

Finally using the second set of keywords reveals that I'm supposed to
#systemctl daemon-reload
whenever i edit fstab.

Seems like the fstab file is not always authoritative anymore and the
authoritative configuration
is kept god-knows elsewhere and these might not be in sync and depend on
god-knows what and if you don't know that it's now allowed to automatically
unmount a perfectly good working filesystem from under you without any
warning. A quick review of fstab header, man fstab, man mount etc. does
not reveal any information about this newish behavior. Also no commands
that got me to this point gave any errors or indication that this will be
happening.

It might be something else I did incorrectly or distribution specific (RHEL
7.2) or a bug already fixed. Most certainly I have not learned enough of
the the new ways of systemd (and selinux)

Andre
Andrei Borzenkov
2017-06-14 17:23:47 UTC
Permalink
Raw Message
Post by Andre Maasikas
systemd: Unit xx.mount is bound to inactive unit dev-mapper-xx.device.
Stopping, too.
systemd: Unmounting /mountpoint...
kernel: XFS (dm-22): Unmounting Filesystem
systemd: Unmounted /mountpoint.
systemd: Unit xxx.mount entered failed state.
(dev-mapper-xx being the old/removed device-mapper device)
Welcome to the club.
Post by Andre Maasikas
Finally using the second set of keywords reveals that I'm supposed to
#systemctl daemon-reload
whenever i edit fstab.
Seems like the fstab file is not always authoritative anymore and the
authoritative configuration
No, it is not. Various configuration files are translated by systemd
generators into native units, afterwards these configuration files
become irrelevant. Which is pretty much clear, given that systemd was
created with a single task in mind - start services during system boot,
and so has no provision for run-time changes. It has workaround
(daemon-reload) but as you never know which generators exist (nor have
any way to discover it) you are bound to call this after editing near to
every file on your system ...
Post by Andre Maasikas
is kept god-knows elsewhere and these might not be in sync and depend on
god-knows what and if you don't know that it's now allowed to automatically
unmount a perfectly good working filesystem from under you without any
warning. A quick review of fstab header, man fstab, man mount etc. does
not reveal any information about this newish behavior. Also no commands
that got me to this point gave any errors or indication that this will be
happening.
It might be something else I did incorrectly or distribution specific (RHEL
7.2) or a bug already fixed. Most certainly I have not learned enough of
the the new ways of systemd (and selinux)
You did not do anything wrong and I do not see how it can really be
fixed. Note that even native units have exactly the same issue -
changing on-disk unit definition is not noticed by systemd until
daemon-reload (and then what is actually running may not match unit
definition anymore).
Michael Chapman
2017-06-14 20:40:31 UTC
Permalink
Raw Message
Post by Andrei Borzenkov
Post by Andre Maasikas
systemd: Unit xx.mount is bound to inactive unit dev-mapper-xx.device.
Stopping, too.
systemd: Unmounting /mountpoint...
kernel: XFS (dm-22): Unmounting Filesystem
systemd: Unmounted /mountpoint.
systemd: Unit xxx.mount entered failed state.
(dev-mapper-xx being the old/removed device-mapper device)
Welcome to the club.
Post by Andre Maasikas
Finally using the second set of keywords reveals that I'm supposed to
#systemctl daemon-reload
whenever i edit fstab.
Seems like the fstab file is not always authoritative anymore and the
authoritative configuration
No, it is not.
As far as I know, it *is* if the mount utility is used directly, since it
just invokes the mount(2) syscall. systemd will track the new point just
as if any other program had created one. It shouldn't matter if systemd's
generated unit files are out-of-date.

What seems to be the problem here is:

systemd: Unit xx.mount is bound to inactive unit dev-mapper-xx.device.
Stopping, too.

I don't know enough about multipath devices or their management to know
what's going on there though.
Post by Andrei Borzenkov
Various configuration files are translated by systemd
generators into native units, afterwards these configuration files
become irrelevant. Which is pretty much clear, given that systemd was
created with a single task in mind - start services during system boot,
and so has no provision for run-time changes. It has workaround
(daemon-reload) but as you never know which generators exist (nor have
any way to discover it) you are bound to call this after editing near to
every file on your system ...
Post by Andre Maasikas
is kept god-knows elsewhere and these might not be in sync and depend on
god-knows what and if you don't know that it's now allowed to automatically
unmount a perfectly good working filesystem from under you without any
warning. A quick review of fstab header, man fstab, man mount etc. does
not reveal any information about this newish behavior. Also no commands
that got me to this point gave any errors or indication that this will be
happening.
It might be something else I did incorrectly or distribution specific (RHEL
7.2) or a bug already fixed. Most certainly I have not learned enough of
the the new ways of systemd (and selinux)
You did not do anything wrong and I do not see how it can really be
fixed. Note that even native units have exactly the same issue -
changing on-disk unit definition is not noticed by systemd until
daemon-reload (and then what is actually running may not match unit
definition anymore).
I'm pretty it should only be necessary to call daemon-reload before using
systemctl. It warns if any source files (unit files, dropins and anything
mentioned by SourcePath= in a unit) have been updated since the last
reload. The fstab generator should be using SourcePath=.

- Michael
Jérémy Rosen
2017-06-15 07:39:40 UTC
Permalink
Raw Message
Post by Michael Chapman
I'm pretty it should only be necessary to call daemon-reload before using
systemctl. It warns if any source files (unit files, dropins and
anything mentioned by SourcePath= in a unit) have been updated since
the last reload. The fstab generator should be using SourcePath=.
Would it be theoretically possible to do a .path that looked at
/etc/fstab and triggered a daemon-reload, or would that be a bad idea
for reasons I don't imagine ?

Jeremy
Post by Michael Chapman
- Michael
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
Logo <http://www.smile.fr/>

20 rue des Jardins
92600 AsniÚres-sur-Seine
www.smile.fr <http://www.smile.fr/>
*Jérémy ROSEN*
Architecte technique
Email : ***@smile.fr <mailto:***@smile.fr>
Tel : +33141402967

Facebook <https://www.facebook.com/smileopensource> Google%2B
<http://fr.slideshare.net/SmileOpenSource/presentations> LinkedIn
<https://www.linkedin.com/company/smile> Twitter
<https://twitter.com/GroupeSmile>


bandeaux_mail
<http://www.smile.fr/Offres-services/Offres/Ingenierie?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÚte, n'imprimez ce mail que si c'est nécessaire
Michael Chapman
2017-06-15 08:05:50 UTC
Permalink
Raw Message
Post by Michael Chapman
I'm pretty it should only be necessary to call daemon-reload before using
systemctl. It warns if any source files (unit files, dropins and anything
mentioned by SourcePath= in a unit) have been updated since the last
reload. The fstab generator should be using SourcePath=.
Would it be theoretically possible to do a .path that looked at /etc/fstab
and triggered a daemon-reload, or would that be a bad idea for reasons I
don't imagine ?
I guess you could do that.

I'm happy enough to not have config loaded automatically, though -- it
just seems neater to have a whole bunch of changes queued up and to have
them all applied at once. I've never found manual reloads to be a big
problem.

On the other hand, I can't think of any big problems with having new
config applied automatically. It's not as if systemd will automatically
start any new units or stop any that disappear under it, at least at the
time of the reload itself. The one concern though is that it can affect
future jobs, as new dependencies between units may have been automatically
added. But maybe that's what the sysadmin expected.
Andre Maasikas
2017-06-15 08:30:27 UTC
Permalink
Raw Message
Post by Michael Chapman
I'm pretty it should only be necessary to call daemon-reload before using
systemctl. It warns if any source files (unit files, dropins and anything
mentioned by SourcePath= in a unit) have been updated since the last
reload. The fstab generator should be using SourcePath=.
Would it be theoretically possible to do a .path that looked at /etc/fstab
and triggered a daemon-reload, or would that be a bad idea for reasons I
don't imagine ?
Hi,
That wont help too much as I see it.
Currently if i manually unmount a filesystem some configuration remains and
later when i remove the md device it tries to
do automatic actions based on that invalid configuration.

To break this link either
1. removing the md device should not umount the filesystem or at least
check that it is still the right one
or more correct
2. umounting should break the link, update configuration so removing the
device doesn't do funny things

Andre
Andrei Borzenkov
2017-06-15 15:24:14 UTC
Permalink
Raw Message
Post by Andre Maasikas
OK, let's mount manually,
mount /dev/mapper/newdev /mountpoint
no errors, seems ok
still df, mount and /proc/mounts show nothing...
WTF moment
Well, opposite case (it is mounted when it should not) is equally true

https://github.com/systemd/systemd/issues/6066

Same reason, same cause. It is assumed that configuration cannot change
since boot (or to view it differently that configuration change requires
reboot).
Mantas Mikulėnas
2017-06-15 15:49:18 UTC
Permalink
Raw Message
This reminds me of a bug (misfeature) that *I think* was fixed in recent
releases...

Mount points are bound to the corresponding .device so that they'd get
cleaned up when the device disappears, but the way it was implemented meant
they'd also get immediately unmounted if the device wasn't there in the
first place.

E.g. in emergency mode when udev wasn't running so systemd thought all
devices were inactive. (State-based when it ought to have been event-based.)

I haven't tested specifically, but according to commit log, that shouldn't
be a problem in 23x versions.
Post by Andre Maasikas
Hi,
Having done this numerous times over the years I proceeded to move our
data to new storage array, first time on a new os version though.
Goes always like this,
* attach array, create LUNS, create multipath conf etc...
* umount old filesystem, mount new filesystems, mount old filesys to temp
location, copy data over, * update fstab, unmount temporary/old filesystem.
DONE
A day later ... Now to cleanly remove the old array/devices I did
# multipath -f olddev
# echo 1 > /sys/block/sdx/device/delete
# echo 1 > /sys/block/sdy/device/delete
etc..
After double checking I see that none of the new filesystems are mounted !
OOPS moment. I estimate I have about 10 minutes now to figure this out
before the transaction logs fill up and lots of revenue goes down. Probably
doesn't look good for me as I discovered the issue after i executed the
removal on most production systems and all clustered nodes
OK, let's mount manually,
mount /dev/mapper/newdev /mountpoint
no errors, seems ok
still df, mount and /proc/mounts show nothing...
WTF moment
/dev/newdev does not contain SELinux labels.
You just mounted an file system that supports labels which does not
contain labels, onto an SELinux box. It is likely that confined
applications will generate AVC messages and not be allowed access to
this file system. For more details see restorecon(8) and mount(8).
Searching google for 3 minutes if I may be confined to a space where I am
not allowed to see the filesystem anymore proved futile.
dmesg is filled with messages about how the filesystem is busy for umount
kernel: XFS (dm-22): Mounting V4 Filesystem
kernel: XFS (dm-22): Ending clean mount
systemd: Unit xx.mount is bound to inactive unit dev-mapper-xx.device.
Stopping, too.
systemd: Unmounting /mountpoint...
kernel: XFS (dm-22): Unmounting Filesystem
systemd: Unmounted /mountpoint.
systemd: Unit xxx.mount entered failed state.
(dev-mapper-xx being the old/removed device-mapper device)
Finally using the second set of keywords reveals that I'm supposed to
#systemctl daemon-reload
whenever i edit fstab.
Seems like the fstab file is not always authoritative anymore and the
authoritative configuration
is kept god-knows elsewhere and these might not be in sync and depend on
god-knows what and if you don't know that it's now allowed to automatically
unmount a perfectly good working filesystem from under you without any
warning. A quick review of fstab header, man fstab, man mount etc. does
not reveal any information about this newish behavior. Also no commands
that got me to this point gave any errors or indication that this will be
happening.
It might be something else I did incorrectly or distribution specific
(RHEL 7.2) or a bug already fixed. Most certainly I have not learned enough
of the the new ways of systemd (and selinux)
Andre
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
Mantas Mikulėnas <***@gmail.com>
Sent from my phone
Loading...