Discussion:
[PATCH] rules: block - add dm devices to whitelist
Add Reply
David Disseldorp
2017-07-05 11:01:45 UTC
Reply
Permalink
Raw Message
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
---
rules/60-persistent-storage.rules | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rules/60-persistent-storage.rules b/rules/60-persistent-storage.rules
index 5ab03fc27..bc0721f32 100644
--- a/rules/60-persistent-storage.rules
+++ b/rules/60-persistent-storage.rules
@@ -6,7 +6,7 @@
ACTION=="remove", GOTO="persistent_storage_end"

SUBSYSTEM!="block", GOTO="persistent_storage_end"
-KERNEL!="loop*|mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|nvme*|sd*|sr*|vd*|xvd*|bcache*|cciss*|dasd*", GOTO="persistent_storage_end"
+KERNEL!="loop*|mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|nvme*|sd*|sr*|vd*|xvd*|bcache*|cciss*|dasd*|dm*", GOTO="persistent_storage_end"

# ignore partitions that span the entire disk
TEST=="whole_disk", GOTO="persistent_storage_end"
--
2.12.3
Lennart Poettering
2017-07-10 08:38:38 UTC
Reply
Permalink
Raw Message
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.

It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.

I hope that makes sense,

Lennart
--
Lennart Poettering, Red Hat
Tomasz Torcz
2017-07-10 08:56:16 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.
It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.
Ceph does not create any devices (except /dev/rbd/* but it's not the case here).
Ceph block storage uses any kind of block device for its operation, be it plain
partition, LUKS encrypted storage, LVM, dm-multipath devices and so on.

Ceph block storage contains few parts - actual storage, write-ahead log,
journal etc. Any of those parts can utilize block devices, so ceph
uses symlinks to point to proper block device:

# ls -l /var/lib/ceph/osd/ceph-11/block
lrwxrwxrwx. 1 root root 33 Jul 10 10:51 /var/lib/ceph/osd/ceph-11/block -> /dev/disk/by-partuuid/43bdcb85-06


Nb. ceph already ships rules for autodetecting if given partition belongs
to ceph (discriminating by partition type, see
https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules ), but 'by-partuuid'
links should be created by earlier udev rules - like they are for some whitelisted
set of device nodes names.
--
Tomasz Torcz "Never underestimate the bandwidth of a station
xmpp: ***@chrome.pl wagon filled with backup tapes." -- Jim Gray
David Disseldorp
2017-07-10 09:37:03 UTC
Reply
Permalink
Raw Message
Thanks for the feedback, Lennart...
Post by Lennart Poettering
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.
It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.
We can certainly do this in a Ceph specific manner via the existing
95-ceph-osd.rules, but my impression was that the by-partuuid symlinks
are "owned" by 60-persistent-storage.rules .

If you don't think the existence of these symlinks for dm paths will be
of use to others then I'll go ahead and propose it as a Ceph only
change.

Cheers, David
Lennart Poettering
2017-07-10 09:53:03 UTC
Reply
Permalink
Raw Message
Post by David Disseldorp
Thanks for the feedback, Lennart...
Post by Lennart Poettering
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.
It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.
We can certainly do this in a Ceph specific manner via the existing
95-ceph-osd.rules, but my impression was that the by-partuuid symlinks
are "owned" by 60-persistent-storage.rules .
If you don't think the existence of these symlinks for dm paths will be
of use to others then I'll go ahead and propose it as a Ceph only
change.
Hmmm, so thinking about this again: "partuuid" is actually for GPT
partition UUIDs if I recall recorrectly, they wouldn't be generated
for DM devices anyway, since they are one layer removed from the GPT,
so are you even sure this will do what you are asking for?

But even if this actually works: DM links so far are created by the
LVM/libdevicemapper/ packages, not by udev, and I don't think this
should change. Hence, please talk to the LVM/libdevicemapper folks
about this and ask them for including it.

Lennart
--
Lennart Poettering, Red Hat
Peter Rajnoha
2017-07-10 10:14:41 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Post by David Disseldorp
Thanks for the feedback, Lennart...
Post by Lennart Poettering
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.
It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.
We can certainly do this in a Ceph specific manner via the existing
95-ceph-osd.rules, but my impression was that the by-partuuid symlinks
are "owned" by 60-persistent-storage.rules .
If you don't think the existence of these symlinks for dm paths will be
of use to others then I'll go ahead and propose it as a Ceph only
change.
Hmmm, so thinking about this again: "partuuid" is actually for GPT
partition UUIDs if I recall recorrectly, they wouldn't be generated
for DM devices anyway, since they are one layer removed from the GPT,
so are you even sure this will do what you are asking for?
But even if this actually works: DM links so far are created by the
LVM/libdevicemapper/ packages, not by udev, and I don't think this
should change. Hence, please talk to the LVM/libdevicemapper folks
about this and ask them for including it.
Yes, please, any rules for symlinks which should be created under
/dev/disk for DM devices (including all its subsystems like LVM,
mpath...) should go into 13-dm-disk.rules that is part of LVM/DM source
tree:

https://sourceware.org/git/?p=lvm2.git;a=blob;f=udev/13-dm-disk.rules.in

Now, when we create a partition over a DM device, there's a new mapping
created on top for each partition (either by calling kpartx manually or
by having it created by partitioning tool directly if it supports that).
So in this case, it's not the kernel directly who creates the
partitions, but they're simply another DM devices created on top of the
underlying DM device to represent these partitions. But I think that
doesn't matter - we should still create those symlinks for people to
still have a possibility to reference the device by its part uuid - I'll
fix 13-dm-disk.rules to include this.
--
Peter
Peter Rajnoha
2017-07-10 10:47:24 UTC
Reply
Permalink
Raw Message
Post by Peter Rajnoha
Post by Lennart Poettering
Post by David Disseldorp
Thanks for the feedback, Lennart...
Post by Lennart Poettering
Post by David Disseldorp
Ceph relies on by-partuuid symlinks, in order to locate the journal
partition from a given OSD partition. For details, see
http://tracker.ceph.com/issues/19489.
This appears way too broad, as it would apply to all LVM and all other
devices.
It appears to me Ceph should do the same as LVM does for this, and
ship its own set of rules, and be careful to only match against the
actual devices it creates.
We can certainly do this in a Ceph specific manner via the existing
95-ceph-osd.rules, but my impression was that the by-partuuid symlinks
are "owned" by 60-persistent-storage.rules .
If you don't think the existence of these symlinks for dm paths will be
of use to others then I'll go ahead and propose it as a Ceph only
change.
Hmmm, so thinking about this again: "partuuid" is actually for GPT
partition UUIDs if I recall recorrectly, they wouldn't be generated
for DM devices anyway, since they are one layer removed from the GPT,
so are you even sure this will do what you are asking for?
But even if this actually works: DM links so far are created by the
LVM/libdevicemapper/ packages, not by udev, and I don't think this
should change. Hence, please talk to the LVM/libdevicemapper folks
about this and ask them for including it.
Yes, please, any rules for symlinks which should be created under
/dev/disk for DM devices (including all its subsystems like LVM,
mpath...) should go into 13-dm-disk.rules that is part of LVM/DM source
https://sourceware.org/git/?p=lvm2.git;a=blob;f=udev/13-dm-disk.rules.in
Now, when we create a partition over a DM device, there's a new mapping
created on top for each partition (either by calling kpartx manually or
by having it created by partitioning tool directly if it supports that).
So in this case, it's not the kernel directly who creates the
partitions, but they're simply another DM devices created on top of the
underlying DM device to represent these partitions. But I think that
doesn't matter - we should still create those symlinks for people to
still have a possibility to reference the device by its part uuid - I'll
fix 13-dm-disk.rules to include this.
Fixed here:

https://sourceware.org/git/?p=lvm2.git;a=commit;h=c48149cf80c6582c2369bc7f8a33d794021d9dae
--
Peter
David Disseldorp
2017-07-10 11:18:22 UTC
Reply
Permalink
Raw Message
...
Post by Peter Rajnoha
Post by Peter Rajnoha
Yes, please, any rules for symlinks which should be created under
/dev/disk for DM devices (including all its subsystems like LVM,
mpath...) should go into 13-dm-disk.rules that is part of LVM/DM source
https://sourceware.org/git/?p=lvm2.git;a=blob;f=udev/13-dm-disk.rules.in
Now, when we create a partition over a DM device, there's a new mapping
created on top for each partition (either by calling kpartx manually or
by having it created by partitioning tool directly if it supports that).
So in this case, it's not the kernel directly who creates the
partitions, but they're simply another DM devices created on top of the
underlying DM device to represent these partitions. But I think that
doesn't matter - we should still create those symlinks for people to
still have a possibility to reference the device by its part uuid - I'll
fix 13-dm-disk.rules to include this.
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c48149cf80c6582c2369bc7f8a33d794021d9dae
Looks good and works for me - thanks Peter.

Cheers, David

Loading...