Discussion:
About stable network interface names
Add Reply
Cesare Leonardi
2017-05-29 00:35:12 UTC
Reply
Permalink
Raw Message
Hello, I've some dubts related to predictable network interface names.

If I understand correctly the reference document about this topic is:
https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

Take this paragraph from that document:
-----
We believe it is a good default choice to generalize the scheme
pioneered by "biosdevname". Assigning fixed names based on
firmware/topology/location information has the big advantage that the
names are fully automatic, fully predictable, that they stay fixed even
if hardware is added or removed (i.e. no reenumeration takes place) and
that broken hardware can be replaced seamlessly.
-----

And this one:
-----
Stable interface names even when hardware is added or removed, i.e. no
re-enumeration takes place (to the level the firmware permits this).
-----

They say that no re-enumeration take place if hardware is added or
removed, with the last quote that puts a depends on firmware.

I ask because I've done several tests, with different motherboards,
adding and removing PCI-express cards and that expectation was not
satisfied in many cases.

For example, in one of those tests I initially had this setup:
Integrated NIC: enp9s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp7s0, enp8s0]

Then i inserted a SATA controller in the PCIE2 slot and three NICs got
renamed:
Integrated NIC: enp10s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp8s0, enp9s0]

Why?
Didn't this interface naming scheme supposed to avoid this kind of renaming?
From what i've experimented network names are guaranteed to be stable
across reboots *and* if you doesn't add or remove hardware.

Please, can you clarify on this?

Cesare.
Greg KH
2017-05-29 05:10:17 UTC
Reply
Permalink
Raw Message
I ask because I've done several tests, with different motherboards, adding
and removing PCI-express cards and that expectation was not satisfied in
many cases.
Integrated NIC: enp9s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp7s0, enp8s0]
Then i inserted a SATA controller in the PCIE2 slot and three NICs got
Integrated NIC: enp10s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp8s0, enp9s0]
Do you mean to show that PCIE2 is still empty here?

Anyway, PCI can, and will sometimes, renumber it's devices on booting
again, that's a known issue. It is rare, but as you have found out,
will happen. So anything depending on PCI numbers will change. Nothing
we can really do about that.

thanks,

greg k-h
Cesare Leonardi
2017-05-29 09:34:13 UTC
Reply
Permalink
Raw Message
Post by Greg KH
Post by Cesare Leonardi
Integrated NIC: enp9s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp7s0, enp8s0]
Then i inserted a SATA controller in the PCIE2 slot and three NICs got
Integrated NIC: enp10s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp8s0, enp9s0]
Do you mean to show that PCIE2 is still empty here?
No, sorry, cut and paste error. In the last case PCIE2 was occupied by
the SATA controller.
Post by Greg KH
Anyway, PCI can, and will sometimes, renumber it's devices on booting
again, that's a known issue. It is rare, but as you have found out,
will happen. So anything depending on PCI numbers will change. Nothing
we can really do about that.
Do you mean that it could rarely happen on boot also without doing any
change to the hardware?

So, to avoid surprises, in case of multiple NICs it's highly
recommendable anyway to hook interface naming to MAC address, isn't it?

Thank you for the clarifications, Greg.

Cesare.
Lennart Poettering
2017-05-29 09:40:31 UTC
Reply
Permalink
Raw Message
Post by Greg KH
Post by Cesare Leonardi
Integrated NIC: enp9s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp7s0, enp8s0]
Then i inserted a SATA controller in the PCIE2 slot and three NICs got
Integrated NIC: enp10s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp8s0, enp9s0]
Do you mean to show that PCIE2 is still empty here?
No, sorry, cut and paste error. In the last case PCIE2 was occupied by the
SATA controller.
Post by Greg KH
Anyway, PCI can, and will sometimes, renumber it's devices on booting
again, that's a known issue. It is rare, but as you have found out,
will happen. So anything depending on PCI numbers will change. Nothing
we can really do about that.
Do you mean that it could rarely happen on boot also without doing any
change to the hardware?
Well, the firmware can do whatever it wants at any time, and this is
really up to the firmware. Ideally firmware would keep things strictly
stable, to make this useful, but you know how firmware is...
So, to avoid surprises, in case of multiple NICs it's highly recommendable
anyway to hook interface naming to MAC address, isn't it?
Well, different naming strategies have different advantages and
disadvantages. If you use the MAC address, then replacing hardware
becomes harder, and you can't cover hardware that doesn't have fixed
MAC addresses (or VMs).

Lennart
--
Lennart Poettering, Red Hat
Reindl Harald
2017-05-29 09:49:41 UTC
Reply
Permalink
Raw Message
Post by Lennart Poettering
Well, different naming strategies have different advantages and
disadvantages. If you use the MAC address, then replacing hardware
becomes harder, and you can't cover hardware that doesn't have fixed
MAC addresses (or VMs)
than KVM or whatever you mean with VMs needs to be fixed

on our VMware cluster all nods are Fedora installed 2008 and all are
configured with MAC addresses in the ifcfg-ethX files, where moved
between different vCenter versions and hosts and the MAC is stable
Greg KH
2017-05-29 09:59:51 UTC
Reply
Permalink
Raw Message
Post by Cesare Leonardi
Post by Greg KH
Anyway, PCI can, and will sometimes, renumber it's devices on booting
again, that's a known issue. It is rare, but as you have found out,
will happen. So anything depending on PCI numbers will change. Nothing
we can really do about that.
Do you mean that it could rarely happen on boot also without doing any
change to the hardware?
Yes it can, I used to have a machine that renumbered the PCI buses every
5th boot or something, was fun for testing PCI changes on :)
Post by Cesare Leonardi
So, to avoid surprises, in case of multiple NICs it's highly recommendable
anyway to hook interface naming to MAC address, isn't it?
Yes, or something else that you know will be stable in the system.

good luck!

greg k-h
Martin Wilck
2017-06-06 13:20:28 UTC
Reply
Permalink
Raw Message
Post by Cesare Leonardi
I ask because I've done several tests, with different motherboards,
adding and removing PCI-express cards and that expectation was not
satisfied in many cases.
Integrated NIC: enp9s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp7s0, enp8s0]
Then i inserted a SATA controller in the PCIE2 slot and three NICs got
Integrated NIC: enp10s0
PCIE1 (x1): dual port ethernet card [enp3s0, enp4s0]
PCIE2 (x16): empty
PCIE3 (x1): dual port ethernet card [enp8s0, enp9s0]
Why?
Didn't this interface naming scheme supposed to avoid this kind of renaming?
From what i've experimented network names are guaranteed to be stable
across reboots *and* if you doesn't add or remove hardware.
As others have remarked already, PCI bus-device-function is subject to
change.

Yet this highlights a problem with the current "predictable" network
device naming scheme. "biosdevname" uses information from various
sources, including ACPI _DSM method and SMBIOS type 41 and type 9. The
systemd device naming scheme (or the kernel code in pci-label.c, for
that matter) evaluates only _DSM and SMBIOS type 41, and not SMBIOS
type 9. The latter is necessary for mapping system PCI slot numbers to
bus-device-function tuples. Obviously, the physical slot a controller
is connected to is less likely to change than the bus-device-function
number, so exposing it might make a lot of sense.

All of this requires support on the BIOS/Firmware side - without that,
none of the "predictable" schemes work correctly.

Regards,
Martin
--
Dr. Martin Wilck <***@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Andrei Borzenkov
2017-06-06 18:40:27 UTC
Reply
Permalink
Raw Message
Post by Martin Wilck
As others have remarked already, PCI bus-device-function is subject to
change.
Can device and function really change? My understanding is that device
part is determined by bus physical wiring and function by PCI card
design; this leaves bus as volatile run-time enumeration value.
Martin Wilck
2017-06-09 20:42:37 UTC
Reply
Permalink
Raw Message
Post by Andrei Borzenkov
Can device and function really change? My understanding is that device
part is determined by bus physical wiring and function by PCI card
design; this leaves bus as volatile run-time enumeration value.
For PCIe, that's only true for the "function" part.
https://superuser.com/questions/1060808/how-is-the-device-determined-in
-pci-enumeration-bus-device-function

The systemd docs are a bit misleading for PCIe, as they talk about
"physical/geographical location" for the common enp$Xs$Yf$Z scheme,
which is actually just the BDF. The interface on my laptop is called
enp0s31f6 although the laptop doesn't have "slot 31". (1)

Martin

(1) Well, that's actually because the manufacturer saved the money to
implement the DMI BIOS correctly: there is even a DMI type 41 entry for
onboard LAN, but the PCI device number (sic!) is wrong.
--
Dr. Martin Wilck <***@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Andrei Borzenkov
2017-06-10 06:08:17 UTC
Reply
Permalink
Raw Message
Post by Martin Wilck
Post by Andrei Borzenkov
Can device and function really change? My understanding is that device
part is determined by bus physical wiring and function by PCI card
design; this leaves bus as volatile run-time enumeration value.
For PCIe, that's only true for the "function" part.
https://superuser.com/questions/1060808/how-is-the-device-determined-in
-pci-enumeration-bus-device-function
I do not see anything there that would imply device designation is
random. You have PCIe switch port instead of IDSEL wiring but port is
most likely hardwired and does not change. Actually this article says
the same

--><--
Since each device has its own independent set of wires, the device IDs
are essentially all hard-coded
--><--
Post by Martin Wilck
The systemd docs are a bit misleading for PCIe, as they talk about
"physical/geographical location" for the common enp$Xs$Yf$Z scheme,
which is actually just the BDF. The interface on my laptop is called
enp0s31f6 although the laptop doesn't have "slot 31". (1)
The reason for this scheme is to generate unique names during boot. All
this buzz about "predictability" etc is just usual marketing stuff to
better sell it.
Greg KH
2017-06-10 06:28:29 UTC
Reply
Permalink
Raw Message
Post by Andrei Borzenkov
Post by Martin Wilck
Post by Andrei Borzenkov
Can device and function really change? My understanding is that device
part is determined by bus physical wiring and function by PCI card
design; this leaves bus as volatile run-time enumeration value.
For PCIe, that's only true for the "function" part.
https://superuser.com/questions/1060808/how-is-the-device-determined-in
-pci-enumeration-bus-device-function
I do not see anything there that would imply device designation is
random. You have PCIe switch port instead of IDSEL wiring but port is
most likely hardwired and does not change. Actually this article says
the same
--><--
Since each device has its own independent set of wires, the device IDs
are essentially all hard-coded
--><--
"essentially" does not mean "will never change". Some BIOSes will
renumber them, some will not, as it's not a PCI requirement it can, and
will, happen.

Yes, for lots of people this will be fine and nothing will ever change,
but you can not guarantee it will never happen for all types of
machines, sorry.

That's just the nature of dynamic busses :)

good luck!

greg k-h

Loading...