[systemd-devel] Storing package metadata in ELF objects

Post by Luca Boccassi
Hello,
Cross-posting to the mailing lists of a few relevant projects.
After an initial discussion [0], recently we have been working on a new
specification [0] to encode rich package-level metadata inside ELF
objects, so that it can be included automatically in generated coredump
files. The prototype to parse this in systemd-coredump and store the
information in systemd-journal is ready for testing and merged
upstream. We are now seeking further comments/opinions/suggestions, as
we have a few months before the next release and thus there's plenty of
time to make incompatible changes to the format and implementation, if
required.
A proposal to use this by default for all packages built in Fedora 35
has been submitted [1].
The Fedora Wiki and the systemd.io document have more details, but to
make a long story short, a new .notes.package section with a JSON
payload will be included in ELF objects, encoding various package-
build-time information like distro name&version, package name&version,
etc.
To summarize from the discussion, the main reasons why we believe this
1) minimal containers: the rpm database is not installed in the
containers. The information about build-ids needs to be stored
externally, so package name information is not available immediately,
but only after offline processing. The new note doesn't depend on the
rpm db in any way.
2) handling of a core from a container, where the container and host
have different distros
3) self-built and external packages: unless a lot of care is taken to
keep access to the debuginfo packages, this information may be lost.
The new note is available even if the repository metadata gets lost.
Users can easily provide equivalent information in a format that makes
sense in their own environment. It should work even when rpms and debs
and other formats are mixed, e.g. during container image creation.
Other than in Fedora, we are already making the required code changes
at Microsoft to use the same format&specification for internally-built
binaries, and for tools that parse core files and logs.
Tools for RPM and DEB (debhelper) integration are also available [3].

Wrong Fedora list address - off to a great start already :-)
(fixed now)

--
Kind regards,
Luca Boccassi

Zbigniew Jędrzejewski-Szmek

2021-04-10 18:44:07 UTC

[I'm forwarding the mail from Luca who is not subscribed to fedora-devel]

On Sat, Apr 10, 2021 at 01:38:31PM +0100, Luca Boccassi wrote:

Hello,

Cross-posting to the mailing lists of a few relevant projects.

After an initial discussion [0], recently we have been working on a new
specification [0] to encode rich package-level metadata inside ELF
objects, so that it can be included automatically in generated coredump
files. The prototype to parse this in systemd-coredump and store the
information in systemd-journal is ready for testing and merged
upstream. We are now seeking further comments/opinions/suggestions, as
we have a few months before the next release and thus there's plenty of
time to make incompatible changes to the format and implementation, if
required.

A proposal to use this by default for all packages built in Fedora 35
has been submitted [1].

The Fedora Wiki and the systemd.io document have more details, but to
make a long story short, a new .notes.package section with a JSON
payload will be included in ELF objects, encoding various package-
build-time information like distro name&version, package name&version,
etc.

To summarize from the discussion, the main reasons why we believe this
is useful are as following:

1) minimal containers: the rpm database is not installed in the
containers. The information about build-ids needs to be stored
externally, so package name information is not available immediately,
but only after offline processing. The new note doesn't depend on the
rpm db in any way.

2) handling of a core from a container, where the container and host
have different distros

3) self-built and external packages: unless a lot of care is taken to
keep access to the debuginfo packages, this information may be lost.
The new note is available even if the repository metadata gets lost.
Users can easily provide equivalent information in a format that makes
sense in their own environment. It should work even when rpms and debs
and other formats are mixed, e.g. during container image creation.

Other than in Fedora, we are already making the required code changes
at Microsoft to use the same format&specification for internally-built
binaries, and for tools that parse core files and logs.

Tools for RPM and DEB (debhelper) integration are also available [3].

Post by Luca Boccassi
--
Kind regards,
Luca Boccassi

Luca Boccassi

2021-05-04 13:43:32 UTC

Hi,

Post by Zbigniew JÄdrzejewski-Szmek
[I'm forwarding the mail from Luca who is not subscribed to fedora-
devel]
Cross-posting to the mailing lists of a few relevant projects.

Note that in this version of the email the [N] references in your email
don't seem to point anywhere. I found an older variant of the same
[0] https://github.com/systemd/systemd/issues/18433
[1] https://systemd.io/COREDUMP_PACKAGE_METADATA/
[2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects
[3] https://github.com/systemd/package-notes

Sorry about that! Must have messed up the copy&paste.

Post by Zbigniew JÄdrzejewski-Szmek
After an initial discussion [0], recently we have been working on a new
specification [0] to encode rich package-level metadata inside ELF
objects, so that it can be included automatically in generated coredump
files. The prototype to parse this in systemd-coredump and store the
information in systemd-journal is ready for testing and merged
upstream. We are now seeking further comments/opinions/suggestions, as
we have a few months before the next release and thus there's plenty of
time to make incompatible changes to the format and implementation, if
required.
A proposal to use this by default for all packages built in Fedora 35
has been submitted [1].
The Fedora Wiki and the systemd.io document have more details, but to
make a long story short, a new .notes.package section with a JSON
payload will be included in ELF objects, encoding various package-
build-time information like distro name&version, package
name&version,
etc.

Is there a list of default keys (and their canonical spelling, upper-
lower-Camel_Case, etc.)? If there is, could we have a "debuginfod" key
with as value an URL pointing to the debuginfod server URL where the
embedded build-id executable, debuginfo and sources can be found?
https://sourceware.org/elfutils/Debuginfod.html

The "Implementation" section of the spec lists the "main" fields:

https://systemd.io/COREDUMP_PACKAGE_METADATA/

(source for that is https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md )

Would you like to send a PR to update it and add that field?

Post by Zbigniew JÄdrzejewski-Szmek
To summarize from the discussion, the main reasons why we believe this
1) minimal containers: the rpm database is not installed in the
containers. The information about build-ids needs to be stored
externally, so package name information is not available immediately,
but only after offline processing. The new note doesn't depend on the
rpm db in any way.
2) handling of a core from a container, where the container and host
have different distros
3) self-built and external packages: unless a lot of care is taken to
keep access to the debuginfo packages, this information may be lost.
The new note is available even if the repository metadata gets lost.
Users can easily provide equivalent information in a format that makes
sense in their own environment. It should work even when rpms and debs
and other formats are mixed, e.g. during container image creation.
Other than in Fedora, we are already making the required code changes
at Microsoft to use the same format&specification for internally-built
binaries, and for tools that parse core files and logs.
Tools for RPM and DEB (debhelper) integration are also available [3].

Post by Luca Boccassi
--
Kind regards,
Luca Boccassi

--
Kind regards,
Luca Boccassi

Luca Boccassi

2021-05-06 11:43:13 UTC

Hi Luca,

https://systemd.io/COREDUMP_PACKAGE_METADATA/
(source for that is https://github.com/systemd/systemd/blob/main/docs/COREDUMP_PACKAGE_METADATA.md )
Would you like to send a PR to update it and add that field?

Sorry, I don't have a github account. But attached is a patch for to
document it and one for the package-notes generator to add an --
debuginfod argument (maybe the distro should set a default value for
that?) Hopefully those patches could be applied somehow.

Hi,

Thanks, opened PRs with your commits:

https://github.com/systemd/systemd/pull/19523
https://github.com/systemd/package-notes/pull/8

Yes, if the distro has a debuginfod server, it should definitely be
included.

--
Kind regards,
Luca Boccassi

Guillem Jover

2021-05-14 10:41:07 UTC

This post might be inappropriate. Click to display it.

Luca Boccassi

2021-05-14 13:41:32 UTC

Post by Luca Boccassi
After an initial discussion [0], recently we have been working on a new
specification [0] to encode rich package-level metadata inside ELF
objects, so that it can be included automatically in generated coredump
files. The prototype to parse this in systemd-coredump and store the
information in systemd-journal is ready for testing and merged
upstream. We are now seeking further comments/opinions/suggestions, as
we have a few months before the next release and thus there's plenty of
time to make incompatible changes to the format and implementation, if
required.

I've skimmed over the discussion at [0], and while having this data
seems like it might be "nice", I've to agree with the comments there
voicing that there does not really seem to be an actual need and the
overhead and additional work do not seem worth it, TBH, at least
in the Debian context.

Hi Guillem, thanks for having a look, much appreciated!

Just to clarify, the need is there - this is not an experimental
exercise, but it is borne out of an actual need&requirement, and it is
undergoing testing right now before deployment in a large scale
production infrastructure.
Not _everybody_ will need it, and not everywhere - that's absolutely
fair, and discussions on whether the ovearhead is worth it for
something that is not universally needed, but only in certain use
cases, is perfectly reasonable and welcome. I know Zbigniew is going to
try and get some raw numbers on the kind of overhead we are talking
about, that will hopefully help frame the discussion with more
precision.

Post by Luca Boccassi
The Fedora Wiki and the systemd.io document have more details, but to
make a long story short, a new .notes.package section with a JSON
payload will be included in ELF objects, encoding various package-
build-time information like distro name&version, package name&version,
etc.
To summarize from the discussion, the main reasons why we believe this
1) minimal containers: the rpm database is not installed in the
containers. The information about build-ids needs to be stored
externally, so package name information is not available immediately,
but only after offline processing. The new note doesn't depend on the
rpm db in any way.

In the Debian context, the build-ids data is going to be available
in the affected executables, and in debug symbols packages and the
Packages metaindices listing them, so there's no need for access to
any local dpkg database. Given that someone needing to install debug
packages will need access to those indices (either with outgoing network
access or with a repository mirror), these can be queried at that time.
Not to mention that AFAIR the Debian debug symbol servers make it
possible to query for specific build-ids.

This is not strictly related to debug packages, though? In fact, on
systems where this could be of most use you explicitly do _not_ install
debug packages (or anything at all). Or even if you wanted to, you
could not - corefiles are not handled inside the container, but
outside. Even if you wanted to and were allowed to (which for many
environments it's not the case), you can't install a Debian debug
package on a CoreOS host or Mariner host or a Flatcar host.

Post by Luca Boccassi
2) handling of a core from a container, where the container and host
have different distros

How each distribution handles debug packages and debug symbols is
going to be different, so it seems there will be a need for explicit
handling of these, at which point the above mentioned querying can be
implemented as well, w/o the need to encode the packaging data inside
the executable.

Again, matching to debug symbols is not the main goal here, build-id
works for that. The main goal is to have useful metadata immediately
available in all occasions, regardless of where the core was generated
on the host, without reaching out to external services, so that it is
directly included and collated in the system journal when the core file
is handled.

With a common metadata definition, there's no need to query or
explicitly handle anything - this already works if you use systemd-
coredump built from the main branch, and handle a core file from
different containers running different distros with binaries having
this metadata in the ELF file, and it just works. This is tested, not
theoretical.

Post by Luca Boccassi
3) self-built and external packages: unless a lot of care is taken to
keep access to the debuginfo packages, this information may be lost.
The new note is available even if the repository metadata gets lost.
Users can easily provide equivalent information in a format that makes
sense in their own environment. It should work even when rpms and debs
and other formats are mixed, e.g. during container image creation.

I'm not sure I see the problem here. Either these self-built 3rd-party
packages are kept in repos that also provide the debug symbols
somewhere for all historically released versions or these will not be
accessible anyway. If they are, they can as well be located as per
above from the Packages metaindices, and even if the repository
metadata gets lost, as long as the debug symbol packages are present
(if they are not what's the point anyway) the build-ids can always be
re-scanned from them as they are part of the Build-Ids field in the
.deb control file.

I think you are thinking about being confined to single distro here - I
am talking about mixed environments. Maybe the build-id can be used to
trace it back - maybe it is available on some federated server. Most
likely it is not, in these cases - it would require customers to go and
upload their sources and symbols and information to random third party
services, a bit unlikely. You have no reference that indicates where to
go look for. Just a filename, which can be anything, and a random id
which might never be published anywhere. What do you do with it?

Post by Luca Boccassi
Other than in Fedora, we are already making the required code changes
at Microsoft to use the same format&specification for internally-built
binaries, and for tools that parse core files and logs.
Tools for RPM and DEB (debhelper) integration are also available [3].

So, to conclude, I don't really see the point of this in the Debian
context. (Not to mention the problems with encoding binary versions
that might be wrong, and the busy work involved.)

Sorry, what do you mean here by encoding binary versions?

Post by Luca Boccassi
[0] https://github.com/systemd/systemd/issues/18433
[1] https://systemd.io/COREDUMP_PACKAGE_METADATA/
[2] https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects
[3] https://github.com/systemd/package-notes

Thanks,
Guillem

--
Kind regards,
Luca Boccassi

Guillem Jover

2021-05-19 00:19:02 UTC

Hi Guillem, thanks for having a look, much appreciated!
Just to clarify, the need is there - this is not an experimental
exercise, but it is borne out of an actual need&requirement, and it is
undergoing testing right now before deployment in a large scale
production infrastructure.
Not _everybody_ will need it, and not everywhere - that's absolutely
fair, and discussions on whether the ovearhead is worth it for
something that is not universally needed, but only in certain use
cases, is perfectly reasonable and welcome. I know Zbigniew is going to
try and get some raw numbers on the kind of overhead we are talking
about, that will hopefully help frame the discussion with more
precision.

Sorry, I think I expressed myself sloppily. My impression is not
that there's no need behind this, but that there's no need for this
specific implementation at least in the Debian context, as I think
the information should be already available by other means.

Also when I mentioned overhead here, it was more about packaging and
integration work than say actual size increase, even though that might
have an adverse effect on .udeb's for example, and perhaps a
non-insignificant one on .deb's depending on the amount of objects
included.

This is not strictly related to debug packages, though?

That was actually the impression I was getting from reading the refs
though. :) But I'll expand below.

Post by Luca Boccassi
In fact, on
systems where this could be of most use you explicitly do _not_ install
debug packages (or anything at all). Or even if you wanted to, you
could not - corefiles are not handled inside the container, but
outside. Even if you wanted to and were allowed to (which for many
environments it's not the case), you can't install a Debian debug
package on a CoreOS host or Mariner host or a Flatcar host.

Sure, but precisely dbgsym .deb's are the things that are trivial to
unpack with dpkg-deb (or ar+tar) w/o needing to actually install them.

Post by Luca Boccassi
2) handling of a core from a container, where the container and host
have different distros

Again, matching to debug symbols is not the main goal here, build-id
works for that. The main goal is to have useful metadata immediately
available in all occasions, regardless of where the core was generated
on the host, without reaching out to external services, so that it is
directly included and collated in the system journal when the core file
is handled.
With a common metadata definition, there's no need to query or
explicitly handle anything - this already works if you use systemd-
coredump built from the main branch, and handle a core file from
different containers running different distros with binaries having
this metadata in the ELF file, and it just works. This is tested, not
theoretical.

So this is where I guess I'm missing something. To be able to make
sense of the coredumps there are two things that might end up being
relevant, backtraces and source code. systemd-coredump might already
emit a backtrace, and depending on the information provided it might
be more or less useful. If one needs the actual debug symbols there's
already some external querying/fetching required, and if distribution
specific source code is required because many distributions patch
upstream source, then even more querying/fetching will be required.

Which is why I'm not seeing why this standalone and isolated metadata
would be of much help by itself. As in, the way I see it, either the
information from systemd (w/o the extra metadata) is sufficient to
track down bugs, or that querying/fetching would be needed anyway, at
which point the metadata can be inferred too then?

Oh, I was thinking about those mixed environments, full chroots or
stripped down containers, from different vendors, but affecting Debian
installations. What I'm also probably missing here is how does the
metadata help for a third-party that is not expected to track/keep/upload
debug symbols nor source packages into some repository, because otherwise
I'm not seeing how they'd make use of the cores (if they are insufficient
by themselves) to debug issues? I guess my question back would be what
would they do with the metadata if they do not have the debug symbols
nor the sources readily available? Also assuming of course they exercise
good practices such as never reusing the same package-version-arch tuple
for different builds, and similar. :)

So, to conclude, I don't really see the point of this in the Debian
context. (Not to mention the problems with encoding binary versions
that might be wrong, and the busy work involved.)

Sorry, what do you mean here by encoding binary versions?

(Expanding, although I assume most of this you already know, but just
for context's sake.)

As was mentioned on the systemd issue (AFAIR), the binary version for
a .deb is decided by the packaging and can (and does) vary from the
source version, and that information is generated after the programs
have been compiled.

For a source version 2.0-1, there are two types of diverging binary
versions, one is for binary-only non-maintainer-uploads (binNMU), such
as 2.0-1+b3 (this would be known at compile-time), and the other would
be for a different binary version-space such as 3:4.5-1 (which is
unknown at compile-time). Both can end up being combined too, as in
3:4.5-1+b5.

Thanks,
Guillem

Uoti Urpala

2021-05-19 10:49:23 UTC

I understood Luca's point to be about more basic metadata. Not in the
context of a human analyzing coredumps, but having basic information
about *what* crashed in log files.

So when the system gets a coredump, the goal is to have more
information in the logs than just "some process crashed in the
container" and name of the binary. Instead the logs would contain
information about what package/version the binary is from (or at least
claims to be from). And writing log files should not involve loading
any extra data from the network.

Luca Boccassi

2021-05-24 10:17:49 UTC

Post by Luca Boccassi
Hi Guillem,

Post by Guillem Jover
So this is where I guess I'm missing something. To be able to make
sense of the coredumps there are two things that might end up being
relevant, backtraces and source code. systemd-coredump might already
emit a backtrace, and depending on the information provided it might
be more or less useful. If one needs the actual debug symbols there's
already some external querying/fetching required, and if distribution
specific source code is required because many distributions patch
upstream source, then even more querying/fetching will be required.
Which is why I'm not seeing why this standalone and isolated metadata
would be of much help by itself. As in, the way I see it, either the
information from systemd (w/o the extra metadata) is sufficient to
track down bugs, or that querying/fetching would be needed anyway, at
which point the metadata can be inferred too then?

Because without that metadata you cannot easily figure out where/how to
get the files needed to properly track down the bugs. But using that
metadata you can figure out where the debuginfod server is that can
provide all that information for the binaries/core files (or a distro
specific method if the distributor doesn't have a debuginfod server
yet).

Yes, that is a good use case. Another use case is that the person (or
bot) who first looks at the journal with the crash notification and the
metadata might not necessarily be the one that does the full in-depth
debugging. First level support will look at things and go "oh this is
from package X at version Y, therefore team Z needs to chime in".
Or it can see that it's version A.B.C, which is listed as known buggy,
and ignore the issue. Or a myriad of other combinations.

Then there are automated systems, parsing logs and creating tickets.
Having the structured metadata available in the journal in the crash
message means much more complex and useful querying and ticket creation
systems can be set up. You _cannot_ just go and query random remote
internet servers from these systems, they need to be network-isolated,
as logs can have confidential data from customers.

The main point is, there are many many things that happen between a
crash and the team owning the component looking up debugging symbols
and loading core files. This feature is very very useful for a whole
lot of those things.