Discussion:
Reduce unit-loading time
(too old to reply)
cee1
2015-05-13 09:51:07 UTC
Permalink
Hi all,

We're trying systemd to boot up an ARM board, and find systemd uses
more than one second to load units.

Comparing with the init of Android on the same board, it manages to
boot the system very fast.

We guess following factors are involved:
1. systemd has a much bigger footprint than the init of Android: the
latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
and is linked with libc/libcap/libpthread, etc)

2. systemd spends quiet a while to read/parse unit files.


Any idea to reduce the unit-loading time?
e.g. one-single file contains all units descriptions, or a "compiled
version"(containing resolved dependencies, or even the boot up
sequence)
--
Regards,

- cee1
Hoyer, Marko (ADITG/SW2)
2015-05-13 11:38:18 UTC
Permalink
Hi,
-----Original Message-----
From: systemd-devel [mailto:systemd-devel-
Sent: Wednesday, May 13, 2015 11:52 AM
To: systemd Mailing List
Subject: [systemd-devel] Reduce unit-loading time
Hi all,
We're trying systemd to boot up an ARM board, and find systemd uses
more than one second to load units.
This sounds far a bit too long to me. Our systemd comes up in an arm based system in about 200-300ms from executing init up to the first unit being executed.
Comparing with the init of Android on the same board, it manages to
boot the system very fast.
1. systemd has a much bigger footprint than the init of Android: the
latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
and is linked with libc/libcap/libpthread, etc)
2. systemd spends quiet a while to read/parse unit files.
This depends on the numbers of units involved in the startup (finally connected in the dependency tree that ends in the default.target root). In our case, a comparable large unit set takes by about 40-60ms, not so long, I'd say.
Any idea to reduce the unit-loading time?
e.g. one-single file contains all units descriptions, or a "compiled
version"(containing resolved dependencies, or even the boot up
sequence)
Could you provide me some additional information about your system setup?

- Version of systemd
- Are you starting something in parallel to systemd which might take significant IO?
- Version of the kernel
- The kernel ticker frequency
- The enabled cgroups controllers

The last three points might sound a bit far away, but I've an idea in mind ...

Best regards

Marko Hoyer
Software Group II (ADITG/SW2)

Tel. +49 5121 49 6948
cee1
2015-05-15 16:46:21 UTC
Permalink
Hey,

Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
Post by Hoyer, Marko (ADITG/SW2)
Hi,
-----Original Message-----
From: systemd-devel [mailto:systemd-devel-
Sent: Wednesday, May 13, 2015 11:52 AM
To: systemd Mailing List
Subject: [systemd-devel] Reduce unit-loading time
Hi all,
We're trying systemd to boot up an ARM board, and find systemd uses
more than one second to load units.
This sounds far a bit too long to me. Our systemd comes up in an arm based system in about 200-300ms from executing init up to the first unit being executed.
Comparing with the init of Android on the same board, it manages to
boot the system very fast.
1. systemd has a much bigger footprint than the init of Android: the
latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
and is linked with libc/libcap/libpthread, etc)
2. systemd spends quiet a while to read/parse unit files.
This depends on the numbers of units involved in the startup (finally connected in the dependency tree that ends in the default.target root). In our case, a comparable large unit set takes by about 40-60ms, not so long, I'd say.
Any idea to reduce the unit-loading time?
e.g. one-single file contains all units descriptions, or a "compiled
version"(containing resolved dependencies, or even the boot up
sequence)
Could you provide me some additional information about your system setup?
- Version of systemd
- Are you starting something in parallel to systemd which might take significant IO?
- Version of the kernel
- The kernel ticker frequency
- The enabled cgroups controllers
The last three points might sound a bit far away, but I've an idea in mind ...
Best regards
Marko Hoyer
Software Group II (ADITG/SW2)
Tel. +49 5121 49 6948
--
Regards,

- cee1
Martin Pitt
2015-05-17 09:45:56 UTC
Permalink
Hello cee,
Post by cee1
Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
cee1
2015-05-18 10:24:00 UTC
Permalink
Post by Martin Pitt
Hello cee,
Post by cee1
Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.
Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Hey,

Thanks for the suggestion, IIRC, sequential read is also beneficial
for flash storage.

Does the readahead-*.service shipped with systemd work for you?


BTW, some suggestions and questions :)

Suggestion:
I use the following command to figure out why my service is scheduled
at the time:
"systemctl list-dependencies --after target.service"
and expect it could output "timing info(unit starting time and
started time, etc)" of dependent units.

Question:
How does systemd schedule two services that can be launched in parallel?

I found that, A and B, if specifies "Before=A", B will start first,
otherwise, B will start at a very late time.
--
Regards,

- cee1
Martin Pitt
2015-05-18 10:40:21 UTC
Permalink
Hello cee1,
Post by cee1
Does the readahead-*.service shipped with systemd work for you?
systemd dropped the builtin readahead in 217. It's reasonably easy to
get back by reverting the "drop readahead" patches, but carrying that
patch in packages is fairly intrusive. In Ubuntu we've had
"ureadahead" [1] for many years which is "good enough" for things like
phones or other ARM boards with slow MMC storage, so I just added
systemd units to that. It's a separate project so that we don't need
to install ureadahead everywhere, just where/when we actually need it.

Martin

[1] https://launchpad.net/ubuntu/+source/ureadahead
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
cee1
2015-05-18 15:52:31 UTC
Permalink
Hi Martin,

At the first glance, I find ureadahead has some difference compared
with the readahead once in systemd, IIRC:

1. ureadahead.service is in default.target, which means ureadahead
starts later than systemd's?
2. The original systemd readahead has "collect" and "replay" two
services, and these are done in ureadahead.service?
3. IIRC, The original systemd readahead uses madvise(); ureadahead
uses readahead()
4. The original systemd readahead uses fanotify() to get the list of
accessed files; ureadahead use fsnotify
5. ureadahead has different readahead strategies for ssd and hhd:
5.1 For the former, initiate multi-threads to perform readahead, and
they are running at the lowest IO priority.
5.2 For the later, perform readahead for both inode and file content
at a very high CPU/IO priority (and only support extN filesystem ?)
Post by Martin Pitt
Hello cee1,
Post by cee1
Does the readahead-*.service shipped with systemd work for you?
systemd dropped the builtin readahead in 217. It's reasonably easy to
get back by reverting the "drop readahead" patches, but carrying that
patch in packages is fairly intrusive. In Ubuntu we've had
"ureadahead" [1] for many years which is "good enough" for things like
phones or other ARM boards with slow MMC storage, so I just added
systemd units to that. It's a separate project so that we don't need
to install ureadahead everywhere, just where/when we actually need it.
Martin
[1] https://launchpad.net/ubuntu/+source/ureadahead
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
--
Regards,

- cee1
Martin Pitt
2015-05-19 17:01:53 UTC
Permalink
Hey cee1,
Post by cee1
At the first glance, I find ureadahead has some difference compared
Yes, for sure. systemd's was improved quite a bit. ureadahead is
mostly unmaintained, but it works well enough so we didn't bother to
put work into it until someone actually complains :-)
Post by cee1
1. ureadahead.service is in default.target, which means ureadahead
starts later than systemd's?
That mostly means that it's not started with e. g. emergency or
rescue. It still starts early (DefaultDependencies=false).
Post by cee1
2. The original systemd readahead has "collect" and "replay" two
services, and these are done in ureadahead.service?
Yes.
Post by cee1
3. IIRC, The original systemd readahead uses madvise(); ureadahead
uses readahead()
4. The original systemd readahead uses fanotify() to get the list of
accessed files; ureadahead use fsnotify
I haven't verified these, but this sounds correct. ureadahead is
really old, presumably the newer features like fanotify weren't
available back then.
Right. These were created by some rather wide-scale measurements back
then.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
cee1
2015-05-25 03:41:36 UTC
Permalink
Post by Martin Pitt
Hey cee1,
Post by cee1
At the first glance, I find ureadahead has some difference compared
Yes, for sure. systemd's was improved quite a bit. ureadahead is
mostly unmaintained, but it works well enough so we didn't bother to
put work into it until someone actually complains :-)
Post by cee1
1. ureadahead.service is in default.target, which means ureadahead
starts later than systemd's?
That mostly means that it's not started with e. g. emergency or
rescue. It still starts early (DefaultDependencies=false).
Post by cee1
2. The original systemd readahead has "collect" and "replay" two
services, and these are done in ureadahead.service?
Yes.
Post by cee1
3. IIRC, The original systemd readahead uses madvise(); ureadahead
uses readahead()
4. The original systemd readahead uses fanotify() to get the list of
accessed files; ureadahead use fsnotify
I haven't verified these, but this sounds correct. ureadahead is
really old, presumably the newer features like fanotify weren't
available back then.
I tried ureadahead, but got following error:

"""write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
while tracing: No such file or directory"""

Needs an out-of-tree kernel patch?
--
Regards,

- cee1
Filipe Brandenburger
2015-05-26 16:49:39 UTC
Permalink
Hi,
Post by cee1
"""write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
while tracing: No such file or directory"""
Needs an out-of-tree kernel patch?
Yes, ureadahead needs an out-of-tree kernel patch to add trace events
for open(), exec() and uselib() syscalls that take file paths.

http://bugs.launchpad.net/bugs/462111

AFAICT, this never went upstream because upstream was discussing a
generic approach of tracing any system calls, but I believe that never
really panned out...

HTH,
Filipe
Lennart Poettering
2015-05-26 17:05:12 UTC
Permalink
Post by Filipe Brandenburger
Hi,
Post by cee1
"""write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
while tracing: No such file or directory"""
Needs an out-of-tree kernel patch?
Yes, ureadahead needs an out-of-tree kernel patch to add trace events
for open(), exec() and uselib() syscalls that take file paths.
http://bugs.launchpad.net/bugs/462111
AFAICT, this never went upstream because upstream was discussing a
generic approach of tracing any system calls, but I believe that never
really panned out...
systemd-readahead used fanotify() which requires no kernel
patching. fanotify is completely generic and perfect for this
purpose...

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-05-18 16:24:36 UTC
Permalink
Post by cee1
Post by Martin Pitt
Hello cee,
Post by cee1
Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.
Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Hey,
Thanks for the suggestion, IIRC, sequential read is also beneficial
for flash storage.
Does the readahead-*.service shipped with systemd work for you?
We removed the readahead logic from systemd a while back, since it had
no maintainer, and nobody wanted to pick it up.
Post by cee1
How does systemd schedule two services that can be launched in
parallel?
It's not defined. systemd will fork things of in an undefined order if
ther are multiple units runnable at the same time.

As mentioned elsewhere, I'd be willing to merge a patchat that changes
this and allows control of what service is forked off first, via some
per-unit Priority= setting or so.

Lennart
--
Lennart Poettering, Red Hat
Cristian Rodríguez
2015-05-19 20:39:44 UTC
Permalink
Post by Martin Pitt
Hello cee,
Post by cee1
Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.
You could also

posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL);
in the bits that load the unit file..the kernel is free to ignore that
advice however.
Cristian Rodríguez
2015-05-19 21:35:36 UTC
Permalink
On Tue, May 19, 2015 at 5:39 PM, Cristian Rodríguez
Post by Cristian Rodríguez
Post by Martin Pitt
Hello cee,
Post by cee1
Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.
You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.
You could also
posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL);
in the bits that load the unit file..the kernel is free to ignore that
advice however.
This however.. won't be of any help, as the default readhead window is
128kb.. which is way bigger than any unit file.
Lennart Poettering
2015-05-13 16:04:06 UTC
Permalink
Post by cee1
Hi all,
We're trying systemd to boot up an ARM board, and find systemd uses
more than one second to load units.
Comparing with the init of Android on the same board, it manages to
boot the system very fast.
1. systemd has a much bigger footprint than the init of Android: the
latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
and is linked with libc/libcap/libpthread, etc)
2. systemd spends quiet a while to read/parse unit files.
Any idea to reduce the unit-loading time?
e.g. one-single file contains all units descriptions, or a "compiled
version"(containing resolved dependencies, or even the boot up
sequence)
Well, before starting to optimize things one should always profile
this step to identify what precisely is slow. This starts from
questions like whether this is IO bound or CPU bound to precisely
tracing things down to individual files.

systemd unit loading is not particularly optimized, we distribute
everything into small unit files, which generally isn't the best way
to get quick access times for many file systems. You can improve the
situation with read-ahead for these files, but before you do that you
really need to spend some time to figure out what precisely is slow.

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...