Discussion:
Parallel startup with sockets and without killing the machine?
(too old to reply)
Scott James Remnant
2011-05-09 16:10:50 UTC
Permalink
Hey, I've been reading through the documentation on systemd and the Mailing
List Archives and have had some questions, so I figured I'd post them here.
These might be fairly complicated or unusual cases, and there may be no good
answers now, and maybe no good answers ever, but I'd still like to get an
idea of what you think.

First up, parallel startup using sockets.

We know that the theory of this goes something like this: you have a service
that many other services and processes depend on, and connect to via a
socket. Rather than manually express that dependency in the configuration,
you instead have the socket created by the init daemon in advance. Now you
simply start the service and all of the other services and processes
together, and let the blocking nature of those sockets (or the software
polling a non-blocking socket) sort out the ordering for you.

This seems to assume that it's a free cost to start a service or process and
have it block on a socket. But that isn't true, to get to the point where it
connects to the socket, that service or process first has to be loaded from
disk, its shared libraries mapped and relocated, it probably has to do a
bunch of initialization including reading its own configuration files, etc.

That's fine for "low priority" services out of the critical path for user
interaction, but not for services we need as fast as possible.

The best example I can think is the X server itself. It's pretty much the
definition of the critical path. Everything started before the X server are
those basic system components needed to start the X server, I understand in
systemd that this core part would be a different target to the X part and a
dep of it. Everything started after the X server is up are components of the
user's session that aren't totally critical for them to login or use their
computer.

It's during the X startup that things get interesting. There are a few
discreet phases of it.

Firstly we have to spawn the X server, load the executable, map and relocate
shared libraries, all of that jazz. You don't want anything else happening
on the system alongside this, and you most certainly don't want every X
client (Chromium/Firefox!) being loaded at the same time. The default
behaviour of systemd seems to be to do just this :-(

Secondly you have the period after the X server has "loaded" while
it initializes but before it's "ready" to receive clients. In a perfect
world this would be fast, but we all know that it isn't. In practice the X
server takes countable seconds to do this work. This is a prime place in
boot optimization to start other services alongside while X does its thang.
But you wouldn't want to flood the system, as you don't want to delay X's
startup, just use the spare resource to get more done. Can systemd discern
"X has been spawned" and "X has initialized". Can systemd serialize services
in this period (perhaps limited to one or two starting at a time)? Can it
prioritize the order of those services in the X target?

Lastly there's the processes you'd start after X is up and services are
connecting to it; again I assume this would be a different target - how
would systemd know that X, and the critical X services/processes are fully
up and that it can begin starting these. The important thing here is user
latency - the startup of these services need to not noticeably impact system
performance - not as simple as prioritization because the service itself may
well need a different priority once its up. Any thoughts here?

Thanks,

Scott
Lennart Poettering
2011-05-09 18:04:02 UTC
Permalink
Post by Scott James Remnant
This seems to assume that it's a free cost to start a service or process and
have it block on a socket. But that isn't true, to get to the point where it
connects to the socket, that service or process first has to be loaded from
disk, its shared libraries mapped and relocated, it probably has to do a
bunch of initialization including reading its own configuration files, etc.
That's fine for "low priority" services out of the critical path for user
interaction, but not for services we need as fast as possible.
We can now boot a resonably complete GNOME userspace in less than
1s. Given that that is the *total* these days, I really don't
understand why you are concerned about the startup speed of individual
service, and fear timeouts there.
Post by Scott James Remnant
The best example I can think is the X server itself. It's pretty much the
definition of the critical path. Everything started before the X server are
those basic system components needed to start the X server, I understand in
systemd that this core part would be a different target to the X part and a
dep of it. Everything started after the X server is up are components of the
user's session that aren't totally critical for them to login or use their
computer.
X itself is really fast these days. While there's room for improvement
it's far from the biggest speed bump right now. (LVM is)
Post by Scott James Remnant
It's during the X startup that things get interesting. There are a few
discreet phases of it.
I don't find X very interesting here at all.
Post by Scott James Remnant
Firstly we have to spawn the X server, load the executable, map and relocate
shared libraries, all of that jazz. You don't want anything else happening
on the system alongside this, and you most certainly don't want every X
client (Chromium/Firefox!) being loaded at the same time. The default
behaviour of systemd seems to be to do just this :-(
Well, there's more than 30 years research in modern CPU and IO scheduler
technology for the kernel. Instead of second guessing those from
userspace I'd just entrust them that they pick something like the
optimal order. And if they don't, then go and optimize them for the new
workloads, don't bypass them by scheduling things in userspace.

Of course, we could microoptimize things and pass along to the kernel
scheduler useful priority data gathered from our dep tree. But right now
I am more interested in harvesting the easier and bigger improvements we
can do. Like getting rid of shell scripts on the full distro, and making
sure the initrd isn't delayed by DDC probing. And after we have done
that, we can focus on passing on prio information.
Post by Scott James Remnant
Secondly you have the period after the X server has "loaded" while
it initializes but before it's "ready" to receive clients. In a perfect
world this would be fast, but we all know that it isn't. In practice the X
server takes countable seconds to do this work. This is a prime place in
That is not what I am seeing here.
Post by Scott James Remnant
boot optimization to start other services alongside while X does its thang.
But you wouldn't want to flood the system, as you don't want to delay X's
startup, just use the spare resource to get more done. Can systemd discern
"X has been spawned" and "X has initialized". Can systemd serialize services
in this period (perhaps limited to one or two starting at a time)? Can it
prioritize the order of those services in the X target?
No, we emphasize parallelization. We want the IO and the CPU scheduler
of the kernel to schedule things for us. They are much better at these
things. If we tried this in userspace most of the time the CPU/IO would
be less than fully utilized.
Post by Scott James Remnant
Lastly there's the processes you'd start after X is up and services are
connecting to it; again I assume this would be a different target - how
would systemd know that X, and the critical X services/processes are fully
up and that it can begin starting these. The important thing here is user
latency - the startup of these services need to not noticeably impact system
performance - not as simple as prioritization because the service itself may
well need a different priority once its up. Any thoughts here?
Ignoring that X currently is not socket activatable I see no issue here
at all.

Also, note that launchd on MacOS does the same thing (including spawning
X like this for X clients). They build their whole OS like this. And it
works really really really well there. All experience with launchd and
systemd tells the same story: socket activation is awesome. Plain awesome.

We boot userspace in < 1s. Really. I mean, doesn't this give a pretty
strong signal that things are working perfectly with the approach we
chose?

Lennart
--
Lennart Poettering - Red Hat, Inc.
Paul Menzel
2011-05-09 19:29:05 UTC
Permalink
Am Montag, den 09.05.2011, 20:04 +0200 schrieb Lennart Poettering:

[
]
We can now boot a reasonably complete GNOME userspace in less than
1s.
Is there a demo image for that? What are the hardware specs?

Will you be on the LinuxTag? coreboot will be demoing systems, e. g.,
Lenovo X60/T60, where hardware initialization is also done in less than
one second [1]. Maybe we could demo a system with less than two seconds
from button push to a GNOME system? That would be good publicity.

[
]


Thanks,

Paul


[1] http://www.coreboot.org/pipermail/coreboot/2011-May/064913.html
Gustavo Sverzut Barbieri
2011-05-09 19:47:33 UTC
Permalink
On Mon, May 9, 2011 at 4:29 PM, Paul Menzel
Post by Paul Menzel
[…]
We can now boot a reasonably complete GNOME userspace in less than
1s.
Is there a demo image for that? What are the hardware specs?
Will you be on the LinuxTag? coreboot will be demoing systems, e. g.,
Lenovo X60/T60, where hardware initialization is also done in less than
one second [1]. Maybe we could demo a system with less than two seconds
from button push to a GNOME system? That would be good publicity.
[…]
If you want to be fast, then I can help you with E17 setup, it's
faster than starting GNOME by a great margin ;-)

All I'd ask you is an SSD to make it faster, since loading stuff from
disk takes time. But that's not mandatory. Software stack just use
slim (DM, in autologin mode) with standard Enlightenment E17 (if you
get it from SVN we collapsed/merged some modules/.so so it's even
faster as there is less runtime work).

I'll not be at LinuxTag, but I'll ask other developers and will let you know.

--
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN: ***@gmail.com
Skype: gsbarbieri
Mobile: +55 (19) 9225-2202
Peter Stuge
2011-05-09 20:19:58 UTC
Permalink
Post by Gustavo Sverzut Barbieri
If you want to be fast, then I can help you with E17 setup, it's
faster than starting GNOME by a great margin ;-)
Help is welcome. GNOME might have the benefit of being slightly more
familiar to visitors though. Many end users at LinuxTag.
Post by Gustavo Sverzut Barbieri
All I'd ask you is an SSD to make it faster, since loading stuff
from disk takes time.
Yes sure, we are very much used to flash in coreboot land, because we
do not want to wait for hard drive spindles.
Post by Gustavo Sverzut Barbieri
slim (DM, in autologin mode) with standard Enlightenment E17 (if you
get it from SVN we collapsed/merged some modules/.so so it's even
faster as there is less runtime work).
The ideal would be to have a ready-to-boot-sans-LILO system tarball.
(Think Gentoo stage4.)


//Peter
Peter Stuge
2011-05-09 19:45:16 UTC
Permalink
Post by Paul Menzel
We can now boot a reasonably complete GNOME userspace in less than
1s.
Is there a demo image for that? What are the hardware specs?
Yes, a short and sweet way to reproduce these results would be
lovely. I was hoping to build a userspace with systemd for the
machines, but have not found time yet. It would be an impressive
demo. (We have Thinkpad X60, T60 and an AMD Fusion board.)

If it helps I'll gladly use sneakernet for file transfer.


//Peter
Lennart Poettering
2011-05-09 20:17:05 UTC
Permalink
Post by Paul Menzel
[…]
We can now boot a reasonably complete GNOME userspace in less than
1s.
Is there a demo image for that? What are the hardware specs?
No image. Measured on Kay's X300 (a few years old) with SSD, a few weeks
ago. It's a OpenSUSE with a number of fixes and cleanups.

Note that these results are fluctuating. i.e. you won't get the exact
same results on each boot. Also: by bringing up userspace I was simply
referring to the time systemd spends doing that. What the services still
do is ignored by this value. it's just that systemd goes idle, and its
queues are empty and fully dispatched.
Post by Paul Menzel
Will you be on the LinuxTag? coreboot will be demoing systems, e. g.,
Lenovo X60/T60, where hardware initialization is also done in less than
one second [1]. Maybe we could demo a system with less than two seconds
from button push to a GNOME system? That would be good publicity.
I'll be at LinuxTag, I live in Berlin. But quite frankly I won't have
the time to do spend time on that, sorry.

Lennart
--
Lennart Poettering - Red Hat, Inc.
Paul Menzel
2011-05-09 20:35:27 UTC
Permalink
Post by Lennart Poettering
Post by Paul Menzel
[
]
We can now boot a reasonably complete GNOME userspace in less than
1s.
Is there a demo image for that? What are the hardware specs?
No image. Measured on Kay's X300 (a few years old) with SSD, a few weeks
ago. It's a OpenSUSE with a number of fixes and cleanups.
Well, maybe Kay still has that image and can share it or even plug in
his SSD.
Post by Lennart Poettering
Note that these results are fluctuating. i.e. you won't get the exact
same results on each boot. Also: by bringing up userspace I was simply
referring to the time systemd spends doing that. What the services still
do is ignored by this value. it's just that systemd goes idle, and its
queues are empty and fully dispatched.
Understood.
Post by Lennart Poettering
Post by Paul Menzel
Will you be on the LinuxTag? coreboot will be demoing systems, e. g.,
Lenovo X60/T60, where hardware initialization is also done in less than
one second [1]. Maybe we could demo a system with less than two seconds
from button push to a GNOME system? That would be good publicity.
I'll be at LinuxTag, I live in Berlin.
I heard rumors about that. ;-)
Post by Lennart Poettering
But quite frankly I won't have the time to do spend time on that, sorry.
Surely you will come by for a few minutes to see with your own eyes what
you hear from all the visitors about that awesome user experience at the
coreboot booth. ;-) We want to take a picture with you and get a short
comment for a press release of one of the main systemd developers saying
how awesome coreboot is (you can mention systemd of course). ;-)


Thanks,

Paul
Lennart Poettering
2011-05-09 20:39:09 UTC
Permalink
Post by Paul Menzel
Post by Lennart Poettering
But quite frankly I won't have the time to do spend time on that, sorry.
Surely you will come by for a few minutes to see with your own eyes what
you hear from all the visitors about that awesome user experience at the
coreboot booth. ;-) We want to take a picture with you and get a short
comment for a press release of one of the main systemd developers saying
how awesome coreboot is (you can mention systemd of course). ;-)
That we certainly can do! ;-)

Lennart
--
Lennart Poettering - Red Hat, Inc.
Kay Sievers
2011-05-09 21:00:39 UTC
Permalink
Post by Lennart Poettering
Post by Paul Menzel
Post by Lennart Poettering
But quite frankly I won't have the time to do spend time on that, sorry.
Surely you will come by for a few minutes to see with your own eyes what
you hear from all the visitors about that awesome user experience at the
coreboot booth. ;-) We want to take a picture with you and get a short
comment for a press release of one of the main systemd developers saying
how awesome coreboot is (you can mention systemd of course). ;-)
That we certainly can do! ;-)
I'll be there Fri/Sat too. And the above mentioned laptop will be there too. :)

Kay
Scott James Remnant
2011-05-10 02:49:01 UTC
Permalink
Thanks for the reply!

On Mon, May 9, 2011 at 11:04 AM, Lennart Poettering
Post by Lennart Poettering
Post by Scott James Remnant
This seems to assume that it's a free cost to start a service or process
and
Post by Scott James Remnant
have it block on a socket. But that isn't true, to get to the point where
it
Post by Scott James Remnant
connects to the socket, that service or process first has to be loaded
from
Post by Scott James Remnant
disk, its shared libraries mapped and relocated, it probably has to do a
bunch of initialization including reading its own configuration files, etc.
That's fine for "low priority" services out of the critical path for user
interaction, but not for services we need as fast as possible.
We can now boot a resonably complete GNOME userspace in less than
1s. Given that that is the *total* these days, I really don't
understand why you are concerned about the startup speed of individual
service, and fear timeouts there.
Because that's on a machine with plenty of L1/L2 Cache, take that away and
this becomes a big concern. Especially if 1s is a target boot time ;-)
Post by Lennart Poettering
The best example I can think is the X server itself. It's pretty much the
Post by Scott James Remnant
definition of the critical path. Everything started before the X server
are
Post by Scott James Remnant
those basic system components needed to start the X server, I understand
in
Post by Scott James Remnant
systemd that this core part would be a different target to the X part and
a
Post by Scott James Remnant
dep of it. Everything started after the X server is up are components of
the
Post by Scott James Remnant
user's session that aren't totally critical for them to login or use
their
Post by Scott James Remnant
computer.
X itself is really fast these days. While there's room for improvement
it's far from the biggest speed bump right now. (LVM is)
LVM isn't on our system. X is the biggest bump along with Chromium.
Firstly we have to spawn the X server, load the executable, map and relocate
Post by Scott James Remnant
shared libraries, all of that jazz. You don't want anything else
happening
Post by Scott James Remnant
on the system alongside this, and you most certainly don't want every X
client (Chromium/Firefox!) being loaded at the same time. The default
behaviour of systemd seems to be to do just this :-(
Well, there's more than 30 years research in modern CPU and IO scheduler
technology for the kernel. Instead of second guessing those from
userspace I'd just entrust them that they pick something like the
optimal order. And if they don't, then go and optimize them for the new
workloads, don't bypass them by scheduling things in userspace.
I'm glad you answered this way, because my thinking was in this direction
too. The scheduler isn't that great in Linux at times, as we all know,
because it lacks information about what's important. Since the kernel is
able to schedule efficiently by cgroup, the ideal seems to be to start
everything at once *but with different priorities* so that the services come
up in something approaching a sane order.

We did something like this back in Ubuntu, all fsck processes are started
simultaneously and we used I/O priorities to ensure only one was running at
a time for any given physical disk. This worked awesomely well.

Obviously this "startup priority" isn't the same as the eventual priority
you'd want the service to settle at, you'd want it to be pretty flexible -
perhaps even changing a few times.
Post by Lennart Poettering
Lastly there's the processes you'd start after X is up and services are
Post by Scott James Remnant
connecting to it; again I assume this would be a different target - how
would systemd know that X, and the critical X services/processes are
fully
Post by Scott James Remnant
up and that it can begin starting these. The important thing here is user
latency - the startup of these services need to not noticeably impact
system
Post by Scott James Remnant
performance - not as simple as prioritization because the service itself
may
Post by Scott James Remnant
well need a different priority once its up. Any thoughts here?
Ignoring that X currently is not socket activatable I see no issue here
at all.
Really, this came as a surprise to me - don't all clients initially connect
to X via its socket?
Post by Lennart Poettering
Also, note that launchd on MacOS does the same thing (including spawning
X like this for X clients). They build their whole OS like this. And it
works really really really well there. All experience with launchd and
systemd tells the same story: socket activation is awesome. Plain awesome.
You've never had to reboot an iPhone, have you?
You need other entertainment while it does so, a copy of the Lord of the
Rings sometimes suffices.

Scott
Lennart Poettering
2011-05-10 15:54:08 UTC
Permalink
Post by Lennart Poettering
Post by Lennart Poettering
We can now boot a resonably complete GNOME userspace in less than
1s. Given that that is the *total* these days, I really don't
understand why you are concerned about the startup speed of individual
service, and fear timeouts there.
Because that's on a machine with plenty of L1/L2 Cache, take that away and
this becomes a big concern. Especially if 1s is a target boot time ;-)
Not to my knowledge. On my crappy Atom netbook with HDD I can boot in
13s now. Which is much better than the 40s or so before. And it's not
even particularly optimized.
Post by Lennart Poettering
Post by Lennart Poettering
Well, there's more than 30 years research in modern CPU and IO scheduler
technology for the kernel. Instead of second guessing those from
userspace I'd just entrust them that they pick something like the
optimal order. And if they don't, then go and optimize them for the new
workloads, don't bypass them by scheduling things in userspace.
I'm glad you answered this way, because my thinking was in this direction
too. The scheduler isn't that great in Linux at times, as we all know,
because it lacks information about what's important. Since the kernel is
able to schedule efficiently by cgroup, the ideal seems to be to start
everything at once *but with different priorities* so that the services come
up in something approaching a sane order.
We did something like this back in Ubuntu, all fsck processes are started
simultaneously and we used I/O priorities to ensure only one was running at
a time for any given physical disk. This worked awesomely well.
fsck nowadays locks the originating block devices itself now on HDD,
thus serializing fscks on rotating disks.
Post by Lennart Poettering
Obviously this "startup priority" isn't the same as the eventual priority
you'd want the service to settle at, you'd want it to be pretty flexible -
perhaps even changing a few times.
Yes, as mentioned elsewhere in the thread such a model of supplying the
kernel with prio data extracted from the dep tree is relatively easy to
do, and perfectly in line with systemd's design.
Post by Lennart Poettering
Post by Lennart Poettering
Ignoring that X currently is not socket activatable I see no issue here
at all.
Really, this came as a surprise to me - don't all clients initially connect
to X via its socket?
We haven't patched X for this yet. We hope to do this for the F16
cycle. See http://lwn.net/Articles/441328/
Post by Lennart Poettering
Post by Lennart Poettering
Also, note that launchd on MacOS does the same thing (including spawning
X like this for X clients). They build their whole OS like this. And it
works really really really well there. All experience with launchd and
systemd tells the same story: socket activation is awesome. Plain awesome.
You've never had to reboot an iPhone, have you?
You need other entertainment while it does so, a copy of the Lord of the
Rings sometimes suffices.
Well, I think we should discuss these issues when they come up in systemd
in real life. So far they are theoretical. And if they do show up we
have a large toolbox of things we could do, including the mentioned
startup CPU/IO priorization based on the dep tree. So, I think it's not
unreasonable to just lean back, and wait what comes in the knowledge
that there's a lot of low-hanging fruit.

Lennart
--
Lennart Poettering - Red Hat, Inc.
microcai
2011-05-09 18:16:43 UTC
Permalink
Post by Scott James Remnant
Hey, I've been reading through the documentation on systemd and the Mailing
List Archives and have had some questions, so I figured I'd post them here.
These might be fairly complicated or unusual cases, and there may be no good
answers now, and maybe no good answers ever, but I'd still like to get an
idea of what you think.
ok, i'm trying to express.
Post by Scott James Remnant
First up, parallel startup using sockets.
We know that the theory of this goes something like this: you have a service
that many other services and processes depend on, and connect to via a
socket. Rather than manually express that dependency in the configuration,
you instead have the socket created by the init daemon in advance. Now you
simply start the service and all of the other services and processes
together, and let the blocking nature of those sockets (or the software
polling a non-blocking socket) sort out the ordering for you.
This seems to assume that it's a free cost to start a service or process and
have it block on a socket. But that isn't true, to get to the point where it
connects to the socket, that service or process first has to be loaded from
disk, its shared libraries mapped and relocated, it probably has to do a
bunch of initialization including reading its own configuration files, etc.
No, we don't. It does not matter how long your server takes to fully
start up.
Post by Scott James Remnant
That's fine for "low priority" services out of the critical path for user
interaction, but not for services we need as fast as possible.
As fast as possible, this can only be done in systemd way. And service
won't lost any request. In traditional way, it lost.
Post by Scott James Remnant
The best example I can think is the X server itself. It's pretty much the
definition of the critical path. Everything started before the X server are
those basic system components needed to start the X server, I understand in
systemd that this core part would be a different target to the X part and a
dep of it. Everything started after the X server is up are components of the
user's session that aren't totally critical for them to login or use their
computer.
Currently, gdm wait for X to be ready before move on.
And to be worse, they use busy pull, not by some notification way.
In systemd way, gdm don't have to wait. And don't even have to X itself.
Just connect to X and X is there. No matter if X is currently running at
all.
Post by Scott James Remnant
It's during the X startup that things get interesting. There are a few
discreet phases of it.
Firstly we have to spawn the X server, load the executable, map and relocate
shared libraries, all of that jazz. You don't want anything else happening
on the system alongside this, and you most certainly don't want every X
client (Chromium/Firefox!) being loaded at the same time. The default
behaviour of systemd seems to be to do just this :-(
No, systemd will not do this anyway. And even it really do this way,
what's the point? why X must been the only running process will it is
initializing ?
Post by Scott James Remnant
Secondly you have the period after the X server has "loaded" while
it initializes but before it's "ready" to receive clients. In a perfect
world this would be fast, but we all know that it isn't. In practice the X
server takes countable seconds to do this work. This is a prime place in
boot optimization to start other services alongside while X does its thang.
But you wouldn't want to flood the system, as you don't want to delay X's
startup, just use the spare resource to get more done. Can systemd discern
"X has been spawned" and "X has initialized". Can systemd serialize services
in this period (perhaps limited to one or two starting at a time)? Can it
prioritize the order of those services in the X target?
Yeah , we have the *period* for every server. In upstart way, it use
complex method to indicate that stages. But with systemd ? We don't
even have to care about that.
Post by Scott James Remnant
Lastly there's the processes you'd start after X is up and services are
connecting to it; again I assume this would be a different target - how
would systemd know that X, and the critical X services/processes are fully
up and that it can begin starting these. The important thing here is user
latency - the startup of these services need to not noticeably impact system
performance - not as simple as prioritization because the service itself may
well need a different priority once its up. Any thoughts here?
This is no such thing called, XX service is initializing, or XX service
is up. No, systemd pressent us only one thing: There is XX service that
is connect-able. And the first connection may took long to response. But
you don't have to wait for the "running" stage any more. The XX servie
is connect-able throughout the way. You don't have to care if XX is
running or not. Just connect to it.
Post by Scott James Remnant
Thanks,
Scott
Gustavo Sverzut Barbieri
2011-05-09 19:16:25 UTC
Permalink
Hi Scott,

Some answers below, but consider reading Lennart's reply first. I'll try to
cover some points he did not in his reply.
Post by Scott James Remnant
Hey, I've been reading through the documentation on systemd and the Mailing
List Archives and have had some questions, so I figured I'd post them here.
These might be fairly complicated or unusual cases, and there may be no good
answers now, and maybe no good answers ever, but I'd still like to get an
idea of what you think.
First up, parallel startup using sockets.
We know that the theory of this goes something like this: you have a
service that many other services and processes depend on, and connect to via
a socket. Rather than manually express that dependency in the configuration,
you instead have the socket created by the init daemon in advance. Now you
simply start the service and all of the other services and processes
together, and let the blocking nature of those sockets (or the software
polling a non-blocking socket) sort out the ordering for you.
This seems to assume that it's a free cost to start a service or process
and have it block on a socket. But that isn't true, to get to the point
where it connects to the socket, that service or process first has to be
loaded from disk, its shared libraries mapped and relocated, it probably has
to do a bunch of initialization including reading its own configuration
files, etc.
Yes and that's exactly the reason it's good to start them before the other
service is fully ready to be used. Say that while DBus is starting due
UDisks request, UDisks may be parsing useful information from system (udev
tables? -- this is speculation on what is possible, I don't know the code of
udisks and if it uses this benefit)

As Lennart said, the feeding the CPU and IO to kernel will let it optimize
the load. Be grouping disk access or scheduling another process during IO
waits for the other. See below for one of my ideas on how to feed more
information to kernel.
Post by Scott James Remnant
That's fine for "low priority" services out of the critical path for user
interaction, but not for services we need as fast as possible.
The best example I can think is the X server itself. It's pretty much the
definition of the critical path. Everything started before the X server are
those basic system components needed to start the X server, I understand in
systemd that this core part would be a different target to the X part and a
dep of it. Everything started after the X server is up are components of the
user's session that aren't totally critical for them to login or use their
computer.
It's during the X startup that things get interesting. There are a few
discreet phases of it.
Firstly we have to spawn the X server, load the executable, map and
relocate shared libraries, all of that jazz. You don't want anything else
happening on the system alongside this, and you most certainly don't want
every X client (Chromium/Firefox!) being loaded at the same time. The
default behaviour of systemd seems to be to do just this :-(
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to GDM/KDM...).

But using your case, just like I said above, you really want that. You'd
chromium and X11 to be loaded at the same time, because the kernel will
handle the load and may load chromium executable while X11 is doing EDID
read, then may let the dynamic linker work while X11 asks for input device,
etc.

Also, ideally the socket activation lets you avoid cruft (unused
applications and services) being loaded. Let's say you have in your machine
Apache, MySQL, Squid for local development. None of these would be loaded,
because you never did a request to Apache that would make it to run, in turn
it would never talk to MySQL that would not start either. If you did no try
to access squid, it would not start either. Same for cups, avahi, ssh and
all dbus activated services... you can see that since less is activated,
less work is left to do.
Post by Scott James Remnant
Secondly you have the period after the X server has "loaded" while
it initializes but before it's "ready" to receive clients. In a perfect
world this would be fast, but we all know that it isn't. In practice the X
server takes countable seconds to do this work. This is a prime place in
boot optimization to start other services alongside while X does its thang.
But you wouldn't want to flood the system, as you don't want to delay X's
startup, just use the spare resource to get more done. Can systemd discern
"X has been spawned" and "X has initialized". Can systemd serialize services
in this period (perhaps limited to one or two starting at a time)? Can it
prioritize the order of those services in the X target?
It can't know. But it could be changed to be like udev-settle service and
force it artificially. But our approach is different and trying to fix the
actual issues are better, with X11, kernel or elsewhere.

But I do agree with you an a way to state our priorities would be awesome.
This is what Lennart said about a future way to feed it to kernel. I
suggested some hackish way some time ago:

http://lists.freedesktop.org/archives/systemd-devel/2011-March/001716.html

A hackish(?) solution would be to have a BootPriority=(True|False), set to
False by default and True for services we care most. Lower priority services
would be set to "background" priority in IO, CPU and others, then being
reset to actual values when systemd is notified. Problem is we need to
notify Systemd of that, as it's not a matter of just starting "gdm", but
actually gdm being in a "usable" state (defined by gdm itself) or desktop
being ready if users use autologin (like I do). This could also be stated as
"system is idle for X seconds", which would be monitored by systemd itself
and then no manual notification is required.


But its discussion is being delayed as at the moment things are going "fast
enough" without it (my userspace takes 1.2s in my Gentoo with SSD disk on a
MacBook Pro).


Lastly there's the processes you'd start after X is up and services are
Post by Scott James Remnant
connecting to it; again I assume this would be a different target - how
would systemd know that X, and the critical X services/processes are fully
up and that it can begin starting these. The important thing here is user
latency - the startup of these services need to not noticeably impact system
performance - not as simple as prioritization because the service itself may
well need a different priority once its up. Any thoughts here?
During LPC last year Lennart, Arjan and others discussed this. Arjan
suggested doing this artificial throttling to not trigger kernel fork-bomb
protections. Lennart argued that this should not happen given the number of
things we start and if that's the case, it should be fixed in the kernel.
Given that MeeGo is now moving to systemd, I wonder if this problem happens
or if it's so small it was ignored. During this discussion we wondered
about changing kernel parameters, like making the system less interactive
during startup to speed up boot, reverting to fully interactive later, but
it proved to be more work than what it was worth.

Hope it help to clarify :-)

Regards,
--
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN: ***@gmail.com
Skype: gsbarbieri
Mobile: +55 (19) 9225-2202
Lennart Poettering
2011-05-09 20:01:33 UTC
Permalink
Post by Gustavo Sverzut Barbieri
But I do agree with you an a way to state our priorities would be awesome.
This is what Lennart said about a future way to feed it to kernel. I
I actually discussed this with a couple of GENIVI folks. They try to
optimize things for booting car computers with systemd. Here's what I
proposed to them:

systemd already has an elaborate dependency system. It's currently
mostly used for early boot, as for late boot we want people to rely on
implicit deps via socket activation. However the dep system is there,
and for embedded uses it might make sense to make a couple of late boot
deps explicit. With that information systemd would then be able to
deduce which units in the dep tree are "the most waited for". These
could then get an IO/CPU boost as long as they haven't fully started up
yet. We'd just iterate through the cgroup to bump their cpu/io sched
parms up, and bump them down when they signal that they started fully up
(i.e. in this case they'd have to use Type=notify instead of
Typo=simple).

So yeah, we already provide most of the tools for the finegrained
optimizing of the boot, except for the actual boosting of the perms and
the algorithm to determine "the most waited for" (which however is basic
graph theory), but that would probably be a patch of 100 lines or
less. I'd be happy to merge such a patch, but I'd like to ask everybody
looking into this, to do proper profiling first.

My guess is that this kind of microoptimization might help, but we have
a lot of other things to fix first (LVM, Ply, DDC, gdm, ...)

Lennart
--
Lennart Poettering - Red Hat, Inc.
Lennart Poettering
2011-05-09 20:21:07 UTC
Permalink
Post by Gustavo Sverzut Barbieri
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to
GDM/KDM...).
That's going to change in the F16 cycle hopefully. We'll spawn a
per-user instance for most logins, much like launchd.

Lennart
--
Lennart Poettering - Red Hat, Inc.
Gustavo Sverzut Barbieri
2011-05-09 21:25:12 UTC
Permalink
On Mon, May 9, 2011 at 5:21 PM, Lennart Poettering
Post by Lennart Poettering
Post by Gustavo Sverzut Barbieri
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to
GDM/KDM...).
That's going to change in the F16 cycle hopefully. We'll spawn a
per-user instance for most logins, much like launchd.
I know, but even then you're in another level of the tree as you're
executed from xsession (or similar) thus it would be chained on DM
itself.
--
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN: ***@gmail.com
Skype: gsbarbieri
Mobile: +55 (19) 9225-2202
Scott James Remnant
2011-05-10 02:52:56 UTC
Permalink
On Mon, May 9, 2011 at 12:16 PM, Gustavo Sverzut Barbieri <
Post by Gustavo Sverzut Barbieri
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to GDM/KDM...).
Really?
A major feature of systemd I understood was that it would also manage user
sessions as a seamless part of boot along with the system session. Is that
not the case?
But using your case, just like I said above, you really want that. You'd
chromium and X11 to be loaded at the same time, because the kernel will
handle the load and may load chromium executable while X11 is doing EDID
read, then may let the dynamic linker work while X11 asks for input device,
etc.
Or the kernel might load Chromium completely before giving X any time at
all; between picking the right thing and the wrong thing, without
information, the kernel seems to more usually pick the latter.

Scott
Gustavo Sverzut Barbieri
2011-05-10 14:41:11 UTC
Permalink
Post by Scott James Remnant
On Mon, May 9, 2011 at 12:16 PM, Gustavo Sverzut Barbieri <
Post by Gustavo Sverzut Barbieri
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to GDM/KDM...).
Really?
A major feature of systemd I understood was that it would also manage user
sessions as a seamless part of boot along with the system session. Is that
not the case?
Not yet. It is planned but we're trying to get the system-wide part done
first. The system part is what is in Fedora 15 and all other distros using
it.

The desktop part will need some investigation. Lennart does have some ideas,
but even he said he will need to analyze what exists now, some "why?" and
then do a ConsoleKit-free desktop, see if that's enough and get the actual
patches out to interested peers (Gnome in his case, I'll do stuff for
Enlightenment, etc).

In his conversion he will rework the concept of session. The session will
not be a shell login or X11 login, instead it will be the whole lifetime of
user in the platform, if you login from tty1/getty then x11 you have the
same session, then logout from tty1 your session will still alive until the
last logs out. This is likely to require some PAM help and will require
changes to DBus and maybe others.

I'm not following all the user-session discussion, but AFAIU this management
will be done by the same code, but not same process (it's not PID1). PAM (or
another entity that has a role in the login/logout) will talk to systemd
(PID1) and ask for a new service to be started to handle this user-session.
This will still be the same code, but of course on the user context and not
handle things like actual reboot/halt of the machine, mounting system
filesystems. But again, I'm not following it closely and I might be wrong
here.
Post by Scott James Remnant
But using your case, just like I said above, you really want that. You'd
chromium and X11 to be loaded at the same time, because the kernel will
handle the load and may load chromium executable while X11 is doing EDID
read, then may let the dynamic linker work while X11 asks for input device,
etc.
Or the kernel might load Chromium completely before giving X any time at
all; between picking the right thing and the wrong thing, without
information, the kernel seems to more usually pick the latter.
If that happens then not a big problem as on X connection Chromium would
block or would sleep while it's ready, leaving time for X11 to run.

Yes, this would not respect your "priority request", but I don't think the
kernel would starve X11 load in benefit for Chromium load for no good.
Either X11 was sleeping on some probe or something like that.
--
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN: ***@gmail.com
Skype: gsbarbieri
Mobile: +55 (19) 9225-2202
Scott James Remnant
2011-05-10 16:25:20 UTC
Permalink
On Tue, May 10, 2011 at 7:41 AM, Gustavo Sverzut Barbieri <
Post by Gustavo Sverzut Barbieri
I'm not following all the user-session discussion, but AFAIU this
management will be done by the same code, but not same process (it's not
PID1). PAM (or another entity that has a role in the login/logout) will talk
to systemd (PID1) and ask for a new service to be started to handle this
user-session. This will still be the same code, but of course on the user
context and not handle things like actual reboot/halt of the machine,
mounting system filesystems. But again, I'm not following it closely and I
might be wrong here.
That's a shame. Having it as a separate process would mean the user session
would be started all at once in one go at the end of the boot, rather than
with the rest of the boot :-(

Scott
Lennart Poettering
2011-05-10 16:39:10 UTC
Permalink
Post by Scott James Remnant
On Tue, May 10, 2011 at 7:41 AM, Gustavo Sverzut Barbieri <
Post by Gustavo Sverzut Barbieri
I'm not following all the user-session discussion, but AFAIU this
management will be done by the same code, but not same process (it's not
PID1). PAM (or another entity that has a role in the login/logout) will talk
to systemd (PID1) and ask for a new service to be started to handle this
user-session. This will still be the same code, but of course on the user
context and not handle things like actual reboot/halt of the machine,
mounting system filesystems. But again, I'm not following it closely and I
might be wrong here.
That's a shame. Having it as a separate process would mean the user session
would be started all at once in one go at the end of the boot, rather than
with the rest of the boot :-(
We have considered doing all of this in the same process, however
ultimiately stepped away from it for security reasons. Adopting a
PID1-for-everything scheme also means that you need to add a fairly
complex authorization and credentials system to systemd, because you
need to validate every single bus call and other operation and map it to
a specific user context.

Since the kernel and D-Bus are fairly good at doing access control and
privilege separation between users we hence decided to make the best of
it in systemd and went for the one-systemd-for-each-user scheme.

Note that you need to delay execution of user code after the base system
is set up anyway, in order to ensure that the right perms are set on the
volatile and other directories. That means having a single transaction
for both user and system services effectively wins you very little.

Lennart
--
Lennart Poettering - Red Hat, Inc.
Scott James Remnant
2011-05-12 15:55:30 UTC
Permalink
On Tue, May 10, 2011 at 9:39 AM, Lennart Poettering
Post by Lennart Poettering
Note that you need to delay execution of user code after the base system
is set up anyway, in order to ensure that the right perms are set on the
volatile and other directories. That means having a single transaction
for both user and system services effectively wins you very little.
Really, what permissions? The only volatile directories on the system are
all tmpfs these days so get passed the permissions in their mount options.

Scott
Lennart Poettering
2011-05-15 21:30:38 UTC
Permalink
Post by Scott James Remnant
On Tue, May 10, 2011 at 9:39 AM, Lennart Poettering
Post by Lennart Poettering
Note that you need to delay execution of user code after the base system
is set up anyway, in order to ensure that the right perms are set on the
volatile and other directories. That means having a single transaction
for both user and system services effectively wins you very little.
Really, what permissions? The only volatile directories on the system are
all tmpfs these days so get passed the permissions in their mount options.
Not quite. We have a couple of subdirs which need to be created with the
right perms and right users, for example /run/user/ (i.e. the
XDG_RUNTIME_DIR hierarchy).

Lennart
--
Lennart Poettering - Red Hat, Inc.
Lennart Poettering
2011-05-10 15:58:02 UTC
Permalink
Post by Scott James Remnant
On Mon, May 9, 2011 at 12:16 PM, Gustavo Sverzut Barbieri <
Post by Gustavo Sverzut Barbieri
Well, right now let's make clear that Firefox/Chromium or other X clients
won't be started by systemd as they are user session applications, and
systemd itself will just deal with system context (ie: up to GDM/KDM...).
Really?
A major feature of systemd I understood was that it would also manage user
sessions as a seamless part of boot along with the system session. Is that
not the case?
Right now you can run systemd as the user, and it will startup and work
fine. Which is very handy for debugging purposes. However, we have not
integrated that with D-Bus or GNOME or anything yet. This is something
we want to do for F16:

http://lwn.net/Articles/441328/

Lennart
--
Lennart Poettering - Red Hat, Inc.
Continue reading on narkive:
Loading...