Discussion:
SysVInit service migration to systemd
(too old to reply)
Lesley Kimmel
2015-06-26 13:15:40 UTC
Permalink
Hi all;

I've been working with RHEL5/6 for the past several years and have developed many init scripts/services which generally use lock files and PID files to allow for tracking of the service status. We are moving to RHEL7 (systemd) in the near future and I am looking for instruction or tutorials on how to effectively migrate these scripts to work with systemd.

I found https://wiki.archlinux.org/index.php/Systemd#Service_types which seems somewhat promising but it is fairly high-level. It looks like I may be able to use the 'forking' type with the 'pidfile' parameter to somewhat mimic what the scripts to today. However, I have a couple of questions:

-How does systemd track whether it should be stopping a service at shutdown (analogous to the /var/lock/subsys files in SysVInit)?
-Are there merits to using the notify or dbus types? If so does anyone know of a tutorial I could use to get to that point? (FYI, I'm not a developer but I learn complicated things quickly).
-If the current service logs to a custom, dedicated log is there a way to get systemd to provide the same type of output that it does for the built-in system services or must I make some modifications to integrate with journald?

Thanks ahead of time,
Lesley Kimmel, RHCE
Linux/Unix Systems Engineer
Lukáš Nykrýn
2015-06-26 13:58:35 UTC
Permalink
Lesley Kimmel píše v Pá 26. 06. 2015 v 08:15 -0500:
> Hi all;
>
> I've been working with RHEL5/6 for the past several years and have
> developed many init scripts/services which generally use lock files
> and PID files to allow for tracking of the service status. We are
> moving to RHEL7 (systemd) in the near future and I am looking for
> instruction or tutorials on how to effectively migrate these scripts
> to work with systemd.

http://0pointer.de/blog/projects/systemd-for-admins-3.html

>
> I found https://wiki.archlinux.org/index.php/Systemd#Service_types
> which seems somewhat promising but it is fairly high-level. It looks
> like I may be able to use the 'forking' type with the 'pidfile'
> parameter to somewhat mimic what the scripts to today. However, I have
> a couple of questions:
>
> -How does systemd track whether it should be stopping a service at
> shutdown (analogous to the /var/lock/subsys files in SysVInit)?

Systemd tracks the services using cgroups, so it knows that the service
is running and needs to be stopped.

> -Are there merits to using the notify or dbus types? If so does anyone
> know of a tutorial I could use to get to that point? (FYI, I'm not a
> developer but I learn complicated things quickly).
> -If the current service logs to a custom, dedicated log is there a way
> to get systemd to provide the same type of output that it does for the
> built-in system services or must I make some modifications to
> integrate with journald?

Look at the Administrators Blog Series and Developers Series
https://wiki.freedesktop.org/www/Software/systemd/

>
> Thanks ahead of time,
> Lesley Kimmel, RHCE
> Linux/Unix Systems Engineer
>
> _______________________________________________
> systemd-devel mailing list
> systemd-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Regards
Lukas
Lesley Kimmel
2015-06-26 15:02:41 UTC
Permalink
Thanks for the information. I've seen that blog before. Unfortunately, it only describes a starting a service that already has a good level of integration with some of the underlying infrastructure of systemd (e.g. dbus). Let me be a little more specific about what I'm trying to accomplish and see if anyone has any thoughts on how systemd could help (or impede me).

I have a collection of servers hosting many processes such as Apache HTTPD, a database, and Java application servers. Using init these servers would: a) need to be started in a specific order and b) take a long time to start. To improve both of these scenarios I created a Python service which took an XML configuration file describing the dependencies of the various components. The Python service is started by init and forks so as to not stop the boot process. The forked process then does some basic dependency checking (including remote tests, mostly telnet or pings) before starting local services using init scripts that are not configured to be started by init.

I'm wondering, with systemd, if this Python "control" daemon would be required at all. Does systemd have the ability to check the status of remote servers? I get the impression that dbus may have this type of functionality but I've never really dug into it before.

Lesley Kimmel, RHCE
Unix/Linux Systems Engineer

> Subject: Re: [systemd-devel] SysVInit service migration to systemd
> From: ***@redhat.com
> To: ***@hotmail.com
> CC: systemd-***@lists.freedesktop.org
> Date: Fri, 26 Jun 2015 15:58:35 +0200
>
> Lesley Kimmel pí¹e v Pá 26. 06. 2015 v 08:15 -0500:
> > Hi all;
> >
> > I've been working with RHEL5/6 for the past several years and have
> > developed many init scripts/services which generally use lock files
> > and PID files to allow for tracking of the service status. We are
> > moving to RHEL7 (systemd) in the near future and I am looking for
> > instruction or tutorials on how to effectively migrate these scripts
> > to work with systemd.
>
> http://0pointer.de/blog/projects/systemd-for-admins-3.html
>
> >
> > I found https://wiki.archlinux.org/index.php/Systemd#Service_types
> > which seems somewhat promising but it is fairly high-level. It looks
> > like I may be able to use the 'forking' type with the 'pidfile'
> > parameter to somewhat mimic what the scripts to today. However, I have
> > a couple of questions:
> >
> > -How does systemd track whether it should be stopping a service at
> > shutdown (analogous to the /var/lock/subsys files in SysVInit)?
>
> Systemd tracks the services using cgroups, so it knows that the service
> is running and needs to be stopped.
>
> > -Are there merits to using the notify or dbus types? If so does anyone
> > know of a tutorial I could use to get to that point? (FYI, I'm not a
> > developer but I learn complicated things quickly).
> > -If the current service logs to a custom, dedicated log is there a way
> > to get systemd to provide the same type of output that it does for the
> > built-in system services or must I make some modifications to
> > integrate with journald?
>
> Look at the Administrators Blog Series and Developers Series
> https://wiki.freedesktop.org/www/Software/systemd/
>
> >
> > Thanks ahead of time,
> > Lesley Kimmel, RHCE
> > Linux/Unix Systems Engineer
> >
> > _______________________________________________
> > systemd-devel mailing list
> > systemd-***@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
> Regards
> Lukas
>
>
Andrei Borzenkov
2015-06-26 16:14:36 UTC
Permalink
В Fri, 26 Jun 2015 10:02:41 -0500
Lesley Kimmel <***@hotmail.com> пишет:

> Thanks for the information. I've seen that blog before. Unfortunately, it only describes a starting a service that already has a good level of integration with some of the underlying infrastructure of systemd (e.g. dbus). Let me be a little more specific about what I'm trying to accomplish and see if anyone has any thoughts on how systemd could help (or impede me).
>
> I have a collection of servers hosting many processes such as Apache HTTPD, a database, and Java application servers. Using init these servers would: a) need to be started in a specific order and b) take a long time to start. To improve both of these scenarios I created a Python service which took an XML configuration file describing the dependencies of the various components. The Python service is started by init and forks so as to not stop the boot process. The forked process then does some basic dependency checking (including remote tests, mostly telnet or pings) before starting local services using init scripts that are not configured to be started by init.

That sounds exactly like what pacemaker does.

>
> I'm wondering, with systemd, if this Python "control" daemon would be required at all. Does systemd have the ability to check the status of remote servers?

No.
Lesley Kimmel
2015-07-06 18:18:18 UTC
Permalink
You mention 'pacemaker'. Does anyone know of any other open-source projects that might accomplish the type of remote service dependency checking that I'm trying to accomplish? For example, if service A on server A depends on service B on server B is there a project that makes this type of checking possible/easy?

> Date: Fri, 26 Jun 2015 19:14:36 +0300
> From: ***@gmail.com
> To: ***@hotmail.com
> CC: systemd-***@lists.freedesktop.org; ***@redhat.com
> Subject: Re: [systemd-devel] SysVInit service migration to systemd
>
> ÷ Fri, 26 Jun 2015 10:02:41 -0500
> Lesley Kimmel <***@hotmail.com> ÐÉÛÅÔ:
>
> > Thanks for the information. I've seen that blog before. Unfortunately, it only describes a starting a service that already has a good level of integration with some of the underlying infrastructure of systemd (e.g. dbus). Let me be a little more specific about what I'm trying to accomplish and see if anyone has any thoughts on how systemd could help (or impede me).
> >
> > I have a collection of servers hosting many processes such as Apache HTTPD, a database, and Java application servers. Using init these servers would: a) need to be started in a specific order and b) take a long time to start. To improve both of these scenarios I created a Python service which took an XML configuration file describing the dependencies of the various components. The Python service is started by init and forks so as to not stop the boot process. The forked process then does some basic dependency checking (including remote tests, mostly telnet or pings) before starting local services using init scripts that are not configured to be started by init.
>
> That sounds exactly like what pacemaker does.
>
> >
> > I'm wondering, with systemd, if this Python "control" daemon would be required at all. Does systemd have the ability to check the status of remote servers?
>
> No.
Graham Cantin
2015-07-06 19:22:56 UTC
Permalink
One I might point out is Spotify's "Luigi" python framework.
http://luigi.readthedocs.org/en/latest/index.html
At first glance, it might look like it's really hadoop specific -- and it
used to be, but the hadoop stuff's been scooped off to a contrib module.
Don't let that fool you though, it's actually quite generic and useful for
many things -- it's a lot more like GNU make where you describe the state
things should be in to proceed.

If you're stuck doing complex dependency control; expressing it to luigi is
far less of a headache.



On Mon, Jul 6, 2015 at 11:18 AM, Lesley Kimmel <***@hotmail.com>
wrote:

> You mention 'pacemaker'. Does anyone know of any other open-source
> projects that might accomplish the type of remote service dependency
> checking that I'm trying to accomplish? For example, if service A on
> server A depends on service B on server B is there a project that makes
> this type of checking possible/easy?
>
> > Date: Fri, 26 Jun 2015 19:14:36 +0300
> > From: ***@gmail.com
> > To: ***@hotmail.com
> > CC: systemd-***@lists.freedesktop.org; ***@redhat.com
> > Subject: Re: [systemd-devel] SysVInit service migration to systemd
> >
> > В Fri, 26 Jun 2015 10:02:41 -0500
> > Lesley Kimmel <***@hotmail.com> пОшет:
> >
> > > Thanks for the information. I've seen that blog before. Unfortunately,
> it only describes a starting a service that already has a good level of
> integration with some of the underlying infrastructure of systemd (e.g.
> dbus). Let me be a little more specific about what I'm trying to accomplish
> and see if anyone has any thoughts on how systemd could help (or impede me).
> > >
> > > I have a collection of servers hosting many processes such as Apache
> HTTPD, a database, and Java application servers. Using init these servers
> would: a) need to be started in a specific order and b) take a long time to
> start. To improve both of these scenarios I created a Python service which
> took an XML configuration file describing the dependencies of the various
> components. The Python service is started by init and forks so as to not
> stop the boot process. The forked process then does some basic dependency
> checking (including remote tests, mostly telnet or pings) before starting
> local services using init scripts that are not configured to be started by
> init.
> >
> > That sounds exactly like what pacemaker does.
> >
> > >
> > > I'm wondering, with systemd, if this Python "control" daemon would be
> required at all. Does systemd have the ability to check the status of
> remote servers?
> >
> > No.
>
> _______________________________________________
> systemd-devel mailing list
> systemd-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
>


--
Graham Cantin | (408) 890-7463
Mantas Mikulėnas
2015-06-26 14:00:34 UTC
Permalink
On Fri, Jun 26, 2015 at 4:15 PM, Lesley Kimmel <***@hotmail.com>
wrote:

> Hi all;
>
> I've been working with RHEL5/6 for the past several years and have
> developed many init scripts/services which generally use lock files and PID
> files to allow for tracking of the service status. We are moving to RHEL7
> (systemd) in the near future and I am looking for instruction or tutorials
> on how to effectively migrate these scripts to work with systemd.
>
> I found https://wiki.archlinux.org/index.php/Systemd#Service_types which
> seems somewhat promising but it is fairly high-level. It looks like I may
> be able to use the 'forking' type with the 'pidfile' parameter to somewhat
> mimic what the scripts to today. However, I have a couple of questions:
>

Usually systemd detects the main process even without PIDFile, using
cgroups to keep track.


> -How does systemd track whether it should be stopping a service at
> shutdown (analogous to the /var/lock/subsys files in SysVInit)?
>

All service control requests go through the always-running init process, so
it can just keep track of service status internally without having to write
it out.

[This also means services run in a clean environment and can't ask for
interactive input when starting. Upstart and even /etc/inittab also work
the same way.]

Systemd also monitors whether the service processes are actually running –
by putting each into a separate cgroup – so it generally avoids issues such
as stale pidfiles.


> -Are there merits to using the notify or dbus types? If so does anyone
> know of a tutorial I could use to get to that point? (FYI, I'm not a
> developer but I learn complicated things quickly).
>

Similar to Type=forking, they both let systemd know whether the service is
ready or still initializing (which can't be done with the default
Type=simple).

Type=dbus might be best for creating DBus-activatable services compatible
with regular dbus-daemon, since it has mostly the same expectations as
dbus-daemon: the service doesn't fork, and is considered ready as soon as
it finally claims the configured BusName.

Type=notify is somewhat more lightweight – the service just needs to send
READY=1 over a Unix socket. (The service can also show some additional
information in `systemctl status` – e.g. lldpd could send "STATUS=No
neighbors"). libsystemd has sd_notify(3)
<http://www.freedesktop.org/software/systemd/man/sd_notify.html> for this,
though it could be done using regular socket API.

I suppose the same notify socket could be exposed to all other service
types. I don't know why it isn't yet.


> -If the current service logs to a custom, dedicated log is there a way to
> get systemd to provide the same type of output that it does for the
> built-in system services or must I make some modifications to integrate
> with journald?
>

`systemctl status` only reads from the journal.

The journal, however, also receives messages sent using the traditional
syslog API or written to the kernel log – in addition to its native
sd_journal_send(3)
<http://www.freedesktop.org/software/systemd/man/sd_journal_send.html> – so
that already covers most services.

You could also use rsyslogd to feed messages from custom text logs into the
journal, using its omjournal module.

--
Mantas Mikulėnas <***@gmail.com>
David Timothy Strauss
2015-06-26 21:54:06 UTC
Permalink
On Fri, Jun 26, 2015 at 7:00 AM Mantas Mikulėnas <***@gmail.com> wrote:

> On Fri, Jun 26, 2015 at 4:15 PM, Lesley Kimmel <***@hotmail.com>
> wrote:
>
>> Hi all;
>>
>> I've been working with RHEL5/6 for the past several years and have
>> developed many init scripts/services which generally use lock files and PID
>> files to allow for tracking of the service status. We are moving to RHEL7
>> (systemd) in the near future and I am looking for instruction or tutorials
>> on how to effectively migrate these scripts to work with systemd.
>>
>> I found https://wiki.archlinux.org/index.php/Systemd#Service_types which
>> seems somewhat promising but it is fairly high-level. It looks like I may
>> be able to use the 'forking' type with the 'pidfile' parameter to somewhat
>> mimic what the scripts to today. However, I have a couple of questions:
>>
>
> Usually systemd detects the main process even without PIDFile, using
> cgroups to keep track.
>

Explicitly setting the PIDFIle is always the superior option, though. It
removes ambiguity, ensures more accurate service startup completion
detection, and (usually) ensures that systemd is definitely sending signals
to the right PID for modes like "mixed."
Jonathan de Boyne Pollard
2015-06-28 13:29:40 UTC
Permalink
Lesley Kimmel:
> I've been working with RHEL5/6 for the past several years and have
> developed many init scripts/services which generally use lock files
> and PID files to allow for tracking of the service status. We are
> moving to RHEL7 (systemd) in the near future and I am looking for
> instruction or tutorials on how to effectively migrate these scripts
> to work with systemd. [...] It looks like I may be able to use the
> 'forking' type with the 'pidfile' parameter to somewhat mimic what the
> scripts to today.

You don't do such migration. You understand how your daemons are
started, and you write service (and other) units to describe that. You
do not start with the assumption that you migrate straight from one to
the other, particularly if your existing mechanisms involve the
dangerous and rickety things (e.g. PID files) that proper service
management is designed to avoid in the first place, or things such as
"subsystem" lock files which proper service management has no need of by
its very nature. "Type=forking" specifies a quite specific readiness
protocol that your daemon has to enact, lest it be regarded as failed.
It's isn't a catch-all for anything that might fork in any possible
way. And service managers know, by dint of simply remembering, what
processes they started and whether they've already started them.

I've been collecting case studies of people who have migrated to systemd
exceedingly badly, and constructed some quite horrendous systems,
because they've done so without either consideration of, or
understanding of, how their sytems actually work. Here's one candidate
that I have yet to add, because I'm hoping that now they've been told
that they are going wrong they will correct themselves, whose errors are
good to learn from.

There's a computer game called "ARK: Survival Evolved". It's daemon is
a program named ShooterGameServer. To run this program continually as a
daemon, someone named Federico Zivolo and a couple of other unidentified
people came up with the somewhat bizarre idea of running it under
screen, and using screen's commands to send its pseudo-terminal an
interrupt character, in order to send a SIGINT to the daemon when it
came time to shut it down. They wrapped this up in a System 5 rc
script, taking the conventional "start" and "stop" verbs, named
"arkmanager". Its prevent-multiple-instances system was not lock files,
but grepping the process table.

Wrapped around this they put another System 5 rc script, named
"arkdaemon", which also took the conventional "start" and "stop" verbs,
and which treated the wrapper "arkmanager" script as the daemon,
recording the process ID of the shell interpreter for the "arkmanager"
script in a PID file, as if it were the actual daemon's process ID. It
also did various other bad things that proper service managers
eliminate, including grepping the process table (again), abusing su to
drop privileges, using arbitrary 5 second sleeps to make the timing
almost work, and (sic) hardwired ECMA-48 SGR sequences to change the
colour of output that isn't going to a terminal in the first place.

Then they wrote a systemd service unit file, "arkdeamon.service" (sic),
that did this:

> ExecStart=/etc/init.d/arkdaemon start
> ExecStop=/etc/init.d/arkdaemon stop
> Type=forking

A couple of days ago, I pointed out the errors of even starting down
this route to them, and wrote a systemd unit file for them that actually
starts the daemon directly from systemd; no PID files, abuses of su,
racy grepping of the process table, abritrary sleeps, or incorrect
"forking" readiness protocol necessary. Set the account with User= ;
set the environment with Environment= ; and run the program with
ExecStart=/home/steam/ARK/ShooterGame/Binaries/Linux/ShooterGameServer
plus some options. Other stuff, like software upgrades, one then builds
on top of the service management layer, not beneath it. I also pointed
out to them that I had been beaten in this regard by some 4 days by the
(again unfortunately unidentified) people who wrote
http://ark.gamepedia.com/Dedicated_Server_Setup#Automatic_Startup ,
whose service unit was pretty much the same as the one that I had
independently written (as one would expect, indeed).

If you understand how your daemon is run in the first place, then you
end up with what I and the gamepedia people did. If you just think that
you wrap what you already have, using "forking" to imperfectly paper
over the cracks, you oftentimes end up with an utterly horrendous mess.
Lesley Kimmel
2015-06-29 13:58:11 UTC
Permalink
Jonathan;

Thanks for the background and information. Since you clearly seem to have a grasp of systemd please humour me with a few more questions (some of them slightly ignorant):

a) Why are PID bad?
b) Why are lock files bad?
c) If a/b are so bad why did they persist for so many years in SysVInit?
d) Generically, how would you prescribe to use systemd to start Java processes (for Java application servers) that are typically started from a set of relatively complex scripts that are used to set up the environment before launching the Java process? It seems that you are advocating to call, as directly as possible, the target service/daemon. However, some things don't seem so straight-forward.

Thanks again!
Lesley Kimmel, RHCE
Unix/Linux Systems Engineer

> Date: Sun, 28 Jun 2015 14:29:40 +0100
> From: J.deBoynePollard-***@NTLWorld.com
> To: systemd-***@lists.freedesktop.org
> Subject: Re: [systemd-devel] SysVInit service migration to systemd
>
> Lesley Kimmel:
> > I've been working with RHEL5/6 for the past several years and have
> > developed many init scripts/services which generally use lock files
> > and PID files to allow for tracking of the service status. We are
> > moving to RHEL7 (systemd) in the near future and I am looking for
> > instruction or tutorials on how to effectively migrate these scripts
> > to work with systemd. [...] It looks like I may be able to use the
> > 'forking' type with the 'pidfile' parameter to somewhat mimic what the
> > scripts to today.
>
> You don't do such migration. You understand how your daemons are
> started, and you write service (and other) units to describe that. You
> do not start with the assumption that you migrate straight from one to
> the other, particularly if your existing mechanisms involve the
> dangerous and rickety things (e.g. PID files) that proper service
> management is designed to avoid in the first place, or things such as
> "subsystem" lock files which proper service management has no need of by
> its very nature. "Type=forking" specifies a quite specific readiness
> protocol that your daemon has to enact, lest it be regarded as failed.
> It's isn't a catch-all for anything that might fork in any possible
> way. And service managers know, by dint of simply remembering, what
> processes they started and whether they've already started them.
>
> I've been collecting case studies of people who have migrated to systemd
> exceedingly badly, and constructed some quite horrendous systems,
> because they've done so without either consideration of, or
> understanding of, how their sytems actually work. Here's one candidate
> that I have yet to add, because I'm hoping that now they've been told
> that they are going wrong they will correct themselves, whose errors are
> good to learn from.
>
> There's a computer game called "ARK: Survival Evolved". It's daemon is
> a program named ShooterGameServer. To run this program continually as a
> daemon, someone named Federico Zivolo and a couple of other unidentified
> people came up with the somewhat bizarre idea of running it under
> screen, and using screen's commands to send its pseudo-terminal an
> interrupt character, in order to send a SIGINT to the daemon when it
> came time to shut it down. They wrapped this up in a System 5 rc
> script, taking the conventional "start" and "stop" verbs, named
> "arkmanager". Its prevent-multiple-instances system was not lock files,
> but grepping the process table.
>
> Wrapped around this they put another System 5 rc script, named
> "arkdaemon", which also took the conventional "start" and "stop" verbs,
> and which treated the wrapper "arkmanager" script as the daemon,
> recording the process ID of the shell interpreter for the "arkmanager"
> script in a PID file, as if it were the actual daemon's process ID. It
> also did various other bad things that proper service managers
> eliminate, including grepping the process table (again), abusing su to
> drop privileges, using arbitrary 5 second sleeps to make the timing
> almost work, and (sic) hardwired ECMA-48 SGR sequences to change the
> colour of output that isn't going to a terminal in the first place.
>
> Then they wrote a systemd service unit file, "arkdeamon.service" (sic),
> that did this:
>
> > ExecStart=/etc/init.d/arkdaemon start
> > ExecStop=/etc/init.d/arkdaemon stop
> > Type=forking
>
> A couple of days ago, I pointed out the errors of even starting down
> this route to them, and wrote a systemd unit file for them that actually
> starts the daemon directly from systemd; no PID files, abuses of su,
> racy grepping of the process table, abritrary sleeps, or incorrect
> "forking" readiness protocol necessary. Set the account with User= ;
> set the environment with Environment= ; and run the program with
> ExecStart=/home/steam/ARK/ShooterGame/Binaries/Linux/ShooterGameServer
> plus some options. Other stuff, like software upgrades, one then builds
> on top of the service management layer, not beneath it. I also pointed
> out to them that I had been beaten in this regard by some 4 days by the
> (again unfortunately unidentified) people who wrote
> http://ark.gamepedia.com/Dedicated_Server_Setup#Automatic_Startup ,
> whose service unit was pretty much the same as the one that I had
> independently written (as one would expect, indeed).
>
> If you understand how your daemon is run in the first place, then you
> end up with what I and the gamepedia people did. If you just think that
> you wrap what you already have, using "forking" to imperfectly paper
> over the cracks, you oftentimes end up with an utterly horrendous mess.
> _______________________________________________
> systemd-devel mailing list
> systemd-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Mantas Mikulėnas
2015-06-29 14:50:36 UTC
Permalink
On Jun 29, 2015 16:58, "Lesley Kimmel" <***@hotmail.com> wrote:
>
> Jonathan;
>
> Thanks for the background and information. Since you clearly seem to have
a grasp of systemd please humour me with a few more questions (some of them
slightly ignorant):
>
> a) Why are PID bad?
> b) Why are lock files bad?
> c) If a/b are so bad why did they persist for so many years in SysVInit?

init itself *doesn't need* either, though it only supports non-forking
services, and in practice the only service it ever manages is agetty. (And
sometimes xdm.)

But the init.d scripts, they're just she'll scripts which aren't capable of
any persistent monitoring. They cannot rely on pid1 to do any monitoring.
So pidfiles and lockfiles are the only (or at least the simplest) way for
these scripts keep state from "init.d/foo start" across to "init.d/foo
stop".

Personally I would say they persisted in sysvinit because they pretty much
/define/ the sysvinit design (i.e. pid1 that does almost nothing, and
short-lived external scripts).

Though there's also openrc, which also runs on top of sysv pid1, but that's
really all I know about it. I wonder how it tracks processes. I heard it
uses cgroups?
Andrei Borzenkov
2015-06-29 16:09:03 UTC
Permalink
В Mon, 29 Jun 2015 08:58:11 -0500
Lesley Kimmel <***@hotmail.com> пишет:

> Jonathan;
>
> Thanks for the background and information. Since you clearly seem to have a grasp of systemd please humour me with a few more questions (some of them slightly ignorant):
>
> a) Why are PID bad?

PID files as the primary way to track running service are too
unreliable - it is easy for a service to die without removing PID file.
PID file as a mean to track service startup the way systemd does it is
IMHO just fine. Actually it is the only way for a service to announce
its readiness without adding explicit systemd support (Type=notify).

> b) Why are lock files bad?
> c) If a/b are so bad why did they persist for so many years in SysVInit?
> d) Generically, how would you prescribe to use systemd to start Java processes (for Java application servers) that are typically started from a set of relatively complex scripts that are used to set up the environment before launching the Java process? It seems that you are advocating to call, as directly as possible, the target service/daemon. However, some things don't seem so straight-forward.
>
> Thanks again!
> Lesley Kimmel, RHCE
> Unix/Linux Systems Engineer
>
> > Date: Sun, 28 Jun 2015 14:29:40 +0100
> > From: J.deBoynePollard-***@NTLWorld.com
> > To: systemd-***@lists.freedesktop.org
> > Subject: Re: [systemd-devel] SysVInit service migration to systemd
> >
> > Lesley Kimmel:
> > > I've been working with RHEL5/6 for the past several years and have
> > > developed many init scripts/services which generally use lock files
> > > and PID files to allow for tracking of the service status. We are
> > > moving to RHEL7 (systemd) in the near future and I am looking for
> > > instruction or tutorials on how to effectively migrate these scripts
> > > to work with systemd. [...] It looks like I may be able to use the
> > > 'forking' type with the 'pidfile' parameter to somewhat mimic what the
> > > scripts to today.
> >
> > You don't do such migration. You understand how your daemons are
> > started, and you write service (and other) units to describe that. You
> > do not start with the assumption that you migrate straight from one to
> > the other, particularly if your existing mechanisms involve the
> > dangerous and rickety things (e.g. PID files) that proper service
> > management is designed to avoid in the first place, or things such as
> > "subsystem" lock files which proper service management has no need of by
> > its very nature. "Type=forking" specifies a quite specific readiness
> > protocol that your daemon has to enact, lest it be regarded as failed.
> > It's isn't a catch-all for anything that might fork in any possible
> > way. And service managers know, by dint of simply remembering, what
> > processes they started and whether they've already started them.
> >
> > I've been collecting case studies of people who have migrated to systemd
> > exceedingly badly, and constructed some quite horrendous systems,
> > because they've done so without either consideration of, or
> > understanding of, how their sytems actually work. Here's one candidate
> > that I have yet to add, because I'm hoping that now they've been told
> > that they are going wrong they will correct themselves, whose errors are
> > good to learn from.
> >
> > There's a computer game called "ARK: Survival Evolved". It's daemon is
> > a program named ShooterGameServer. To run this program continually as a
> > daemon, someone named Federico Zivolo and a couple of other unidentified
> > people came up with the somewhat bizarre idea of running it under
> > screen, and using screen's commands to send its pseudo-terminal an
> > interrupt character, in order to send a SIGINT to the daemon when it
> > came time to shut it down. They wrapped this up in a System 5 rc
> > script, taking the conventional "start" and "stop" verbs, named
> > "arkmanager". Its prevent-multiple-instances system was not lock files,
> > but grepping the process table.
> >
> > Wrapped around this they put another System 5 rc script, named
> > "arkdaemon", which also took the conventional "start" and "stop" verbs,
> > and which treated the wrapper "arkmanager" script as the daemon,
> > recording the process ID of the shell interpreter for the "arkmanager"
> > script in a PID file, as if it were the actual daemon's process ID. It
> > also did various other bad things that proper service managers
> > eliminate, including grepping the process table (again), abusing su to
> > drop privileges, using arbitrary 5 second sleeps to make the timing
> > almost work, and (sic) hardwired ECMA-48 SGR sequences to change the
> > colour of output that isn't going to a terminal in the first place.
> >
> > Then they wrote a systemd service unit file, "arkdeamon.service" (sic),
> > that did this:
> >
> > > ExecStart=/etc/init.d/arkdaemon start
> > > ExecStop=/etc/init.d/arkdaemon stop
> > > Type=forking
> >
> > A couple of days ago, I pointed out the errors of even starting down
> > this route to them, and wrote a systemd unit file for them that actually
> > starts the daemon directly from systemd; no PID files, abuses of su,
> > racy grepping of the process table, abritrary sleeps, or incorrect
> > "forking" readiness protocol necessary. Set the account with User= ;
> > set the environment with Environment= ; and run the program with
> > ExecStart=/home/steam/ARK/ShooterGame/Binaries/Linux/ShooterGameServer
> > plus some options. Other stuff, like software upgrades, one then builds
> > on top of the service management layer, not beneath it. I also pointed
> > out to them that I had been beaten in this regard by some 4 days by the
> > (again unfortunately unidentified) people who wrote
> > http://ark.gamepedia.com/Dedicated_Server_Setup#Automatic_Startup ,
> > whose service unit was pretty much the same as the one that I had
> > independently written (as one would expect, indeed).
> >
> > If you understand how your daemon is run in the first place, then you
> > end up with what I and the gamepedia people did. If you just think that
> > you wrap what you already have, using "forking" to imperfectly paper
> > over the cracks, you oftentimes end up with an utterly horrendous mess.
> > _______________________________________________
> > systemd-devel mailing list
> > systemd-***@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
Reindl Harald
2015-06-29 14:52:38 UTC
Permalink
Am 29.06.2015 um 15:58 schrieb Lesley Kimmel:
> Jonathan;
>
> Thanks for the background and information. Since you clearly seem to
> have a grasp of systemd please humour me with a few more questions (some
> of them slightly ignorant):
>
> a) Why are PID bad?

what are they good for when the supervisor knows the PID to monitor?

> b) Why are lock files bad?

what are they good for when the supervisor knows the state of each service

> c) If a/b are so bad why did they persist for so many years in SysVInit?

because SysVInit did have no other way to know the PID or if a service
is running and things like lockfiles are far away from safe when the
application crashs and don't remove them - often enough that a restart
of services failed because of that

> d) Generically, how would you prescribe to use systemd to start Java
> processes (for Java application servers) that are typically started from
> a set of relatively complex scripts that are used to set up the
> environment before launching the Java process? It seems that you are
> advocating to call, as directly as possible, the target service/daemon.
> However, some things don't seem so straight-forward.

most complexity in that scripts is because the way SysVInit worked

a daemon should read each configuration files and not need more than the
argument of the config file as param
Cristian Rodríguez
2015-06-30 03:21:29 UTC
Permalink
On Mon, Jun 29, 2015 at 10:58 AM, Lesley Kimmel <***@hotmail.com> wrote:
> Jonathan;
>
> Thanks for the background and information. Since you clearly seem to have a
> grasp of systemd please humour me with a few more questions (some of them
> slightly ignorant):
>
> a) Why are PID bad?

Because they pretend to work but they really don't.
This is because only a tiny portion of software implements pid file
creation correctly,
this is in part due to the lack of a FREEBSD-like pidfile_*()
interface that at least tries to be correct.

> b) Why are lock files bad?

Mostly because at least till the *very recent* advent of File-private
POSIX locks (un-POSIX locks)
the OS facilities were terrible.

> c) If a/b are so bad why did they persist for so many years in SysVInit?

Because sysvinit is unable to track processes, in that case you need
at least to know what is the PID of the deamon, in order to be able to
kill it:
Continue reading on narkive:
Loading...