Discussion:
Is this list still active? Where can I get basic help with systemd
(too old to reply)
Doug Snyder
2017-12-03 01:40:33 UTC
Permalink
Raw Message
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and I
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd is
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.


My previous email:

I am trying to create a systemd service to manage the celery task queue

My service file is :
[Unit]
Description=Celery bf Service
After=network.target

[Service]
Type=forking
User=doug
Group=doug
WorkingDirectory=/home/doug/www/my_project
ExecStart=/home/doug/.virtualenvs/bf-env/bin/celery -A tasks worker
--app=tasks.celery_app:app --logfile=/var/log/celery/bf/%n%I.log
--loglevel=INFO -Ofair --time-limit=300

When I run:
$ systemctl start celery_bf.service
I get:
Job for celery_bf.service failed because a configured resource limit was
exceeded. See "systemctl status celery_bf.service" and "journalctl -xe" for
details.

$ systemctl status celery_bf.service
● celery_bf.service - Celery bf Service
Loaded: loaded (/etc/systemd/system/celery_bf.service; enabled; vendor
preset: disabled)
Active: failed (Result: resources)

Nov 20 13:57:20 stower systemd[1]: celery_bf.service failed.
Nov 20 13:57:20 stower systemd[1]: Starting Celery bf Service...
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run 'start'
task: Operation not supported
Nov 20 13:58:33 stower systemd[1]: Failed to start Celery bf Service.
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed.
Nov 20 13:58:33 stower systemd[1]: Starting Celery bf Service...
Nov 20 14:06:03 stower systemd[1]: celery_bf.service failed to run 'start'
task: Operation not supported
Nov 20 14:06:03 stower systemd[1]: Failed to start Celery bf Service.
Nov 20 14:06:03 stower systemd[1]: celery_bf.service failed.
Nov 20 14:06:03 stower systemd[1]: Starting Celery bf Service...

$ journalctl -xe
Nov 20 14:05:12 stower polkitd[29785]: Unregistered Authentication Agent
for unix-process:11449:78569047 (system bus name :1.1184, object path
/org/freedesktop
Nov 20 14:06:00 stower polkitd[29785]: Registered Authentication Agent for
unix-process:11476:78574149 (system bus name :1.1187 [/usr/bin/pkttyagent
--notify-f
Nov 20 14:06:03 stower polkitd[29785]: Operator of
unix-process:11476:78574149 successfully authenticated as unix-user:doug to
gain ONE-SHOT authorization for
Nov 20 14:06:03 stower systemd[1]: celery_bf.service failed to run 'start'
task: Operation not supported
Nov 20 14:06:03 stower systemd[1]: Failed to start Celery bf Service.
-- Subject: Unit celery_bf.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit celery_bf.service has failed.
Andrei Borzenkov
2017-12-03 07:00:05 UTC
Permalink
Raw Message
Post by Doug Snyder
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and I
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd is
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.
I am trying to create a systemd service to manage the celery task queue
[Unit]
Description=Celery bf Service
After=network.target
[Service]
Type=forking
User=doug
Group=doug
WorkingDirectory=/home/doug/www/my_project
ExecStart=/home/doug/.virtualenvs/bf-env/bin/celery -A tasks worker
--app=tasks.celery_app:app --logfile=/var/log/celery/bf/%n%I.log
--loglevel=INFO -Ofair --time-limit=300
$ systemctl start celery_bf.service
Job for celery_bf.service failed because a configured resource limit was
This cryptic message was changed more than a year ago which suggests you
are running older distribution release.
Post by Doug Snyder
exceeded. See "systemctl status celery_bf.service" and "journalctl -xe" for
details.
$ systemctl status celery_bf.service
● celery_bf.service - Celery bf Service
Loaded: loaded (/etc/systemd/system/celery_bf.service; enabled; vendor
preset: disabled)
Active: failed (Result: resources)
Nov 20 13:57:20 stower systemd[1]: celery_bf.service failed.
Nov 20 13:57:20 stower systemd[1]: Starting Celery bf Service...
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run 'start'
task: Operation not supported
This is really catch-all message. Try to enable systemd debug output, it
may provide more information; anything in application logs; anything in
journal around this event (you show only messages directly related to
service, but execution may be blocked by apparmor, selinux etc).
Lennart Poettering
2017-12-05 14:36:51 UTC
Permalink
Raw Message
Post by Doug Snyder
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and I
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd is
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.
Please always indicate which systemd version you are using, which
distribution and which kernel.
Post by Doug Snyder
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run 'start'
task: Operation not supported
Hmm, this indicates that some kernel functionality required is not
available in the kernel. I am not sure which one that is, and logging
is a bit too brief for that.

If you run "systemd-analyze set-log-level debug", and run this again,
what do you see then?

Lennart
--
Lennart Poettering, Red Hat
Doug Snyder
2017-12-05 16:17:32 UTC
Permalink
Raw Message
Also I have turned off SELinux for now, so that shouldn't be the issue
Post by Lennart Poettering
Post by Doug Snyder
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and I
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd is
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.
Please always indicate which systemd version you are using, which
distribution and which kernel.
Post by Doug Snyder
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run
'start'
Post by Doug Snyder
task: Operation not supported
Hmm, this indicates that some kernel functionality required is not
available in the kernel. I am not sure which one that is, and logging
is a bit too brief for that.
If you run "systemd-analyze set-log-level debug", and run this again,
what do you see then?
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-12-05 17:57:02 UTC
Permalink
Raw Message
On Di, 05.12.17 11:10, Doug Snyder (***@gmail.com) wrote:

Please keep discussions like this on the mailing list. Thanks!
(Readded the CC)

So, I think I figured it out: your service is not a template is it? If
so, you can't use %I. if you drop that, does it work then? (and make
sure to issue systemctl daemon-reload after editing the file).

But yeah, we should generate more useful messages in such a case (but
I am pretty sure we already do)
I'm using a CentOs-7 x86 minimal install ( I have updated yum )
https://www.freedesktop.org/software/systemd/man/systemd.html
It doesn't work. There's no systemd command to get the version from
Hmm, yeah, we need to update that man page to specify the full
path. Prepped a fix for that here:

https://github.com/systemd/systemd/pull/7550
BOOT_IMAGE=/vmlinuz-3.10.0-693.5.2.el7.x86_64
root=/dev/mapper/cl_stower-root ro crashkernel=auto rd.lvm.lv=cl_stower/root
rd.lvm.lv=cl_stower/swap rhgb quiet LANG=en_US.UTF-8
https://freedesktop.org/wiki/Software/systemd/Debugging/#index3h1
$ /usr/bin/systemd --test --system --log-level=debug > systemd-test.txt 2>&1
-bash: /usr/bin/systemd: No such file or directory
Oh, uh. That path is out of date. It should be /usr/lib/systemd. Fixed
that now.

Lennart
--
Lennart Poettering, Red Hat
Doug Snyder
2017-12-05 18:04:32 UTC
Permalink
Raw Message
I'm trying to get you the debug output as described in
https://freedesktop.org/wiki/Software/systemd/Debugging/#index3h1
I'm sending the emails but they aren't showing up in the email list at:
https://lists.freedesktop.org/archives/systemd-devel/2017-December/thread.html#39934
with or without email attachments

At this point two people have said I have an updated systemd which is
strange because I thought I used the latest CentOS minimal install
$ systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
+GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
So maybe it makes sense for me to just update and see if that resolves this.

I don't know how to do that though.
I found these instruciton but they don't work anymore:
https://www.certdepot.net/new-systemd-version/
Post by Lennart Poettering
Post by Doug Snyder
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and I
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd is
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.
Please always indicate which systemd version you are using, which
distribution and which kernel.
Post by Doug Snyder
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run
'start'
Post by Doug Snyder
task: Operation not supported
Hmm, this indicates that some kernel functionality required is not
available in the kernel. I am not sure which one that is, and logging
is a bit too brief for that.
If you run "systemd-analyze set-log-level debug", and run this again,
what do you see then?
Lennart
--
Lennart Poettering, Red Hat
Reindl Harald
2017-12-05 18:10:19 UTC
Permalink
Raw Message
Post by Doug Snyder
At this point two people have said I have an updated systemd which is
strange because I thought I used the latest CentOS minimal install
$ systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
+LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS
+KMOD +IDN
as expected on RHEL7 currently

rpm -q systemd
systemd-219-42.el7_4.4.x86_64
Post by Doug Snyder
So maybe it makes sense for me to just update and see if that resolves this.
I don't know how to do that though.
https://www.certdepot.net/new-systemd-version/
you shouldn't on a some years old distribution with a ton of other not
recent software on which systemd depends - that's the price you pay for
a LTS distribution and expect for 3 machines out auf 25 hence i prefer
to pay the price of Fedora nd two dist-upgrades per year
Doug Snyder
2017-12-05 18:33:26 UTC
Permalink
Raw Message
Getting rid of the template syntax changed the behavior but now it appears
there's another problem I don't understand.
it tries to start the unit and it shows some encouraging signs but fails
because a timeout was exceeded.
Celery starts and shows healthy output but then sends a SIGCHLD and then
dies
I don't know if you know what celery is but its a task queue so it does
create child processes
I thought the Type=forking line in the service unit would let systemd about
that.
Or maybe its dying for some other reason?
It works fine when run without systemd
I don't think I can send you detailed output until we get this issue
resolved with the mailing list and why its not excepting emails with debug
output.
Short emails seem to post. Long ones don't. There's no indication of whats
going on
I'll try including the journalctl -xe output and see if that crashes the
email list:

Dec 05 13:20:30 stower celery[5675]: ---- **** -----
Dec 05 13:20:30 stower celery[5675]: --- * *** * --
Linux-3.10.0-693.5.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core 2017-12-05
13:19:02
Dec 05 13:20:30 stower celery[5675]: -- * - **** ---
Dec 05 13:20:30 stower celery[5675]: - ** ---------- [config]
Dec 05 13:20:30 stower celery[5675]: - ** ---------- .> app:
tasks:0x1e15c90
Dec 05 13:20:30 stower celery[5675]: - ** ---------- .> transport:
amqp://guest:**@localhost:5672//
Dec 05 13:20:30 stower celery[5675]: - ** ---------- .> results: rpc://
Dec 05 13:20:30 stower celery[5675]: - *** --- * --- .> concurrency: 2
(prefork)
Dec 05 13:20:30 stower celery[5675]: -- ******* ---- .> task events: OFF
(enable -E to monitor tasks in this worker)
Dec 05 13:20:30 stower celery[5675]: --- ***** -----
Dec 05 13:20:30 stower celery[5675]: -------------- [queues]
Dec 05 13:20:30 stower celery[5675]: .> celery
exchange=celery(direct) key=celery
Dec 05 13:20:30 stower celery[5675]: [tasks]
Dec 05 13:20:30 stower celery[5675]: . tasks.test.add
Dec 05 13:20:30 stower celery[5675]: . tasks.twt_api.tasks.twt_followers_ids
Dec 05 13:20:30 stower celery[5675]: . tasks.twt_api.tasks.twt_friends_ids
Dec 05 13:20:31 stower systemd[1]: Received SIGCHLD from PID 5675 (celery).
Dec 05 13:20:31 stower systemd[1]: Child 5675 (celery) died (code=exited,
status=0/SUCCESS)
Dec 05 13:20:31 stower systemd[1]: Child 5675 belongs to celery_bf.service
Dec 05 13:20:31 stower systemd[1]: celery_bf.service: control process
exited, code=exited status=0
Dec 05 13:20:31 stower polkitd[684]: Unregistered Authentication Agent for
unix-process:5659:1185247 (system bus name :1.53, object path
/org/freedesktop/Polic
Dec 05 13:20:31 stower systemd[1]: celery_bf.service got final SIGCHLD for
state final-sigterm
Dec 05 13:20:31 stower systemd[1]: celery_bf.service changed final-sigterm
-> failed
Dec 05 13:20:31 stower systemd[1]: Job celery_bf.service/start finished,
result=failed
Dec 05 13:20:31 stower systemd[1]: Failed to start Celery bf Service.
-- Subject: Unit celery_bf.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit celery_bf.service has failed.
--
-- The result is failed.
Dec 05 13:20:31 stower systemd[1]: Sent message type=signal sender=n/a
destination=n/a object=/org/freedesktop/systemd1
interface=org.freedesktop.systemd1.Mana
Dec 05 13:20:31 stower systemd[1]: Sent message type=method_call sender=n/a
destination=org.freedesktop.DBus object=/org/freedesktop/DBus
interface=org.freedes
Dec 05 13:20:31 stower systemd[1]: Unit celery_bf.service entered failed
state.
Dec 05 13:20:31 stower systemd[1]: celery_bf.service failed.
Dec 05 13:20:31 stower systemd[1]: celery_bf.service: cgroup is empty
Dec 05 13:20:31 stower systemd[1]: Sent message type=signal sender=n/a
destination=n/a object=/org/freedesktop/systemd1/unit/celery_5fbf_2eservice
interface=or
Dec 05 13:20:31 stower systemd[1]: Sent message type=signal sender=n/a
destination=n/a object=/org/freedesktop/systemd1/unit/celery_5fbf_2eservice
interface=or
Dec 05 13:20:31 stower systemd[1]: Got message type=method_call
sender=:1.54 destination=org.freedesktop.systemd1
object=/org/freedesktop/systemd1/unit/celery_
Dec 05 13:20:31 stower systemd[1]: Sent message type=method_call sender=n/a
destination=org.freedesktop.DBus object=/org/freedesktop/DBus
interface=org.freedes
Dec 05 13:20:31 stower systemd[1]: Sent message type=method_call sender=n/a
destination=org.freedesktop.DBus object=/org/freedesktop/DBus
interface=org.freedes
Dec 05 13:20:31 stower systemd[1]: Sent message type=method_call sender=n/a
destination=org.freedesktop.DBus object=/org/freedesktop/DBus
interface=org.freedes
Dec 05 13:20:31 stower systemd[1]: SELinux access check
scon=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
tcon=unconfined_u:object_r:systemd_unit_file
Dec 05 13:20:31 stower systemd[1]: Sent message type=method_return
sender=n/a destination=:1.54 object=n/a interface=n/a member=n/a
cookie=1317 reply_cookie=7
Dec 05 13:20:31 stower systemd[1]: Got cgroup empty notification for:
/system.slice/celery_bf.service
Dec 05 13:20:31 stower systemd[1]: Sent message type=signal sender=n/a
destination=n/a object=/org/freedesktop/systemd1/agent
interface=org.freedesktop.systemd
Dec 05 13:20:31 stower systemd[1]: Got message type=signal
sender=org.freedesktop.DBus destination=n/a object=/org/freedesktop/DBus
interface=org.freedesktop.
Post by Doug Snyder
I'm trying to get you the debug output as described in
https://freedesktop.org/wiki/Software/systemd/Debugging/#index3h1
https://lists.freedesktop.org/archives/systemd-devel/
2017-December/thread.html#39934
with or without email attachments
At this point two people have said I have an updated systemd which is
strange because I thought I used the latest CentOS minimal install
$ systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
+GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
So maybe it makes sense for me to just update and see if that resolves this.
I don't know how to do that though.
https://www.certdepot.net/new-systemd-version/
Post by Lennart Poettering
Post by Doug Snyder
I'm trying to write a basic systemd service. The ExecStart command I'm
using works. The systemd service doesn't. It throws a cryptic error and
I
Post by Doug Snyder
can't find any documentation on the web that makes any sense of it. I
posted on this list and got no responses. Is this list active? If not is
there somewhere I can get basic help for setting up a service. Systemd
is
Post by Doug Snyder
the default tool daemonizing on most Linux distros right? There must be
some way to get help to get a basic example working.
Please always indicate which systemd version you are using, which
distribution and which kernel.
Post by Doug Snyder
Nov 20 13:58:33 stower systemd[1]: celery_bf.service failed to run
'start'
Post by Doug Snyder
task: Operation not supported
Hmm, this indicates that some kernel functionality required is not
available in the kernel. I am not sure which one that is, and logging
is a bit too brief for that.
If you run "systemd-analyze set-log-level debug", and run this again,
what do you see then?
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-12-05 18:50:54 UTC
Permalink
Raw Message
Post by Doug Snyder
Getting rid of the template syntax changed the behavior but now it appears
there's another problem I don't understand.
it tries to start the unit and it shows some encouraging signs but fails
because a timeout was exceeded.
Celery starts and shows healthy output but then sends a SIGCHLD and then
dies
Uh, I am not sure what celery is, but if celery is started but exits
on its own then I am not sure I can help you very much.
Post by Doug Snyder
I don't know if you know what celery is but its a task queue so it does
create child processes
I thought the Type=forking line in the service unit would let systemd about
that.
Or maybe its dying for some other reason?
It works fine when run without systemd
Does it detach on its own? i.e. double fork? if yes, then type=forking
is the right choice, otherwise you need a different Type=.
Post by Doug Snyder
I don't think I can send you detailed output until we get this issue
resolved with the mailing list and why its not excepting emails with debug
output.
Short emails seem to post. Long ones don't. There's no indication of whats
going on
There are message size limits on this mailing list. Please condense
your logs to only include relevant bits hence, or use fpaste.org or
something like that, and drop the URL.

Lennart
--
Lennart Poettering, Red Hat
Doug Snyder
2017-12-05 19:19:43 UTC
Permalink
Raw Message
Well, it runs fine by itself using the exact same command on the command
line.
Its when systemd is added to the mix that the problems occur
The people that put out celery have docs on how to use it with systemd, its
just that they don't work:
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
I based my service off theirs but the Type=forking was copied right from
theirs
Maybe if I use their version more exactly it might work.
But their version doesn't work at all ssince I suspect they are using
templating
( I copied the %I from their file )
They reference config variables in their template:

/etc/systemd/system/celery.service:

[Unit]Description=Celery ServiceAfter=network.target
[Service]Type=forkingUser=celeryGroup=celeryEnvironmentFile=-/etc/conf.d/celeryWorkingDirectory=/opt/celeryExecStart=/bin/sh
-c '${CELERY_BIN} multi start ${CELERYD_NODES} \ -A ${CELERY_APP}
--pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE}
--loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'ExecStop=/bin/sh -c
'${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'ExecReload=/bin/sh -c '${CELERY_BIN}
multi restart ${CELERYD_NODES} \ -A ${CELERY_APP}
--pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE}
--loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]WantedBy=multi-user.target

/etc/default/celeryd:

# Names of nodes to start# most people will only start one
node:CELERYD_NODES="worker1"# but you can also start multiple and
configure settings# for each in CELERYD_OPTS#CELERYD_NODES="worker1
worker2 worker3"# alternatively, you can specify the number of nodes
to start:#CELERYD_NODES=10
# Absolute or relative path to the 'celery'
command:CELERY_BIN="/usr/local/bin/celery"#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use# comment out this line if you don't use an
appCELERY_APP="proj"# or fully qualified:#CELERY_APP="proj.tasks:app"
# Where to chdir at start.CELERYD_CHDIR="/opt/Myproject/"
# Extra command-line arguments to the
workerCELERYD_OPTS="--time-limit=300 --concurrency=8"# Configure
node-specific settings by appending node name to
arguments:#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3
2 -Ofair:worker1"
# Set logging level to DEBUG#CELERYD_LOG_LEVEL="DEBUG"
# %n will be replaced with the first part of the
nodename.CELERYD_LOG_FILE="/var/log/celery/%n%I.log"CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.# You need to create
this user manually (or you can choose# a user/group combination that
already exists (e.g.,
nobody).CELERYD_USER="celery"CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,# and
owned by the userid/group configured.CELERY_CREATE_DIRS=1

But nowhere in their documentation is any mention of the fact that
they are using templates or how how make a service a template.

Is there docs on this you can point me to? Or if not just explain it?
Post by Doug Snyder
Post by Doug Snyder
Getting rid of the template syntax changed the behavior but now it
appears
Post by Doug Snyder
there's another problem I don't understand.
it tries to start the unit and it shows some encouraging signs but fails
because a timeout was exceeded.
Celery starts and shows healthy output but then sends a SIGCHLD and then
dies
Uh, I am not sure what celery is, but if celery is started but exits
on its own then I am not sure I can help you very much.
Post by Doug Snyder
I don't know if you know what celery is but its a task queue so it does
create child processes
I thought the Type=forking line in the service unit would let systemd
about
Post by Doug Snyder
that.
Or maybe its dying for some other reason?
It works fine when run without systemd
Does it detach on its own? i.e. double fork? if yes, then type=forking
is the right choice, otherwise you need a different Type=.
Post by Doug Snyder
I don't think I can send you detailed output until we get this issue
resolved with the mailing list and why its not excepting emails with
debug
Post by Doug Snyder
output.
Short emails seem to post. Long ones don't. There's no indication of
whats
Post by Doug Snyder
going on
There are message size limits on this mailing list. Please condense
your logs to only include relevant bits hence, or use fpaste.org or
something like that, and drop the URL.
Lennart
--
Lennart Poettering, Red Hat
Andrei Borzenkov
2017-12-05 19:48:38 UTC
Permalink
Raw Message
Post by Doug Snyder
Well, it runs fine by itself using the exact same command on the command
line.
Its when systemd is added to the mix that the problems occur
The people that put out celery have docs on how to use it with systemd, its
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
I based my service off theirs but the Type=forking was copied right from
theirs
Maybe if I use their version more exactly it might work.
So - did you try first using example that is known to work before
modifying it?
Post by Doug Snyder
But their version doesn't work at all
How can you claim it if you did not apparently try their version?
Post by Doug Snyder
ssince I suspect they are using
templating
( I copied the %I from their file )
Systemd unit on this site does not use template nor does it use %I in
unit definition. They use %I in variable assignment in configuration
file which is entirely different story.

Nor does your start command match what is shown on the site you refer to.
Doug Snyder
2017-12-05 22:46:11 UTC
Permalink
Raw Message
---------- Forwarded message ----------
From: Doug Snyder <***@gmail.com>
Date: Tue, Dec 5, 2017 at 5:42 PM
Subject: Re: [systemd-devel] Is this list still active? Where can I get
basic help with systemd
To: Doug Snyder <***@gmail.com>


*Andrei, *
*yes I did try to their solution and the reason I claimed that it didn't
work is because it doesn't*
*Of course thats the first thing I tried and it fails because It doesn't
reference the variables in the config file.*
*I tried to take mostly the same code without the config file since it
wasn't working*
*That caused problems above since I had an %I in my service file and I was
told that this was because I wasn't using a template*
*( with no further explanation or any documentation I can see on
freedesktop.org <http://freedesktop.org> )*
*I assumed maybe the fact that I wasn't using a template was why the
variables being templated from the config file weren't being resolved,*
*but asked for further clarification ...*
*but really how would I know?*
*The problem seems to be rooted in the fact that I'm not a Linux
enthusiast, I'm a data scientist that needs to run a simple task queue*
*but none of the instructions I find work.*
*Its true I've modified the ExecStart command from the example the celery
docs give.*
*I used the command that's listed in different part of the documentation to
run the task queue.*
*The command works fine in the command line, so I thought it would work
when run by systemd*
*I said that clearly in my first post.*
*As far as I know I'm trying to just get systemd to run a command at start
up in its own process*
*If that's not how things work, I'd like to know about it*
*That's why I'm here ... to learn how systemd works*
*I tried to learn about templates myself
here: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
<https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files>*
*and found something useful about forking untis which seems to be relevant:*
*For forking services you are supposed to define the PIDFile:*

- PIDFile=: If the service type is marked as "forking", this directive
is used to set the path of the file that should contain the process ID
number of the main child that should be monitored.

*celery's docs service does have that defined and mine doesn't.*
*But they have it defined in the Environment file using %n to define the
pid file*

# %n will be replaced with the first part of the nodename.

CELERYD_PID_FILE="/var/run/celery/%n.pid"

I can't do that without using templates apparently, so I'm not sure if I
can accomplish the same thing without getting a template working
and if I should get the first part of the nodename some other way or I
should try to get the templating working.
I'm assuming that the node name is the celery node not the systemd node ...
or is there something in systemd called a node?
Assuming this is a celery defined variable %n, how is it defined in the
envirnment file. I just see it being referenced.
How are these variables defined in systemd templates?
Post by Doug Snyder
Well, it runs fine by itself using the exact same command on the command
line.
Its when systemd is added to the mix that the problems occur
The people that put out celery have docs on how to use it with systemd,
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
I based my service off theirs but the Type=forking was copied right from
theirs
Maybe if I use their version more exactly it might work.
But their version doesn't work at all ssince I suspect they are using
templating
( I copied the %I from their file )
[Unit]Description=Celery ServiceAfter=network.target
[Service]Type=forkingUser=celeryGroup=celeryEnvironmentFile=-/etc/conf.d/celeryWorkingDirectory=/opt/celeryExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \ -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \ --pidfile=${CELERYD_PID_FILE}'ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \ -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \ --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]WantedBy=multi-user.target
# Names of nodes to start# most people will only start one node:CELERYD_NODES="worker1"# but you can also start multiple and configure settings# for each in CELERYD_OPTS#CELERYD_NODES="worker1 worker2 worker3"# alternatively, you can specify the number of nodes to start:#CELERYD_NODES=10
# Absolute or relative path to the 'celery' command:CELERY_BIN="/usr/local/bin/celery"#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use# comment out this line if you don't use an appCELERY_APP="proj"# or fully qualified:#CELERY_APP="proj.tasks:app"
# Where to chdir at start.CELERYD_CHDIR="/opt/Myproject/"
# Extra command-line arguments to the workerCELERYD_OPTS="--time-limit=300 --concurrency=8"# Configure node-specific settings by appending node name to arguments:#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"
# Set logging level to DEBUG#CELERYD_LOG_LEVEL="DEBUG"
# %n will be replaced with the first part of the nodename.CELERYD_LOG_FILE="/var/log/celery/%n%I.log"CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.# You need to create this user manually (or you can choose# a user/group combination that already exists (e.g., nobody).CELERYD_USER="celery"CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,# and owned by the userid/group configured.CELERY_CREATE_DIRS=1
But nowhere in their documentation is any mention of the fact that they are using templates or how how make a service a template.
Is there docs on this you can point me to? Or if not just explain it?
Post by Doug Snyder
Post by Doug Snyder
Getting rid of the template syntax changed the behavior but now it
appears
Post by Doug Snyder
there's another problem I don't understand.
it tries to start the unit and it shows some encouraging signs but fails
because a timeout was exceeded.
Celery starts and shows healthy output but then sends a SIGCHLD and then
dies
Uh, I am not sure what celery is, but if celery is started but exits
on its own then I am not sure I can help you very much.
Post by Doug Snyder
I don't know if you know what celery is but its a task queue so it does
create child processes
I thought the Type=forking line in the service unit would let systemd
about
Post by Doug Snyder
that.
Or maybe its dying for some other reason?
It works fine when run without systemd
Does it detach on its own? i.e. double fork? if yes, then type=forking
is the right choice, otherwise you need a different Type=.
Post by Doug Snyder
I don't think I can send you detailed output until we get this issue
resolved with the mailing list and why its not excepting emails with
debug
Post by Doug Snyder
output.
Short emails seem to post. Long ones don't. There's no indication of
whats
Post by Doug Snyder
going on
There are message size limits on this mailing list. Please condense
your logs to only include relevant bits hence, or use fpaste.org or
something like that, and drop the URL.
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-12-06 11:02:00 UTC
Permalink
Raw Message
Post by Doug Snyder
Well, it runs fine by itself using the exact same command on the command
line.
Its when systemd is added to the mix that the problems occur
The people that put out celery have docs on how to use it with systemd, its
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html
I based my service off theirs but the Type=forking was copied right from
theirs
Maybe if I use their version more exactly it might work.
But their version doesn't work at all ssince I suspect they are using
templating
( I copied the %I from their file )
Uh. I figure part of the issue here stems from the fact that "celery"
has its own specifier expansion using "%", and that conflicts with
systemd's own. i.e. if you us "%I" in a systemd unit file, then
systemd will replace it by the instance ID, but this doesn't work if
the service is not templated. Now, the celery sysv init scripts on
that page also use "%I" which suggests that "%I" is also a construct
celery understands on its own. That means that if you have to 'escape'
the "%" character properly here, to bypass systemd's expansion so that
celery's own expansion can be used. Escaping is done by duplicating
the "%". I.e. "%I%n" → "%%I%%n".
Post by Doug Snyder
Is there docs on this you can point me to? Or if not just explain it?
systemd unit templates are described here:

https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description

Look for the paragraph about "Optionally, units may be instantiated
from a template file at runtime…" in the "Description" section.

That said, I doubt unit templates/instances is what you want here,
this is just caused by the confusion that both celery know %I and %n
and systemd units do, and hence both want to expand that.

Lennart
--
Lennart Poettering, Red Hat
Doug Snyder
2017-12-06 01:34:48 UTC
Permalink
Raw Message
Andrei,
yes I did try to their solution and the reason I claimed that it didn't
work is because it doesn't
Of course thats the first thing I tried and it fails because It doesn't
reference the variables in the config file.
I tried to simplify the example posted on celery's site to restrict the
many possible sources of errors
That caused problems above since I had an %I in my service file and I was
told that this was because I wasn't using a template
( with no further explanation or any documentation I can see on
freedesktop.org )
I assumed maybe the fact that I wasn't using a template was why the
variables being templated from the config file weren't being resolved,
but asked for further clarification ...
but really, how would I know?
The problem seems to be rooted in the fact that I'm not a Linux enthusiast,
I'm a data scientist that needs to run a simple task queue
but none of the instructions I find work.
Its true I've modified the ExecStart command from the example the celery
docs give.
I used the command that's listed in different part of the documentation to
run the task queue.
The command works fine in the command line, so I thought it would work when
run by systemd
I said that clearly in my first post.
As far as I know I'm trying to just get systemd to run a command at start
up in its own process
The only working systemd service I had was one I wrote for couchdb and so I
made the assumption it would work in a similar way
and tried to work off code that was working since systemd is so bad about
letting you know where errors originate
If that's not how things work, I'd like to know about it
That's why I'm here ... to learn how systemd works

While searching for something else I found the answer that I needed:
https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
For forking services you are supposed to define the PIDFile:

- PIDFile=: If the service type is marked as "forking", this directive
is used to set the path of the file that should contain the process ID
number of the main child that should be monitored.

This was what was causing the error even though the error said nothing of
the sort.
I had to get the templating working and define PIDFile using the %n
variable that was used on celery's site
Its all working now.
Post by Doug Snyder
Post by Doug Snyder
Getting rid of the template syntax changed the behavior but now it
appears
Post by Doug Snyder
there's another problem I don't understand.
it tries to start the unit and it shows some encouraging signs but fails
because a timeout was exceeded.
Celery starts and shows healthy output but then sends a SIGCHLD and then
dies
Uh, I am not sure what celery is, but if celery is started but exits
on its own then I am not sure I can help you very much.
Post by Doug Snyder
I don't know if you know what celery is but its a task queue so it does
create child processes
I thought the Type=forking line in the service unit would let systemd
about
Post by Doug Snyder
that.
Or maybe its dying for some other reason?
It works fine when run without systemd
Does it detach on its own? i.e. double fork? if yes, then type=forking
is the right choice, otherwise you need a different Type=.
Post by Doug Snyder
I don't think I can send you detailed output until we get this issue
resolved with the mailing list and why its not excepting emails with
debug
Post by Doug Snyder
output.
Short emails seem to post. Long ones don't. There's no indication of
whats
Post by Doug Snyder
going on
There are message size limits on this mailing list. Please condense
your logs to only include relevant bits hence, or use fpaste.org or
something like that, and drop the URL.
Lennart
--
Lennart Poettering, Red Hat
Jonathan de Boyne Pollard
2017-12-07 18:56:12 UTC
Permalink
Raw Message
Uh, I am not sure what celery is, but if celery is started but exitson
its own then I am not sure I can help you very much.
Does it detach on its own? i.e. double fork? if yes, then type=forking
is the right choice, otherwise you need a different Type=.
A tip from one of my service bundles:

# As of version 3.1, by default Celery does not "backgroundize" and logs
to standard error, and one stops it with SIGKILL.
# See also https://github.com/celery/celery/issues/3459 .

Note in particular that Celery has its own %n%I system, as explained there.
Loading...