Discussion:
how to debug failures when trying to lock down services
(too old to reply)
Michael Biebl
2017-11-30 03:27:26 UTC
Permalink
Hi,

today I tried to lock down the rsyslog.service that I have on my system.

For that I first created an override.conf that contained

[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes

ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg

CapabilityBoundingSet=CAP_SYSLOG
CapabilityBoundingSet=CAP_NET_BIND_SERVICE


Unfortunately, rsyslog.service failed to start:
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
vendor preset: enabled)
Drop-In: /etc/systemd/system/rsyslog.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET; 2s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=1/FAILURE)
Main PID: 2734 (code=exited, status=1/FAILURE)

Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Service hold-off
time over, scheduling restart.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Scheduled restart
job, restart counter is at 5.
Nov 30 04:25:03 pluto systemd[1]: Stopped System Logging Service.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Start request
repeated too quickly.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Failed with result
'exit-code'.
Nov 30 04:25:03 pluto systemd[1]: Failed to start System Logging Service.


The journal doesn't contain anything useful.
Any hints how I can further debug this why rsyslog fails to start?
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Mantas Mikulėnas
2017-11-30 05:52:35 UTC
Permalink
Post by Michael Biebl
Hi,
today I tried to lock down the rsyslog.service that I have on my system.
For that I first created an override.conf that contained
[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg
Are you using imklog or imkmsg? The latter would require the new /dev/kmsg
interface (which probably conflicts with PrivateDevices= above).
Post by Michael Biebl
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
vendor preset: enabled)
Drop-In: /etc/systemd/system/rsyslog.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET; 2s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited,
status=1/FAILURE)
Main PID: 2734 (code=exited, status=1/FAILURE)
Well, it does say that the failure comes from rsyslogd itself, not from the
namespace setup...
Post by Michael Biebl
The journal doesn't contain anything useful.
I'm guessing rsyslog will log its own errors to /var/log/syslog rather than
stderr.
Post by Michael Biebl
Any hints how I can further debug this why rsyslog fails to start?
rsyslogd -d -d -d

strace
--
Mantas Mikulėnas <***@gmail.com>
Michael Biebl
2017-11-30 08:31:50 UTC
Permalink
Post by Mantas Mikulėnas
Post by Michael Biebl
Hi,
today I tried to lock down the rsyslog.service that I have on my system.
For that I first created an override.conf that contained
[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg
Are you using imklog or imkmsg? The latter would require the new /dev/kmsg
interface (which probably conflicts with PrivateDevices= above).
I suspect it's related to ProtectSystem=strict, as with
ProtectSystem=full rsyslog seems to start successfully. But this is
just trial and error.
Post by Mantas Mikulėnas
Post by Michael Biebl
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
vendor preset: enabled)
Drop-In: /etc/systemd/system/rsyslog.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET; 2s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited,
status=1/FAILURE)
Main PID: 2734 (code=exited, status=1/FAILURE)
Well, it does say that the failure comes from rsyslogd itself, not from the
namespace setup...
Post by Michael Biebl
The journal doesn't contain anything useful.
I'm guessing rsyslog will log its own errors to /var/log/syslog rather than
stderr.
I don't have anyting in /var/log/syslog
Post by Mantas Mikulėnas
Post by Michael Biebl
Any hints how I can further debug this why rsyslog fails to start?
rsyslogd -d -d -d
Already tried that, doesn't produce any useful logs.
Post by Mantas Mikulėnas
strace
Already tried
ExecStart=
ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n

but this didn't produce any /var/log/strace log file.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Mantas Mikulėnas
2017-11-30 08:35:26 UTC
Permalink
Post by Mantas Mikulėnas
Post by Mantas Mikulėnas
Post by Michael Biebl
Hi,
today I tried to lock down the rsyslog.service that I have on my system.
For that I first created an override.conf that contained
[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg
Are you using imklog or imkmsg? The latter would require the new
/dev/kmsg
Post by Mantas Mikulėnas
interface (which probably conflicts with PrivateDevices= above).
I suspect it's related to ProtectSystem=strict, as with
ProtectSystem=full rsyslog seems to start successfully. But this is
just trial and error.
[
]
Post by Mantas Mikulėnas
Already tried
ExecStart=
ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n
but this didn't produce any /var/log/strace log file.
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only... I think I've seen other people have that problem
recently.

Take a look with `ExecStartPre=/usr/bin/findmnt`.
--
Mantas Mikulėnas <***@gmail.com>
Michael Biebl
2017-11-30 15:12:04 UTC
Permalink
Post by Mantas Mikulėnas
Post by Michael Biebl
Post by Michael Biebl
[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg
I suspect it's related to ProtectSystem=strict, as with
ProtectSystem=full rsyslog seems to start successfully. But this is
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only... I think I've seen other people have that problem
recently.
*facepalm*
rsyslog.service by default uses StandardOutput=null, so I didn't see
the error messages in debug mode.
After fixing that, it was rather obvious.
*double facepalm*
rsyslog writes a pid file in /run and fails to start if it can't write
the pidfile. I will raise this upstream that maybe writing a pidfile
in socket activation / sd_notify mode is not really necessary and it
should stop doing that.
Filed https://github.com/rsyslog/rsyslog/issues/2143 for that
For now I used RuntimeDirectory=rsyslog and
ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
So the complete rsyslog.service now looks like
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/
[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n
StandardOutput=null
Restart=on-failure
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg
ReadWritePaths=/tmp
RuntimeDirectory=rsyslog
CapabilityBoundingSet=CAP_SYSLOG
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
[Install]
WantedBy=multi-user.target
Alias=syslog.service
Feedback welcome on how to reasonably lock down rsyslog by default
without breaking commonly used functionality (like remote syslog)
Regards,
Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2017-11-30 17:24:30 UTC
Permalink
Post by Mantas Mikulėnas
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only...
Hmm, it does? It really shouldn't.

I thought the issues were mostly around InaccessiblePaths= not
permitting exclusions, not about ProtectSystem/ReadOnlyPaths...

Have a link?

Lennart
--
Lennart Poettering, Red Hat
Michael Biebl
2017-12-01 20:05:04 UTC
Permalink
Post by Lennart Poettering
Post by Mantas Mikulėnas
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only...
Hmm, it does? It really shouldn't.
I thought the issues were mostly around InaccessiblePaths= not
permitting exclusions, not about ProtectSystem/ReadOnlyPaths...
Have a link?
I think I can reproduce that with v235. I think it works with git
master, but I need to verify that.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Loading...