Discussion:
how to debug failures when trying to lock down services
(too old to reply)
Michael Biebl
2017-11-30 03:27:26 UTC
Permalink
Raw Message
Hi,

today I tried to lock down the rsyslog.service that I have on my system.

For that I first created an override.conf that contained

[Service]
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes

ProtectSystem=strict
ReadWritePaths=/var/log
ReadWritePaths=/var/spool/rsyslog
ReadWritePaths=/proc/kmsg

CapabilityBoundingSet=CAP_SYSLOG
CapabilityBoundingSet=CAP_NET_BIND_SERVICE


Unfortunately, rsyslog.service failed to start:
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
vendor preset: enabled)
Drop-In: /etc/systemd/system/rsyslog.service.d
└─override.conf
Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET; 2s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=1/FAILURE)
Main PID: 2734 (code=exited, status=1/FAILURE)

Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Service hold-off
time over, scheduling restart.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Scheduled restart
job, restart counter is at 5.
Nov 30 04:25:03 pluto systemd[1]: Stopped System Logging Service.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Start request
repeated too quickly.
Nov 30 04:25:03 pluto systemd[1]: rsyslog.service: Failed with result
'exit-code'.
Nov 30 04:25:03 pluto systemd[1]: Failed to start System Logging Service.


The journal doesn't contain anything useful.
Any hints how I can further debug this why rsyslog fails to start?

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Mantas Mikulėnas
2017-11-30 05:52:35 UTC
Permalink
Raw Message
On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl <***@gmail.com> wrote:

> Hi,
>
> today I tried to lock down the rsyslog.service that I have on my system.
>
> For that I first created an override.conf that contained
>
> [Service]
> ProtectHome=yes
> PrivateTmp=yes
> PrivateDevices=yes
>
> ProtectSystem=strict
> ReadWritePaths=/var/log
> ReadWritePaths=/var/spool/rsyslog
> ReadWritePaths=/proc/kmsg
>

Are you using imklog or imkmsg? The latter would require the new /dev/kmsg
interface (which probably conflicts with PrivateDevices= above).


> Unfortunately, rsyslog.service failed to start:
> ● rsyslog.service - System Logging Service
> Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
> vendor preset: enabled)
> Drop-In: /etc/systemd/system/rsyslog.service.d
> └─override.conf
> Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET;
> 2s ago
> Docs: man:rsyslogd(8)
> http://www.rsyslog.com/doc/
> Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited,
> status=1/FAILURE)
> Main PID: 2734 (code=exited, status=1/FAILURE)
>

Well, it does say that the failure comes from rsyslogd itself, not from the
namespace setup...


> The journal doesn't contain anything useful.
>

I'm guessing rsyslog will log its own errors to /var/log/syslog rather than
stderr.


> Any hints how I can further debug this why rsyslog fails to start?
>

rsyslogd -d -d -d

strace

--
Mantas Mikulėnas <***@gmail.com>
Michael Biebl
2017-11-30 08:31:50 UTC
Permalink
Raw Message
2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas <***@gmail.com>:
> On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl <***@gmail.com> wrote:
>>
>> Hi,
>>
>> today I tried to lock down the rsyslog.service that I have on my system.
>>
>> For that I first created an override.conf that contained
>>
>> [Service]
>> ProtectHome=yes
>> PrivateTmp=yes
>> PrivateDevices=yes
>>
>> ProtectSystem=strict
>> ReadWritePaths=/var/log
>> ReadWritePaths=/var/spool/rsyslog
>> ReadWritePaths=/proc/kmsg
>
>
> Are you using imklog or imkmsg? The latter would require the new /dev/kmsg
> interface (which probably conflicts with PrivateDevices= above).

I suspect it's related to ProtectSystem=strict, as with
ProtectSystem=full rsyslog seems to start successfully. But this is
just trial and error.

>>
>> Unfortunately, rsyslog.service failed to start:
>> ● rsyslog.service - System Logging Service
>> Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled;
>> vendor preset: enabled)
>> Drop-In: /etc/systemd/system/rsyslog.service.d
>> └─override.conf
>> Active: failed (Result: exit-code) since Thu 2017-11-30 04:25:03 CET;
>> 2s ago
>> Docs: man:rsyslogd(8)
>> http://www.rsyslog.com/doc/
>> Process: 2734 ExecStart=/usr/sbin/rsyslogd -n (code=exited,
>> status=1/FAILURE)
>> Main PID: 2734 (code=exited, status=1/FAILURE)
>
>
> Well, it does say that the failure comes from rsyslogd itself, not from the
> namespace setup...
>
>>
>> The journal doesn't contain anything useful.
>
>
> I'm guessing rsyslog will log its own errors to /var/log/syslog rather than
> stderr.

I don't have anyting in /var/log/syslog

>>
>> Any hints how I can further debug this why rsyslog fails to start?
>
>
> rsyslogd -d -d -d

Already tried that, doesn't produce any useful logs.


> strace

Already tried
ExecStart=
ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n

but this didn't produce any /var/log/strace log file.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Mantas Mikulėnas
2017-11-30 08:35:26 UTC
Permalink
Raw Message
On Thu, Nov 30, 2017 at 10:31 AM, Michael Biebl <***@gmail.com> wrote:

> 2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas <***@gmail.com>:
> > On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl <***@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> today I tried to lock down the rsyslog.service that I have on my system.
> >>
> >> For that I first created an override.conf that contained
> >>
> >> [Service]
> >> ProtectHome=yes
> >> PrivateTmp=yes
> >> PrivateDevices=yes
> >>
> >> ProtectSystem=strict
> >> ReadWritePaths=/var/log
> >> ReadWritePaths=/var/spool/rsyslog
> >> ReadWritePaths=/proc/kmsg
> >
> >
> > Are you using imklog or imkmsg? The latter would require the new
> /dev/kmsg
> > interface (which probably conflicts with PrivateDevices= above).
>
> I suspect it's related to ProtectSystem=strict, as with
> ProtectSystem=full rsyslog seems to start successfully. But this is
> just trial and error.

[
]
> Already tried
> ExecStart=
> ExecStart=/usr/bin/strace -f -o /var/log/strace /usr/sbin/rsyslogd -n
>
> but this didn't produce any /var/log/strace log file.
>
>
Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
/var/log read-only... I think I've seen other people have that problem
recently.

Take a look with `ExecStartPre=/usr/bin/findmnt`.

--
Mantas Mikulėnas <***@gmail.com>
Michael Biebl
2017-11-30 15:12:04 UTC
Permalink
Raw Message
2017-11-30 16:07 GMT+01:00 Michael Biebl <***@gmail.com>:
> 2017-11-30 9:35 GMT+01:00 Mantas Mikulėnas <***@gmail.com>:
>> On Thu, Nov 30, 2017 at 10:31 AM, Michael Biebl <***@gmail.com> wrote:
>>>
>>> 2017-11-30 6:52 GMT+01:00 Mantas Mikulėnas <***@gmail.com>:
>>> > On Thu, Nov 30, 2017 at 5:27 AM, Michael Biebl <***@gmail.com> wrote:
>>> >>
>>> >> [Service]
>>> >> ProtectHome=yes
>>> >> PrivateTmp=yes
>>> >> PrivateDevices=yes
>>> >>
>>> >> ProtectSystem=strict
>>> >> ReadWritePaths=/var/log
>>> >> ReadWritePaths=/var/spool/rsyslog
>>> >> ReadWritePaths=/proc/kmsg
>>> >
>
>>>
>>> I suspect it's related to ProtectSystem=strict, as with
>>> ProtectSystem=full rsyslog seems to start successfully. But this is
>
>> Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
>> /var/log read-only... I think I've seen other people have that problem
>> recently.
>
>
> *facepalm*
> rsyslog.service by default uses StandardOutput=null, so I didn't see
> the error messages in debug mode.
>
> After fixing that, it was rather obvious.
>
> *double facepalm*
> rsyslog writes a pid file in /run and fails to start if it can't write
> the pidfile. I will raise this upstream that maybe writing a pidfile
> in socket activation / sd_notify mode is not really necessary and it
> should stop doing that.

Filed https://github.com/rsyslog/rsyslog/issues/2143 for that

> For now I used RuntimeDirectory=rsyslog and
> ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
>
> So the complete rsyslog.service now looks like
>
> [Unit]
> Description=System Logging Service
> Requires=syslog.socket
> Documentation=man:rsyslogd(8)
> Documentation=http://www.rsyslog.com/doc/
>
> [Service]
> Type=notify
> ExecStart=/usr/sbin/rsyslogd -n
> StandardOutput=null
> Restart=on-failure
>
> PrivateTmp=yes
> PrivateDevices=yes
> ProtectHome=yes
> ProtectSystem=strict
> ReadWritePaths=/var/log
> ReadWritePaths=/var/spool/rsyslog
> ReadWritePaths=/proc/kmsg
> ReadWritePaths=/tmp
>
> RuntimeDirectory=rsyslog
>
> CapabilityBoundingSet=CAP_SYSLOG
> CapabilityBoundingSet=CAP_NET_BIND_SERVICE
>
> ExecStart=/usr/sbin/rsyslogd -n -i /run/rsyslog/rsyslogd.pid
>
> [Install]
> WantedBy=multi-user.target
> Alias=syslog.service
>
> Feedback welcome on how to reasonably lock down rsyslog by default
> without breaking commonly used functionality (like remote syslog)
>
> Regards,
> Michael
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?



--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Lennart Poettering
2017-11-30 17:24:30 UTC
Permalink
Raw Message
On Do, 30.11.17 10:35, Mantas Mikulėnas (***@gmail.com) wrote:

> Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
> /var/log read-only...

Hmm, it does? It really shouldn't.

I thought the issues were mostly around InaccessiblePaths= not
permitting exclusions, not about ProtectSystem/ReadOnlyPaths...

Have a link?

Lennart

--
Lennart Poettering, Red Hat
Michael Biebl
2017-12-01 20:05:04 UTC
Permalink
Raw Message
2017-11-30 18:24 GMT+01:00 Lennart Poettering <***@poettering.net>:
> On Do, 30.11.17 10:35, Mantas Mikulėnas (***@gmail.com) wrote:
>
>> Then I'm guessing ProtectSystem=strict overrides ReadWritePaths and makes
>> /var/log read-only...
>
> Hmm, it does? It really shouldn't.
>
> I thought the issues were mostly around InaccessiblePaths= not
> permitting exclusions, not about ProtectSystem/ReadOnlyPaths...
>
> Have a link?

I think I can reproduce that with v235. I think it works with git
master, but I need to verify that.




--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Loading...