[systemd-devel] Native Journal source vs syslog forwarding

[Gergely Nagy]

Hi!

I have an interesting situation here, which I'm trying to wrap my head
around and solve. The problem is that I have a syslog daemon (syslog-ng
3.6.1) that has a native Journal source, meaning it can pull entries
from the Journal directly, and does not need the syslog forwarding
socket - and this is the default when running on a systemd-enabled
machine.

This works beautifully, except there's one problem:

Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages.

On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

I'm not sure how to solve this problem. As far as I see, I have the
following options:

1) Drop the native journal source and use syslog forwarding.

This is trivial to do, but I loose the extra fields and info the
Journal collects. I'd rather not do this.

2) Have a dummy listener on /run/systemd/journal/syslog, that just reads
everything and drops it on the floor.

This sounds fishy, and is a bit awkward to implement in the config.
This would also be an ugly hack, not a real solution.

3) Disable syslog forwarding if syslog-ng is installed

Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

I'd appreciate any hints. (Disabling syslog forwarding by default is not
an option.)

[Colin Guthrie]

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

I think there were some patches recently that talked about added ".d/"
dir support for such config files. Thus syslog-ng could ship a
/etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or
preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it
follows best practice guidelines for stateless systems)

See the patches by Josh Triplett:

[PATCH 1/2] Introduce CONF_DIRS_NULSTR helper to define standard conf dirs
[PATCH 2/2] logind: Support logind.conf.d directories in the usual
search paths

The latter patch just adds support in logind, but mentions the likes of
journald too as further work if approved.

I think this approach would solve your problem fairly cleanly.

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Gergely Nagy]

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

Colin> I think there were some patches recently that talked about added ".d/"
Colin> dir support for such config files. Thus syslog-ng could ship a
Colin> /etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or
Colin> preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it
Colin> follows best practice guidelines for stateless systems)

Thanks, this sounds like a good start!

[Lennart Poettering]

Post by Gergely Nagy

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

Colin> I think there were some patches recently that talked about added ".d/"
Colin> dir support for such config files. Thus syslog-ng could ship a
Colin> /etc/systemd/journal.conf.d/syslog-ng.conf file which did this (or
Colin> preferably a /usr/lib/systemd/journal.conf.d/syslog-ng.conf file so it
Colin> follows best practice guidelines for stateless systems)
Thanks, this sounds like a good start!

Please don't tape over things. Let's figure out the cause for the
message first. You really shouldn't get the message if everything was
working right...

Lennart

[Jóhann B. Guðmundsson]

Post by Gergely Nagy
Hi!
I have an interesting situation here, which I'm trying to wrap my head
around and solve. The problem is that I have a syslog daemon (syslog-ng
3.6.1) that has a native Journal source, meaning it can pull entries
from the Journal directly, and does not need the syslog forwarding
socket - and this is the default when running on a systemd-enabled
machine.
Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages.

This happens when the socket buffer is full for the syslogd which is
being forwarding to.
( try tweaking those settings )

Post by Gergely Nagy
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.
I'm not sure how to solve this problem. As far as I see, I have the
1) Drop the native journal source and use syslog forwarding.
This is trivial to do, but I loose the extra fields and info the
Journal collects. I'd rather not do this.
2) Have a dummy listener on /run/systemd/journal/syslog, that just reads
everything and drops it on the floor.
This sounds fishy, and is a bit awkward to implement in the config.
This would also be an ugly hack, not a real solution.
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

As of systemd 216 forwarding to another syslogd has been disabled by
default so no need to fiddle with this setting.

Post by Gergely Nagy
I'd appreciate any hints. (Disabling syslog forwarding by default is not
an option.)

Why is that not an option since there is nothing by your own account
listening to /run/systemd/journal/syslog?

JBG

[Gergely Nagy]

Post by Gergely Nagy
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

[...]

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

Jóhann> As of systemd 216 forwarding to another syslogd has been disabled by
Jóhann> default so no need to fiddle with this setting.

Forwarding is enabled by default on Debian, as I wrote in my original
mail. I have no control over the default, and I have no desire to argue
for changing it. There are other syslogds in Debian (including the
default one) that do not read from the Journal, but rely on forwarding.
If forwarding was turned off, they'd stop working. Older versions of
syslog-ng would, too.

[Jóhann B. Guðmundsson]

Post by Gergely Nagy

Post by Gergely Nagy
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

[...]

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

Jóhann> As of systemd 216 forwarding to another syslogd has been disabled by
Jóhann> default so no need to fiddle with this setting.
Forwarding is enabled by default on Debian, as I wrote in my original
mail. I have no control over the default, and I have no desire to argue
for changing it. There are other syslogds in Debian (including the
default one) that do not read from the Journal, but rely on forwarding.
If forwarding was turned off, they'd stop working. Older versions of
syslog-ng would, too.

Right but you are going against downstream distribution policy and
unwilling yourself ( or have the consumer of the syslog-ng package do
that ) to change the default manually ( via journald.conf ) or advocate
for that change ( which would be simply advocating for following
upstream ).

If you are the syslog-ng maintainer in Debian then I guess you will have
to introduce conflicts with other sysloggers in other components and
have them remove the default syslogger ( assuming the journal has not
been made the default yet in Debian and something like rsyslog is )
because if you go the snipped route you would overwrite the default thus
break rsyslog ( and others if they exist ) in the process.

What is Jessie using 215? if so you are probably stuck with 215 for the
reminder of Jessie which requires you to convince the Debian systemd
maintainers to backport the relevant patch(es) to make that work.

Does Debian have two syslog-ng components one tweak for journal as in
uses systemd-journal() along with filters and default journal tweaked
send declaration and another one that is tweaked for $other systemd init
systems since there are more tweaks than just systemd-journal() which
should be made as an default in that process?

Or is there a single component that detects which init system is
installed and install configuration files based on that? ( if so could
he not simply s/ForwardToSyslog=/ from yes to no in the process? )

JBG

[Gergely Nagy]

Jóhann> Right but you are going against downstream distribution policy and
Jóhann> unwilling yourself ( or have the consumer of the syslog-ng package do
Jóhann> that ) to change the default manually ( via journald.conf ) or
Jóhann> advocate for that change ( which would be simply advocating for
Jóhann> following upstream ).

As I said, changing the default is not my decision. That belongs to the
systemd package. A local administrator can, of course, override it.

But from the syslog-ng package, I am not allowed to touch another
package's config file. I could drop a snippet to journald.conf.d/, if it
were supported, and that would be the ideal solution.

Jóhann> If you are the syslog-ng maintainer in Debian then I guess you will
Jóhann> have to introduce conflicts with other sysloggers in other components
Jóhann> and have them remove the default syslogger ( assuming the journal has
Jóhann> not been made the default yet in Debian and something like rsyslog is
Jóhann> ) because if you go the snipped route you would overwrite the default
Jóhann> thus break rsyslog ( and others if they exist ) in the
Jóhann> process.

The syslogds already conflict with each other, so this part is covered.

Jóhann> What is Jessie using 215? if so you are probably stuck with 215 for
Jóhann> the reminder of Jessie which requires you to convince the Debian
Jóhann> systemd maintainers to backport the relevant patch(es) to make that
Jóhann> work.

Yep, but Jessie has syslog-ng 3.5, which uses the forwarder. (3.6 is
only in experimental, so plenty of time and opportunity to solve this
issue).

Jóhann> Does Debian have two syslog-ng components one tweak for journal as in
Jóhann> uses systemd-journal() along with filters and default journal tweaked
Jóhann> send declaration and another one that is tweaked for $other systemd
Jóhann> init systems since there are more tweaks than just systemd-journal()
Jóhann> which should be made as an default in that process?

That would be an option, but I'd rather have one component that
auto-detects what to use. One thing I've been discussing with syslog-ng
upstream is to use the forwarder if available, but emit a warning that
suggests turning it off, and using the native journal source. At least
until we can drop a snippet in journald.conf.d/.

Anyhow, possible solutions were found, thanks everyone for the swift
help and suggestions!

[Martin Pitt]

Hey Gergely,

Post by Gergely Nagy
Forwarding is enabled by default on Debian, as I wrote in my
original mail. I have no control over the default, and I have no
desire to argue for changing it.

I'm just packaging systemd 217, and will revert the disabled
forwarding by default (i. e. retain the behaviour of 215). As a
systemd package maintainer I'm also not in a position to unilaterally
changing the default and breaking rsyslog and friends. So this
requires some coordination indeed (and either way, none of this is a
matter for the frozen jessie).

So we either need the journal.conf.d/ feature and have journal-pulling
sysloggers disable forwarding along the way, or we need to wait until
all packaged sysloggers can read from the journal before we turn off
forwarding by default.

Thanks,

Martin

[Gergely Nagy]

Martin> So we either need the journal.conf.d/ feature and have journal-pulling
Martin> sysloggers disable forwarding along the way, or we need to wait until
Martin> all packaged sysloggers can read from the journal before we turn off
Martin> forwarding by default.

For what its worth, syslog-ng upstream suggested another solution, that
appears to fix the issue (but I've yet to test it myself): if the
syslog-ng.service file does *not* Require=syslog.socket, then the
syslog.socket does not get started, and if it doesn't get started, the
journal will not do forwarding, either, and thus, the error messages
disappear too.

If this works (and I was assured it does), then no change is neccessary
in systemd itself, and I can trivially patch up the syslog-ng.service
file.

[Lennart Poettering]

Post by Gergely Nagy
Martin> So we either need the journal.conf.d/ feature and have journal-pulling
Martin> sysloggers disable forwarding along the way, or we need to wait until
Martin> all packaged sysloggers can read from the journal before we turn off
Martin> forwarding by default.
For what its worth, syslog-ng upstream suggested another solution, that
appears to fix the issue (but I've yet to test it myself): if the
syslog-ng.service file does *not* Require=syslog.socket, then the
syslog.socket does not get started, and if it doesn't get started, the
journal will not do forwarding, either, and thus, the error messages
disappear too.

Ah, yeah! It shouldn't require it. If it pulls it in, that's what it gets...

Lennart

[Lennart Poettering]

Post by Martin Pitt
Hey Gergely,

Post by Gergely Nagy
Forwarding is enabled by default on Debian, as I wrote in my
original mail. I have no control over the default, and I have no
desire to argue for changing it.

I'm just packaging systemd 217, and will revert the disabled
forwarding by default (i. e. retain the behaviour of 215). As a
systemd package maintainer I'm also not in a position to unilaterally
changing the default and breaking rsyslog and friends. So this
requires some coordination indeed (and either way, none of this is a
matter for the frozen jessie).
So we either need the journal.conf.d/ feature and have journal-pulling
sysloggers disable forwarding along the way, or we need to wait until
all packaged sysloggers can read from the journal before we turn off
forwarding by default.

On Fedora all loggers apparently read from the journal directly now,
to my knowledge. So the code at least is there.

Lennart

[Lennart Poettering]

Post by Gergely Nagy

Post by Gergely Nagy
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

[...]

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

Jóhann> As of systemd 216 forwarding to another syslogd has been disabled by
Jóhann> default so no need to fiddle with this setting.
Forwarding is enabled by default on Debian, as I wrote in my original
mail. I have no control over the default, and I have no desire to argue
for changing it. There are other syslogds in Debian (including the
default one) that do not read from the Journal, but rely on forwarding.
If forwarding was turned off, they'd stop working. Older versions of
syslog-ng would, too.

"other syslogds"? Which ones just out of curiosity? If both syslog-ng
and rsyslog can now read directly from the journal I wonder what other
syslog implementation debian wants to support there...

Lennart

[Gergely Nagy]

Post by Gergely Nagy
Forwarding is enabled by default on Debian, as I wrote in my original
mail. I have no control over the default, and I have no desire to argue
for changing it. There are other syslogds in Debian (including the
default one) that do not read from the Journal, but rely on forwarding.
If forwarding was turned off, they'd stop working. Older versions of
syslog-ng would, too.

Lennart> "other syslogds"? Which ones just out of curiosity? If both syslog-ng
Lennart> and rsyslog can now read directly from the journal I wonder what other
Lennart> syslog implementation debian wants to support there...

There's a handful of syslogds, including busybox-syslogd, and the
ancient sysklogd, among other things. Furthermore, we want to support
upgrades that continue running sysvinit, so we can't make the journal
reader default (unless the syslogd can - like syslog-ng - figure out
which one to use at run time). (iirc the default rsyslog.conf uses
/dev/log, and rsyslog translates that to /run/systemd/journal/syslog if
it detects systemd, thus preserving compatibility.)

[Lennart Poettering]

Post by Gergely Nagy

Post by Gergely Nagy
Forwarding is enabled by default on Debian, as I wrote in my original
mail. I have no control over the default, and I have no desire to argue
for changing it. There are other syslogds in Debian (including the
default one) that do not read from the Journal, but rely on forwarding.
If forwarding was turned off, they'd stop working. Older versions of
syslog-ng would, too.

Lennart> "other syslogds"? Which ones just out of curiosity? If both syslog-ng
Lennart> and rsyslog can now read directly from the journal I wonder what other
Lennart> syslog implementation debian wants to support there...
There's a handful of syslogds, including busybox-syslogd, and the
ancient sysklogd, among other things. Furthermore, we want to support
upgrades that continue running sysvinit, so we can't make the journal
reader default (unless the syslogd can - like syslog-ng - figure out
which one to use at run time). (iirc the default rsyslog.conf uses
/dev/log, and rsyslog translates that to /run/systemd/journal/syslog if
it detects systemd, thus preserving compatibility.)

Note that in very recent systemd /dev/log is actually a symlink to the
real socket in /run/systemd/. This concept allows compatibility with
those ancient syslog implementations by simply making them replace the
symlink and own /dev/log directly. That way the the journal is
bypassed for syslog messages, but I figure this might be a better
compat model for those implementations, after all they'd need patching
anyway to read from the forwarder socket or the journal...

Lennart

[Lennart Poettering]

Post by Gergely Nagy
Hi!
I have an interesting situation here, which I'm trying to wrap my head
around and solve. The problem is that I have a syslog daemon (syslog-ng
3.6.1) that has a native Journal source, meaning it can pull entries
from the Journal directly, and does not need the syslog forwarding
socket - and this is the default when running on a systemd-enabled
machine.
Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages.
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

Hmmm, we only increase the counter when we get EAGAIN from the
socket. Normally, if nobody is listening on the other side we
shouldn't get EAGAIN, we should get ECONNREFUSED or so.

Most likely for some reason "syslog.socket" gets started (which
establishes the socket), but nothing is then started behind it? That
sounds wrong.

Note that in systemd upstream we don't do forward-to-syslog anymore,
simply because it turned out to be really slow when nobody is
listening on the socket, and rsyslog didn't use it anymore. Does
Debian turn this on after all?

Post by Gergely Nagy
3) Disable syslog forwarding if syslog-ng is installed
Not sure how this could be achieved, because journald.conf does not
belong to the syslog-ng package, therefore I can't fiddle its
settings from there. (Technically, I could, but I won't, that'd be
extremely rude.)

We really should figure out why journald gets EAGAIN when writing to
that socket. Is syslog.socket started? If so, why?

Lennart

[Gergely Nagy]

Post by Gergely Nagy
Nov 26 10:41:05 eowyn systemd-journal[14843]: Forwarding to syslog missed 1343 messages.
On Debian, syslog forwarding is enabled by default, and since syslog-ng
reads from the journal, there's nothing listening on
/run/systemd/journal/syslog, and I get spammed with messages like the
one above.

Lennart> Hmmm, we only increase the counter when we get EAGAIN from the
Lennart> socket. Normally, if nobody is listening on the other side we
Lennart> shouldn't get EAGAIN, we should get ECONNREFUSED or so.

Lennart> Most likely for some reason "syslog.socket" gets started (which
Lennart> establishes the socket), but nothing is then started behind it? That
Lennart> sounds wrong.

Yep, it turns out that was the issue. syslog-ng.service had a
Requires=syslog.socket, a left over from an older version which did not
support reading from the journal. It appears that removing that line
disabled starting the socket, and the forwarding errors stopped too.

Lennart> Note that in systemd upstream we don't do forward-to-syslog anymore,
Lennart> simply because it turned out to be really slow when nobody is
Lennart> listening on the socket, and rsyslog didn't use it anymore. Does
Lennart> Debian turn this on after all?

Yes, because rsyslog (the default syslogd in debian) does not have the
journald reader enabled (or at least, not as a default), as far as I
remember.