[systemd-devel] journald syslog forwarding

What Jóhann describes should work, even though it is a bit hacky.
There is also a TODO item to add the functionality to send syslog packets
over the network to either systemd-journald or a different tool.

Zbyszek

Post by JÃ³hann B. GuÃ°mundsson
Does adding systemd.journald.forward_to_kmsg= ( or configure it in
journald.conf via ForwardToKMsg=yes )
kernel command line work for you?

Holger Winkelmann [TP]

2014-12-11 13:56:24 UTC

Hi,

However there is a need to forward log messages to remote syslog.
We try to avoid to introduce a full blown syslog solution
like i.e. rsyslog to just forward log messages to a remote syslog server.

As we have worked on a ZeroMQ Journal gateway [1], which is a bit more advanced
like the HTTP gateway and allows to "query" the journal apart of just forwarding,
we could imagine to contribute to this work. As well of i.e. adding native GELF [2]
forwarding to a Graylog2 server.

Are there any plans to follow? I.e, having all protocols in a generic gateway,
or having one gateways per protocol? How the should we define which field
form the journal should be forwarded to syslog?

Post by Zbigniew JÄdrzejewski-Szmek
Zbyszek

Post by JÃ³hann B. GuÃ°mundsson
Does adding systemd.journald.forward_to_kmsg= ( or configure it in
journald.conf via ForwardToKMsg=yes )
kernel command line work for you?

[1] https://github.com/travelping/zmq-journal-gatewayd
[2] https://www.graylog2.org/resources/gelf

--
Holger Winkelmann

email: ***@travelping.com

Zbigniew Jędrzejewski-Szmek

2014-12-11 14:34:43 UTC

Post by Holger Winkelmann [TP]
Hi,

However there is a need to forward log messages to remote syslog.
We try to avoid to introduce a full blown syslog solution
like i.e. rsyslog to just forward log messages to a remote syslog server.

As we have worked on a ZeroMQ Journal gateway [1], which is a bit more advanced
like the HTTP gateway and allows to "query" the journal apart of just forwarding,
we could imagine to contribute to this work. As well of i.e. adding native GELF [2]
forwarding to a Graylog2 server.
Are there any plans to follow? I.e, having all protocols in a generic gateway,
or having one gateways per protocol? How the should we define which field
form the journal should be forwarded to syslog?

IIUC, Lennart wanted to add this funcitonality to systemd-journal-upload or
a new tool. I think it might be nicer to add it directly to systemd-journald,
even though it would then use the network. The details are fuzzy atm.

Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

I think that additional forwarders (e.g. GELF) should stay as separate projects.

Zbyszek

Holger Winkelmann [TP]

2014-12-11 14:54:18 UTC

HI,

Post by Holger Winkelmann [TP]
Are there any plans to follow? I.e, having all protocols in a generic gateway,
or having one gateways per protocol? How the should we define which field
form the journal should be forwarded to syslog?

upload sounds a bit of a batch process, but thats just cosmetic wording.

Having this in systems-journald and extend the forward to syslog config with the target
host was our expectation anyway.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

it would be great to have at least the following format to send to syslog:

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msg%\n"

described as rsyslog configuration. All the meta infos are there IMHO.

Post by Zbigniew JÄdrzejewski-Szmek
I think that additional forwarders (e.g. GELF) should stay as separate projects.

I think so as well.

Post by Zbigniew JÄdrzejewski-Szmek
Zbyszek

--
Holger Winkelmann

email: ***@travelping.com

Zbigniew Jędrzejewski-Szmek

2014-12-11 15:31:09 UTC

Post by Holger Winkelmann [TP]
HI,

upload sounds a bit of a batch process, but thats just cosmetic wording.
Having this in systems-journald and extend the forward to syslog config with the target
host was our expectation anyway.

The difference is in how the logs are accessed: if journald itself does the jobs,
they would be forwarded "live". If anything else, the uploader would be a client
which reads the files in /var/log/journal/. The are advantages to both solutions:
the first one might be more robust if writing the logs fails or stops for whatever
reason. The second one will probably send more logs, because sending of logs can
be delayed until the network is up. In the second version, the uploader can also
forward logs from other machines (containers). Now that I spelled it out, the second
version seems nicer.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other fields.

Post by Zbigniew JÄdrzejewski-Szmek
I think that additional forwarders (e.g. GELF) should stay as separate projects.

I think so as well.

Zbyszek

Holger Winkelmann [TP]

2014-12-11 15:39:56 UTC

Hi,

Post by Holger Winkelmann [TP]
upload sounds a bit of a batch process, but thats just cosmetic wording.
Having this in systems-journald and extend the forward to syslog config with the target
host was our expectation anyway.

The difference is in how the logs are accessed: if journald itself does the jobs,
they would be forwarded "live". If anything else, the uploader would be a client
the first one might be more robust if writing the logs fails or stops for whatever
reason. The second one will probably send more logs, because sending of logs can
be delayed until the network is up. In the second version, the uploader can also
forward logs from other machines (containers). Now that I spelled it out, the second
version seems nicer.

I agree with your findings, and basically thats how the zmq journal gateway works as
well. And thanks to bring the "wait for network" up here, you would miss important
boot log entries here.

Would the upload tool learn a new URL for this purpose i.e. syslog://<ip>:514 ?

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name%
%procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other fields.

I agreed as well mapping then into the "struct syslog format" wold be a config
pain I assume. However the one we listed above should be there ?!?!... If people
correlating syslog messages on a central server, time, hostname etc. are meaningful.

Holger

--
Holger Winkelmann

email: ***@travelping.com

Zbigniew Jędrzejewski-Szmek

2014-12-11 16:16:43 UTC

Post by Holger Winkelmann [TP]
Hi,

I agree with your findings, and basically thats how the zmq journal gateway works as
well. And thanks to bring the "wait for network" up here, you would miss important
boot log entries here.
Would the upload tool learn a new URL for this purpose i.e. syslog://<ip>:514 ?

Or a broadcast address, so no configuration is required.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name%
%procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other fields.

I agreed as well mapping then into the "struct syslog format" wold be a config
pain I assume. However the one we listed above should be there ?!?!...

That's rfc5424, right? Then yes.

Post by Holger Winkelmann [TP]
If people
correlating syslog messages on a central server, time, hostname etc. are meaningful.

Zbyszek

Lennart Poettering

2014-12-11 16:18:58 UTC

Post by Holger Winkelmann [TP]
I agree with your findings, and basically thats how the zmq journal gateway works as
well. And thanks to bring the "wait for network" up here, you would miss important
boot log entries here.
Would the upload tool learn a new URL for this purpose i.e. syslog://<ip>:514 ?

Or a broadcast address, so no configuration is required.

I'd really like to see broadcast delivery I must say.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name%
%procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other fields.

I agreed as well mapping then into the "struct syslog format" wold be a config
pain I assume. However the one we listed above should be there ?!?!...

That's rfc5424, right? Then yes.

Is that RFC actually widely adopted? I'd stay as conservative as
possible with all of this.

Lennart

--
Lennart Poettering, Red Hat

Zbigniew Jędrzejewski-Szmek

2014-12-11 16:23:07 UTC

Post by Lennart Poettering

Or a broadcast address, so no configuration is required.

I'd really like to see broadcast delivery I must say.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name%
%procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other
fields.

I agreed as well mapping then into the "struct syslog format" wold be a config
pain I assume. However the one we listed above should be there ?!?!...

That's rfc5424, right? Then yes.

Is that RFC actually widely adopted? I'd stay as conservative as
possible with all of this.

Ooops, sorry, wrong RFC. I think 3164 is the right one.

Zbyszek

Holger Winkelmann [TP]

2014-12-11 16:29:23 UTC

HI,

Post by Lennart Poettering

Post by Zbigniew JÄdrzejewski-Szmek
Or a broadcast address, so no configuration is required.

I'd really like to see broadcast delivery I must say.

as mentioned, why not if it can switched off and unicasted to the IP:514

Post by Lennart Poettering

Post by Zbigniew JÄdrzejewski-Szmek
That's rfc5424, right? Then yes.

Is that RFC actually widely adopted? I'd stay as conservative as
possible with all of this.

Ooops, sorry, wrong RFC. I think 3164 is the right one.

RFC 5324 obsoletes RFC 3164, but if it comes to the "format" of the messages most of
the resources point back to RFC3164, so both is slightly right.

in reply to Lennart, at least I know a couple of people correlating and sorting messages based
on the format mentioned above. As already said I would be conservative either and not try to put
all the structure in structured syslog. but adding the few Meta data to the syslog message would
help.

Holger

--
Holger Winkelmann

email: ***@travelping.com

Holger Winkelmann [TP]

2014-12-11 16:21:50 UTC

HI,

Post by Holger Winkelmann [TP]
Would the upload tool learn a new URL for this purpose i.e. syslog://<ip>:514 ?

Or a broadcast address, so no configuration is required.

Hmmm. Admin guys would not really like broadcast here. But anyway would not argue
much if it can be configured.

Post by Zbigniew JÄdrzejewski-Szmek
Initial plan was to implement the most straighforward syslog forwarding,
so only the MESSAGE field would be sent.

"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name%
%procid% %msg%\n"
described as rsyslog configuration. All the meta infos are there IMHO.

Yes. We just wouldn't go into "structured" syslog messages to carry other fields.

I agreed as well mapping then into the "struct syslog format" wold be a config
pain I assume. However the one we listed above should be there ?!?!...

That's rfc5424, right? Then yes.

Yes, it is

--
Holger Winkelmann

email: ***@travelping.com

Jóhann B. Guðmundsson

2014-12-11 16:09:54 UTC

Post by Zbigniew JÄdrzejewski-Szmek
The difference is in how the logs are accessed: if journald itself does the jobs,
they would be forwarded "live". If anything else, the uploader would be a client
the first one might be more robust if writing the logs fails or stops for whatever
reason. The second one will probably send more logs, because sending of logs can
be delayed until the network is up. In the second version, the uploader can also
forward logs from other machines (containers). Now that I spelled it out, the second
version seems nicer.

I'm not quite following what you said there but I would actually think
the former as in "forward it live" is better, just define a host and a
port in journald.conf as well as perhaps the format of the logs being
sent. native journal, bsd-syslog, json ( or not and just send it
natively ) and perhaps the ability to send just specific journal types
as in...

system journal --> system.journal
User journal --> user-x.journal
Container journal --> container-x.journal
etc.

I personally dont think we should write any "clients" or uploaders other
then perhaps a listener that accepts only native journal output being
sent to it, and probably should rotate those files on tcp disconnects
and stores those "machine/host files" under relevant journal path.

We already have existing log solution like syslog-ng that natively
reads, filters locally and forwards those filtered logs over the wire
and or to a local ( running on the same host ) centralized syslog server
which takes care of anything including and beyond simply sending the log
over the wire...

JBG

Zbigniew Jędrzejewski-Szmek

2014-12-11 16:19:44 UTC

I'm not quite following what you said there but I would actually
think the former as in "forward it live" is better, ju

Journal carries messages from the initramfs. We cannot send them from
the initramfs, unless we bring up the network then, which we don't want
to do just for this purpose. But those messages are stored in /run/log,
and then flushed to /var/log, and the uploader tool can forward them
after the network is established.

Post by JÃ³hann B. GuÃ°mundsson
host and a port in journald.conf as well as perhaps the format of
the logs being sent. native journal, bsd-syslog, json ( or not and
just send it natively ) and perhaps the ability to send just
specific journal types as in...

I don't think this is a good idea. rsyslog and friends already do
such forwarding quite well. We should just aim to be simple replacement
for the common case where RFC5424 is enough.

Zbyszek

Post by JÃ³hann B. GuÃ°mundsson
system journal --> system.journal
User journal --> user-x.journal
Container journal --> container-x.journal
etc.
I personally dont think we should write any "clients" or uploaders
other then perhaps a listener that accepts only native journal
output being sent to it, and probably should rotate those files on
tcp disconnects and stores those "machine/host files" under relevant
journal path.
We already have existing log solution like syslog-ng that natively
reads, filters locally and forwards those filtered logs over the
wire and or to a local ( running on the same host ) centralized
syslog server which takes care of anything including and beyond
simply sending the log over the wire...
JBG

Jóhann B. Guðmundsson

2014-12-11 16:35:24 UTC

I'm not quite following what you said there but I would actually
think the former as in "forward it live" is better, ju

Right but I thought that might be controlled via socket and once the
network would become available it would dump the content of the socket
buffer on the wire...

JBG

Zbigniew Jędrzejewski-Szmek

2014-12-11 16:38:17 UTC

I'm not quite following what you said there but I would actually
think the former as in "forward it live" is better, ju

Right but I thought that might be controlled via socket and once the
network would become available it would dump the content of the
socket buffer on the wire...

You need an address to establish a socket. And the socket queue will not
be enough if you have copious debug or kernel messages.

Zbyszek

Jóhann B. Guðmundsson

2014-12-11 16:43:25 UTC

Post by Zbigniew JÄdrzejewski-Szmek
You need an address to establish a socket. And the socket queue will not
be enough if you have copious debug or kernel messages.

Cant we just tweak those values to suit our needs?

JBG

Zbigniew Jędrzejewski-Szmek

2014-12-11 16:46:52 UTC

Post by JÃ³hann B. GuÃ°mundsson
Cant we just tweak those values to suit our needs?

Post by Zbigniew JÄdrzejewski-Szmek
You need an address to establish a socket.

No.

Post by JÃ³hann B. GuÃ°mundsson
And the socket queue will not

Post by Zbigniew JÄdrzejewski-Szmek
be enough if you have copious debug or kernel messages.

Possibly.

Zbyszek

Holger Winkelmann [TP]

2014-12-11 20:11:32 UTC

Post by JÃ³hann B. GuÃ°mundsson
I'm not quite following what you said there but I would actually
think the former as in "forward it live" is better, ju

Right but I thought that might be controlled via socket and once the
network would become available it would dump the content of the socket
buffer on the wire...

I was thinking about the same, but what about the size Limitation? Is there
not a long standing issue that the socket buffer size is global and not
per socket? I remember a discussion around one of the plumbers conference?

Anyway what abut a two fold approach? the journal learns plain live forwarding
to a remote host or even broadcast. and the messages can be forwarded once
the network is available. If socket controlled works you may win a few more messages
but you never know you get all messages from boot. This is current behavior
of remote syslog forwarding I assume.

If you want more there could be a independent tool works like the systems-journal-gatewayd
or systemd-journal-uplaod and can be started later once network available. This will
read the log and forward all messages from the current boot and following.

Holger

--
Holger Winkelmann
email: ***@travelping.com

Holger Winkelmann [TP]

2014-12-17 20:30:41 UTC

This post might be inappropriate. Click to display it.

Zbigniew Jędrzejewski-Szmek

2014-12-17 20:42:28 UTC

Post by Holger Winkelmann [TP]
Hi, sorry for delay in reply..