Discussion:
[systemd-nspawn] nginx: [emerg] open() "/dev/stderr" failed (6: No such device or address)
(too old to reply)
Lennart Poettering
2015-04-24 13:24:13 UTC
Permalink
Hi,
I run nginx in a CentOS 7.0 container via systemd-nspawn. nginx logs to
stderr/stdout via configuration to capture logs via journald.
nginx.conf
error_log /dev/stderr warn;
If I use systemd 219-1 (-1 is the package number of Arch Linux) which seems
to be a non-patched systemd 219, everything is fine. If I upgrade to systemd
219-6, nginx cannot be started via systemd-nspawn. systemd 219-6 includes
this patch "https://projects.archlinux.org/svntogit/packages.git/tree/repos/core-x86_64/0001-nspawn-when-connected-to-pipes-for-stdin-stdout-pass.patch?h=packages/systemd".
BTW: I see the same error if I use systemd-git-HEAD.
I see the following errors in journal - I tried bot "stderr" and "stdout".
Apr 24 04:48:12 server systemd-nspawn[421]: nginx: [emerg] open()
"/dev/stdout" failed (6: No such device or address)
Apr 24 04:48:45 server systemd-nspawn[496]: nginx: [emerg] open()
"/dev/stderr" failed (6: No such device or address)
Any idea what the precise syscall is that triggers that? i.e. what
strace says?
If I run the container with
sudo /usr/bin/systemd-nspawn --register=no -M docker-centos-nginx
What happens if you use "nsenter" instead to join all namespaces of
the running nginx container and invoke a shell there, and then try to
acess /dev/stderr? Does this also work?

What happens if you use "dd" to write to /dev/stdout? Does that work,
too? (i think that bash handles /dev/stderr specially when you use it
with redirection, that's why I am asking).

Lennart
--
Lennart Poettering, Red Hat
Mantas Mikulėnas
2015-04-24 13:51:27 UTC
Permalink
On Fri, Apr 24, 2015 at 4:24 PM, Lennart Poettering
Post by Lennart Poettering
Hi,
I run nginx in a CentOS 7.0 container via systemd-nspawn. nginx logs to
stderr/stdout via configuration to capture logs via journald.
nginx.conf
error_log /dev/stderr warn;
If I use systemd 219-1 (-1 is the package number of Arch Linux) which seems
to be a non-patched systemd 219, everything is fine. If I upgrade to systemd
219-6, nginx cannot be started via systemd-nspawn. systemd 219-6 includes
this patch "https://projects.archlinux.org/svntogit/packages.git/tree/repos/core-x86_64/0001-nspawn-when-connected-to-pipes-for-stdin-stdout-pass.patch?h=packages/systemd".
BTW: I see the same error if I use systemd-git-HEAD.
I see the following errors in journal - I tried bot "stderr" and "stdout".
Apr 24 04:48:12 server systemd-nspawn[421]: nginx: [emerg] open()
"/dev/stdout" failed (6: No such device or address)
Apr 24 04:48:45 server systemd-nspawn[496]: nginx: [emerg] open()
"/dev/stderr" failed (6: No such device or address)
Any idea what the precise syscall is that triggers that? i.e. what
strace says?
It kind of makes sense when stdout is a socket, since
open(/dev/stdout) or open(/proc/self/fd/*) doesn't just dup that fd,
it tries to open the file anew (including permission checks and
everything). A bit annoying.
--
Mantas Mikulėnas <***@gmail.com>
Lennart Poettering
2015-04-24 13:53:49 UTC
Permalink
Post by Mantas Mikulėnas
On Fri, Apr 24, 2015 at 4:24 PM, Lennart Poettering
Post by Lennart Poettering
Hi,
I run nginx in a CentOS 7.0 container via systemd-nspawn. nginx logs to
stderr/stdout via configuration to capture logs via journald.
nginx.conf
error_log /dev/stderr warn;
If I use systemd 219-1 (-1 is the package number of Arch Linux) which seems
to be a non-patched systemd 219, everything is fine. If I upgrade to systemd
219-6, nginx cannot be started via systemd-nspawn. systemd 219-6 includes
this patch "https://projects.archlinux.org/svntogit/packages.git/tree/repos/core-x86_64/0001-nspawn-when-connected-to-pipes-for-stdin-stdout-pass.patch?h=packages/systemd".
BTW: I see the same error if I use systemd-git-HEAD.
I see the following errors in journal - I tried bot "stderr" and "stdout".
Apr 24 04:48:12 server systemd-nspawn[421]: nginx: [emerg] open()
"/dev/stdout" failed (6: No such device or address)
Apr 24 04:48:45 server systemd-nspawn[496]: nginx: [emerg] open()
"/dev/stderr" failed (6: No such device or address)
Any idea what the precise syscall is that triggers that? i.e. what
strace says?
It kind of makes sense when stdout is a socket, since
open(/dev/stdout) or open(/proc/self/fd/*) doesn't just dup that fd,
it tries to open the file anew (including permission checks and
everything). A bit annoying.
Well, but it's not a socket here, is it? Peter?

Lennart
--
Lennart Poettering, Red Hat
Mantas Mikulėnas
2015-04-24 14:33:12 UTC
Permalink
On Fri, Apr 24, 2015 at 4:53 PM, Lennart Poettering
Post by Lennart Poettering
Post by Mantas Mikulėnas
On Fri, Apr 24, 2015 at 4:24 PM, Lennart Poettering
Post by Lennart Poettering
Hi,
I run nginx in a CentOS 7.0 container via systemd-nspawn. nginx logs to
stderr/stdout via configuration to capture logs via journald.
nginx.conf
error_log /dev/stderr warn;
If I use systemd 219-1 (-1 is the package number of Arch Linux) which seems
to be a non-patched systemd 219, everything is fine. If I upgrade to systemd
219-6, nginx cannot be started via systemd-nspawn. systemd 219-6 includes
this patch "https://projects.archlinux.org/svntogit/packages.git/tree/repos/core-x86_64/0001-nspawn-when-connected-to-pipes-for-stdin-stdout-pass.patch?h=packages/systemd".
BTW: I see the same error if I use systemd-git-HEAD.
I see the following errors in journal - I tried bot "stderr" and "stdout".
Apr 24 04:48:12 server systemd-nspawn[421]: nginx: [emerg] open()
"/dev/stdout" failed (6: No such device or address)
Apr 24 04:48:45 server systemd-nspawn[496]: nginx: [emerg] open()
"/dev/stderr" failed (6: No such device or address)
Any idea what the precise syscall is that triggers that? i.e. what
strace says?
It kind of makes sense when stdout is a socket, since
open(/dev/stdout) or open(/proc/self/fd/*) doesn't just dup that fd,
it tries to open the file anew (including permission checks and
everything). A bit annoying.
Well, but it's not a socket here, is it? Peter?
Hmm, I'm pretty sure the default StandardOutput=journal means stdout
will be a socket connection to journald, doesn't it?

(And since it's a process-specific thing, "echo "asdf" > /dev/stdout"
from an interactive shell will merely test the shell's stdout (which
is a tty), not nginx's stdout...)
--
Mantas Mikulėnas <***@gmail.com>
Lennart Poettering
2015-04-24 14:40:07 UTC
Permalink
Post by Mantas Mikulėnas
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Lennart Poettering
Any idea what the precise syscall is that triggers that? i.e. what
strace says?
It kind of makes sense when stdout is a socket, since
open(/dev/stdout) or open(/proc/self/fd/*) doesn't just dup that fd,
it tries to open the file anew (including permission checks and
everything). A bit annoying.
Well, but it's not a socket here, is it? Peter?
Hmm, I'm pretty sure the default StandardOutput=journal means stdout
will be a socket connection to journald, doesn't it?
Ah, true!
Post by Mantas Mikulėnas
(And since it's a process-specific thing, "echo "asdf" > /dev/stdout"
from an interactive shell will merely test the shell's stdout (which
is a tty), not nginx's stdout...)
Indeed.

I figure /dev/stderr is simply not compatible with sockets, regardless
if nspawn is in the mix or not... WHich actually came up before, and I
think is someething to accept...

People should really use the shell construct "2>" instead of ">
/dev/stderr" if they want the redirect to work always.

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-04-27 13:39:52 UTC
Permalink
BTW: I did the `echo "asdf" > /dev/stderr`-thing just to test if
`/dev/stderr` worked as expected.
/dev/stderr does not work for socket fds, and that's a kernel
limitation, systemd can't do much bout it.
What I do not understand is: What changed in systemd that the
service-unit-thing worked in some earlier versions?
We changed nspawn so that it may be included in shell
pipelines. Effectively this meant passing through the original
stdin/stdout that nspawn got all the way down to PID 1 inside the
container. We do so now if we are invoked non-interactively, i.e. with
stdin/stdout not being a tty.

Previously, we would never pass through fds, but always create a pty
inside the container and automatically forward all bytes of
stdin/stdout from outside the container to it and back. However, that
broke shell pipelines, since it ate up the independent EOF on stdin
and stdout: ptys cannot signal those individually (there's only a
hangup that terminates both directions at once), but that's a property
you inherently need for any kind of pipelines.

I am pretty sure that the new behaviour is a ton more correct though:
with this you get the same behaviour if you start a process
non-intractively as a service or inside an nspawn container, the same
fds, and hence the same (broken) /dev/stderr semantics.
And what can I do to make it work again? There seems to be no other
logging target _today_ both for nginx and apache which makes them
compatible with journald.
Do not use /dev/stderr. If you are in a shell script replace this:

echo foobar > /dev/stderr

with this

echo foobar 1>&2

The latter will just duplicate stdin to stderr, the former will reopen
stdin as stderr. Which is a difference, though an non-obvious one,
that is further complicated that GNU bash (though not necessarily
other shells) actually automatically do the second command if you pass
it the first command. The first command does not work (in non-bash
shells..) if stdout is a socket, the second command does.

Lennart
--
Lennart Poettering, Red Hat
Mantas Mikulėnas
2015-04-27 13:59:47 UTC
Permalink
Post by Lennart Poettering
BTW: I did the `echo "asdf" > /dev/stderr`-thing just to test if
`/dev/stderr` worked as expected.
/dev/stderr does not work for socket fds, and that's a kernel
limitation, systemd can't do much bout it.
What I do not understand is: What changed in systemd that the
service-unit-thing worked in some earlier versions?
We changed nspawn so that it may be included in shell
pipelines. Effectively this meant passing through the original
stdin/stdout that nspawn got all the way down to PID 1 inside the
container. We do so now if we are invoked non-interactively, i.e. with
stdin/stdout not being a tty.
Previously, we would never pass through fds, but always create a pty
inside the container and automatically forward all bytes of
stdin/stdout from outside the container to it and back. However, that
broke shell pipelines, since it ate up the independent EOF on stdin
and stdout: ptys cannot signal those individually (there's only a
hangup that terminates both directions at once), but that's a property
you inherently need for any kind of pipelines.
with this you get the same behaviour if you start a process
non-intractively as a service or inside an nspawn container, the same
fds, and hence the same (broken) /dev/stderr semantics.
And what can I do to make it work again? There seems to be no other
logging target _today_ both for nginx and apache which makes them
compatible with journald.
echo foobar > /dev/stderr
with this
echo foobar 1>&2
The latter will just duplicate stdin to stderr, the former will reopen
stdin as stderr. Which is a difference, though an non-obvious one,
that is further complicated that GNU bash (though not necessarily
other shells) actually automatically do the second command if you pass
it the first command. The first command does not work (in non-bash
shells..) if stdout is a socket, the second command does.
I'm guessing from the error message that it's not a shell script but nginx
itself configured to use "/dev/stderr" as its log file, so there's no >&
that could be used...
Post by Lennart Poettering
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-04-27 14:09:34 UTC
Permalink
Post by Mantas Mikulėnas
I'm guessing from the error message that it's not a shell script but nginx
itself configured to use "/dev/stderr" as its log file, so there's no >&
that could be used...
If this indeed is the case, try using /dev/console instead, this is
also forwarded to stderr by nspawn...

Lennart
--
Lennart Poettering, Red Hat
Mantas Mikulėnas
2015-04-27 18:25:15 UTC
Permalink
Post by Mantas Mikulėnas
I'm guessing from the error message that it's not a shell script but
nginx
Post by Mantas Mikulėnas
itself configured to use "/dev/stderr" as its log file, so there's no >&
that could be used...
Correct - see http://nginx.org/en/docs/ngx_core_module.html
Syntax: error_log file | stderr |
syslog:server=address[,parameter=value] | memory:size [debug |
info |
notice | warn | error | crit | alert | emerg];
Default: error_log logs/error.log error;
Context: main, http, stream, server, location
Hmm, but that already lists a native config keyword for "stderr"?
--
Mantas Mikulėnas <***@gmail.com>
Lennart Poettering
2015-04-28 09:28:33 UTC
Permalink
Post by Mantas Mikulėnas
Hmm, but that already lists a native config keyword for "stderr"?
Yes, I saw that too late. I copied the default configuration of the Arch Linux
nginx package and used that.
So, does it work if you use the config keyword "stderr" there?

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-04-27 19:43:34 UTC
Permalink
Post by Mantas Mikulėnas
I'm guessing from the error message that it's not a shell script but nginx
itself configured to use "/dev/stderr" as its log file, so there's no >&
that could be used...
Correct - see http://nginx.org/en/docs/ngx_core_module.html
Syntax: error_log file | stderr |
syslog:server=address[,parameter=value] | memory:size [debug | info |
notice | warn | error | crit | alert | emerg];
Default: error_log logs/error.log error;
Context: main, http, stream, server, location
What precisely is the setting you picked?

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-04-27 13:25:03 UTC
Permalink
Maybe syslog will do the trick?
Well, the journal will do the trick, if you run systemd inside your
container. If you don't, then bind mounting the syslog socket might
suffice.
Do I need a syslog daemon to receive messages on UDP 514, or is/will be
systemd-journal-remote able to handle this? Didn't found a clue about
that in the man-page.
No, journald does not cover that. Use rsyslog or syslog-ng if you care
about classic BSD syslog-over-UDP.

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...