Discussion:
Environment-variable security?
(too old to reply)
David Parsley
2018-11-12 19:49:06 UTC
Permalink
It's a fairly common practice to configure services and provide secrets
with environment variables. For instance, both Hubot (made by Github) and
Gopherbot (made by me) can get their Slack token from an environment
variable. In my case, github.com/lnxjedi/ansible-role-gopherbot stores the
Slack bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by marking the
unit file world-inaccessible. I was dismayed to see the log warning about
values being accessible via the API, though super glad that my unprivileged
user couldn't fetch it with a simple systemctl cat gopherbot. I know very
little about DBUS or any APIs for systemd, so wanted to ask - is there some
means by which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon on
server systems?

Thanks,
-David
aleivag
2018-11-12 20:41:52 UTC
Permalink
You can define those secrets on /etc/robotsecret.txt, and then on your unit
you do `EnvironmentFile=/etc/robotsecret.txt`

then you protect /etc/robotsecret.txt as you would normally do

Alvaro Leiva Geisse
Post by David Parsley
It's a fairly common practice to configure services and provide secrets
with environment variables. For instance, both Hubot (made by Github) and
Gopherbot (made by me) can get their Slack token from an environment
variable. In my case, github.com/lnxjedi/ansible-role-gopherbot stores
the Slack bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the
systemd unit file. I had hoped to keep this info to the robot user by
marking the unit file world-inaccessible. I was dismayed to see the log
warning about values being accessible via the API, though super glad that
my unprivileged user couldn't fetch it with a simple systemctl cat
gopherbot. I know very little about DBUS or any APIs for systemd, so
wanted to ask - is there some means by which a non-privileged user can
access the values provided with "Environment=..." lines? Can I disable
this by disabling dbus-daemon on server systems?
Thanks,
-David
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Reindl Harald
2018-11-12 20:54:31 UTC
Permalink
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your
unit you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
and how does that protect anything?

on a webserver running php it's just a one-liner to get $_ENV which is
why sensitive data don't belong there and should never exposed that way
like passwords in teh commandline are plain wrong
Post by aleivag
It's a fairly common practice to configure services and provide
secrets with environment variables. For instance, both Hubot (made
by Github) and Gopherbot (made by me) can get their Slack token from
an environment variable. In my case,
github.com/lnxjedi/ansible-role-gopherbot
<http://github.com/lnxjedi/ansible-role-gopherbot> stores the Slack
bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by
marking the unit file world-inaccessible. I was dismayed to see the
log warning about values being accessible via the API, though super
glad that my unprivileged user couldn't fetch it with a
simple |systemctl cat gopherbot|. I know very little about DBUS or
any APIs for systemd, so wanted to ask - is there some means by
which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon
on server systems?
aleivag
2018-11-12 21:23:35 UTC
Permalink
hi Reindl: I was protecting against "systemctl cat/show" disclosure of
information, as stated in the question; but i agree with you, and there are
always risk in passing credential in env variables, but you can always try
to mitigate them (e.g. the main process can read the variables and then
remove them from the env).
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your
unit you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
and how does that protect anything?

on a webserver running php it's just a one-liner to get $_ENV which is
why sensitive data don't belong there and should never exposed that way
like passwords in teh commandline are plain wrong
Post by aleivag
It's a fairly common practice to configure services and provide
secrets with environment variables. For instance, both Hubot (made
by Github) and Gopherbot (made by me) can get their Slack token from
an environment variable. In my case,
github.com/lnxjedi/ansible-role-gopherbot
<http://github.com/lnxjedi/ansible-role-gopherbot> stores the Slack
bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by
marking the unit file world-inaccessible. I was dismayed to see the
log warning about values being accessible via the API, though super
glad that my unprivileged user couldn't fetch it with a
simple |systemctl cat gopherbot|. I know very little about DBUS or
any APIs for systemd, so wanted to ask - is there some means by
which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon
on server systems?
David Parsley
2018-11-12 22:18:03 UTC
Permalink
I already scrub the environment when executing external scripts, and I've
found that even after os.Unsetenv(...) the full environment is available to
all processes owned by the robot in /proc/<pid>/environ. I'm fleshing out a
solution where the process consumes the environment then scrubs it from
proc w/ execve before starting the main loop that can run external scripts.

I don't know what mechanism made the environment available via API, so I
wasn't sure if using an EnvironmentFile protected against that same
mechanism. If effective, that would be a nice solution that parallels
operation of the dockerized version.

So, anybody know what API calls an unprivileged user can make to get that
information from the unit file? It'd be nice to test before and after to
make sure the measure is effective.

Regards,
-David
Post by aleivag
hi Reindl: I was protecting against "systemctl cat/show" disclosure of
information, as stated in the question; but i agree with you, and there are
always risk in passing credential in env variables, but you can always try
to mitigate them (e.g. the main process can read the variables and then
remove them from the env).
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your
unit you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
and how does that protect anything?
on a webserver running php it's just a one-liner to get $_ENV which is
why sensitive data don't belong there and should never exposed that way
like passwords in teh commandline are plain wrong
Post by aleivag
It's a fairly common practice to configure services and provide
secrets with environment variables. For instance, both Hubot (made
by Github) and Gopherbot (made by me) can get their Slack token from
an environment variable. In my case,
github.com/lnxjedi/ansible-role-gopherbot
<http://github.com/lnxjedi/ansible-role-gopherbot> stores the Slack
bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by
marking the unit file world-inaccessible. I was dismayed to see the
log warning about values being accessible via the API, though super
glad that my unprivileged user couldn't fetch it with a
simple |systemctl cat gopherbot|. I know very little about DBUS or
any APIs for systemd, so wanted to ask - is there some means by
which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon
on server systems?
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
aleivag
2018-11-12 22:53:29 UTC
Permalink
If you use EnvironmentFile the only thing a user could do is systemctl
show, and that will tell them that what environment file was used , but not
it's content...

As long as you unset the env, you should be fine (but I'm not a expert on
this)

El lun., 12 de noviembre de 2018 7:18 p. m., David Parsley <
Post by David Parsley
I already scrub the environment when executing external scripts, and I've
found that even after os.Unsetenv(...) the full environment is available to
all processes owned by the robot in /proc/<pid>/environ. I'm fleshing out a
solution where the process consumes the environment then scrubs it from
proc w/ execve before starting the main loop that can run external scripts.
I don't know what mechanism made the environment available via API, so I
wasn't sure if using an EnvironmentFile protected against that same
mechanism. If effective, that would be a nice solution that parallels
operation of the dockerized version.
So, anybody know what API calls an unprivileged user can make to get that
information from the unit file? It'd be nice to test before and after to
make sure the measure is effective.
Regards,
-David
Post by aleivag
hi Reindl: I was protecting against "systemctl cat/show" disclosure of
information, as stated in the question; but i agree with you, and there are
always risk in passing credential in env variables, but you can always try
to mitigate them (e.g. the main process can read the variables and then
remove them from the env).
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your
unit you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
and how does that protect anything?
on a webserver running php it's just a one-liner to get $_ENV which is
why sensitive data don't belong there and should never exposed that way
like passwords in teh commandline are plain wrong
Post by aleivag
It's a fairly common practice to configure services and provide
secrets with environment variables. For instance, both Hubot (made
by Github) and Gopherbot (made by me) can get their Slack token from
an environment variable. In my case,
github.com/lnxjedi/ansible-role-gopherbot
<http://github.com/lnxjedi/ansible-role-gopherbot> stores the Slack
bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by
marking the unit file world-inaccessible. I was dismayed to see the
log warning about values being accessible via the API, though super
glad that my unprivileged user couldn't fetch it with a
simple |systemctl cat gopherbot|. I know very little about DBUS or
any APIs for systemd, so wanted to ask - is there some means by
which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon
on server systems?
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Lennart Poettering
2018-11-13 10:26:49 UTC
Permalink
Post by aleivag
If you use EnvironmentFile the only thing a user could do is systemctl
show, and that will tell them that what environment file was used , but not
it's content...
As long as you unset the env, you should be fine (but I'm not a expert on
this)
No. Really, don't do this.

The problem reamains that env vars are generally not considered
secrets, and hence are unsuitable for passing secrets.

I already mentioned the problem of propagation. But there's also this:
if for some reason the admin turned on debug logging in systemd you'll
see messages up passed env vars in the debug logs, and no, they are
likely readable by unpriv users, because they go to the console, kmsg,


Again, don't put the env vars there in the first place.

Lennart
--
Lennart Poettering, Red Hat
Michael Chapman
2018-11-14 06:39:33 UTC
Permalink
Post by David Parsley
I already scrub the environment when executing external scripts, and I've
found that even after os.Unsetenv(...) the full environment is available to
all processes owned by the robot in /proc/<pid>/environ.
I'm a bit hesitent to enter this already-too-heated discussion, but I
think this point here is important.

Yes, a process can clear or sanitize its environment, and once done that
is done you can be confident the variables won't be propagated to child
processes. It is important that this be done as early as possible, of
course.

But this sanitization does not necessarily clean up the _original_
environment passed in from the kernel, and this memory is still readable
to other processes running as that user through that /proc/$pid/environ
file. Certainly, glibc's environment-manipulation functions (i.e.
setenv(3), putenv(3), unsetenv(3) and clearenv(3)) do not overwrite it.

If a process wants to hide this content, it needs to explicitly overwrite
this memory or change its address range using prctl(2). Maybe you can
investigate whether this is possible in Python.
Lennart Poettering
2018-11-13 10:18:22 UTC
Permalink
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your unit
you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
Don't do this. This is only partially secure, and that only by
coincidence, not by design. env vars are generally not considered
secrets, and will still propagate down the tree.

If you have secrets pick a place where they are strictly access
controlled, and where this access control is built into the concept
itself. Files on disk work (with their age old UNIX access mode) and
kernel keyrings work too (they have been designed just for this
purpose). env vars do not qualify. Neither in understanding of its
users, not in actual code.

Lennart
--
Lennart Poettering, Red Hat
David Parsley
2018-11-13 12:49:36 UTC
Permalink
I disagree; privacy of environment variables to individual users on the
system is as fundamental as Unix file permissions. If a privileged process
(systemd) is configured to start a service and provide environment
variables to an unprivileged service account, it is a reasonable
expectation that said environment is only available to root and the service
account (and it's child processes), and not other arbitrary
users/processes. From a system security engineering perspective, it would
be better if systemd didn't start a service at all with 0600 on the unit
file, rather than violate the principle of Unix environment privacy, and in
fact should actually just check the world-read bit.

Thanks aleivag; "systemctl show" was what I was looking for; unprivileged,
I was able to see the "Environment=" values, but not the contents of
/etc/gopherbot.env. I'm going to go ahead and update the Ansible role to
operate that way.

Regards,
-David
Post by Lennart Poettering
Post by aleivag
You can define those secrets on /etc/robotsecret.txt, and then on your
unit
Post by aleivag
you do `EnvironmentFile=/etc/robotsecret.txt`
then you protect /etc/robotsecret.txt as you would normally do
Don't do this. This is only partially secure, and that only by
coincidence, not by design. env vars are generally not considered
secrets, and will still propagate down the tree.
If you have secrets pick a place where they are strictly access
controlled, and where this access control is built into the concept
itself. Files on disk work (with their age old UNIX access mode) and
kernel keyrings work too (they have been designed just for this
purpose). env vars do not qualify. Neither in understanding of its
users, not in actual code.
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2018-11-13 14:17:19 UTC
Permalink
Post by David Parsley
I disagree; privacy of environment variables to individual users on the
system is as fundamental as Unix file permissions. If a privileged process
(systemd) is configured to start a service and provide environment
variables to an unprivileged service account, it is a reasonable
expectation that said environment is only available to root and the service
account (and it's child processes), and not other arbitrary
users/processes. From a system security engineering perspective, it would
be better if systemd didn't start a service at all with 0600 on the unit
file, rather than violate the principle of Unix environment privacy, and in
fact should actually just check the world-read bit.
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.

You appear dead set on using env vars for this. It's a very bad choice
however, it's a pity you ignore comments that don't fit in your view
of the world though.

Note that even docker got this right, and their "docker secrets"
feature, stores them in a file, not in an env var:

https://docs.docker.com/engine/swarm/secrets/#how-docker-manages-secrets

I mean, there's a lot to complain in what Docker does, but the way it
looks, at least that they did get right...
Post by David Parsley
Thanks aleivag; "systemctl show" was what I was looking for; unprivileged,
I was able to see the "Environment=" values, but not the contents of
/etc/gopherbot.env. I'm going to go ahead and update the Ansible role to
operate that way.
Urks. I really don't hope this catches on. You are doing it wrong.

Lennart
--
Lennart Poettering, Red Hat
David Parsley
2018-11-13 17:22:27 UTC
Permalink
Google "twelve-factor app". Excerpt from a random article on medium.com:
"... most frameworks support the use of System Environment variables inside
configuration files. For instance, in the Spring Boot yaml file you can
write:
Security.oauth2.client.clientSecret: ${CL_SERVICE_PASSWORD}"

You're correct that's not a Unix security standard, and I couldn't find one
- it's just a fairly common practice because environment variables are easy
to use. Other systems engineers that I work with also rely on their
environment being private from other users on large, multi-user research
computing systems like the ones we manage. It's also a reasonable
expectation that an unprivileged process can't trivially obtain the
contents of a 0600:root:root file under /etc, or that a user-level exploit
of an unrelated service could also trivially obtain that data.

I'll just go ahead and concede that I'm wrong, and that GOPHER_SLACK_TOKEN
and HUBOT_SLACK_TOKEN shouldn't be in an env var, and that my co-workers
are also wrong to store a particular password in their environment.
Regardless, that doesn't make it OK for systemd to hand out the contents of
of that file or make service environment vars available to unprivileged
users. You can think I'm a stubborn damned ignoramus if you like - but I'd
be surprised if you think I'm the *only* damned ignoramus who thinks that
environment data should be private to the process owner and root. I'm just
one of the few who happened to notice the warning in the logs and
investigated it.

I think the real point here is the information disclosure vulnerability.
I'm going to suggest to my team that we consider making
/run/dbus/system_bus_socket 440 or 400 - no reason for these non-privileged
users to have access to that, anyway.

Thanks for giving this some thought.

Regards,

-David
Post by aleivag
Post by David Parsley
I disagree; privacy of environment variables to individual users on the
system is as fundamental as Unix file permissions. If a privileged
process
Post by David Parsley
(systemd) is configured to start a service and provide environment
variables to an unprivileged service account, it is a reasonable
expectation that said environment is only available to root and the
service
Post by David Parsley
account (and it's child processes), and not other arbitrary
users/processes. From a system security engineering perspective, it would
be better if systemd didn't start a service at all with 0600 on the unit
file, rather than violate the principle of Unix environment privacy, and
in
Post by David Parsley
fact should actually just check the world-read bit.
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
You appear dead set on using env vars for this. It's a very bad choice
however, it's a pity you ignore comments that don't fit in your view
of the world though.
Note that even docker got this right, and their "docker secrets"
https://docs.docker.com/engine/swarm/secrets/#how-docker-manages-secrets
I mean, there's a lot to complain in what Docker does, but the way it
looks, at least that they did get right...
Post by David Parsley
Thanks aleivag; "systemctl show" was what I was looking for;
unprivileged,
Post by David Parsley
I was able to see the "Environment=" values, but not the contents of
/etc/gopherbot.env. I'm going to go ahead and update the Ansible role to
operate that way.
Urks. I really don't hope this catches on. You are doing it wrong.
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2018-11-13 17:59:06 UTC
Permalink
Maybe following "random articles" blindly is not really the best
approach to computer security...
Post by David Parsley
You're correct that's not a Unix security standard, and I couldn't find one
- it's just a fairly common practice because environment variables are easy
to use. Other systems engineers that I work with also rely on their
environment being private from other users on large, multi-user research
computing systems like the ones we manage. It's also a reasonable
expectation that an unprivileged process can't trivially obtain the
contents of a 0600:root:root file under /etc, or that a user-level exploit
of an unrelated service could also trivially obtain that data.
Well, if you place the secrets in files with proper access modes, then
that's excellent, that's exactly what I suggested you to do.

I am just saying that then converting these then to env vars is a
really bad idea. Make the relevant tools read the files directly
instead of expecting them in an env var.
Post by David Parsley
I'll just go ahead and concede that I'm wrong, and that GOPHER_SLACK_TOKEN
and HUBOT_SLACK_TOKEN shouldn't be in an env var, and that my co-workers
are also wrong to store a particular password in their environment.
Regardless, that doesn't make it OK for systemd to hand out the contents of
of that file or make service environment vars available to unprivileged
users. You can think I'm a stubborn damned ignoramus if you like - but I'd
be surprised if you think I'm the *only* damned ignoramus who thinks that
environment data should be private to the process owner and root. I'm just
one of the few who happened to notice the warning in the logs and
investigated it.
Well, it's not as easy as you suggest. it's not just the dbus
interface that is built like this, it's all over the place. in
systemd, and other tools too.

I am sorry, but that ship has sailed. You can try to close all the
holes you can find, but you started out with a very leaky concept
which is env vars, that have been around since the 1970s, and are used
all over the placein a myriad of ways, and exposed in gazillions
programs. You don't get to redefine them the way you need them to,
they are already used in ways incompatible with what you are trying to
do.

Or let's say this differently, let's talk again when you convinced the
kernel folks that they flush out all env vars whenever there's a
security transition, i.e. setuid, setgid, selinux context, ….
Post by David Parsley
I think the real point here is the information disclosure vulnerability.
I'm going to suggest to my team that we consider making
/run/dbus/system_bus_socket 440 or 400 - no reason for these non-privileged
users to have access to that, anyway.
I mean, knock yourself out, but uh, oh. This is not how this all works.

Lennart
--
Lennart Poettering, Red Hat
Kenneth Porter
2018-11-13 20:55:08 UTC
Permalink
--On Tuesday, November 13, 2018 6:59 PM +0100 Lennart Poettering
Post by Lennart Poettering
Maybe following "random articles" blindly is not really the best
approach to computer security...
OTOH, most of us learned from reading the writings of others in the
industry. So perhaps the wisdom of this thread should get captured in a
blog post by a security expert so it can be shared around on multiple
mailing lists and forums.

The Unit file documentation should also discourage putting secrets in those
files.
Lennart Poettering
2018-11-13 22:37:18 UTC
Permalink
Post by Kenneth Porter
The Unit file documentation should also discourage putting secrets in those
files.
True. Here's the PR:

https://github.com/systemd/systemd/pull/10764

Lennart
--
Lennart Poettering, Red Hat
Jonathan de Boyne Pollard
2018-11-16 10:33:06 UTC
Permalink
Post by Kenneth Porter
OTOH, most of us learned from reading the writings of others in the
industry. So perhaps the wisdom of this thread should get captured in
a blog post by a security expert so it can be shared around on
multiple mailing lists and forums.
One can learn from the example of Daniel J. Bernstein's checkpassword
design, for starters. It has been around for (if memory serves) just
under two decades.

* https://unix.stackexchange.com/questions/302948/
Reindl Harald
2018-11-13 22:35:26 UTC
Permalink
Post by David Parsley
You're correct that's not a Unix security standard, and I couldn't find
one - it's just a fairly common practice because environment variables
are easy to use.
"common practice" and "easy to use" is in most case sthe opposite of
security, even one of both, but when you combine both you can be pretty
sure it#s everything but secure
Marek Howard
2018-11-14 00:35:38 UTC
Permalink
Post by Lennart Poettering
Post by David Parsley
I disagree; privacy of environment variables to individual users on the
system is as fundamental as Unix file permissions. If a privileged process
(systemd) is configured to start a service and provide environment
variables to an unprivileged service account, it is a reasonable
expectation that said environment is only available to root and the service
account (and it's child processes), and not other arbitrary
users/processes. From a system security engineering perspective, it would
be better if systemd didn't start a service at all with 0600 on the unit
file, rather than violate the principle of Unix environment privacy, and in
fact should actually just check the world-read bit.
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.

It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
Marek Howard
2018-11-14 01:17:02 UTC
Permalink
Post by Marek Howard
Post by Lennart Poettering
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.
It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
If you want some examples:

borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD

Again, it's not so common, but it's not unusual and it's not insecure
if you know what you're doing (which you usually are when you have
powers to create system services).
Reindl Harald
2018-11-14 01:20:26 UTC
Permalink
Post by Marek Howard
Post by Marek Howard
It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD
makes it not better
Post by Marek Howard
Again, it's not so common, but it's not unusual and it's not insecure
if you know what you're doing (which you usually are when you have
powers to create system services)
don't get me wrong, but with systemd a trained monkey can write a system
service in whatever scripting language he wants

* type=simple
* <?php while(true){serviceloop}?>
* done
Tomasz Torcz
2018-11-14 05:13:57 UTC
Permalink
Post by Marek Howard
Post by Marek Howard
Post by Lennart Poettering
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.
It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD
Again, it's not so common, but it's not unusual and it's not insecure
if you know what you're doing (which you usually are when you have
powers to create system services).
Generally, storing secret data in environment is common in
web microservices world, popularised by https://12factor.net/config
But those apps are supposed to be run by Kubernetes or other
container runtime - with dedicated clusters, PID namespaces and so on.
Running them as plain unix (systemd) services is the wrong way
to run them ;)
--
Tomasz Torcz There exists no separation between gods and men:
xmpp: ***@chrome.pl one blends softly casual into the other.
Ryan Gonzalez
2018-11-14 05:28:52 UTC
Permalink
FWIW Kubernetes also supports mounting files containing secrets, which I've
personally found to be easier to work with than environment variables.

--
Ryan (ラむアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/
Post by Tomasz Torcz
Post by Marek Howard
Post by Marek Howard
Post by Lennart Poettering
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.
It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD
Again, it's not so common, but it's not unusual and it's not insecure
if you know what you're doing (which you usually are when you have
powers to create system services).
Generally, storing secret data in environment is common in
web microservices world, popularised by https://12factor.net/config
But those apps are supposed to be run by Kubernetes or other
container runtime - with dedicated clusters, PID namespaces and so on.
Running them as plain unix (systemd) services is the wrong way
to run them ;)
--
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Matthew Hannigan
2018-11-19 04:50:33 UTC
Permalink
Yes, sadly https://12factor.net/ has a lot of currency.

The first time I read that config section I thought
1. it doesn't answer the question; how/where/when do those env vars get
defined?
2. it's not secure

Matt
Post by Tomasz Torcz
Post by Marek Howard
Post by Marek Howard
Post by Lennart Poettering
Well, you are of course welcome to ignore whatever I say, but again,
environment blocks are leaky, they propagate down the process tree,
and are *not* generally understood as being secret.
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.
It's quite handy way to pass secrets and as I said above, there's
really no risk if it's done in cases where it makes sense. Of course
systemd leaking it to everyone makes it not usable with systemd, but
that's not really a problem with environment variables.
borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD
Again, it's not so common, but it's not unusual and it's not insecure
if you know what you're doing (which you usually are when you have
powers to create system services).
Generally, storing secret data in environment is common in
web microservices world, popularised by https://12factor.net/config
But those apps are supposed to be run by Kubernetes or other
container runtime - with dedicated clusters, PID namespaces and so on.
Running them as plain unix (systemd) services is the wrong way
to run them ;)
--
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Lennart Poettering
2018-11-14 08:43:23 UTC
Permalink
Post by Marek Howard
Post by Marek Howard
It is not *that* common to pass secrets via environment variable but
it's nothing unusual, and many programs offer this interface. OpenVPN
comes to bind. Where such interface is offered, propagating down the
process tree is usually not a concern, because such programs usually
don't fork "untrusted" programs.
Well, what's "trusted" or "untrusted" is in the eye of the beholder,
and you never know what your libraries do in the background.

What is common or not is orthogonal to what is a good idea and what is
not.
Post by Marek Howard
borgbackup - BORG_PASSPHRASE
restic - RESTIC_PASSWORD
openssl - env:var
rsync - RSYNC_PASSWORD
hub - GITHUB_PASSWORD, GITHUB_TOKEN
rclone - RCLONE_CONFIG_PASS
smbclient - PASSWD
Well, if you look at those, at least some of them even take the
password from the command line (for example: smbclient). And as
hopefully everyone knows any information included in the command line
is readily visible to everybody else (including unprivileged) on the
system with "ps". And yes, tools doing that tend to override them
quickly after reading, but that's still awfully racy.

I mean, seriously, people do lots of stuff. It doesn't mean that all
what people do is actually a good idea or just safe.

Lennart
--
Lennart Poettering, Red Hat
David Parsley
2018-11-14 13:45:25 UTC
Permalink
Post by Lennart Poettering
I mean, seriously, people do lots of stuff. It doesn't mean that all
what people do is actually a good idea or just safe.
Certainly agreed on this point. It is my belief, however, that system
software, where possible, should implement controls to head-off security
mistakes of this kind made by those people - like me. Right now I think
systemd doesn't go far enough - I was lucky that I saw that warning in the
logs, and the contents of my unit file were insecure when I thought it was
secured by file permissions. I still believe systemd should refuse to start
the service to draw attention to this error in thinking, and that the log
entry should probably reference the doc where the not-uncommon practice of
putting secrets in environment variables is discouraged - certainly with
systemd managed services.

The practice will continue, however; TravisCI, CircleCI, and my own
GopherCI all have mechanisms for providing e.g a GITHUB_PASSWORD
environment variable for automated build publishing. No shock that a DevOps
engineer might make the mistake of applying the same principle with systemd.

Regards,
-David
Marek Howard
2018-11-30 13:25:14 UTC
Permalink
Post by Lennart Poettering
I mean, seriously, people do lots of stuff. It doesn't mean that all
what people do is actually a good idea or just safe.
Certainly agreed on this point. It is my belief, however, that system software, where possible, should implement controls to head-off security mistakes of this kind made by those people - like me. Right now I think systemd doesn't go far enough - I was lucky that I saw that warning in the logs, and the contents of my unit file were insecure when I thought it was secured by file permissions. I still believe systemd should refuse to start the service to draw attention to this error in thinking, and that the log entry should probably reference the doc where the not-uncommon practice of putting secrets in environment variables is discouraged - certainly with systemd managed services.
The practice will continue, however; TravisCI, CircleCI, and my own GopherCI all have mechanisms for providing e.g a GITHUB_PASSWORD environment variable for automated build publishing. No shock that a DevOps engineer might make the mistake of applying the same principle with systemd.
I still don't see how that mistake.

- Lennart mentions that some of tools that take secrets via environment
variables also do so via command line, which is truly insecure. But
that doesn't say anything about insecurity of passing secrets via
environment variables.

- Lennart keeps repeating that passing secrets via environment variable
is insecure because they are passed down the process tree. They are, if
you choose so in execve(), they are also readable by other processes
running under same user from /proc/$PID/environ just like your
~/.bashrc or ~/.netrc. (Don't even start telling me that ~/.netrc is
insecure please. Of course it is once you let other users read it.)

- Worries about untrusted programs and libraries were mentioned,
suggesting that passing secrets via environment variables is insecure,
because not all libraries or programs used by the main program could be
trusted, completely neglecting the fact that reading environment
variable is least worrisome once you let "untrusted" code run as part
of it.

Give me one solid reason why it's insecure to pass passwords via
environment variables please. So far it seems that systemd just broke
this concept for no valid reason and now you try to defend with
completely unrelated reasons. There were certainly more users expecting
that programs sourcing non-world readable environment variables will
keep the variables to themselves, than users wanting to check out what
environment variables systemd set for its processes.
Marek Howard
2018-11-30 13:30:18 UTC
Permalink
Post by Marek Howard
Give me one solid reason why it's insecure to pass passwords via
environment variables please. So far it seems that systemd just broke
this concept for no valid reason and now you try to defend with
completely unrelated reasons. There were certainly more users expecting
that programs sourcing non-world readable environment variables will
keep the variables to themselves, than users wanting to check out what
environment variables systemd set for its processes.
And just to be clear, I'm talking about

EnvironmentFile=/foo/bar.env

and not variables set via unit files.
Lennart Poettering
2018-11-30 13:53:57 UTC
Permalink
Post by Marek Howard
- Lennart keeps repeating that passing secrets via environment variable
is insecure because they are passed down the process tree. They are, if
you choose so in execve(), they are also readable by other processes
running under same user from /proc/$PID/environ just like your
~/.bashrc or ~/.netrc. (Don't even start telling me that ~/.netrc is
insecure please. Of course it is once you let other users read it.)
Well, they are propagated down the process tree *by default*. That's
the problem. Almost nothing in this world sanitizes env vars. su/sudo
do, but everything passes them on, including across suid/sgid/fcaps
priv boundaries.

So, it doesn't matter if you *can* suppress them. Fact is that they
generally are *not* suppressed, and you can stick your head in the
sand as much as you like, but that's not going to change.

Why do you think the per-session, per-user, per-process, per-thread
kernel keyring was added if env vars would be fine? Precisely because
you can limit exactly how the passwords are propagated, and every
access to them requires a priv check. And that's what you want here:
control of propagation and access checks on read.

If you store the files in some file, and set an env var to the file
you'll get the access checks at least (though still no propagation
controler), which is why I am proposing that to you.

Lennart
--
Lennart Poettering, Red Hat
Marek Howard
2018-11-30 16:04:39 UTC
Permalink
Post by Lennart Poettering
Post by Marek Howard
- Lennart keeps repeating that passing secrets via environment variable
is insecure because they are passed down the process tree. They are, if
you choose so in execve(), they are also readable by other processes
running under same user from /proc/$PID/environ just like your
~/.bashrc or ~/.netrc. (Don't even start telling me that ~/.netrc is
insecure please. Of course it is once you let other users read it.)
Well, they are propagated down the process tree *by default*. That's
the problem. Almost nothing in this world sanitizes env vars. su/sudo
do, but everything passes them on, including across suid/sgid/fcaps
priv boundaries.
So, it doesn't matter if you *can* suppress them. Fact is that they
generally are *not* suppressed, and you can stick your head in the
sand as much as you like, but that's not going to change.
I understand, but that's by design and there's nothing wrong with that.
It's even useful for the case where you want wrap a thing with a
script.

I still don't understand why this is a problem. If a program expects a
secret being passed via environment variable, you don't expect this
program to spawn an executable which can do malicious execution (e.g.
that could be controlled by network) and if it really does, then that's
a bug in the program and reading a password from an environment
variable is least severe of the problems that come from it.
Lennart Poettering
2018-11-30 17:16:10 UTC
Permalink
Post by Marek Howard
Post by Lennart Poettering
Post by Marek Howard
- Lennart keeps repeating that passing secrets via environment variable
is insecure because they are passed down the process tree. They are, if
you choose so in execve(), they are also readable by other processes
running under same user from /proc/$PID/environ just like your
~/.bashrc or ~/.netrc. (Don't even start telling me that ~/.netrc is
insecure please. Of course it is once you let other users read it.)
Well, they are propagated down the process tree *by default*. That's
the problem. Almost nothing in this world sanitizes env vars. su/sudo
do, but everything passes them on, including across suid/sgid/fcaps
priv boundaries.
So, it doesn't matter if you *can* suppress them. Fact is that they
generally are *not* suppressed, and you can stick your head in the
sand as much as you like, but that's not going to change.
I understand, but that's by design and there's nothing wrong with that.
It's even useful for the case where you want wrap a thing with a
script.
I still don't understand why this is a problem. If a program expects a
secret being passed via environment variable, you don't expect this
program to spawn an executable which can do malicious execution (e.g.
that could be controlled by network) and if it really does, then that's
a bug in the program and reading a password from an environment
variable is least severe of the problems that come from it.
Well, you don't know what libraries and code you use do in the
background. You know, what you you are doing is simply not how you do
security. When you do security you restrict access as much as you can,
you limit propagation. Env vars are the opposite of that.

But anyway, I think this discussion is pointless. I get the impression
that whatever I tell you you'll ignore it anyway, and keep asking
"why, why?".

But maybe it helps if it's not me who tells you this, but some other
web people:

https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/
http://movingfast.io/articles/environment-variables-considered-harmful/
https://blog.fortrabbit.com/how-to-keep-a-secret

That's just what a quick Google reveals.

Anyway, let's leave this discussion with that.

Lennart
--
Lennart Poettering, Red Hat
Marek Howard
2018-11-30 18:57:04 UTC
Permalink
Post by Lennart Poettering
Post by Marek Howard
I understand, but that's by design and there's nothing wrong with that.
It's even useful for the case where you want wrap a thing with a
script.
I still don't understand why this is a problem. If a program expects a
secret being passed via environment variable, you don't expect this
program to spawn an executable which can do malicious execution (e.g.
that could be controlled by network) and if it really does, then that's
a bug in the program and reading a password from an environment
variable is least severe of the problems that come from it.
Well, you don't know what libraries and code you use do in the
background. You know, what you you are doing is simply not how you do
security. When you do security you restrict access as much as you can,
you limit propagation. Env vars are the opposite of that.
But anyway, I think this discussion is pointless. I get the impression
that whatever I tell you you'll ignore it anyway, and keep asking
"why, why?".
I get the same impression of you. I've repeated several times that
having libraries read environment variables is least sever concern if
you really expect code you execute to be malicious.

If using dedicated configuration file with secrets is supposed to
remedy it, then please explain to me how please. Because I've the
impression that if a malicious library can read environment variables,
it can pretty much read that configuration file as well.

And if you're that security conscious that you restrict system calls,
what are the odds that you forget to clean the environment?
Lennart Poettering
2018-11-30 19:36:01 UTC
Permalink
Post by Marek Howard
Post by Lennart Poettering
Well, you don't know what libraries and code you use do in the
background. You know, what you you are doing is simply not how you do
security. When you do security you restrict access as much as you can,
you limit propagation. Env vars are the opposite of that.
But anyway, I think this discussion is pointless. I get the impression
that whatever I tell you you'll ignore it anyway, and keep asking
"why, why?".
I get the same impression of you. I've repeated several times that
having libraries read environment variables is least sever concern if
you really expect code you execute to be malicious.
Well, there's a lot more shades between "not malicious" and
"malicious". Come on, read a book about computer security. It will
tell you that everything should get as little access to stuff as
reasonably possible. Least privileges and things. UNIX isn't really
designed with that too much in mind, but we do intend to do things
that way today. The opposite of doing things like that is placing
secrets in env vars and giving anything you start or any of the libs,
helpers, child processes you start free access to those, because you
are lazy.
Post by Marek Howard
If using dedicated configuration file with secrets is supposed to
remedy it, then please explain to me how please. Because I've the
impression that if a malicious library can read environment variables,
it can pretty much read that configuration file as well.
it's very easy: getenv() makes no access check. open() does. Files
have access modes, i.e. those rwxrwxrwx things, and
selinux/smack/apparmor on top. Environment variables do not. Whenever
you open a file for reading the kernel hence checks for you whether
you actually are allowed to have access. For environment blocks that
doesn't happen, everybody gets the full thing dropped on its feet, by
default, for free, without *any* restrictions.

Again, let me compare this for you in a pretty table:

| propagation control | life-cycle control | access control
---------------+---------------------+--------------------+----------------
kernel keyring | YES YES YES YES YES | YES YES YES YES YE | YES YES YES YE
files | NO NO NO NO NO NO N | NO NO NO NO NO NO | YES YES YES YE
environment | NO NO NO NO NO NO N | NO NO NO NO NO NO | NO NO NO NO NO

of course, you could extend this table, and mention paging control
(i.e. whether passwords can be swapped out or not), and other
concepts, but let's keep things simple.
Post by Marek Howard
And if you're that security conscious that you restrict system calls,
what are the odds that you forget to clean the environment?
We do that in systemd. All services invoked by systemd start with a
clean environment, and don't inherit things from the invoking client
(which is very different from sysvinit btw, where the env block for
daemons was inherited from the user session that ran
"/etc/init.d/foobar start").

Lennart
--
Lennart Poettering, Red Hat
Cristian Rodríguez
2018-11-13 14:48:37 UTC
Permalink
Post by David Parsley
I disagree; privacy of environment variables to individual users on the
system is as fundamental as Unix file permissions.
Please find us ONE reference to a relevant, current *nix standard that
says environment variables are either secret, suitable for secrets,
private or the system is expected to treat them as such.

Breaking news.. no such requirement or even suggestion exists,
environment variables just work this way. No need to be so stubborn when
the world does not work the way you want to..just find a different
proper solution to fix the problem.
Lennart Poettering
2018-11-13 10:15:09 UTC
Permalink
Post by David Parsley
It's a fairly common practice to configure services and provide secrets
with environment variables.
Passing secrets through env vars is marginally better than passing
them in via cmdline params, but still bad, bad, bad
practice. Seriously, don't do that.

Env vars suck for this. They are generally not considered secret, and
thus they leak everywhere, as programs generally don't assume them to
be. As one example: "systemctl show -p Environment …" will show to
unpriv users the env vars you pass to any service of your choice.

Moreover, env vars are by default inherited down the process tree, and
this means code that really shouldn't see them might end up seeing
them. The lifecycle of secrets needs to be tightly controlled, and as
limited as possible. Env vars are the opposite of that. They are
propagated agressively by default, even across security boundaries
(setuid/setgid!) and everything.

Seriously, don't do it that way. If you do it like that, then you are
building an insecure system.

There are other options available. The optimal way would be to stick
them in the kernel keyring. In the kernel keyring you get some
guarantees about lifecycle, access control, even paging that you
otherwise generally don't get. If the keyring doesn't work, then stick
them in a regular file in your $RUNTIME_DIRECTORY and make sure that
files has 0600 access mode or so. You can even set an env var to the
path of such a file if you really want to use the env vars for
something. But please, never stick passwords directly there.

Another option is to allocate a pipe(), write the password in there,
then pass over the reading side of the pipe to the process you
fork. It's not ideal since the fd would theoretically be propagated
further down the tree, but it does have nice properties in that you
can read the pw off the pipe only once, and then it's gone. It also
has the benefit over the file solution suggested above that it won't
leave artifacts in the fs by design.
Post by David Parsley
For instance, both Hubot (made by Github) and
Gopherbot (made by me) can get their Slack token from an environment
variable. In my case, github.com/lnxjedi/ansible-role-gopherbot stores the
Slack bot token with "Environtment=GOPHER_SLACK_TOKEN=xxx" in the systemd
unit file. I had hoped to keep this info to the robot user by marking the
unit file world-inaccessible. I was dismayed to see the log warning about
values being accessible via the API, though super glad that my unprivileged
user couldn't fetch it with a simple systemctl cat gopherbot. I know very
little about DBUS or any APIs for systemd, so wanted to ask - is there some
means by which a non-privileged user can access the values provided with
"Environment=..." lines? Can I disable this by disabling dbus-daemon on
server systems?
Yes, the bus API is generally open for any clients. One way to access
it is "systemctl show -p Environment …", another is directly with busctl.

And no, you cannot reasonably disable this. It's the wrong place. It's
a swiss cheese.

Seriously, the right approach is not to pass these secrets in the env
vars in the first place, not then to try to play hide and seek to make
them hopefully not leak.

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...