[systemd-devel] [HEADSUP] /var/lock and /var/lock/lockdev

[Lennart Poettering]

Heya,

A few minutes ago there was a discussion on #systemd involving lots of
folks (from various distros even), regarding the chaotic setup of
/var/lock on most distros. As it turns out every distro set this up
differently, and almost all of them insecurely in one way or another.

I'd like to see this cleaned up and standardized in a secure way among
the distros, so here's what I propose for adoption:

/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.

/var/lock/lockdev should be root:lock 0775 and the place for
LCK..xxx style device lock files.

Why?

- We need to separate system service locks (i.e. root-only locks as
stored in /var/lock/lvm/* or /var/lock/subsys/*) from LCK..xxx device
locks (locks which user programs may acquire). Otherwise it is
possible for users which somehow manage to get into the lock group to
replace files like /var/lock/lvm or
/var/lock/subsys by their own files, for example symlinks, which can
be used to to exploit the system if later on an unexpecting root
process accesses the dir again. This is explained in more detail here:

https://bugzilla.redhat.com/show_bug.cgi?id=581884

- We cannot use the sticky bit on the LCK..xxx dir, because user apps
must be able to remove stale lock files of other users:

https://bugzilla.redhat.com/show_bug.cgi?id=145264#c1

- We should not create another world-writable dir, hence limit access to
a group "lock", so that only apps which are in that group can create
an LCK..xxx lock. it is recommended to do that with a library such as
liblockdev, which comes with a little setgid helper which creates the
actual file.

I have also filed a bug against the (dead, but hey, let's think positive)
FHS in the hope this gets standardized one day:

http://bugs.freestandards.org/show_bug.cgi?id=719

I have also changed the default tmpfiles file in systemd
accordingly. Distros may deviate from this by patching this downstream,
but by shipping this as secure default I do hope to gently push
everybody in the same direction.

What does that mean for you, distro packager?

1. File a bug against your distro copying
https://bugzilla.redhat.com/show_bug.cgi?id=581884 if it is
vulnerable, since this is a security hole that should be fixed.

2. Two options:

A: Patch our new defaults out, and ignore our suggestions and do your
own stuff.

B: Adapt the suggested scheme in your distro: introduce a "lock"
group, and change all apps writing LCK..xxx lock files to
do so in /var/lock/lockdev/ and that after becoming a member of
"lock" somehow, in case the process is not privileged.

Ideas? Comments? Suggestions?

Thanks everybody who explained the intricacies of all of this to me,
Michal, Karel, Michael in particular.

Lennart

[Zbigniew Jędrzejewski-Szmek]

Post by Lennart Poettering
Heya,
A few minutes ago there was a discussion on #systemd involving lots of
folks (from various distros even), regarding the chaotic setup of
/var/lock on most distros. As it turns out every distro set this up
differently, and almost all of them insecurely in one way or another.
I'd like to see this cleaned up and standardized in a secure way among
/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.

Hi,
why not /run/lock ?

Zbyszek

[Lennart Poettering]

Post by Zbigniew Jędrzejewski-Szmek

Post by Lennart Poettering
Heya,
A few minutes ago there was a discussion on #systemd involving lots of
folks (from various distros even), regarding the chaotic setup of
/var/lock on most distros. As it turns out every distro set this up
differently, and almost all of them insecurely in one way or another.
I'd like to see this cleaned up and standardized in a secure way among
/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.

Hi,
why not /run/lock ?

Well, both are equivalent. Since the FHS mentions /var/lock only so far
I referred to this directory via /var/lock.

Lennart

--
Lennart Poettering - Red Hat, Inc.

[Ludwig Nussel]

Post by Lennart Poettering
I'd like to see this cleaned up and standardized in a secure way among
/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.
/var/lock/lockdev should be root:lock 0775 and the place for
LCK..xxx style device lock files.

Those strangely named lock files in /var/lock only exist because of
legacy programs. If they are no longer put in /var/lock you're
throwing away the legacy compatibility and you don't actually need
to create those files at all anymore.
So if legacy compat isn't needed one could just use fcntl for
locking the devices (maybe time to fix the kernel in case that
doesn't work already). Programs using liblockdev would transparently
use the new method, we'd have one less ugly setgid program, the stale
lock file problem wouldn't exist anymore and locking would work for
chroots too.

cu
Ludwig

[Lennart Poettering]

Post by Ludwig Nussel

Post by Lennart Poettering
I'd like to see this cleaned up and standardized in a secure way among
/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.
/var/lock/lockdev should be root:lock 0775 and the place for
LCK..xxx style device lock files.

Those strangely named lock files in /var/lock only exist because of
legacy programs. If they are no longer put in /var/lock you're
throwing away the legacy compatibility and you don't actually need
to create those files at all anymore.

Well, there's not doubt that LCK..xxx locks are evil. However, many many
tools use them, hence we need support them. In most packages the path
for these locks can be configured at build time, a task easily
accomplished by distros as shown by the Fedora example. However,
patching this kind of locking out and replacing it by BSD locks or
something like that is a massive amount of work.

I think it is a good idea to ensure that no new software invents new
lock files or directories beneath /var/run. But for legacy software and
for software involving ttyS0 handling there are only two options: a) go
on with the insecure solution most distros have adopted or b) fix
things, and acknowledge that LCK.. will continue to exist for a while,
but at least create them somewhat securely.

Post by Ludwig Nussel
So if legacy compat isn't needed one could just use fcntl for
locking the devices (maybe time to fix the kernel in case that
doesn't work already). Programs using liblockdev would transparently
use the new method, we'd have one less ugly setgid program, the stale
lock file problem wouldn't exist anymore and locking would work for
chroots too.

Don't use fcntl() style locks please. They are evil.

http://0pointer.de/blog/projects/locking
http://0pointer.de/blog/projects/locking2

BSD locks FTW!

Recent fsck versions use bsd locks on device nodes to serialize fscking
on rotating media. That's how things should be done. However, in the
fsck case it is sufficient to lock fsck against fsck. and there was no
pre-existing locking scheme for that. That's different for serial
devices.

Lennart

[Ludwig Nussel]

Post by Lennart Poettering

Post by Ludwig Nussel

Post by Lennart Poettering
I'd like to see this cleaned up and standardized in a secure way among
/var/lock should be root:root 0755 and the place for various system
service lock directories such as /var/lock/subsys or /var/lock/lvm.
/var/lock/lockdev should be root:lock 0775 and the place for
LCK..xxx style device lock files.

Those strangely named lock files in /var/lock only exist because of
legacy programs. If they are no longer put in /var/lock you're
throwing away the legacy compatibility and you don't actually need
to create those files at all anymore.

Well, there's not doubt that LCK..xxx locks are evil. However, many many
tools use them, hence we need support them. In most packages the path
for these locks can be configured at build time, a task easily
accomplished by distros as shown by the Fedora example. However,
patching this kind of locking out and replacing it by BSD locks or
something like that is a massive amount of work.

Chickening out already? Come on :-)
Speaking only of openSUSE no unprivileged program could ever write
to /var/lock without setgid which in turn is centrally tracked. So
programs are either linked to liblockdev (11.3+), are called uucp
itself or are broken and noone noticed or cared anyways.
Fixing the central place liblockdev should be easy.

In any case declaring some directory as the standard place for lock
files doesn't fully solve the problem anyways. The exact lock file
naming isn't quite standardized either. There's at least the FHS
"LCK..name" method and the SVr4 "LK.dev.maj.min" method. Both have
their problem. Ie what if the device name contains slashes? What if
you want to lock a block and a character device with the same
major/minor?
So I'd rather recommend the ttylock.h interface as standard and hide
the implementation details of whether a setgid helper, flock or
whatever is used. Heck, you could even route the ttylock calls
through systemd to have it create the legacy files in /var/lock as
root without the need group writeable stuff (good old resmgr did
that).

The other option would be to leave that legacy crap alone and only
deprecate use of subdirectories in /var/lock. FHS doesn't say
anything about subdirectories there anyways.

cu
Ludwig

[Kay Sievers]

Post by Ludwig Nussel
The other option would be to leave that legacy crap alone and only
deprecate use of subdirectories in /var/lock.

That wold probably the best option. lockdev could just go straight
into /run/lockdev/ then, and leave the silly shared directory with
magic filenames alone.

It's kind of weird, to invent any new subdir there, keeping in mind
how broken the entire concept is. I really doubt, that any other
software will or should be changed to look directly for files in this
new directory.

All in all I don't think /var/lock/lockdev/ makes much sense at all,
and I don't think it should be done in systemd.

Kay

[Lennart Poettering]

Post by Kay Sievers

Post by Ludwig Nussel
The other option would be to leave that legacy crap alone and only
deprecate use of subdirectories in /var/lock.

That wold probably the best option. lockdev could just go straight
into /run/lockdev/ then, and leave the silly shared directory with
magic filenames alone.

I see no value in making these legacy locks even more prominent by
giving them their own directory on level up.

Post by Kay Sievers
It's kind of weird, to invent any new subdir there, keeping in mind
how broken the entire concept is. I really doubt, that any other
software will or should be changed to look directly for files in this
new directory.

Well, it's not invented now, it's how Fedora has been doing things for a
while.

You have three options basically:

1) Leave things as they are with major security holes

2) Go all the way, and remove support for LCK.. lock files from all
programs, port them to flock() instead.

3) Separate the LCK..xxx lock files from the rest, fixing the security
hole, and simply pass a different compile time switch to various
programs.

I don't think we should do #1. #2 is the right thing to do, but it's
nothing I believe we can pull off just like that. The patches are more
than one-liners, and it's a lot of software that needs to be patched.

Leaves #3.

As mentioned I have now placed the rules for /var/lock/lockdev into a
separate tmpfile legacy.conf which hopefully emphasizes that this crap
should go in the long run (and I added comments explaining that to
it). Also, this file is no longer installed if you disable legacy
support in systemd during build time.

I fear LCK.. is much like SysV init scripts something that will take
time to get rid of, and that we will have to support for a while in
systemd. But we can mark it as "legacy" and add support for builds
without it.

Lennart

[Kay Sievers]

Post by Lennart Poettering
I fear LCK.. is much like SysV init scripts something that will take
time to get rid of, and that we will have to support for a while in
systemd. But we can mark it as "legacy" and add support for builds
without it.

Fine. As long as this will go away some day, and is not on set up on
systems by default which don't need it, sounds good to me.

Kay

[Lennart Poettering]

Post by Ludwig Nussel
Speaking only of openSUSE no unprivileged program could ever write
to /var/lock without setgid which in turn is centrally tracked. So
programs are either linked to liblockdev (11.3+), are called uucp
itself or are broken and noone noticed or cared anyways.
Fixing the central place liblockdev should be easy.

I am not sure it is actually.

I believe the most elegant and simple way to lock a tty device is by
just calling flock() on the tty fd. The lockdev API is not suitable for
such an API, since it takes no fd, only a device name. (And it must be
the fd the app actually uses that is locked, because the lock goes away
automatically on close()).

If we do things cleanly, then liblockdev should go, people can just call
flock() directly, no need for an extra dep.

Post by Ludwig Nussel
In any case declaring some directory as the standard place for lock
files doesn't fully solve the problem anyways. The exact lock file
naming isn't quite standardized either. There's at least the FHS
"LCK..name" method and the SVr4 "LK.dev.maj.min" method.

I have never seen LK.x.y.z files. Documentation link?

Post by Ludwig Nussel
Both have
their problem. Ie what if the device name contains slashes? What if
you want to lock a block and a character device with the same
major/minor?
So I'd rather recommend the ttylock.h interface as standard and hide
the implementation details of whether a setgid helper, flock or
whatever is used. Heck, you could even route the ttylock calls
through systemd to have it create the legacy files in /var/lock as
root without the need group writeable stuff (good old resmgr did
that).

ttylock.h seems to be a header that maps things directly onto lockdev.h
calls. It also suffers by the problem that flock() cannot be mapped to
it, since it takes no fd.

Post by Ludwig Nussel
The other option would be to leave that legacy crap alone and only
deprecate use of subdirectories in /var/lock. FHS doesn't say
anything about subdirectories there anyways.

Too many packages use those dirs. I do agree though it's a bad idea to
use them, and people should keep their internal locks in /run (or
/var/run) instead, placing them in /var/lock/ is a bad idea.

So here's what I changed now in git:

I moved the creation of /var/lock/subsys and /var/lock/lockdev into a
seperate "legacy.conf" tmpfiles file, which is only installed when you
enable the legacy/SysV support in systemd at compile time.

On a legacy-free system, especially embedded ones these two directories
won't exist at all hence.

Eventually we will drop these legacy features from systemd. I just doubt
that we can pull it off to remove /var/lock right now, even if we really
should. Simply because too many things use it and there's no strong
incentive for people to fix it, and I don't want to patch them all
myself.

It's one thing extending the base that the FHS describes in a distro,
it's another thing dropping things that it defines, such as the
/var/lock directory.

Now, the only improvement on the brokeness of LCK..xxx style locks we
can pull off easily is seperating them from the other stuff that is
stored in /var/lock. And this is usually simple (compile time switches
in various programs), and actually implemented in Fedora.

I hope this makes some sense.

Lennart

[Ludwig Nussel]

Post by Lennart Poettering

Post by Ludwig Nussel
In any case declaring some directory as the standard place for lock
files doesn't fully solve the problem anyways. The exact lock file
naming isn't quite standardized either. There's at least the FHS
"LCK..name" method and the SVr4 "LK.dev.maj.min" method.

I have never seen LK.x.y.z files. Documentation link?

http://www.airs.com/ian/uucp-doc/uucp_7.html#SEC86
http://www.columbia.edu/kermit/ckuins.html#x10

Post by Lennart Poettering
It's one thing extending the base that the FHS describes in a distro,
it's another thing dropping things that it defines, such as the
/var/lock directory.
Now, the only improvement on the brokeness of LCK..xxx style locks we
can pull off easily is seperating them from the other stuff that is
stored in /var/lock. And this is usually simple (compile time switches
in various programs), and actually implemented in Fedora.
I hope this makes some sense.

If you moving the lock files to a different place you are silently
breaking legacy applications. Sure they can still put the lock files
to /var/lock but other programs in the system just don't honor them
anymore. So as long as there are no inherently unsolvable problems
with lockdev using /var/lock directly I see no need to go the a half
solution /var/lock/lockdev. I'd rather add an rpmlint check to ban
use of /var/lock/subsys on openSUSE, there are only a few packages
using it anyways.

How many packages in Fedora that did not use lockdev already were
actually patched to use /var/lock/lockdev anyways?

cu
Ludwig

[Lennart Poettering]

Post by Ludwig Nussel

Post by Lennart Poettering

Post by Ludwig Nussel
In any case declaring some directory as the standard place for lock
files doesn't fully solve the problem anyways. The exact lock file
naming isn't quite standardized either. There's at least the FHS
"LCK..name" method and the SVr4 "LK.dev.maj.min" method.

I have never seen LK.x.y.z files. Documentation link?

http://www.airs.com/ian/uucp-doc/uucp_7.html#SEC86
http://www.columbia.edu/kermit/ckuins.html#x10

Post by Lennart Poettering
It's one thing extending the base that the FHS describes in a distro,
it's another thing dropping things that it defines, such as the
/var/lock directory.
Now, the only improvement on the brokeness of LCK..xxx style locks we
can pull off easily is seperating them from the other stuff that is
stored in /var/lock. And this is usually simple (compile time switches
in various programs), and actually implemented in Fedora.
I hope this makes some sense.

So as long as there are no inherently unsolvable problems
with lockdev using /var/lock directly I see no need to go the a half
solution /var/lock/lockdev.

There are. A lot of software creates subdirectories beneath
/var/lock, for example LVM. If you allow creation of lockfiles in
/var/lock, then this enables the same programs to break LVM (and
everything else creating subdirs there), and even use LVM to break the
system even further.

That's the point that https://bugzilla.redhat.com/show_bug.cgi?id=581884
tries to make.

Post by Ludwig Nussel
I'd rather add an rpmlint check to ban
use of /var/lock/subsys on openSUSE, there are only a few packages
using it anyways.

On Fedora and a lot of other distros /var/lock/subsys is used by the
main RC script to synchronize execution of SysV scripts.

Post by Ludwig Nussel
How many packages in Fedora that did not use lockdev already were
actually patched to use /var/lock/lockdev anyways?

No idea, this happened before my time.

Lennart

[Ludwig Nussel]

Post by Lennart Poettering

Post by Ludwig Nussel
So as long as there are no inherently unsolvable problems
with lockdev using /var/lock directly I see no need to go the a half
solution /var/lock/lockdev.

There are. A lot of software creates subdirectories beneath
/var/lock, for example LVM. If you allow creation of lockfiles in
/var/lock, then this enables the same programs to break LVM (and
everything else creating subdirs there), and even use LVM to break the
system even further.
That's the point that https://bugzilla.redhat.com/show_bug.cgi?id=581884
tries to make.

Well, that's not nice but not an immediate problem either. You'd
have to exploit a bug in lockdev to gain access to the lock group
first. Same risk as with any other setuid program.

Post by Lennart Poettering

Post by Ludwig Nussel
How many packages in Fedora that did not use lockdev already were
actually patched to use /var/lock/lockdev anyways?

No idea, this happened before my time.

I'd guess not many. At least one of the more obvious candidates,
pppd still puts it's lock file in /var/lock.

cu
Ludwig

[Lennart Poettering]

Post by Ludwig Nussel

Post by Lennart Poettering
There are. A lot of software creates subdirectories beneath
/var/lock, for example LVM. If you allow creation of lockfiles in
/var/lock, then this enables the same programs to break LVM (and
everything else creating subdirs there), and even use LVM to break the
system even further.
That's the point that https://bugzilla.redhat.com/show_bug.cgi?id=581884
tries to make.

Well, that's not nice but not an immediate problem either. You'd
have to exploit a bug in lockdev to gain access to the lock group
first. Same risk as with any other setuid program.

But it defeats the point of the "lock" group. Because it enables code
that runs under that GID to destroy the system as if it was root.

Lennart

[Ludwig Nussel]

Post by Lennart Poettering

Post by Ludwig Nussel

Post by Lennart Poettering
There are. A lot of software creates subdirectories beneath
/var/lock, for example LVM. If you allow creation of lockfiles in
/var/lock, then this enables the same programs to break LVM (and
everything else creating subdirs there), and even use LVM to break the
system even further.
That's the point that https://bugzilla.redhat.com/show_bug.cgi?id=581884
tries to make.

Well, that's not nice but not an immediate problem either. You'd
have to exploit a bug in lockdev to gain access to the lock group
first. Same risk as with any other setuid program.

But it defeats the point of the "lock" group. Because it enables code
that runs under that GID to destroy the system as if it was root.

Tjo. The system just isn't thought out well in several ways.

cu
Ludwig