[systemd-devel] Permission/updating problems; different behaviour of two identical nspawn containers

Discussion:

Olaf the Lost Viking

2017-08-30 15:24:56 UTC

Hi ML,

currently I am seeing differences between two, what I consider identical,
nspawn-containers which prevents me to update one of them. (Lots of) details
are at the end of the mail.

I set up two (hopefully) identical debian containers in nspawn for a single
service (DNS) on a debian host. Today's "apt upgrade" now throws permissions
problem on _one_ of the containers (ns4 fails, all others still work - ns3
should be identical but some service data):

***@ns4:~# apt upgrade
...
75 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.0 MB of archives.
After this operation, 313 kB of additional disk space will be used.
W: chown to _apt:root of directory /var/cache/apt/archives/partial failed -
_ SetupAPTPartialDirectory (1: Operation not permitted)
Do you want to continue? [Y/n]

Downloading works, but then moving the archives fails:

...
E: Failed to fetch http://security.debian.org/pool/updates/main/p/
_ postgresql-9.6/postgresql-9.6_9.6.4-0+deb9u1_amd64.deb rename failed,
_ Permission denied (/var/cache/apt/archives/partial/
_ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb -> /var/cache/apt/archives/
_ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb).
E: Unable to fetch some archives, maybe run apt-get update or try with --
_ fix-missing?
***@ns4:~#

I also cannot set the correct container group on the host! (Please see an
example at the very end of the mail.) Neither in the HOST, nor in the ns4
journal anything is shown.

Following I try to give as much information I consider as relevant as I can.
Please do not hesitate to ask for more details. The system is not critical and
can be rebooted (which I already did) or whatever.

Thanks a lot!

== Host
***@HOST:~# cat /etc/debian_version
9.1
***@HOST:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
***@HOST:~# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
ns3 container systemd-nspawn debian 9 10.225.32.1...
ns4 container systemd-nspawn debian 9 10.225.64.1...
nsrec2 container systemd-nspawn debian 9 10.225.1.1...

3 machines listed.
***@HOST:~#

== nspawn container 1 (ns3) ==
***@ns3:~# cat /etc/debian_version
9.1
***@ns3:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

== nspawn container 2 (ns4) ==
***@ns4:~# cat /etc/debian_version
9.1
***@ns4:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

The configuration of both containers look the same to me:

== nspawn config
***@HOST:~# cat /etc/systemd/nspawn/ns3.nspawn
[Exec]
# -> guid parse bug in the kernel
#PrivateUsers=false

[Files]
# -> dynamic uid mounts apt w/o root access
#Bind=/var/cache/apt/
#Bind=/var/lib/apt/

***@HOST:~# diff /etc/systemd/nspawn/ns3.nspawn /etc/systemd/nspawn/
ns4.nspawn
***@HOST:~#

== mount config
***@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3.mount
[Unit]
Before=local-fs.target

[Install]
WantedBy=local-fs.target

[Mount]
What=/dev/disk/by-label/virt
Where=/var/lib/machines/ns3/
Type=btrfs
Options=noatime,nodiratime,subvol=vm-***@active
***@HOST:~# cat /etc/systemd/system/var-lib-machines-ns3-var-cache.mount
[Unit]
Before=local-fs.target

[Install]
WantedBy=local-fs.target

[Mount]
What=/dev/disk/by-label/virt
Where=/var/lib/machines/ns3//var/cache
Type=btrfs
Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var-
_ ***@active
***@HOST:~#

***@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3.mount /etc/
_ systemd/system/var-lib-machines-ns4.mount
9c9
< Where=/var/lib/machines/ns3/
---

Where=/var/lib/machines/ns4/

11c11
< Options=noatime,nodiratime,subvol=vm-***@active
---
***@HOST:~# diff /etc/systemd/system/var-lib-machines-ns3-var-cache.mount /
_ etc/systemd/system/var-lib-machines-ns4-var-cache.mount
9c9
< Where=/var/lib/machines/ns3//var/cache
---

Where=/var/lib/machines/ns4//var/cache

11c11
< Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var-
_ ***@active
---

Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns4_var-

_ ***@active
***@HOST:~#

***@HOST:~# mount | grep 'ns[34].*/cache'
/dev/mapper/volg-virt on /var/lib/machines/ns4/var/cache type btrfs
_ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=331,subvol=/
_ vm-ns4_var-***@active)
/dev/mapper/volg-virt on /var/lib/machines/ns3/var/cache type btrfs
_ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=350,subvol=/
_ vm-ns3_var-***@active)
***@HOST:~#

***@HOST:~# btrfs subvolume list /var/lib/btrfs/ | grep
_ 'ns[34].*cache'
ID 331 gen 68872 top level 5 path vm-ns4_var-***@active
ID 350 gen 67791 top level 5 path vm-ns3_var-***@active
***@HOST:~#

== file permissions
***@HOST:~# ls -l /var/lib/machines/ns3/
total 4
...
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 100 Apr 26 12:33 var
***@HOST:~# ls -l /var/lib/machines/ns3/var/
total 8
...
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 120 May 1 20:48 cache
...
***@HOST:~# ls -l /var/lib/machines/ns3/var/cache/
total 4
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 70 May 1 20:47 apt
...
***@HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/
total 1044
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 22 Aug 30 14:48 archives
-rw-r--r-- 1 vu-ns3-0 vg-ns3-0 641725 Apr 29 12:31 pkgcache.bin
-rw-r--r-- 1 vu-ns3-0 vg-ns3-0 425316 Apr 29 12:31 srcpkgcache.bin
***@HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/archives/
total 0
-rw-r----- 1 vu-ns3-0 vg-ns3-0 0 May 1 20:47 lock
drwx------ 1 vu-ns3-104 vg-ns3-0 0 Aug 30 14:41 partial
***@HOST:~#

***@HOST:~# ls -l /var/lib/machines/ns4
total 4
...
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 100 Apr 26 12:33 var
***@HOST:~# ls -l /var/lib/machines/ns4/var/
total 8
...
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 120 Apr 28 22:07 cache
...
***@HOST:~# ls -l /var/lib/machines/ns4/var/cache/
total 4
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 70 Apr 29 12:31 apt
...
***@HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/
total 51920
drwxr-xr-x 1 vu-ns4-0 root 22 Aug 30 14:49 archives
-rw-r--r-- 1 vu-ns4-0 root 26581616 Apr 29 12:31 pkgcache.bin
-rw-r--r-- 1 vu-ns4-0 root 26581534 Apr 29 12:31 srcpkgcache.bin
***@HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/archives/
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
***@HOST:~#

== Problems

As you could see the few lines above, the groups in ns4 aren't correct for
certain files/directories. But correcting them in the guest as well as the
host fails:

***@ns4:/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 root root 0 Apr 28 22:04 lock
drwx------ 1 _apt nogroup 5000 Aug 30 17:01 partial
***@ns4:/var/cache/apt/archives# chgrp root partial/
chgrp: changing group of 'partial/': Operation not permitted
***@ns4:/var/cache/apt/archives#

***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0
_ partial/
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives#

Lennart Poettering

2017-08-31 16:41:38 UTC

Permalink

Post by Olaf the Lost Viking
Hi ML,
currently I am seeing differences between two, what I consider identical,
nspawn-containers which prevents me to update one of them. (Lots of) details
are at the end of the mail.
I set up two (hopefully) identical debian containers in nspawn for a single
service (DNS) on a debian host. Today's "apt upgrade" now throws permissions
problem on _one_ of the containers (ns4 fails, all others still work - ns3

Most likely something went wrong with the userns UID mapping... Not
sure what though...

Post by Olaf the Lost Viking
As you could see the few lines above, the groups in ns4 aren't correct for
certain files/directories. But correcting them in the guest as well as the
total 0
-rw-r----- 1 root root 0 Apr 28 22:04 lock
drwx------ 1 _apt nogroup 5000 Aug 30 17:01 partial
chgrp: changing group of 'partial/': Operation not permitted
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
_ partial/
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial

Are you suggesting that doing this on the host has no effect at all?
That's seriously strange...

When you ran this, was the container running?

Lennart

--
Lennart Poettering, Red Hat

Olaf the Lost Viking

2017-09-04 14:06:01 UTC

Permalink

[Sorry for not answering to Lennart's answer directly - it somehow got lost so
I have to copy/paste it from the archive.]

Post by Lennart Poettering

Post by Olaf the Lost Viking
I set up two (hopefully) identical debian containers in nspawn for a single
service (DNS) on a debian host. Today's "apt upgrade" now throws
permissions problem on _one_ of the containers (ns4 fails, all others still

Most likely something went wrong with the userns UID mapping... Not
sure what though...

Post by Olaf the Lost Viking
As you could see the few lines above, the groups in ns4 aren't correct for
certain files/directories. But correcting them in the guest as well as the

Are you suggesting that doing this on the host has no effect at all?
That's seriously strange...

Yes, that's the case - at least for the group ownership. And yes, I agree it's
strange ;-)

Post by Lennart Poettering
When you ran this, was the container running?

Yes, it is running:

***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0
_ partial
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# echo $?
0
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
ns3 container systemd-nspawn debian 9 10.225.32.1...
ns4 container systemd-nspawn debian 9 10.225.64.1...
nsrec2 container systemd-nspawn debian 9 10.225.1.1...

3 machines listed.
***@HOST:/var/lib/machines/ns4/var/cache/apt/archives#

Thanks for having a look!

Olaf the Lost Viking

2017-09-04 19:53:25 UTC

Permalink

[Sorry for not answering to Lennart's mail directly - it somehow got lost on
my side so I have to copy/paste it from the archive.]

Post by Lennart Poettering

Most likely something went wrong with the userns UID mapping... Not
sure what though...

Post by Olaf the Lost Viking
As you could see the few lines above, the groups in ns4 aren't correct for
certain files/directories. But correcting them in the guest as well as the

Are you suggesting that doing this on the host has no effect at all?
That's seriously strange...

Yes, that's the case - at least for the group ownership. And yes, I agree it's
strange...

Post by Lennart Poettering
When you ran this, was the container running?