Discussion:
systemd user instance and raising limits
(too old to reply)
Jeff Solomon
2017-11-19 19:52:40 UTC
Permalink
Raw Message
Hi,

Two questions.

I want to raise the "number of files" limits for the user instance.

First, I set DefaultLimitNOFILE to something higher than the global system
default in /etc/systemd/user.conf and I rebooted.

Then I confirmed that the setting has taken effect:

"systemctl --user show" showed the new DefaultLimitNOFILE and the unit
itself showed the higher setting of LimitNOFILE when I ran "systemctl
--user show foo".

So far everything worked as expected.

However, when I checked "cat /proc/<pid>/limits" on the ExecStart process
of foo.service, I don't see the "number of files" limit has changed.

What did I do wrong?

Second question: if I want to raise the limit just for a single user, how
would I go about it?

Making a change in user.conf would make it apply in all user instances
(assuming I could get it to work).

I have found that if I create /etc/systemd/system/user@<uid>.service and
add LimitNOFILE to the [Service] section of that file, then it will do two
things. First, it actually works whereas editing user.conf did not. Second,
the change only applies to user <uid> and not all users.

I assume I'm not getting how systemd is supposed to work. So please
enlighten me.

Thanks,

Jeff

Machine stats (although I see the same behavior on Ubuntu and on Centos7.3):

$ systemctl --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
+GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

$ uname -a
Linux foo-ubuntu-vm1 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
LSB Version:
core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
Mantas Mikulėnas
2017-11-19 20:22:34 UTC
Permalink
Raw Message
Post by Jeff Solomon
Hi,
Two questions.
I want to raise the "number of files" limits for the user instance.
First, I set DefaultLimitNOFILE to something higher than the global system
default in /etc/systemd/user.conf and I rebooted.
"systemctl --user show" showed the new DefaultLimitNOFILE and the unit
itself showed the higher setting of LimitNOFILE when I ran "systemctl
--user show foo".
So far everything worked as expected.
However, when I checked "cat /proc/<pid>/limits" on the ExecStart process
of foo.service, I don't see the "number of files" limit has changed.
What did I do wrong?
Second question: if I want to raise the limit just for a single user, how
would I go about it?
Making a change in user.conf would make it apply in all user instances
(assuming I could get it to work).
add LimitNOFILE to the [Service] section of that file, then it will do two
things. First, it actually works whereas editing user.conf did not. Second,
the change only applies to user <uid> and not all users.
I assume I'm not getting how systemd is supposed to work. So please
enlighten me.
Thanks,
Jeff
First reason:

Limit* in ***@.service is set by init before it starts the user instance.
Init is privileged and can raise limits above the current hard limit. (The
same could be done via pam_limit.)

DefaultLimit* in user.conf is set by the user instance itself, which runs
under your uid and does not have any special privileges. It cannot raise
the limits beyond the current hard limit, just as the `ulimit` command
cannot.

Second reason:

The defaults are for units – not for the service manager itself.

So although the defaults are *read* successfully, they will only be applied
when you start a service.

If you want to raise limits for all users, best to do that via pam_limits.
(Letting --user services have different limits than directly launched
programs is likely to result in confusion.)

If you want to override ***@.service, you *do not have* to create an
instance for every uid; you can just have "/etc/systemd/system/***@.service";
that's how it looks in /usr/lib anyway.

Though a better method is to use drop-in configuration to only extend the
service with your new options, while still loading the rest from /usr.
Search the systemd.unit manpage for "drop-in", and put your extensions
Post by Jeff Solomon
--
Mantas Mikulėnas <***@gmail.com>
Sent from my phone
Jeff Solomon
2017-11-20 00:27:16 UTC
Permalink
Raw Message
Understood.

I didn't think that systemd paid one bit of attention to the settings
controlled by pam_limits?

I'm only interested in a user instance that is lingering and operates
outside of a session.

My goal is that the child processes of the user instance will have limits
that I set. If I understand correctly, if those limits are to be higher
than the system's hard limits, then the user instance itself must have
those higher limits set on it, yes?

I appreciate that the user instance doesn't need higher limits itself and
that it is weird that the children of the user instance will have
difference limits than a logged in user, but that's fine.

It seems to me that the only use of the defaults set in
/etc/systemd/user.conf is to further restrict the user instance's children
beyond whatever restrictions are set by the system itself. I had
mistakenly believed that since /etc/systemd/user.conf was a restricted
file, that setting it would override system settings but it doesn't because
it's read by the user instance after it's already running at the user.

Thanks for the tip regarding /etc/systemd/system/user@
.service.d/whatever.conf.

I did previously know that ***@.service would work. For my application,
I'm interested in a single special user only however. I don't want these
customizations to apply to all users.
Post by Jeff Solomon
Hi,
Two questions.
I want to raise the "number of files" limits for the user instance.
First, I set DefaultLimitNOFILE to something higher than the global
system default in /etc/systemd/user.conf and I rebooted.
"systemctl --user show" showed the new DefaultLimitNOFILE and the unit
itself showed the higher setting of LimitNOFILE when I ran "systemctl
--user show foo".
So far everything worked as expected.
However, when I checked "cat /proc/<pid>/limits" on the ExecStart process
of foo.service, I don't see the "number of files" limit has changed.
What did I do wrong?
Second question: if I want to raise the limit just for a single user, how
would I go about it?
Making a change in user.conf would make it apply in all user instances
(assuming I could get it to work).
add LimitNOFILE to the [Service] section of that file, then it will do two
things. First, it actually works whereas editing user.conf did not. Second,
the change only applies to user <uid> and not all users.
I assume I'm not getting how systemd is supposed to work. So please
enlighten me.
Thanks,
Jeff
instance. Init is privileged and can raise limits above the current hard
limit. (The same could be done via pam_limit.)
DefaultLimit* in user.conf is set by the user instance itself, which runs
under your uid and does not have any special privileges. It cannot raise
the limits beyond the current hard limit, just as the `ulimit` command
cannot.
The defaults are for units – not for the service manager itself.
So although the defaults are *read* successfully, they will only be
applied when you start a service.
If you want to raise limits for all users, best to do that via pam_limits.
(Letting --user services have different limits than directly launched
programs is likely to result in confusion.)
that's how it looks in /usr/lib anyway.
Though a better method is to use drop-in configuration to only extend the
service with your new options, while still loading the rest from /usr.
Search the systemd.unit manpage for "drop-in", and put your extensions
Post by Jeff Solomon
--
Sent from my phone
Mantas Mikulėnas
2017-11-20 00:36:21 UTC
Permalink
Raw Message
Post by Jeff Solomon
Understood.
I didn't think that systemd paid one bit of attention to the settings
controlled by pam_limits?
The user@ instance runs user-controlled processes, much like cron would, so
its service unit has PAM enabled as well.
Post by Jeff Solomon
I'm only interested in a user instance that is lingering and operates
outside of a session.
They always run outside of a session, even if the start was triggered by
session creation.
Post by Jeff Solomon
My goal is that the child processes of the user instance will have limits
that I set. If I understand correctly, if those limits are to be higher
than the system's hard limits, then the user instance itself must have
those higher limits set on it, yes?
Yes, since it's a regular user process with the same user privileges as
those child processes.
--
Mantas Mikulėnas <***@gmail.com>
Sent from my phone
Jeff Solomon
2017-11-20 00:57:07 UTC
Permalink
Raw Message
Post by Jeff Solomon
I didn't think that systemd paid one bit of attention to the settings
Post by Jeff Solomon
controlled by pam_limits?
so its service unit has PAM enabled as well.
When I change pam_limits for a user via a file /etc/security/limits.d/, and
then restart the user instance, neither the user instance itself nor the
children of that instance are affected by those settings. OTOH, when I
login again as that user, that login session does have those custom limits
set.

Based on your previous comment, I would have expected the user instance and
its child to show those custom limits. What did am I getting wrong?
Lennart Poettering
2017-11-20 12:10:27 UTC
Permalink
Raw Message
Post by Jeff Solomon
Post by Jeff Solomon
I didn't think that systemd paid one bit of attention to the settings
Post by Jeff Solomon
controlled by pam_limits?
so its service unit has PAM enabled as well.
When I change pam_limits for a user via a file /etc/security/limits.d/, and
then restart the user instance, neither the user instance itself nor the
children of that instance are affected by those settings. OTOH, when I
login again as that user, that login session does have those custom limits
set.
Based on your previous comment, I would have expected the user instance and
its child to show those custom limits. What did am I getting wrong?
Note that ***@.service is only restarted if you fully log out
(i.e. all your sessions) and then login back again. And only when it
is restarted the new limits will be applied to systemd --user.

if you use lingering, then not even this will work, since after all
you declare that way that for your user the ***@.service instance
shall stick around for system boot-up till shutdown. In that case,
please just explicitly issue "systemctl restart user@….service" as
root, so that the service is restarted.

Lennart
--
Lennart Poettering, Red Hat
Jeff Solomon
2017-11-20 16:32:56 UTC
Permalink
Raw Message
I am using lingering and I have issued "systemctl restart user@<uid>" and
then seen the instance restart with a new PID. So I think I am restarting
the user instance.

When Limit* directives are applied in "***@.service" or in
"/etc/systemd/system/***@.service.d/whatever.conf" I see that they are
respected in the user instance itself and the child processes it starts.

However, I do NOT see settings applied through pam_limits
(/etc/security/limits.d etc etc) respected in the user instance although
Mantas implied that I should. Is this expected?
Post by Mantas Mikulėnas
Post by Jeff Solomon
Post by Jeff Solomon
I didn't think that systemd paid one bit of attention to the settings
Post by Jeff Solomon
controlled by pam_limits?
would,
Post by Jeff Solomon
Post by Jeff Solomon
so its service unit has PAM enabled as well.
When I change pam_limits for a user via a file /etc/security/limits.d/,
and
Post by Jeff Solomon
then restart the user instance, neither the user instance itself nor the
children of that instance are affected by those settings. OTOH, when I
login again as that user, that login session does have those custom
limits
Post by Jeff Solomon
set.
Based on your previous comment, I would have expected the user instance
and
Post by Jeff Solomon
its child to show those custom limits. What did am I getting wrong?
(i.e. all your sessions) and then login back again. And only when it
is restarted the new limits will be applied to systemd --user.
if you use lingering, then not even this will work, since after all
shall stick around for system boot-up till shutdown. In that case,
root, so that the service is restarted.
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-11-20 16:47:33 UTC
Permalink
Raw Message
Post by Jeff Solomon
then seen the instance restart with a new PID. So I think I am restarting
the user instance.
respected in the user instance itself and the child processes it starts.
However, I do NOT see settings applied through pam_limits
(/etc/security/limits.d etc etc) respected in the user instance although
Mantas implied that I should. Is this expected?
When systemd executes a service that has PAM enabled, it will will
first start the PAM session, which is where pam_limits does its
thing. It then goes on setting up the execution environment for the
service, and if resource limits are configured for the unit, they'll
be put into effect. This means that any settings configured in the
unit file they take precedence over the pam_limits settings.

Lennart
--
Lennart Poettering, Red Hat
Jeff Solomon
2017-11-20 17:20:11 UTC
Permalink
Raw Message
Lennart,

Your explanation sounds great but it's just not what I'm seeing.

My ***@.service has "PAMName=systemd-user" in the [Service] section.

I have setup limits for the user in /etc/security/limits.d/foo.conf.

I have no other limit overrides in any other systemd file.

Whether I reboot or "systemctl restart user@<uid>" I see the same thing.
That is, the limits set through pam_limits are not respected.

I consistently see that if I login as that user, then "ulimit -a" shows the
values I expect from pam_limits while "cat /proc/<pid>/limits" for the user
instance process or its children do not.
Post by Lennart Poettering
and
Post by Jeff Solomon
then seen the instance restart with a new PID. So I think I am restarting
the user instance.
respected in the user instance itself and the child processes it starts.
However, I do NOT see settings applied through pam_limits
(/etc/security/limits.d etc etc) respected in the user instance although
Mantas implied that I should. Is this expected?
When systemd executes a service that has PAM enabled, it will will
first start the PAM session, which is where pam_limits does its
thing. It then goes on setting up the execution environment for the
service, and if resource limits are configured for the unit, they'll
be put into effect. This means that any settings configured in the
unit file they take precedence over the pam_limits settings.
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-11-20 17:23:05 UTC
Permalink
Raw Message
Post by Jeff Solomon
Lennart,
Your explanation sounds great but it's just not what I'm seeing.
I have setup limits for the user in /etc/security/limits.d/foo.conf.
I have no other limit overrides in any other systemd file.
That is, the limits set through pam_limits are not respected.
I consistently see that if I login as that user, then "ulimit -a" shows the
values I expect from pam_limits while "cat /proc/<pid>/limits" for the user
instance process or its children do not.
Is pam_limits even enabled for the "systemd-user" PAM fragment on your
distro?

Lennart
--
Lennart Poettering, Red Hat
Jeff Solomon
2017-11-20 17:47:37 UTC
Permalink
Raw Message
I guess the answer is "no." :)

This is Ubuntu 16.04. On CentOS7.3, pam_limits is part of systemd-user
through system-auth

Here is /etc/pam.d/systemd-user from my Ubuntu system:

# This file is part of systemd.
#
# Used by systemd --user instances.

@include common-account

session required pam_selinux.so close
session required pam_selinux.so nottys open
@include common-session-noninteractive
session optional pam_systemd.so

So on RHEL systems, it doesn't matter that is works because user instances
are officially not included and it just doesn't work on Ubuntu because
pam_limits is not used by systemd-user.

I find it odd that two major distros differ in this behavior.
Post by Lennart Poettering
Post by Jeff Solomon
Lennart,
Your explanation sounds great but it's just not what I'm seeing.
I have setup limits for the user in /etc/security/limits.d/foo.conf.
I have no other limit overrides in any other systemd file.
That is, the limits set through pam_limits are not respected.
I consistently see that if I login as that user, then "ulimit -a" shows
the
Post by Jeff Solomon
values I expect from pam_limits while "cat /proc/<pid>/limits" for the
user
Post by Jeff Solomon
instance process or its children do not.
Is pam_limits even enabled for the "systemd-user" PAM fragment on your
distro?
Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2017-11-20 17:49:53 UTC
Permalink
Raw Message
Post by Jeff Solomon
I guess the answer is "no." :)
This is Ubuntu 16.04. On CentOS7.3, pam_limits is part of systemd-user
through system-auth
# This file is part of systemd.
#
# Used by systemd --user instances.
@include common-account
session required pam_selinux.so close
session required pam_selinux.so nottys open
@include common-session-noninteractive
session optional pam_systemd.so
Have you checked the snippets listed in the @include lines? Maybe they
pull it in?
Post by Jeff Solomon
So on RHEL systems, it doesn't matter that is works because user instances
are officially not included and it just doesn't work on Ubuntu because
pam_limits is not used by systemd-user.
I find it odd that two major distros differ in this behavior.
PAM is a mess. Setups and syntax vary wildly between distros. It's sad.

Lennart
--
Lennart Poettering, Red Hat
Jeff Solomon
2017-11-20 17:52:37 UTC
Permalink
Raw Message
I have checked the snippets. "common-account" only deal with account
settings. "common-session-interactive" does not include a pam_limits entry.
Post by Lennart Poettering
Post by Jeff Solomon
I guess the answer is "no." :)
This is Ubuntu 16.04. On CentOS7.3, pam_limits is part of systemd-user
through system-auth
# This file is part of systemd.
#
# Used by systemd --user instances.
@include common-account
session required pam_selinux.so close
session required pam_selinux.so nottys open
@include common-session-noninteractive
session optional pam_systemd.so
pull it in?
Post by Jeff Solomon
So on RHEL systems, it doesn't matter that is works because user
instances
Post by Jeff Solomon
are officially not included and it just doesn't work on Ubuntu because
pam_limits is not used by systemd-user.
I find it odd that two major distros differ in this behavior.
PAM is a mess. Setups and syntax vary wildly between distros. It's sad.
Lennart
--
Lennart Poettering, Red Hat
Michael Biebl
2017-11-20 18:26:41 UTC
Permalink
Raw Message
https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/debian/extra/pam.d/systemd-user?id=b3238e9604fa61c7ec45a2d0acc1f8b40728cd87

This might be relevant to you.

See how the pam config contains pam_limits
Post by Lennart Poettering
Post by Jeff Solomon
I guess the answer is "no." :)
This is Ubuntu 16.04. On CentOS7.3, pam_limits is part of systemd-user
through system-auth
# This file is part of systemd.
#
# Used by systemd --user instances.
@include common-account
session required pam_selinux.so close
session required pam_selinux.so nottys open
@include common-session-noninteractive
session optional pam_systemd.so
pull it in?
Post by Jeff Solomon
So on RHEL systems, it doesn't matter that is works because user instances
are officially not included and it just doesn't work on Ubuntu because
pam_limits is not used by systemd-user.
I find it odd that two major distros differ in this behavior.
PAM is a mess. Setups and syntax vary wildly between distros. It's sad.
Lennart
--
Lennart Poettering, Red Hat
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Michael Biebl
2017-11-20 18:29:48 UTC
Permalink
Raw Message
Post by Michael Biebl
https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/debian/extra/pam.d/systemd-user?id=b3238e9604fa61c7ec45a2d0acc1f8b40728cd87
This might be relevant to you.
See how the pam config contains pam_limits
This was a result of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838191
Jeff Solomon
2017-11-20 19:07:54 UTC
Permalink
Raw Message
Good to know! Thanks for googling for me!
Post by Michael Biebl
Post by Michael Biebl
https://anonscm.debian.org/git/pkg-systemd/systemd.git/
commit/debian/extra/pam.d/systemd-user?id=b3238e9604fa61c7ec45a2d0acc1f8
b40728cd87
Post by Michael Biebl
This might be relevant to you.
See how the pam config contains pam_limits
This was a result of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838191
Loading...