Discussion:
binding tmpfiles.d to unit startup
(too old to reply)
Colin Walters
2014-03-01 14:46:45 UTC
Permalink
Hi,

So for OSTree I am trying to move to a model where services populate
the contents of /var on *start*. See previous discussion here:

https://www.mail-archive.com/systemd-***@lists.freedesktop.org/msg07859.html

The really great part about this is that one is then able to totally
reset OS state at any time by simply just doing a shutdown of services,
then "rm -rf /var/*", then reboot. (You can also reset /etc, that's a
separate discussion)

Now while patching the service code itself got me pretty far,
ultimately there are a lot of RPMs and such out there...so what I ended
up doing for rpm-ostree was to generate tmpfiles.d snippets from the
/var content:

https://github.com/cgwalters/rpm-ostree/commit/f8ddf38aa2dce207399d5e485842168eb2540e5f

Basically we end up with a gigantic
/usr/lib/tmpfiles.d/rpm-ostree-autovar.conf that has stuff like:

d /var/spool/mail 0775 0 12 - -
d /var/lib/dnsmasq 0755 0 0 - -
...

The problem with this is it all happens on boot, and it doesn't really
scale. I want the RPMs (and packages in general) to do this by default.

So what's the problem with patching the world to do mkdir() on bootup?

1) Lots of work
2) Loss of association between package data and directory. While this
is
pretty minor, it *is* useful to be able to do rpm -qf /var/kerberos
and see that it comes from the "krb5-libs" package.
3) SELinux labeling - any service that creates a directory like this
will
have to be sure it labels it correctly. While the latest SELinux
does have
support for doing this automatically, it'd still be better if it was
done at a more trusted level.

What I really want is an easy way to bind tmpfiles.d snippets to unit
startup.
Something like:

StateDirectory=/var/kerberos 0755 0 0

(Why not full tmpfiles.d syntax? Well I can't think of a good reason
to make
a *file* on startup. There are packages that ship plain files in /var
but
I'd say they're broken.)

With this, it's easy to do without patching the service. It retains the
association (if a bit indirect) between the service and its state
directory,
because you can query ownership of the .service file. And systemd
ensures
SELinux labeling happens correctly.

Thoughts? Should be a pretty easy patch.
Michael Biebl
2014-03-01 18:44:09 UTC
Permalink
Hi,
So for OSTree I am trying to move to a model where services populate the
The really great part about this is that one is then able to totally reset
OS state at any time by simply just doing a shutdown of services, then "rm
-rf /var/*", then reboot. (You can also reset /etc, that's a separate
discussion)
Now while patching the service code itself got me pretty far, ultimately
there are a lot of RPMs and such out there...so what I ended up doing for
https://github.com/cgwalters/rpm-ostree/commit/f8ddf38aa2dce207399d5e485842168eb2540e5f
Basically we end up with a gigantic
d /var/spool/mail 0775 0 12 - -
d /var/lib/dnsmasq 0755 0 0 - -
...
The problem with this is it all happens on boot, and it doesn't really
scale. I want the RPMs (and packages in general) to do this by default.
So what's the problem with patching the world to do mkdir() on bootup?
1) Lots of work
2) Loss of association between package data and directory. While this is
pretty minor, it *is* useful to be able to do rpm -qf /var/kerberos
and see that it comes from the "krb5-libs" package.
3) SELinux labeling - any service that creates a directory like this will
have to be sure it labels it correctly. While the latest SELinux does
have
support for doing this automatically, it'd still be better if it was
done at a more trusted level.
What I really want is an easy way to bind tmpfiles.d snippets to unit
startup.
StateDirectory=/var/kerberos 0755 0 0
(Why not full tmpfiles.d syntax? Well I can't think of a good reason to
make
a *file* on startup. There are packages that ship plain files in /var but
I'd say they're broken.)
With this, it's easy to do without patching the service. It retains the
association (if a bit indirect) between the service and its state directory,
because you can query ownership of the .service file. And systemd ensures
SELinux labeling happens correctly.
Thoughts? Should be a pretty easy patch.
I vaguely remember that we exactly had this discussion a while ago.
Unfortunately I'm not able to find it in the archives right now.

One of the objections I remember was, that this would lead to having
two different ways of setting up runtime directories.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Colin Walters
2014-03-01 18:47:34 UTC
Permalink
Post by Michael Biebl
I vaguely remember that we exactly had this discussion a while ago.
Argh, yes, possibly. The dangers of getting older...
Post by Michael Biebl
Unfortunately I'm not able to find it in the archives right now.
I think if we did talk about it, it was some variant, and then maybe
the conversation was on IRC?
Post by Michael Biebl
One of the objections I remember was, that this would lead to having
two different ways of setting up runtime directories.
Can you elaborate on this? The two different things being tmpfiles.d
and unit files?
Michael Biebl
2014-03-01 19:04:55 UTC
Permalink
Post by Michael Biebl
I vaguely remember that we exactly had this discussion a while ago.
Argh, yes, possibly. The dangers of getting older...
Unfortunately I'm not able to find it in the archives right now.
Found it: http://lists.freedesktop.org/archives/systemd-devel/2013-July/011956.html
Tom Gundersen
2014-03-01 19:18:36 UTC
Permalink
Post by Michael Biebl
Post by Michael Biebl
I vaguely remember that we exactly had this discussion a while ago.
Argh, yes, possibly. The dangers of getting older...
Unfortunately I'm not able to find it in the archives right now.
Found it: http://lists.freedesktop.org/archives/systemd-devel/2013-July/011956.html
And a bit further down that thread there was this proposal from Lennar
(which doesn't seem far from what Colin wants):
<http://lists.freedesktop.org/archives/systemd-devel/2013-July/012024.html>.

-t
Michael Biebl
2014-03-01 19:45:33 UTC
Permalink
Post by Tom Gundersen
Post by Michael Biebl
Post by Michael Biebl
I vaguely remember that we exactly had this discussion a while ago.
Argh, yes, possibly. The dangers of getting older...
Unfortunately I'm not able to find it in the archives right now.
Found it: http://lists.freedesktop.org/archives/systemd-devel/2013-July/011956.html
And a bit further down that thread there was this proposal from Lennar
<http://lists.freedesktop.org/archives/systemd-devel/2013-July/012024.html>.
In Debian we have debhelper addons which generate maintainer scripts
code automatically to run systemd-tmpfiles
/usr/lib/tmpfiles.d/foo.conf upon installation, so this isn't really a
problem anymore for us.

Regarding discoverability, my suggestion in that thread was to add a
new field which allows to reference the associated tmpfiles snippet,
similar as we do for Documentation.

Having such a reference on the tmpfile snippet(s) in the .service file
would also make it possible to run them automatically whenever the
service is started. This could be useful for distros which haven't
automated the invocation of systemd-tmpfiles

Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Colin Walters
2014-03-01 19:46:15 UTC
Permalink
Post by Tom Gundersen
And a bit further down that thread there was this proposal from Lennar
<http://lists.freedesktop.org/archives/systemd-devel/2013-July/012024.html>.
Right...so rereading that, the discussion seemed to peter out after:
http://lists.freedesktop.org/archives/systemd-devel/2013-July/012090.html

I see Michael's point here as well:
http://lists.freedesktop.org/archives/systemd-devel/2013-July/012037.html

Certainly it magnifies the pain for the RPM world in that there's
nothing that scans the build directory and says "hey you installed
something that looks like a tmpfiles.d snippet, let me auto-generate
some postinst shell script".

I think any postinsts like this are still pretty ugly though. It means
the directories are created when the daemon is *installed*, not when
it's run. Which is odd and against the generally dynamic nature of
systemd.

Also, running shell script as root is just full of fail, sorry package
people =) Even if it's auto-generated from a centrally maintained
place.

I admit this is sometimes what you want, e.g. if you need a way to
install postgres and then restore a dump file from backup *before*
starting it. Those cases could be covered of course by an
administrator doing a mkdir manually and if they're careful about
SELinux, using restorecon.

So basically here'd what I'd propose, which is an extension of
Lennart's:

RuntimeDirectory=/run/mydaemon
PersistentStateDirectory=/var/lib/mydaemon

We could cover a lot of simple cases with that - default to being owned
by User= and mode 0700, and have RuntimeDirectoryMode= and
PersistentStateDirectoryMode= to override that last bit.

Note - the previous discussion seemed largely about /run, but I am most
interested in real permanent directories in /var.

The primary goal again for me is that I can rm -rf /var/* and reboot,
and have the system be clean.
Colin Walters
2014-03-01 21:01:06 UTC
Permalink
Post by Colin Walters
RuntimeDirectory=/run/mydaemon
PersistentStateDirectory=/var/lib/mydaemon
Btw, see also this thread:

https://lists.fedoraproject.org/pipermail/server/2014-February/000843.html

Putting these together (and how about we just go ahead and mandate /run
and /var/lib), we could have something like:

RuntimeDirectory=yes

That would auto-instantiate /run/$SYSTEMD_UNIT_NAME.

Similarly,
PersistentStateDirectory=yes
would give you /var/lib/$SYSTEMD_UNIT_NAME

Notably, this would help administrators move away from the package
being the central naming unit and more towards systemd being the basis
for names.

(It is confusing that one has to deal with package names, systemd unit
names, and process names, which can all be the same, closely related,
or wildly different)
Lennart Poettering
2014-03-02 23:03:15 UTC
Permalink
Post by Colin Walters
Post by Colin Walters
RuntimeDirectory=/run/mydaemon
PersistentStateDirectory=/var/lib/mydaemon
https://lists.fedoraproject.org/pipermail/server/2014-February/000843.html
Putting these together (and how about we just go ahead and mandate
RuntimeDirectory=yes
That would auto-instantiate /run/$SYSTEMD_UNIT_NAME.
Would $SYSTEMD_UNIT_NAME according to your suggestion map to %n or to %p
or to %i, followign systemd's specifier notation? (see systemd.unit(5)).
Post by Colin Walters
Notably, this would help administrators move away from the package
being the central naming unit and more towards systemd being the
basis for names.
(It is confusing that one has to deal with package names, systemd
unit names, and process names, which can all be the same, closely
related, or wildly different)
So far we stayed out of the business of being normative on how to name
things. Which I figure is something we can get away with as long as we
focus on building an OS. I wonder though whether unit files like this
would be the right place to start enforcing naming rules for 3rd party
apps...

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2014-03-02 22:57:58 UTC
Permalink
Post by Colin Walters
Certainly it magnifies the pain for the RPM world in that there's
nothing that scans the build directory and says "hey you installed
something that looks like a tmpfiles.d snippet, let me auto-generate
some postinst shell script".
But that is fixable. RPM could do that. The same way it scans .so files
and creates deps from it.
Post by Colin Walters
I think any postinsts like this are still pretty ugly though. It
means the directories are created when the daemon is *installed*,
not when it's run. Which is odd and against the generally dynamic
nature of systemd.
Well, but /var and /etc are by their very nature non-volatile,
non-dynamic. We shouldnt set up more during runtime then we really have
to...

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2014-03-02 22:55:38 UTC
Permalink
Post by Michael Biebl
Post by Michael Biebl
I vaguely remember that we exactly had this discussion a while ago.
Argh, yes, possibly. The dangers of getting older...
Unfortunately I'm not able to find it in the archives right now.
Found it: http://lists.freedesktop.org/archives/systemd-devel/2013-July/011956.html
And a bit further down that thread there was this proposal from Lennart
<http://lists.freedesktop.org/archives/systemd-devel/2013-July/012024.html>.
I am still open for this btw. If somebody wants to hack on that, I
figure this should simply be addded to ExecContext, as a strv of
directory names. In exec_spawn() we'd then just create all those dirs,
right after resolving the UID/GID. When running in system mode we'd then
create the dirs in /run, when running user mode in $XDG_RUNTIME_DIR.

Should be a ~20 line patch or so...

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2014-03-03 17:53:37 UTC
Permalink
Post by Lennart Poettering
I am still open for this btw. If somebody wants to hack on that, I
figure this should simply be addded to ExecContext, as a strv of
directory names. In exec_spawn() we'd then just create all those dirs,
right after resolving the UID/GID. When running in system mode we'd then
create the dirs in /run, when running user mode in $XDG_RUNTIME_DIR.
Should be a ~20 line patch or so...
Just to mention this: I have implemented this now in git.

Lennart
--
Lennart Poettering, Red Hat
Zbigniew Jędrzejewski-Szmek
2014-03-01 19:26:23 UTC
Permalink
Post by Colin Walters
2) Loss of association between package data and directory. While
this is
pretty minor, it *is* useful to be able to do rpm -qf /var/kerberos
and see that it comes from the "krb5-libs" package.
On a related note, I want to generate rpm %files contents from tmpfiles
snippets. I think it should work like this:

--------------------------------

%build
systemd-tmpfiles --generate-rpm-files %{buildroot}/usr/lib/tmpfiles.d/%{name}.conf >.tpmfiles.list

%files -f .tmpfiles.list

%post
systemd-tmpfiles --create /usr/lib/tmpfiles.d/%{name}.conf

--------------------------------

Where the output out tmpfiles --generate-rpm-files [name subject to change],
would be something like:

%ghost /var/name/

The point is that this will simplify rpm packaging by removing the need
to repeat the list in two places.

Zbyszek
Zbigniew Jędrzejewski-Szmek
2014-03-02 02:06:14 UTC
Permalink
---

I'm not sure if this can be actually as useful as I envisioned. At
least in case of systemd.spec there's a lot of special-casing
required, which partially destroys the usefulness of this. Configuration
for "normal" packages will hopefully be more straighforward.

I made the switch name generic, so that other "transforms" can
be added later without complicating the command-line api.

Zbyszek

src/tmpfiles/tmpfiles.c | 144 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 107 insertions(+), 37 deletions(-)

diff --git a/src/tmpfiles/tmpfiles.c b/src/tmpfiles/tmpfiles.c
index 6e36dc7..008bc16 100644
--- a/src/tmpfiles/tmpfiles.c
+++ b/src/tmpfiles/tmpfiles.c
@@ -101,6 +101,10 @@ typedef struct Item {
bool keep_first_level:1;
} Item;

+enum {
+ TRANSFORM_RPM = 1,
+};
+
static Hashmap *items = NULL, *globs = NULL;
static Set *unix_sockets = NULL;

@@ -108,6 +112,7 @@ static bool arg_create = false;
static bool arg_clean = false;
static bool arg_remove = false;
static bool arg_boot = false;
+static int arg_transform = 0;

static char **include_prefixes = NULL;
static char **exclude_prefixes = NULL;
@@ -125,7 +130,13 @@ static const char conf_file_dirs[] =
#define MAX_DEPTH 256

static bool needs_glob(ItemType t) {
- return t == IGNORE_PATH || t == IGNORE_DIRECTORY_PATH || t == REMOVE_PATH || t == RECURSIVE_REMOVE_PATH || t == RELABEL_PATH || t == RECURSIVE_RELABEL_PATH;
+ return IN_SET(t,
+ IGNORE_PATH,
+ IGNORE_DIRECTORY_PATH,
+ REMOVE_PATH,
+ RECURSIVE_REMOVE_PATH,
+ RELABEL_PATH,
+ RECURSIVE_RELABEL_PATH);
}

static struct Item* find_glob(Hashmap *h, const char *match) {
@@ -972,22 +983,55 @@ static int clean_item(Item *i) {
return r;
}

+static int transform_item(Item *i, int transform) {
+ assert(i);
+ assert(transform == TRANSFORM_RPM);
+
+ switch (i->type) {
+
+ case IGNORE_PATH:
+ case IGNORE_DIRECTORY_PATH:
+ case REMOVE_PATH:
+ case RECURSIVE_REMOVE_PATH:
+ case ADJUST_MODE:
+ case TRUNCATE_DIRECTORY:
+ case RELABEL_PATH:
+ case RECURSIVE_RELABEL_PATH:
+ log_debug("%s ignored.", i->path);
+ return 0;
+
+ case CREATE_FILE:
+ case TRUNCATE_FILE:
+ case WRITE_FILE:
+ case CREATE_FIFO:
+ case CREATE_SYMLINK:
+ case CREATE_BLOCK_DEVICE:
+ case CREATE_CHAR_DEVICE:
+ printf("%%ghost %s\n", i->path);
+ break;
+
+ case CREATE_DIRECTORY:
+ printf("%%ghost %%dir %s\n", i->path);
+ break;
+ }
+
+ return 0;
+}
+
static int process_item(Item *i) {
- int r, q, p;
+ int r, q, p, s;

assert(i);

r = arg_create ? create_item(i) : 0;
q = arg_remove ? remove_item(i) : 0;
p = arg_clean ? clean_item(i) : 0;
+ s = arg_transform ? transform_item(i, arg_transform) : 0;

- if (r < 0)
- return r;
-
- if (q < 0)
- return q;
-
- return p;
+ return r < 0 ? r :
+ q < 0 ? q :
+ p < 0 ? p :
+ s;
}

static void item_free(Item *i) {
@@ -1270,14 +1314,15 @@ static int help(void) {

printf("%s [OPTIONS...] [CONFIGURATION FILE...]\n\n"
"Creates, deletes and cleans up volatile and temporary files and directories.\n\n"
- " -h --help Show this help\n"
- " --version Show package version\n"
- " --create Create marked files/directories\n"
- " --clean Clean up marked directories\n"
- " --remove Remove marked files/directories\n"
- " --boot Execute actions only safe at boot\n"
- " --prefix=PATH Only apply rules that apply to paths with the specified prefix\n"
- " --exclude-prefix=PATH Ignore rules that apply to paths with the specified prefix\n",
+ " -h --help Show this help\n"
+ " --version Show package version\n"
+ " --create Create marked files/directories\n"
+ " --clean Clean up marked directories\n"
+ " --remove Remove marked files/directories\n"
+ " --boot Execute actions only safe at boot\n"
+ " --transform=rpm Create files list suitable for RPM %%files section\n"
+ " --prefix=PATH Only apply rules that match the specified prefix\n"
+ " --exclude-prefix=PATH Ignore rules that match the specified prefix\n",
program_invocation_short_name);

return 0;
@@ -1291,19 +1336,21 @@ static int parse_argv(int argc, char *argv[]) {
ARG_CLEAN,
ARG_REMOVE,
ARG_BOOT,
+ ARG_TRANSFORM,
ARG_PREFIX,
ARG_EXCLUDE_PREFIX,
};

static const struct option options[] = {
- { "help", no_argument, NULL, 'h' },
- { "version", no_argument, NULL, ARG_VERSION },
- { "create", no_argument, NULL, ARG_CREATE },
- { "clean", no_argument, NULL, ARG_CLEAN },
- { "remove", no_argument, NULL, ARG_REMOVE },
- { "boot", no_argument, NULL, ARG_BOOT },
- { "prefix", required_argument, NULL, ARG_PREFIX },
- { "exclude-prefix", required_argument, NULL, ARG_EXCLUDE_PREFIX },
+ { "help", no_argument, NULL, 'h' },
+ { "version", no_argument, NULL, ARG_VERSION },
+ { "create", no_argument, NULL, ARG_CREATE },
+ { "clean", no_argument, NULL, ARG_CLEAN },
+ { "remove", no_argument, NULL, ARG_REMOVE },
+ { "transform", required_argument, NULL, ARG_TRANSFORM },
+ { "boot", no_argument, NULL, ARG_BOOT },
+ { "prefix", required_argument, NULL, ARG_PREFIX },
+ { "exclude-prefix", required_argument, NULL, ARG_EXCLUDE_PREFIX },
{}
};

@@ -1336,6 +1383,16 @@ static int parse_argv(int argc, char *argv[]) {
arg_remove = true;
break;

+ case ARG_TRANSFORM:
+ if (streq(optarg, "rpm")) {
+ arg_transform = TRANSFORM_RPM;
+ arg_boot = true;
+ } else {
+ log_error("Unknown transformation '%s'", optarg);
+ return -EINVAL;
+ }
+ break;
+
case ARG_BOOT:
arg_boot = true;
break;
@@ -1358,31 +1415,44 @@ static int parse_argv(int argc, char *argv[]) {
}
}

- if (!arg_clean && !arg_create && !arg_remove) {
- log_error("You need to specify at least one of --clean, --create or --remove.");
+ if ((arg_clean || arg_create || arg_remove) + !!arg_transform != 1) {
+ log_error("You must specify at least one of --clean/--create/--remove, or only --transform.");
+ return -EINVAL;
+ }
+
+ if (arg_transform && optind >= argc) {
+ log_error("Configuration file must be specified with --transform");
return -EINVAL;
}

return 1;
}

-static int read_config_file(const char *fn, bool ignore_enoent) {
+static int read_config_file(const char *fn, bool ignore_enoent, bool real_path) {
_cleanup_fclose_ FILE *f = NULL;
char line[LINE_MAX];
Iterator iterator;
unsigned v = 0;
Item *i;
- int r;
+ int r = 0;

assert(fn);

- r = search_and_fopen_nulstr(fn, "re", conf_file_dirs, &f);
- if (r < 0) {
- if (ignore_enoent && r == -ENOENT)
- return 0;
+ if (real_path) {
+ f = fopen(fn, "re");
+ if (!f) {
+ log_error("Failed to open '%s': %m", fn);
+ return -errno;
+ }
+ } else {
+ r = search_and_fopen_nulstr(fn, "re", conf_file_dirs, &f);
+ if (r < 0) {
+ if (ignore_enoent && r == -ENOENT)
+ return 0;

- log_error("Failed to open '%s', ignoring: %s", fn, strerror(-r));
- return r;
+ log_error("Failed to open '%s', ignoring: %s", fn, strerror(-r));
+ return r;
+ }
}

FOREACH_LINE(line, f, break) {
@@ -1468,7 +1538,7 @@ int main(int argc, char *argv[]) {
int j;

for (j = optind; j < argc; j++) {
- k = read_config_file(argv[j], false);
+ k = read_config_file(argv[j], false, !!arg_transform);
if (k < 0 && r == 0)
r = k;
}
@@ -1484,7 +1554,7 @@ int main(int argc, char *argv[]) {
}

STRV_FOREACH(f, files) {
- k = read_config_file(*f, true);
+ k = read_config_file(*f, true, false);
if (k < 0 && r == 0)
r = k;
}
--
1.8.5.3
Kay Sievers
2014-03-02 05:14:58 UTC
Permalink
On Sun, Mar 2, 2014 at 3:06 AM, Zbigniew Jędrzejewski-Szmek
Post by Zbigniew Jędrzejewski-Szmek
+ " --transform=rpm Create files list suitable for RPM %%files section\n"
Hmm, can't we maybe come up with something that isn't specific to rpm
or dpkg, something that is easily parsed but not bound to the
internals of the format of the files?

It sounds like the-wrong-way-around to encode packaging formats into
high-level systemd tools. In the end, both formats are really a blast
from the past, and represent the status quo, but are surely not the
future of how packaging of of a modern Linux system should look like,
so it might be nicer to make it generic?

Kay
Zbigniew Jędrzejewski-Szmek
2014-03-02 14:38:43 UTC
Permalink
Post by Kay Sievers
On Sun, Mar 2, 2014 at 3:06 AM, Zbigniew Jędrzejewski-Szmek
Post by Zbigniew Jędrzejewski-Szmek
+ " --transform=rpm Create files list suitable for RPM %%files section\n"
Hmm, can't we maybe come up with something that isn't specific to rpm
or dpkg, something that is easily parsed but not bound to the
internals of the format of the files?
It sounds like the-wrong-way-around to encode packaging formats into
high-level systemd tools. In the end, both formats are really a blast
from the past, and represent the status quo, but are surely not the
future of how packaging of of a modern Linux system should look like,
so it might be nicer to make it generic?
Something like this would certainly work too:
output from systemd-tmpfiles would be just
/some/file
/some/dir/
...

and it would be used as

%install
systemd-tmpfiles --transform=list ... | sed 's|.*/$|%dir \0|; s|^/|%ghost /;' > .tmpfiles.list

This makes the "consumer" side slightly more complex, but I guess we could
wrap this is in a macro too.

Zbyszek
Lennart Poettering
2014-03-02 23:08:26 UTC
Permalink
Post by Zbigniew Jędrzejewski-Szmek
Post by Kay Sievers
On Sun, Mar 2, 2014 at 3:06 AM, Zbigniew Jędrzejewski-Szmek
Post by Zbigniew Jędrzejewski-Szmek
+ " --transform=rpm Create files list suitable for RPM %%files section\n"
Hmm, can't we maybe come up with something that isn't specific to rpm
or dpkg, something that is easily parsed but not bound to the
internals of the format of the files?
It sounds like the-wrong-way-around to encode packaging formats into
high-level systemd tools. In the end, both formats are really a blast
from the past, and represent the status quo, but are surely not the
future of how packaging of of a modern Linux system should look like,
so it might be nicer to make it generic?
output from systemd-tmpfiles would be just
/some/file
/some/dir/
...
and it would be used as
%install
systemd-tmpfiles --transform=list ... | sed 's|.*/$|%dir \0|;
s|^/|%ghost /;' > .tmpfiles.list
This could even be turned into a shell read loop I figure... I do like
the general concept.
Post by Zbigniew Jędrzejewski-Szmek
This makes the "consumer" side slightly more complex, but I guess we could
wrap this is in a macro too.
I am pretty sure that whatever we end up adding here, it should come
with an RPM macro from day 1, so that we have some freedom to change
around later on should we need it.

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2014-03-02 22:51:06 UTC
Permalink
On Sat, 01.03.14 14:46, Colin Walters (***@verbum.org) wrote:

Hi!
Post by Colin Walters
So for OSTree I am trying to move to a model where services populate
The really great part about this is that one is then able to totally
reset OS state at any time by simply just doing a shutdown of
services, then "rm -rf /var/*", then reboot. (You can also reset
/etc, that's a separate discussion)
There has been this long-time TODO list item of ours to create something
we called "provisioning" (though the name might be misleading, but
that's the keyword we are currently using for it). The provisioning
scheme would basically consist of another directory in /usr/lib that
contains snippets that are inspired by tmpfiles.d/, but are
different. These files would describe what to do if /etc or /var are
found empty at boot. The would not only list files and directories to
create or copy, but also contain information so that we can reconstruct
/etc/passwd and /etc/group to match UIDs and GIDs used in /usr already.

Maybe something like this:

u root 0
g mail /usr/bin/procmail
g tty /usr/bin/write
d /var/lib/foobar 664 root root
c /etc/sudoers /usr/share/sudo/sudoers.default

This snippet would create one user with UID and call it "root". It would
create a group "mail" with a GID that matches the current gid owner of
/usr/bin/procmail, and one "tty" with the GID that matches the current
gid owner of /usr/bin/write. Then, it would also create a dir in
/var/lib. Finally it would copy a sudoers file into place from some
source in /usr (though maybe we should not allow copying files with
this, so that people don't get too lazy, and instead just provide
symlinks).

Then, we'd add a generator that checks for the existance of
/etc/machine-id or so (which we simply use as a flag file for
uninitialized systems here). If it is missing we boot into a special
boot target "provision.target" or so, which runs our provisioning tool
that simply reconstructs everything according to these files, and then
continues booting into default.target. There could even be a kernel
cmdline option that results in this being executed called
"systemd.provision=1" or so. Since "privision.target" is a target like
any other packages could even pull in their own code snippets from this
target if they really want to, but of course this would be quite
contrary to the entire goal which is to have a declarative description
of the system rather than one in code.

These snippets could then also be hooked up with RPM, so that RPM adds
the files listed therein automatically to its file list, and they can be
executed at package installation time with some RPM macro.

The provisioning tool that applies these files could of course also be
run manually on the command line, and much like tmpfiles support a
scheme to only reconstruct a subset of the file system hierarchy (for
example only /var).

For a container usecase this would allow us to have an OS /usr tree
somewhere which we then can mount into hundreds of containers, and on
their first start-up they would get a fully populated /var and /etc.

Colin, when you ask for doing this setup for /var right before starting
up a service, wouldn't it be nicer if we just did that on package
installation if we can, and on boot if /var is empty? Wouldn't this
"provisioning" concept work for you?
Post by Colin Walters
Thoughts? Should be a pretty easy patch.
I am not convinced that'd would be a good idea to place information
about this into the unit files. The unit files have been designed to be
something we only read on demand, when referenced. However, such
reconstruction logic is this kind of static thing where such an
on-demand concept would be really inappropriate. This makes it "feel" a
bit incomptible, I'd say.

Also, it just sounds wrong to make changes to persistent file systems
during runtime, when they could be done at install time already...

Another problem is the one of NSS. We cannot resolve user names from
PID1, since we cannot block on the network, we cannot be client to other
services and we cannot have dynamic NSS modules loaded into PID 1. Thus
creating files from PID 1 is difficult. (Though not impossible. That's
where the restriction like RuntimeDirectory= would be nice btw, since we
know that at least the backing fs isn't blocking, and we could add this
into the preparation step for executing processes, i.e. right after the
fork() in the child, before we exec() the daemon to start. Or we could
even introduce a new service state that is done before ExecStartPre= and
that to doesn't execute any external binaries, but simply does the NSS
stuff and fs access in a forked off process that does what it needs to
do and quickly exits, and never does exec(). -- but anyway, the
take-away here is probably that it is harder thatn it might sound...)

Lennart
--
Lennart Poettering, Red Hat
Holger Schurig
2014-03-03 14:48:14 UTC
Permalink
Make it more user friendly (e.g. without an open man page). Instead of
Post by Lennart Poettering
u root 0
g mail /usr/bin/procmail
g tty /usr/bin/write
d /var/lib/foobar 664 root root
c /etc/sudoers /usr/share/sudo/sudoers.default
user root 0
setgroup mail /usr/bin/procmail

... and so on.


Hmm, that gave me one thougth: if systemd starts as PID 1 and no
/etc/passwd etc doesn't exist, I can very well understand that, when
compiled with --enable-privioning, it should create those things. But
the "c "-line could be happily handled by a shell script. So my
proposal is to only add things into systemd-provision that absolutely
must be done by pid 1, because without it /bin/dash or most user-space
won't run. But then systemd-provision should just execute provision
shell scripts in /lib/systemd/provision.d (or similar). No need to
re-create "cp", for example. Also it gives overall a bigger
flexibility.
Lennart Poettering
2014-03-03 14:52:25 UTC
Permalink
Post by Holger Schurig
Hmm, that gave me one thougth: if systemd starts as PID 1 and no
/etc/passwd etc doesn't exist, I can very well understand that, when
compiled with --enable-privioning, it should create those things. But
the "c "-line could be happily handled by a shell script. So my
proposal is to only add things into systemd-provision that absolutely
must be done by pid 1, because without it /bin/dash or most user-space
won't run. But then systemd-provision should just execute provision
shell scripts in /lib/systemd/provision.d (or similar). No need to
re-create "cp", for example. Also it gives overall a bigger
flexibility.
The explicit goal here is to have something declarative for recreating
/etc and /var. Something one can easily turn into a file list for rpm or
dpkg. Which means shell scripts are not suitable for this, because they
are imperative...

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...