Discussion:
[PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
(too old to reply)
Pavel Odvody
2015-05-07 15:46:54 UTC
Permalink
Hi,

the attached series of patches add support for pulling from V2 docker
registries, so let me break down first what happened to the format since
V1
- Image is now defined by a JSON manifest
- contains fields like name, tag, schemaVersion ...
- and fsLayers - which is an array of sha256 references to a
*content-addressable FS layers*
- the manifest is now also signed using JWS/JWT (ECDSA p-256 mostly)
- Authentication/Authorization now bearer token only
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
- The whole manifest can be hashed using sha256 to obtain a
"digest", which provides an immutable global identifier of the image,
and can be used instead of a tag when pulling the image (the REST
API endpoints are the same).

So far so good, now what's in the patches, besides the V2 workflow
- lightweight JSON parser, written around json_tokenize
- I've renamed 'tag' to 'reference' to accommodate for the digest
semantics
- all layers are saved in a directory .dkr-$imageid - image id is
resolved from the v1 compatibility section of the manifest
- since the layers are now CAS, we can't assume that the order, or
mere presence of certain layers will be preserved throughout
multitude of images/manifests, and therefore due to the
incremental nature of BTRFS snapshots we need to throw any
intermediary snapshots away.
- small bugfix for the JSON tokenizer (it'd choke after reading
any digit)

This is the bare minimum to pull&run V2 images, since the signature is
now embedded in the manifest, it could now support --verify=signature.
However, I've got one open question - how do we support V1/V2
concurrently (this patch makes V2 the default and only)? Docker first
pings the V2 endpoint and then falls back to V1, but I think that this is
sub optimal, since --verify=signature makes sense only with V2, so I think
something like

--dkr-pull-strategy=v1|v2

as an argument would be the best?

Thanks,

Pavel

Pavel Odvody (5):
shared/import-util: tag renamed to reference to support v2 pull by
digest
shared/json: JSON parser + number tokenizer bugfix
test/test-json: Tests for the JSON parser and the tokenizer bugfix
import/pull: Tag replaced with reference
import/pull-dkr: V2 Image specification + manifest support

src/import/pull-dkr.c | 531 +++++++++++++++++++++++++++++++++++++++++------
src/import/pull-dkr.h | 48 ++++-
src/import/pull.c | 28 ++-
src/shared/import-util.c | 19 ++
src/shared/import-util.h | 1 +
src/shared/json.c | 437 +++++++++++++++++++++++++++++++++++++-
src/shared/json.h | 36 ++++
src/test/test-json.c | 16 ++
8 files changed, 1034 insertions(+), 82 deletions(-)
--
2.1.0
Daurnimator
2015-05-08 01:31:15 UTC
Permalink
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
Vincent Batts
2015-05-08 18:33:00 UTC
Permalink
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).

vb
Pavel Odvody
2015-05-08 22:31:47 UTC
Permalink
Post by Vincent Batts
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,

$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>

$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}

I actually added a little clarification in my 5th patch:

"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)

The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.

I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.

The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the "mirror" registry is received in HTTP headers from the Hub)

Sounds like a topic for another patch?

I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
--
Pavel Odvody <***@redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
Vincent Batts
2015-05-11 14:32:39 UTC
Permalink
Post by Pavel Odvody
Post by Vincent Batts
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}
"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the "mirror" registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
< 1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.

vb
Pavel Odvody
2015-05-11 15:07:34 UTC
Permalink
Post by Vincent Batts
Post by Pavel Odvody
Post by Vincent Batts
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}
"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the "mirror" registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
< 1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
--
Pavel Odvody <***@redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
Vincent Batts
2015-05-11 15:15:54 UTC
Permalink
Post by Pavel Odvody
Post by Vincent Batts
Post by Pavel Odvody
Post by Vincent Batts
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}
"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the "mirror" registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
< 1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.

vb
Vincent Batts
2015-05-14 20:12:44 UTC
Permalink
Post by Vincent Batts
Post by Pavel Odvody
Post by Vincent Batts
Post by Pavel Odvody
Post by Vincent Batts
Post by Daurnimator
Post by Pavel Odvody
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}
"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the "mirror" registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
< 1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.
Cool. My raised issue was heard and acted on. Now the UA header is not
required on the hub either.

$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested
resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]}


vb

Loading...