Discussion:
Will *.network replace resolv.conf? What about "Options single-request"?
(too old to reply)
Christian Brunotte
2015-05-04 12:57:39 UTC
Permalink
Hello

systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.

If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?

(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers cannot
handle queries properly and make the requests time out.")

Best Regards

-christian-
Tom Gundersen
2015-05-15 19:40:37 UTC
Permalink
Post by Christian Brunotte
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
In many setups, yes, but we will not aim at bug-for-bug compatibility
or anything like that. We are open to adding features that make sense
though.
Post by Christian Brunotte
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers cannot
handle queries properly and make the requests time out.")
Hm, this really does not sound like something that should be
configurable. Are you really seeing these bugs in the wild? And if so,
how do other OSs deal with this? I mean, having to add quirks to every
client does not sound very viable... Surely DNS is something that
should 'just work'. I feel like I'm missing some part of the picture
here.

Cheers,

Tom
Mantas Mikulėnas
2015-05-15 19:51:46 UTC
Permalink
Post by Tom Gundersen
Post by Christian Brunotte
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
In many setups, yes, but we will not aim at bug-for-bug compatibility
or anything like that. We are open to adding features that make sense
though.
Post by Christian Brunotte
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers cannot
handle queries properly and make the requests time out.")
Hm, this really does not sound like something that should be
configurable. Are you really seeing these bugs in the wild? And if so,
how do other OSs deal with this? I mean, having to add quirks to every
client does not sound very viable... Surely DNS is something that
should 'just work'. I feel like I'm missing some part of the picture
here.
Yeah, it *is* something that should 'just work', but then there are both
corporate firewalls and embedded devices (i.e. home gateways) which choke
on EDNS, crash upon seeing DNSSEC, truncate all packets to 512 bytes
(manufactured in 2015!), and even outright lie – by making up their own
(incorrect) responses instead of forwarding the query. (If it's not
encrypted, some proxy somewhere will screw it up...)

So, sometimes it "works" in the sense of "Windows doesn't trigger the bugs".
--
Mantas Mikulėnas <***@gmail.com>
Lennart Poettering
2015-05-15 20:02:07 UTC
Permalink
Post by Mantas Mikulėnas
Post by Tom Gundersen
Post by Christian Brunotte
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
In many setups, yes, but we will not aim at bug-for-bug compatibility
or anything like that. We are open to adding features that make sense
though.
Post by Christian Brunotte
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers cannot
handle queries properly and make the requests time out.")
Hm, this really does not sound like something that should be
configurable. Are you really seeing these bugs in the wild? And if so,
how do other OSs deal with this? I mean, having to add quirks to every
client does not sound very viable... Surely DNS is something that
should 'just work'. I feel like I'm missing some part of the picture
here.
Yeah, it *is* something that should 'just work', but then there are both
corporate firewalls and embedded devices (i.e. home gateways) which choke
on EDNS, crash upon seeing DNSSEC, truncate all packets to 512 bytes
(manufactured in 2015!), and even outright lie – by making up their own
(incorrect) responses instead of forwarding the query. (If it's not
encrypted, some proxy somewhere will screw it up...)
So, sometimes it "works" in the sense of "Windows doesn't trigger the bugs".
Well, but in all these cases, we should try to actively learn, and try
a few things before we give up, but certainly not give up completely
and let the user configure things.

For example, if a server doesn't resond to EDNS or DNSSEC or truncates
packets, then we should detect that automatically, fall back and
remember this during runtime.

One of the benefits that we have with resolved is that we are
system-wide and stateful, hence can remember these things relatively
easily, where classic nss-dns can't...

Lennart
--
Lennart Poettering, Red Hat
Marcel Holtmann
2015-05-15 20:28:12 UTC
Permalink
Hi Lennart,
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Tom Gundersen
Post by Christian Brunotte
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
In many setups, yes, but we will not aim at bug-for-bug compatibility
or anything like that. We are open to adding features that make sense
though.
Post by Christian Brunotte
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers cannot
handle queries properly and make the requests time out.")
Hm, this really does not sound like something that should be
configurable. Are you really seeing these bugs in the wild? And if so,
how do other OSs deal with this? I mean, having to add quirks to every
client does not sound very viable... Surely DNS is something that
should 'just work'. I feel like I'm missing some part of the picture
here.
Yeah, it *is* something that should 'just work', but then there are both
corporate firewalls and embedded devices (i.e. home gateways) which choke
on EDNS, crash upon seeing DNSSEC, truncate all packets to 512 bytes
(manufactured in 2015!), and even outright lie – by making up their own
(incorrect) responses instead of forwarding the query. (If it's not
encrypted, some proxy somewhere will screw it up...)
So, sometimes it "works" in the sense of "Windows doesn't trigger the bugs".
Well, but in all these cases, we should try to actively learn, and try
a few things before we give up, but certainly not give up completely
and let the user configure things.
For example, if a server doesn't resond to EDNS or DNSSEC or truncates
packets, then we should detect that automatically, fall back and
remember this during runtime.
so EDNS0 over UDP is something you really should to give up on. In ConnMan we tried hard to make that work and in the end gave up. Funny thing is that home routers in Germany were the most broken ones when it comes to EDNS0 support. The solution is to just jump to DNS over TCP right away and not try to handle larger packets with EDNS0 over UDP.

What this means is if a remote server or home network gateways truncates DNS packets, then switch over to establish a TCP connection for DNS and start using that one. Meaning in case the result is too large to fit in over UDP, make the same request over TCP and deliver that result to the clients. At that point, you can also keep using the TCP connection. I think in ConnMan we used some sort of idle timeout to terminate the TCP connections and fallback to UDP until the next truncated packet required us to re-establish the TCP connection.

And I think that glibc does something similar. If it receives a truncated packet, it tries to get the full packet over TCP and only if the DNS server does not support TCP, it delivers the truncated data.

Regards

Marcel
Mantas Mikulėnas
2015-05-16 11:30:05 UTC
Permalink
Post by Marcel Holtmann
Hi Lennart,
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Tom Gundersen
Post by Christian Brunotte
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
In many setups, yes, but we will not aim at bug-for-bug compatibility
or anything like that. We are open to adding features that make sense
though.
Post by Christian Brunotte
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions"
just
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Tom Gundersen
Post by Christian Brunotte
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
(quoting from the man page: "By default, glibc performs IPv4 and IPv6
lookups in parallel since version 2.9. Some appliance DNS servers
cannot
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Tom Gundersen
Post by Christian Brunotte
handle queries properly and make the requests time out.")
Hm, this really does not sound like something that should be
configurable. Are you really seeing these bugs in the wild? And if so,
how do other OSs deal with this? I mean, having to add quirks to every
client does not sound very viable... Surely DNS is something that
should 'just work'. I feel like I'm missing some part of the picture
here.
Yeah, it *is* something that should 'just work', but then there are both
corporate firewalls and embedded devices (i.e. home gateways) which
choke
Post by Lennart Poettering
Post by Mantas Mikulėnas
on EDNS, crash upon seeing DNSSEC, truncate all packets to 512 bytes
(manufactured in 2015!), and even outright lie – by making up their own
(incorrect) responses instead of forwarding the query. (If it's not
encrypted, some proxy somewhere will screw it up...)
So, sometimes it "works" in the sense of "Windows doesn't trigger the
bugs".
Post by Lennart Poettering
Well, but in all these cases, we should try to actively learn, and try
a few things before we give up, but certainly not give up completely
and let the user configure things.
For example, if a server doesn't resond to EDNS or DNSSEC or truncates
packets, then we should detect that automatically, fall back and
remember this during runtime.
so EDNS0 over UDP is something you really should to give up on. In ConnMan
we tried hard to make that work and in the end gave up. Funny thing is that
home routers in Germany were the most broken ones when it comes to EDNS0
support. The solution is to just jump to DNS over TCP right away and not
try to handle larger packets with EDNS0 over UDP.
What this means is if a remote server or home network gateways truncates
DNS packets, then switch over to establish a TCP connection for DNS and
start using that one. Meaning in case the result is too large to fit in
over UDP, make the same request over TCP and deliver that result to the
clients. At that point, you can also keep using the TCP connection. I think
in ConnMan we used some sort of idle timeout to terminate the TCP
connections and fallback to UDP until the next truncated packet required us
to re-establish the TCP connection.
And I think that glibc does something similar. If it receives a truncated
packet, it tries to get the full packet over TCP and only if the DNS server
does not support TCP, it delivers the truncated data.
Hmm, what happens when the server supports neither? (Retry with 8.8.8.8 or
another fallback?)
--
Mantas Mikulėnas <***@gmail.com>
Lennart Poettering
2015-05-18 15:51:59 UTC
Permalink
Post by Mantas Mikulėnas
Post by Marcel Holtmann
And I think that glibc does something similar. If it receives a truncated
packet, it tries to get the full packet over TCP and only if the DNS server
does not support TCP, it delivers the truncated data.
Hmm, what happens when the server supports neither? (Retry with 8.8.8.8 or
another fallback?)
I'd be very careful with such fallbacks: DNS servers (especially in
companies) tend to define their own local hostnames, and if we'd
automatically fall back to 8.8.8.8 in such cases, those hostnames
would suddently vanish from view.

8.8.8.8 is fine as fallback if no other DNS servers are known, but if
we know one we better do our best to talk to it.

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-05-15 19:59:37 UTC
Permalink
Post by Christian Brunotte
Hello
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
What kind of bugs does this really solve?

DNS servers that can only process one request per client at a time?

I mean, our intention is certainly to make things work by default,
without any manual configuration, and this glibc option is something
I'd really not like to replicate in resolved I must say.

but before we can see what we can do on this, I'd really need to
understand what kind of broken DNS servers this is really supposed to
fix?

Lennart
--
Lennart Poettering, Red Hat
Christian Brunotte
2015-05-15 21:01:25 UTC
Permalink
Post by Lennart Poettering
Post by Christian Brunotte
Hello
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
What kind of bugs does this really solve?
DNS servers that can only process one request per client at a time?
Firewalls notice outgoing UDP packets and allow response packets only within a
configured "UDP session timeout" time span. They need this timeout as UDP has
no opening and closing handshake like TCP. Some firewalls with "application
layer
gateways" try to be especially clever and "understand" that a DNS request packet
only gets exactly one DNS response packet after which they can safely close this
port. In the case of a IPv4+IPv6 dual stack system that is no longer the case,
though.
The resolver can send one DNS request packet (IPv4 or IPv6 doesn't matter) that
contains
queries for both the A and AAAA entries and the resolver may answer them in
separate packets.
Once the first one passes the firewall, the port is closed though. The requestor
now has to wait
some seconds in the hope that he gets the second packet - which never happens.
The GNU libc resolver knows about that problem and switches to single-request
automatically,
but that only happens for the current process which is why it's usually not
noticed. But if you
try to e.g. login via SSH, you encounter the problem every login as a new sshd
process is
spawned each time.

The above is from memory, so it might not be 100% accurate But I found other
links mentioning
the same problem and also with firewalls and IPv6:
* https://access.redhat.com/solutions/58625 (can't access myself but figured you
have an account :))
* http://udrepper.livejournal.com/20948.html
* https://bugzilla.redhat.com/show_bug.cgi?id=505105#c6 (mentioned Juniper
firwall)
* http://kb.juniper.net/InfoCenter/index?page=content&id=KB12312&actp=RSS (not
Linux specific)
Post by Lennart Poettering
I mean, our intention is certainly to make things work by default,
without any manual configuration, and this glibc option is something
I'd really not like to replicate in resolved I must say.
but before we can see what we can do on this, I'd really need to
understand what kind of broken DNS servers this is really supposed to
fix?
It's hard to estimate how many people are still affected by this as surely all
firewall vendors
should have fixed this by now.

But on the other hand, maybe it would'nt have any noticable speed drawback if
your resolved
just defaults to sending one request packet for each query and configure libc6
to do so as well
(I admit I don't understand how systemd-resolved and libc6 NSS work together
exactly).
Post by Lennart Poettering
Lennart
best regards,

-christian-
Alexander E. Patrakov
2015-05-16 10:52:49 UTC
Permalink
Post by Christian Brunotte
The resolver can send one DNS request packet (IPv4 or IPv6 doesn't matter) that
contains
queries for both the A and AAAA entries and the resolver may answer them in
separate packets.
I would be very much interested in seeing such successful conversation
in a pcap file. Here is the reason why I don't really belive you:
Unbound contains code that marks all DNS packets with multiple records
in the query section as invalid. The code is in ./daemon/worker.c,
Post by Christian Brunotte
if(LDNS_QDCOUNT(sldns_buffer_begin(pkt)) != 1) {
verbose(VERB_QUERY, "request wrong nr qd=%d",
LDNS_QDCOUNT(sldns_buffer_begin(pkt)));
return LDNS_RCODE_FORMERR;
}
--
Alexander E. Patrakov
Christian Brunotte
2015-05-16 12:18:08 UTC
Permalink
On Sat, 16 May 2015 15:52:49 +0500
Post by Alexander E. Patrakov
Post by Christian Brunotte
The resolver can send one DNS request packet (IPv4 or IPv6 doesn't matter) that
contains
queries for both the A and AAAA entries and the resolver may answer them in
separate packets.
I would be very much interested in seeing such successful conversation
Unbound contains code that marks all DNS packets with multiple records
in the query section as invalid. The code is in ./daemon/worker.c,
Post by Christian Brunotte
if(LDNS_QDCOUNT(sldns_buffer_begin(pkt)) != 1) {
verbose(VERB_QUERY, "request wrong nr qd=%d",
LDNS_QDCOUNT(sldns_buffer_begin(pkt)));
return LDNS_RCODE_FORMERR;
}
I've just captured one lookup (using "curl debian.netcologne.de").
You'll get the pcap separately, this is the text output:

# Without "single-request", both queries are made almost simultaneously
# and from the same UDP source port!
14:02:56.295863 IP james.intern.48045 > router.intern.domain: 20946+ A? debian.netcologne.de. (38)
14:02:56.295894 IP james.intern.48045 > router.intern.domain: 24543+ AAAA? debian.netcologne.de. (38)
14:02:56.296646 IP router.intern.domain > james.intern.48045: 20946* 2/0/0 CNAME mirror.netcologne.de., A 194.8.197.22 (75)
14:02:56.297064 IP router.intern.domain > james.intern.48045: 24543* 2/0/0 CNAME mirror.netcologne.de., AAAA 2001:4dd0:1234:1::deb (87)

# With "single-request", both queries still use the same source ports
# but are sent in order.
14:03:05.103876 IP james.intern.34759 > router.intern.domain: 19419+ A? debian.netcologne.de. (38)
14:03:05.104547 IP router.intern.domain > james.intern.34759: 19419* 2/0/0 CNAME mirror.netcologne.de., A 194.8.197.22 (75)
14:03:05.104671 IP james.intern.34759 > router.intern.domain: 37358+ AAAA? debian.netcologne.de. (38)
14:03:05.105246 IP router.intern.domain > james.intern.34759: 37358* 2/0/0 CNAME mirror.netcologne.de., AAAA 2001:4dd0:1234:1::deb (87)

So apparently the difference is not exactly as I described it but the
outcome is similar: The broken firewall appliance closes the UDP port after
receiving the first answer packet and drops the second one.

I wonder why it didn't send both requests simultaneously but from
different UDP source ports, shouldn't that be fast and safe?

best regards,

-christian-
Lennart Poettering
2015-05-18 15:48:46 UTC
Permalink
Post by Christian Brunotte
The resolver can send one DNS request packet (IPv4 or IPv6 doesn't matter) that
contains
queries for both the A and AAAA entries and the resolver may answer them in
separate packets.
I would be very much interested in seeing such successful conversation in a
pcap file. Here is the reason why I don't really belive you: Unbound
contains code that marks all DNS packets with multiple records in the query
section as invalid. The code is in ./daemon/worker.c, function
Post by Christian Brunotte
if(LDNS_QDCOUNT(sldns_buffer_begin(pkt)) != 1) {
verbose(VERB_QUERY, "request wrong nr qd=%d",
LDNS_QDCOUNT(sldns_buffer_begin(pkt)));
return LDNS_RCODE_FORMERR;
}
Yes, the DNS protocol does not allow query sections with more than one
question. (mDNS does allow this however).

Also note that doing an ANY query instead of A or AAAA will not work
either, since ANY actually means just "give me any RR you have", but
not actually "give me all you have" that match the rest of the
query... An ANY query might hence yield a response with just the A RR,
with just the AAAA RR or with both, it's up to the server, and the
server has the complete freedom to choose between the three cases...

Lennart
--
Lennart Poettering, Red Hat
Lennart Poettering
2015-05-18 16:59:56 UTC
Permalink
Post by Christian Brunotte
Post by Lennart Poettering
Post by Christian Brunotte
Hello
systemd.network(5) with Options like "DNS=" and "Domains=" looks like
/etc/resolv.conf will soon be superfluous.
If that's the plan, I wonder what happens to "options single-request"
which I had to use on all IPv6 enabled devices. Is "ResolveOptions" just
missing in Systemd or considered so "special" that will stay in libc's
resolv.conf?
What kind of bugs does this really solve?
DNS servers that can only process one request per client at a time?
Firewalls notice outgoing UDP packets and allow response packets only within a
configured "UDP session timeout" time span. They need this timeout as UDP has
no opening and closing handshake like TCP. Some firewalls with "application
layer
gateways" try to be especially clever and "understand" that a DNS request packet
only gets exactly one DNS response packet after which they can safely close this
port. In the case of a IPv4+IPv6 dual stack system that is no longer the case,
though.
The resolver can send one DNS request packet (IPv4 or IPv6 doesn't matter) that
contains
queries for both the A and AAAA entries and the resolver may answer them in
separate packets.
Once the first one passes the firewall, the port is closed though. The requestor
now has to wait
some seconds in the hope that he gets the second packet - which never happens.
Well, this is not possible with DNS (see other mail). But maybe this
really is about doing multiple parallel DNS queries from the same source IP +
port.

Right now all queries resolved does originate from the same IP
port. It has been requested to change this and use a new port number
for every single request, so that the 16 bit of the port can add to
the entropy when attackers want to guess DNS transaction credentials.

I wonder if we implement that if this might as side-effect also make
us more compatible with such firewalls, since unlike glibc we'd then
also have the A and AAAA requests come from a different IP/port pair...

Lennart
--
Lennart Poettering, Red Hat
Continue reading on narkive:
Loading...