This repository was archived by the owner on Oct 21, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 21, 2024. It is now read-only.
DNS resolution broken over IPv6 #280
Description
Hi,
I'm having an issue and after trying to investigate myself I've come to a dead-end.
A container in a local dev env was timing out on making some requests, and it seems to be due to an ipv6 issue.
This command has a delay of ~5s before returning results
docker run --rm -it alpine:3.7 ping google.comwhereas
docker run --rm -it alpine:3.7 ping -4 google.comis almost instant.
I think this is because the DNS resolution is not working over ipv6.
$ docker run --rm -it alpine:3.7 sh -c 'apk add --no-cache bind-tools && dig google.com AAAA'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/4) Installing libgcc (6.4.0-r5)
(2/4) Installing libxml2 (2.9.7-r0)
(3/4) Installing bind-libs (9.11.3-r0)
(4/4) Installing bind-tools (9.11.3-r0)
Executing busybox-1.27.2-r11.trigger
OK: 9 MiB in 17 packages
; <<>> DiG 9.11.3 <<>> google.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOTIMP, id: 8782
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 7dfd9da7da61a45f (echoed)
;; QUESTION SECTION:
;google.com. IN AAAA
;; Query time: 0 msec
;; SERVER: 10.0.2.3#53(10.0.2.3)
;; WHEN: Thu Sep 06 08:48:44 UTC 2018
;; MSG SIZE rcvd: 51$ docker run --rm -it alpine:3.7 sh -c 'apk add --no-cache bind-tools && dig google.com AAAA @1.1.1.1'
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/4) Installing libgcc (6.4.0-r5)
(2/4) Installing libxml2 (2.9.7-r0)
(3/4) Installing bind-libs (9.11.3-r0)
(4/4) Installing bind-tools (9.11.3-r0)
Executing busybox-1.27.2-r11.trigger
OK: 9 MiB in 17 packages
; <<>> DiG 9.11.3 <<>> google.com AAAA @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53178
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;google.com. IN AAAA
;; ANSWER SECTION:
google.com. 229 IN AAAA 2a00:1450:4009:803::200e
;; Query time: 4 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 05 15:49:14 UTC 2018
;; MSG SIZE rcvd: 67Not sure why this is causing ping/curl to timeout, but this seems to be the root of my issue.
Any ideas on how I can fix or work around the issue?
Thanks a lot!
Activity
adamquaile commentedon Sep 6, 2018
I should also say that I'm using the virtualbox VM (xhyve doesn't work for me, no licenses for others) and that I've tried the same steps with Docker for mac / without dinghy and it works as expected.
codekitchen commentedon Sep 6, 2018
Hm I can't reproduce this on my machine using the xhyve backend, I'll have to set up a Virtualbox env when I get a chance. It looks like IPv6 DNS resolution is working fine, but my container doesn't even get an IPv6 address (though the host docker-machine VM does have an IPv6 address). Is there something specific you've done to give the container IPv6 support, or did that happen automatically for you? I'll admit I've never had the need for IPv6 in a docker container yet, so I haven't really looked at how that is configured, etc.
So it's possibly a problem specific to the Virtualbox backend, but I'm not sure why that'd be. If you
dinghy sshinto the docker-machine VM and run dig and ping there outside of a container, do you see the same problem?adamquaile commentedon Sep 6, 2018
I don't think I do have ipv6 support working:
$ docker run --rm -it alpine:3.7 ifconfig eth0 eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:03 inet addr:172.17.0.3 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:110 (110.0 B) TX bytes:0 (0.0 B)The reason I'm doing an IPv6 lookup is not particularly because I need that functionality - it's just that alpine linux seems to do an
AAAAandArequest regardless. There's other issues relating to that project if you search around, e.g. gliderlabs/docker-alpine#153I think this is normally not an issue so long as the AAAA request either succeeds or fails immediately.
I think it will be virtualbox specific. I have done some tests as you suggest and a bit of digging earlier and found a few interesting things.
From within the dinghy VM:
The IP it's using to do that lookup is 10.0.2.3 which I think is something Virtualbox sets up maybe related to its creation of a NAT network adapter.
I also tried to get around that issue by disabling that functionality, by running:
but that didn't seem to help.
This does seem to be the root of the issue though; if I update /etc/resolv.conf in the dinghy VM to 1.1.1.1 then the lookups succeed from inside the VM and inside the alpine containers too.
I understand maybe that's not a permanent solution (would that break *.docker resolution from inside other containers?) but if there's any other steps I can do to help diagnose it let me know.
Also, thanks for your support and the project in general. Having tried my own dnsmasq/nginx-proxy/vagrant hacks, it's a much better executed version of all that and I'm so close to being able to drop all my hacks and still have a nice dev environment on mac 👍
codekitchen commentedon Sep 10, 2018
I'm able to repro the problem using the Virtualbox backend. I tried creating another VM directly with
docker-machine createand couldn't repro there, so that narrowed it down to what Dinghy does during the set up phase.If I remove the
VBoxManage modifyvm "dinghy" --natdnshostresolver1 oncommand that Dinghy runs on setup, that fixes the issue (ping still fails of course, but it resolves google.com over IPv6 immediately). I'm not sure why it didn't fix it when you turned it off after the fact, there must be some sticky state somewhere.Unfortunately, the whole reason we run that command is that resolving our
*.dockeraddresses doesn't work without it, so I don't think it's an acceptable solution unless we find some other good way to make that work under Virtualbox.I'm no expert at this stuff but I'll keep playing around with it a bit, see if I can find any configuration that fixes IPv6 DNS without breaking our resolver.
ImTheDeveloper commentedon Mar 4, 2019
I seem to be hitting the same issue without vbox I tried to document it as well as I could here. Does this look to be an underlying issue?
https://superuser.com/questions/1411111/docker-dns-resolution-slow-for-http-calls-in-node-application-ipv6-bridge-netw