Using unix socket to access docker daemon

[mbaechler]

I try to make GenericContainer launch a container using unix:///var/run/docker.sock as DOCKER_HOST.
I looks like it's the way to go based on https://github.com/docker-java/docker-java documentation, but it fails with :

org.testcontainers.shaded.javax.ws.rs.ProcessingException: org.newsclub.net.unix.AFUNIXSocketException: No such file or directory (socket: /home/matthieu/code/james-gitsvn/server/protocols/jmap-integration-testing/:80)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:490)
    at org.testcontainers.shaded.org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:246)
    at org.testcontainers.shaded.org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
    at org.testcontainers.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.testcontainers.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.testcontainers.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:228)
    at org.testcontainers.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424)
    at org.testcontainers.shaded.org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:679)
    at org.testcontainers.shaded.org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:408)
    at org.testcontainers.shaded.org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:308)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.VersionCmdExec.execute(VersionCmdExec.java:25)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.VersionCmdExec.execute(VersionCmdExec.java:12)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:24)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:75)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:71)
    at org.apache.james.jmap.ContainerTest.<init>(ContainerTest.java:36)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195)
    at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: org.newsclub.net.unix.AFUNIXSocketException: No such file or directory (socket: /home/matthieu/code/james-gitsvn/server/protocols/jmap-integration-testing/:80)
    at org.newsclub.net.unix.NativeUnixSocket.connect(Native Method)
    at org.newsclub.net.unix.AFUNIXSocketImpl.connect(AFUNIXSocketImpl.java:134)
    at org.newsclub.net.unix.AFUNIXSocket.connect(AFUNIXSocket.java:97)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.ApacheUnixSocket.connect(ApacheUnixSocket.java:64)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.UnixConnectionSocketFactory.connectSocket(UnixConnectionSocketFactory.java:73)
    at org.testcontainers.shaded.org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
    at org.testcontainers.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
    at org.testcontainers.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
    at org.testcontainers.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
    at org.testcontainers.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
    at org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
    at org.testcontainers.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
    at org.testcontainers.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
    at org.testcontainers.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:443)
    ... 38 more

Is there something I don't understand ?

[rnorth]

Hi,
This looks odd; to be honest I suspect it's more than likely not your mistake. UNIX socket support isn't properly tested so I wonder if some changes along the line have broken it. Sorry for this.

Just to be sure, please could you paste some more of the logs that run up to this error - Testcontainers should report on the daemon connection settings it's about to use. This would help rule out any other issues.

I need to take an action to (a) give socket support the automated test coverage it needs and (b) fix this.

In the mean time do you have the option to use the docker daemon over HTTP/HTTPS instead of sockets?

Thanks
Richard

[mbaechler]

here are the logs :

[main] INFO org.testcontainers.dockerclient.DockerConfigurationStrategy - Looking for Docker environment. Trying Environment variables, system properties and defaults. Resolved: 
    uri=unix:///var/run/docker.sock
    sslConfig='LocalDirectorySSLConfig{dockerCertPath=/home/matthieu/.docker}'
    version='null'
    username='matthieu'
    password='null'
    email='null'
    serverAddress='https://index.docker.io/v1/'
    dockerCfgPath='/home/matthieu/.dockercfg'

[main] INFO org.testcontainers.dockerclient.DockerConfigurationStrategy - Found docker client settings from environment
[main] INFO org.testcontainers.dockerclient.DockerConfigurationStrategy - Docker host IP address is localhost
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - I/O exception (org.newsclub.net.unix.AFUNIXSocketException) caught when processing request: No such file or directory
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - Retrying request
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - I/O exception (org.newsclub.net.unix.AFUNIXSocketException) caught when processing request: No such file or directory
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - Retrying request
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - I/O exception (org.newsclub.net.unix.AFUNIXSocketException) caught when processing request: No such file or directory
[main] INFO org.testcontainers.shaded.org.apache.http.impl.execchain.RetryExec - Retrying request

It'll use TCP instead for now but I will need unix socket support later.

Thank for your support anad for you library, it looks great !

[mbaechler]

What is really weird is that ping command works :

15:27:56.856 [main] DEBUG org.apache.http.conn.HttpClientConnectionManager - Connecting to localhost/127.0.0.1:80
15:27:56.858 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request GET /_ping HTTP/1.1
15:27:56.858 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
15:27:56.858 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
15:27:56.859 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /_ping HTTP/1.1
15:27:56.859 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Jersey/2.11 (Apache HttpClient 4.3.1)
15:27:56.859 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: localhost:80
15:27:56.859 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
15:27:56.859 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "GET /_ping HTTP/1.1[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Jersey/2.11 (Apache HttpClient 4.3.1)[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: localhost:80[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: Docker/1.9.1 (linux)[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Thu, 11 Feb 2016 14:27:56 GMT[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Length: 2[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: text/plain; charset=utf-8[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
15:27:56.860 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "OK"
15:27:56.863 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK
15:27:56.863 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Server: Docker/1.9.1 (linux)
15:27:56.863 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Thu, 11 Feb 2016 14:27:56 GMT
15:27:56.863 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Length: 2
15:27:56.863 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: text/plain; charset=utf-8

And the next call fails

[rnorth]

Wow - that's really perplexing!
This may be interesting to track down!

[mbaechler]

Ok, I think I got it :
When trying configuration strategies, DockerCmdExecFactoryImpl sanitize the given url from unix:///var/run/docker.sock to unix://localhost:80/ but by doing so, it changes the input configuration.
The second time, the input is the sanitized url is passed to DockerCmdExecFactoryImpl but then it uses this sanitized version to create UnixConnectionSocketFactory. No luck, the constructor simply strip unix://localhost ... and there's no url left.
The good solution would be to prevent DockerCmdExecFactoryImpl to change values into the input configuration.

[mbaechler]

opened on docker-java : docker-java/docker-java#456
will fix it

[rnorth]

Awesome, thank you :)

[mbaechler]

Could you have a look at #67 and even better, release a version with this fix in it ?

[rnorth]

I've just released v1.0.1 with this change merged in. Thanks for your hard (cross-project!) work on this - I hope this fixes the issue you were seeing.
Richard

[mbaechler]

Thank you, that was fast !