testcontainers-java
testcontainers-java copied to clipboard
testcontainers
Handle empty or invalid `$HOME/.docker/config.json` gracefully
Reference: https://issues.apache.org/jira/browse/INFRA-22697
Apparently testcontainers requires $HOME/.docker/config.json to be a valid JSON file.
touch ~/.docker/config.json
// lauch some test containers tests will now fail
echo '{}' > ~/.docker/config.json
// lauch some test containers tests will now succeeds
rm ~/.docker/config.json
// lauch some test containers tests will now succeeds
The problem is that I have little control on the content of this file as provided on the ASF CI environments. I would like testcontainers to ignore it as missing when its content is missing...
Hey @chibenwa, thanks for raising the issue. Can you please provide logs of the errors that occur in Testcontainers if config.json is empty?
I got for instance
java.lang.ExceptionInInitializerError
at org.apache.james.backends.cassandra.DockerCassandraRule.start(DockerCassandraRule.java:46)
at org.apache.james.backends.cassandra.DockerCassandraExtension.beforeAll(DockerCassandraExtension.java:48)
at org.apache.james.backends.cassandra.CassandraClusterExtension.beforeAll(CassandraClusterExtension.java:54)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$8(ClassBasedTestDescriptor.java:368)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:368)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:192)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:136)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Suppressed: java.lang.NullPointerException
at org.apache.james.backends.cassandra.CassandraClusterExtension.afterAll(CassandraClusterExtension.java:93)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeAfterAllCallbacks$14(ClassBasedTestDescriptor.java:434)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeAfterAllCallbacks$15(ClassBasedTestDescriptor.java:434)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeAfterAllCallbacks(ClassBasedTestDescriptor.java:434)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:216)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:78)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:149)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:149)
... 29 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
at org.apache.james.backends.cassandra.DockerCassandra.start(DockerCassandra.java:198)
at org.apache.james.backends.cassandra.DockerCassandraSingleton.<clinit>(DockerCassandraSingleton.java:40)
... 40 more
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=<resolving>, imagePullPolicy=DefaultPullPolicy())
at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1286)
at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:615)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
... 43 more
Caused by: com.github.dockerjava.api.exception.DockerClientException: Failed to parse docker configuration file
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultDockerClientConfig.getDockerConfig(DefaultDockerClientConfig.java:265)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultDockerClientConfig.getAuthConfigurations(DefaultDockerClientConfig.java:301)
at org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig.getAuthConfigurations(AuthDelegatingDockerClientConfig.java:23)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrDockerCmdExec.getBuildAuthConfigs(AbstrDockerCmdExec.java:40)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.BuildImageCmdExec.resourceWithOptionalAuthConfig(BuildImageCmdExec.java:29)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.BuildImageCmdExec.execute0(BuildImageCmdExec.java:126)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.BuildImageCmdExec.execute0(BuildImageCmdExec.java:20)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.execute(AbstrAsyncDockerCmdExec.java:56)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:21)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:12)
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrAsyncDockerCmd.exec(AbstrAsyncDockerCmd.java:21)
at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:119)
at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:37)
at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
at org.testcontainers.shaded.com.google.common.util.concurrent.Futures$3.get(Futures.java:1332)
at org.testcontainers.images.RemoteDockerImage.getImageName(RemoteDockerImage.java:104)
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:63)
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1284)
... 45 more
Caused by: java.io.IOException: Failed to parse docker config.json
at org.testcontainers.shaded.com.github.dockerjava.core.DockerConfigFile.loadCurrentConfig(DockerConfigFile.java:164)
at org.testcontainers.shaded.com.github.dockerjava.core.DockerConfigFile.loadConfig(DockerConfigFile.java:131)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultDockerClientConfig.getDockerConfig(DefaultDockerClientConfig.java:263)
... 66 more
Caused by: org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
at [Source: (File); line: 1, column: 0]
at org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
at org.testcontainers.shaded.com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4360)
at org.testcontainers.shaded.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4205)
at org.testcontainers.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
at org.testcontainers.shaded.com.github.dockerjava.core.DockerConfigFile.loadCurrentConfig(DockerConfigFile.java:162)
... 68 more
Handle empty $HOME/.docker/config.json gracefully
I wonder if it should not be rather Handle invalid $HOME/.docker/config.json gracefully.
As this config is apparently only used to discover private registry if invalid we could adopt a lenient beaviour and behave as if there was no such file (maybe with a log)
This is what docker CLI does:
$ echo "bad" > ~/.docker/config.json
$ cat ~/.docker/config.json
bad
$ docker ps
WARNING: Error loading config file: /home/interview1/.docker/config.json: invalid character 'b' looking for beginning of value
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
...
You could see that even if it complains, the invalid file did not prevent the docker CLI from working normally...
I think such a behaviour would be less specific to my use case but benefit the whole community.
BTW it looks like the exception is coming from docker-java, so a fix would be needed there first and then we just need to update it in Testcontainers.
Hi @chibenwa, can you still reproduce this if invalid config.json file is provided?
Reference: https://issues.apache.org/jira/browse/INFRA-22697
Apparently testcontainers requires $HOME/.docker/config.json to be a valid JSON file.
touch ~/.docker/config.json // lauch some test containers tests will now fail echo '{}' > ~/.docker/config.json // lauch some test containers tests will now succeeds rm ~/.docker/config.json // lauch some test containers tests will now succeedsThe problem is that I have little control on the content of this file as provided on the ASF CI environments. I would like testcontainers to ignore it as missing when its content is missing...
I have tried to reproduce it locally (testcontainers version 1.17.2) by executing test provided here, but I can't (not sure if I'm doing something wrong, or if the "issue" is fixed) 🤔
touch ~/.docker/config.json
// lauch some test containers tests will now fail <-- this is no longer true
21:13:07.484 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Delegate call to effectiveAuthConfig failed with cause: 'Failed to parse docker configuration file'. Resolution of auth config will continue using RegistryAuthLocator.
21:13:07.486 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.3 at registry: https://index.docker.io/v1/
21:13:07.487 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /qwe/user/.docker/config.json (exists) and commandPathPrefix:
21:13:07.487 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.3, configFile: /qwe/user/.docker/config.json. Falling back to docker-java default behaviour. Exception message: No content to map due to end-of-input
at [Source: /qwe/user/.docker/config.json; line: 1, column: 0]
21:13:07.489 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [AuthConfig{username=null, password=blank, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
21:13:07.490 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=null, password=blank, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
2
echo '{}' > ~/.docker/config.json
// lauch some test containers tests will now succeeds <-- this is true
21:16:06.512 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.3 at registry: https://index.docker.io/v1/
21:16:06.514 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /qwe/user/.docker/config.json (exists) and commandPathPrefix:
21:16:06.514 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [https://index.docker.io/v1/] for dockerImageName [testcontainers/ryuk:0.3.3]
21:16:06.515 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
21:16:06.515 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
rm ~/.docker/config.json
// lauch some test containers tests will now succeeds <-- this is true
21:17:47.531 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.3 at registry: https://index.docker.io/v1/
21:17:47.532 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /qwe/user/.docker/config.json (does not exist) and commandPathPrefix:
21:17:47.532 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.3, configFile: /qwe/user/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /qwe/user/.docker/config.json (No such file or directory)
21:17:47.534 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
21:17:47.534 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
echo "bad" > ~/.docker/config.json
// lauch some test containers tests will now fail <-- this is no longer true
21:23:04.943 [main] INFO 🐳 [testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
21:23:04.946 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Delegate call to effectiveAuthConfig failed with cause: 'Failed to parse docker configuration file'. Resolution of auth config will continue using RegistryAuthLocator.
21:23:04.947 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.3 at registry: https://index.docker.io/v1/
21:23:04.948 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /qwe/user/.docker/config.json (exists) and commandPathPrefix:
21:23:04.949 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.3, configFile: /qwe/user/.docker/config.json. Falling back to docker-java default behaviour. Exception message: Unrecognized token 'bad': was expecting ('true', 'false' or 'null')
at [Source: /qwe/user/.docker/config.json; line: 1, column: 5]
21:23:04.950 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [AuthConfig{username=null, password=blank, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
21:23:04.950 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=null, password=blank, auth=blank, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
@chibenwa @kiview hi, let's close this issue I also tried to reproduce the problem locally, but all works without errors
It appeared on 1.18.0 (empty config.json file) (previous releases working fine)
I guess api platform('com.github.docker-java:docker-java-bom:3.2.13') shaded platform('com.github.docker-java:docker-java-bom:3.2.13') api platform('com.github.docker-java:docker-java-bom:3.3.0') shaded platform('com.github.docker-java:docker-java-bom:3.3.0')
cause it
support for docker context was recently introduced in docker-java. So, that's what can cause the issue now.
