testcontainers
HiveMQContainer with reuse on macOS M1 intermittently fails to start
Intermittently, when the HiveMQContainer is already running, our reused container fails to start with the following exception.
...
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hiveMqContainer' defined in class path resource [app/site/config/HiveMqContainerConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.hivemq.HiveMQContainer]: Factory method 'hiveMqContainer' threw exception; nested exception is org.testcontainers.containers.ContainerLaunchException: Container startup failed
at app//org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
at app//org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at app//org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at app//org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at app//org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at app//org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at app//org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
at app//org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
at app//org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
at app//org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
at app//org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
... 106 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.hivemq.HiveMQContainer]: Factory method 'hiveMqContainer' threw exception; nested exception is org.testcontainers.containers.ContainerLaunchException: Container startup failed
at app//org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at app//org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
... 120 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:345)
at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
at app//app.site.config.HiveMqContainerConfig$Companion.createContainer(InfrastructureConfig.kt:130)
at app//app.site.config.HiveMqContainerConfig.hiveMqContainer(InfrastructureConfig.kt:137)
at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c.CGLIB$hiveMqContainer$0(<generated>)
at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c$$FastClassBySpringCGLIB$$87ad679a.invoke(<generated>)
at app//org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at app//org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c.hiveMqContainer(<generated>)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
at app//org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 121 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:338)
... 134 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:537)
at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:340)
at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 135 more
Caused by: org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
at app//org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:70)
at app//org.rnorth.ducttape.timeouts.Timeouts.doWithTimeout(Timeouts.java:60)
at app//org.testcontainers.containers.wait.strategy.WaitAllStrategy.waitUntilReady(WaitAllStrategy.java:53)
at app//org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:926)
at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:480)
... 137 more
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:65)
... 141 more
We haven't been able to pinpoint a clear cause. Third party MQTT clients (https://mqttx.app/) remain connected just fine, while the HiveMQContainer code fails. Vital stats follow.
build.gradle.kts plugins:
plugins {
id("org.springframework.boot") version "2.6.7"
id("io.spring.dependency-management") version "1.0.11.RELEASE"
id("org.jlleitschuh.gradle.ktlint") version "10.3.0"
id("org.jetbrains.dokka") version "1.6.21"
jacoco
kotlin("jvm") version "1.6.21"
kotlin("plugin.spring") version "1.6.21"
kotlin("plugin.allopen") version "1.6.21"
kotlin("kapt") version "1.6.21"
}
build.gradle.kts testcontainer dependencies:
// normally, these would be testImplementation, but this allows us to run locally
// without having to launch containers manually
implementation(platform("org.testcontainers:testcontainers-bom:1.17.2"))
implementation("org.testcontainers:testcontainers")
implementation("org.testcontainers:mongodb")
implementation("org.testcontainers:postgresql")
implementation("org.testcontainers:hivemq")
Docker version:
$ docker version
Client:
Cloud integration: v1.0.24
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:49:20 2022
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Desktop 4.8.2 (79419)
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:44 2022
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
macOS version: 12.3.1 (21E258) (Monterey)
MBP Model Name: MacBook Pro (16-inch, 2021)
MBP Model Number: A2485
MBP Model Identifier: MacBookPro18,2
RAM: 64 Gb
Spring Boot Kotlin config class:
@Configuration
@Profile("default") // should only be activated locally on developers' machines
class HiveMqContainerConfig {
companion object {
val LOG = LoggerFactory.getLogger(HiveMqContainerConfig::class.java)
val REUSE_ID = "app:site-microservice:hivemq"
val IMAGE = DockerImageName.parse("hivemq/hivemq-ce:2021.3")
var CONTAINER: HiveMQContainer? = null
fun createContainer(): HiveMQContainer {
if (CONTAINER === null) {
CONTAINER = HiveMQContainer(IMAGE).apply {
withLogLevel(Level.DEBUG)
// NOTE: you have to set
// testcontainers.reuse.enable=true in local file ~/.testcontainers.properties
// for containers to actually be reused!
withReuse(true)
withLabel("reuse.id", REUSE_ID)
}
LOG.info("starting hivemq container -- you're prolly a dev, right?")
CONTAINER!!.start()
}
return CONTAINER!!
}
}
@Bean(destroyMethod = "")
fun hiveMqContainer(): HiveMQContainer = createContainer()
@Bean
fun hiveMqClient(
@Value("\${mqtt.client.identifier}") identifier: String,
container: HiveMQContainer
): Mqtt5AsyncClient {
return MqttClient
.builder()
.useMqttVersion5()
.identifier(identifier)
.serverHost(container.host)
.serverPort(container.mqttPort)
// TODO: other stuff...
.buildAsync()
}
}
More info: noticed that only the first instantiation of HiveMQContainer with in a JVM fails. All subsequent invocations of the container class appear to succeed. Might be useful.
I've also tried other fixes I've seen in this issue list of adding checks.disable=true and adding versions 5.7.0 & 5.11.0 of JNA, all to no avail.
It turns out that my first statement above (https://github.com/testcontainers/testcontainers-java/issues/5464#issuecomment-1145096572) is a lie. We just saw a first instantiation of a reused container work, then a subsequent instantiation of a reused container (all with the same reuse id) fail. 🤷🏽 This is a real problem for us, because we cannot get through our integration tests....
I ran these tests and spun up the container on a 2020 Intel Mac with Monterey 12.4 and also experienced the same flakiness with the HiveMQ container
Hello @matthewadams!
The exception you present looks like the HiveMQ docker container is unable to start in time. It seems like you are running on arm64, please note that the image hivemq/hivemq-ce:2021.3 only comes for amd64. Normally, it is still possible to run amd64 images on arm64 machines, but from experience we know that this is rather unstable because of the underlying emulation.
As HiveMQ CE is open source there are unofficial arm64 images on dockerhub, that you can use: https://hub.docker.com/r/yannickweber/hivemq-ce.
val IMAGE = DockerImageName.parse("yannickweber/hivemq-ce").asCompatibleSubstituteFor("hivemq/hivemq-ce")
Anyhow, official arm64 support will come in the very near future.
If your problem persists please share the full log output of the container.
Hey @tristanradams!
That indeed seems odd. Would you mind sharing the whole HiveMQ log output and the exception?
@YannickWeber FYI https://github.com/hivemq/hivemq4-docker-images/issues/38
We should follow https://github.com/hivemq/hivemq4-docker-images/issues/38 and wait for an arm image due to is already known that it will not work well on that architecture.
