apache
"Could not acquire lock(s)" during multi-threaded builds (-T 1C)
Affected version
3.9.9
Bug description
Using Maven 3.9.9 or Maven Daemon 1.0.2 (which I understand packages 3.9.9 internally), on a Linux Rocky 8.10 system. The fault occurs sporadically with either type of build.
We've been confused for a while about this issue because we have a master repository whose build is driven by a Makefile, and 5 of the components in that repository are isolated multi-module Maven projects with interdependencies. We've organized the Makefile dependencies so no Maven project will be built at the same time as another one when make -j is used.
However, further research and careful analysis indicates that this error does not occur between component builds, but rather, between modules within a single component build:
...
2025-09-10 12:16:02.872 [INFO] ------------------------------------------------------------------------
2025-09-10 12:16:02.872 [INFO] Reactor Summary for Java Common Project 5.3.0:
2025-09-10 12:16:02.872 [INFO]
2025-09-10 12:16:02.872 [INFO] Java Common Project ................................ SUCCESS [01:15 min]
2025-09-10 12:16:02.872 [INFO] Common Utilities ................................... FAILURE [01:31 min]
2025-09-10 12:16:02.872 [INFO] CSP ReST Client .................................... SKIPPED
2025-09-10 12:16:02.872 [INFO] ICAP Client ........................................ SKIPPED
2025-09-10 12:16:02.872 [INFO] Storage Connectors ................................. SKIPPED
2025-09-10 12:16:02.872 [INFO] HMDB Common Model .................................. SKIPPED
2025-09-10 12:16:02.872 [INFO] KMS Provider Project ............................... SUCCESS [ 5.752 s]
2025-09-10 12:16:02.872 [INFO] KMS Common Model ................................... SKIPPED
2025-09-10 12:16:02.872 [INFO] KMS AWS Provider ................................... SKIPPED
2025-09-10 12:16:02.872 [INFO] KMS NShield HSM Provider ........................... SKIPPED
2025-09-10 12:16:02.872 [INFO] General ReST Clients ............................... SKIPPED
2025-09-10 12:16:02.872 [INFO] NFSv3 RPC Client ................................... SKIPPED
2025-09-10 12:16:02.872 [INFO] Kafka Test Framework ............................... SUCCESS [03:09 min]
2025-09-10 12:16:02.872 [INFO] ------------------------------------------------------------------------
2025-09-10 12:16:02.872 [INFO] BUILD FAILURE
2025-09-10 12:16:02.872 [INFO] ------------------------------------------------------------------------
2025-09-10 12:16:02.872 [INFO] Total time: 04:26 min (Wall Clock)
2025-09-10 12:16:02.872 [INFO] Finished at: 2025-09-10T19:15:57Z
2025-09-10 12:16:02.872 [INFO] ------------------------------------------------------------------------
2025-09-10 12:16:02.872 [ERROR] Could not acquire lock(s)
2025-09-10 12:16:02.872 [ERROR] Common Utilities: Could not acquire lock(s)
2025-09-10 12:16:02.872 [ERROR]
2025-09-10 12:16:02.872 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2025-09-10 12:16:02.872 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2025-09-10 12:16:02.872 [ERROR]
2025-09-10 12:16:02.872 [ERROR] After correcting the problems, you can resume the build with the command
2025-09-10 12:16:02.872 [ERROR] mvn <args> -r
2025-09-10 12:16:02.872 + '[' 1 -ne 0 ']'
2025-09-10 12:16:02.872 + die 'java-common build failed!'
2025-09-10 12:16:02.872 + set +x
2025-09-10 12:16:02.872
2025-09-10 12:16:02.872 java-common build failed!
As you can see here, a few of the modules in this multi-module component succeeded, and then suddenly we get a lock error on the next module. We've carefully verified that, not only is this the only component using the .m2 directory at this time, but it's also the only Maven component being built on this system at this time.
We've experienced this issue sporadically since we upgraded from Java 8 to Java 21 last December (2024) and (in the process) we also upgraded from Maven 3.5.6 to 3.9.9 and from Artifactory to Nexus. We only just decided to file an issue on it because we've noticed that we're not seeing the lock issue if we downgrade to Maven 3.8.9, which I understand has a different locking mechanism. We've also noticed we do not see the issue when using Maven Daemon 1.0.1, though we do see an occasional daemon startup failure in 1.0.1 (different, unrelated issue).
We have also never seen the issue when using -T1 - this only occurs in multi-threaded builds (-T 1C).
Because it's sporadic, we never know when it's going to happen, so we don't have -e enabled during the build. If you need that data, we could temporarily enable it in our master builds, but it makes for some chatty logs.
Thanks for report! Maven version before 3.9 does not use locking at all (local repo is "free for all" [as it gets]). Also, MRM is not involved at this at all.
For start, but I cannot tell "blindly", you may do following:
- increase timeouts on locks, see https://maven.apache.org/resolver-archives/resolver-1.9.24/configuration.html
- maybe disable locking at all, or try some diag collection https://maven.apache.org/resolver-archives/resolver-1.9.24/maven-resolver-named-locks/index.html
Related: https://github.com/apache/maven/issues/9049
Thanks @cstamas for the quick response. I read through that entire other thread. I had seen it before posting this issue, but I assumed because I was on a later version, my issue must be different. I can now see that was an incorrect assumption.
The fact is, we've got a sort of "perfect storm" scenario, that's likely the root cause of our CI builds exposing this problem. First, we're having network and resource problems on our build boxes - they're VMware VMs and our company has been growing rapidly in the last year, so these boxes are underpowered - we're working on that, but this is slowing things down dramatically over the last month or two in a way that was not happening back in January. Second, we build from scratch with empty local maven repositories, so we have to fill those from nexus with every build. Hence, we're much more likely to hit a lock timeout in a CI build than in a regular user build.
I read about the use of these options:
-Daether.syncContext.named.time=600 -Daether.syncContext.named.time.unit=SECONDS
But I'm not an expert on the internals of Maven, so I'm not 100% sure how to use these options most effectively. Can I just export them in the MAVEN_OPTS environment variable before building?
Read here: https://maven.apache.org/configure.html
The MAVEN_OPTS are passed to JVM, so you want MAVEN_ARGS. Or just create .mvn/maven.config in project root (but this will apply to all users building the project, not only CI). Or, if you use CI specific settings.xml you can define properties there.
There "configurations" are tweaking configurations, so feel free to experiment.
@cstamas we used to build these projects in our master repo as completely isolated projects - there was no root-level pom. each project had it's own "root-level" pom. We're slowly migrating to a model where we can ultimately build all of our maven projects from the root using a true root-level pom.xml. Right now, all it does is refer to the sub-projects (modules), and each project's root pom refers to the parent. This way we can start moving our dependency and plugin management sections into that proper root pom.
Since we have this root pom, I added a .mvn/maven.config file to the repository root. I'm assuming that when building one of the components, it will pick up the .mvn/maven.config file in the parent directory, correct? If not, I can put a .mvn/maven.config file in each of the component roots.
Question: How can I tell if this .mvn/maven.config file is being picked up and used? Is there some kind of command line option I can use that will tell me it's being read and applied?
For example you can add -V (show version and continue), which is anyway good thing to have on CI... (unless you have it already on command line CI invokes).
It prints out every essential thing about Maven, for example:
[cstamas@angeleyes maven-3 (maven-3.9.x)]$ mvn -V clean
Apache Maven 3.9.12-SNAPSHOT (76aaa5bd5900b0da89b2c70238ee7f1801e79c6c)
Maven home: /home/cstamas/.sdkman/candidates/maven/latest-3
Java version: 21.0.8, vendor: Eclipse Adoptium, runtime: /home/cstamas/.sdkman/candidates/java/21.0.8-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.16.7-200.fc42.x86_64", arch: "amd64", family: "unix"
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache Maven [pom]
[INFO] Maven Model [jar]
[INFO] Maven Artifact [jar]
[INFO] Maven Plugin API [jar]
[INFO] Maven Builder Support [jar]
@cstamas any idea why I might see this on a build?
2025-09-16 11:37:16.94 ----- End of the daemon output -----
2025-09-16 11:37:16.94
2025-09-16 11:37:16.94 at org.mvndaemon.mvnd.client.DaemonClientConnection.receive(DaemonClientConnection.java:138)
2025-09-16 11:37:16.94 at org.mvndaemon.mvnd.client.DefaultClient.execute(DefaultClient.java:377)
2025-09-16 11:37:16.94 at org.mvndaemon.mvnd.client.DefaultClient.main(DefaultClient.java:169)
2025-09-16 11:37:16.94 at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)
2025-09-16 11:37:16.94 Caused by: java.io.IOException: No message received within 3000ms, daemon may have crashed. You may want to check its status using mvnd --status
2025-09-16 11:37:16.94 at org.mvndaemon.mvnd.client.DaemonClientConnection.receive(DaemonClientConnection.java:119)
2025-09-16 11:37:16.94 ... 3 more
I know I should probably open another ticket for this, but it seems like you may already know what causes this (from some hints on the other ticket).
This is question for https://github.com/apache/maven-mvnd rather 😄
Check ~/.m2/mvnd/registry/$MVND_VERSION for logs that may tell more.
FWIW I live with this issue now for many versions of Maven - currently v3.9.11
Although ./mvn/maven.config
-Daether.syncContext.named.factory=file-lock
-Daether.syncContext.named.nameMapper=file-gav
-Daether.syncContext.named.retry=3
-Daether.syncContext.named.time=60
the resolver doesn't respect the retry config and fails after the first attempt:
[2025-09-20T03:47:00.595Z] [ERROR] Could not acquire lock(s)
[2025-09-20T03:47:00.595Z] java.lang.IllegalStateException: Could not acquire lock(s)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.synccontext.named.NamedLockFactoryAdapter$AdaptedLockSyncContext.acquire (NamedLockFactoryAdapter.java:219)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:276)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:261)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:353)
[2025-09-20T03:47:00.595Z] at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:184)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:227)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:136)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:355)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:314)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
[2025-09-20T03:47:00.595Z] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.TurboBuilder.lambda$createBuildCallable$1 (TurboBuilder.java:218)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.CurrentProjectExecution.doWithCurrentProject (CurrentProjectExecution.java:36)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.TurboBuilder.lambda$createBuildCallable$2 (TurboBuilder.java:217)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.SignalingExecutorCompletionService.lambda$submit$1 (SignalingExecutorCompletionService.java:62)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.OrderedCallable.call (OrderedCallable.java:25)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.FutureTask.run (FutureTask.java:317)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
[2025-09-20T03:47:00.595Z] at java.lang.Thread.run (Thread.java:1570)
[2025-09-20T03:47:00.595Z] Suppressed: java.lang.IllegalStateException: Attempt 1: Could not acquire write lock for 'C:\ws\TEP-2.4.1.1\.m2\repository\.locks\artifact~ch.qos.logback~logback-access~1.4.14.lock' in 60 SECONDS
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.synccontext.named.NamedLockFactoryAdapter$AdaptedLockSyncContext.acquire (NamedLockFactoryAdapter.java:202)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:276)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:261)
[2025-09-20T03:47:00.595Z] at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:353)
[2025-09-20T03:47:00.595Z] at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:184)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:227)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:136)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:355)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:314)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
[2025-09-20T03:47:00.595Z] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
[2025-09-20T03:47:00.595Z] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.TurboBuilder.lambda$createBuildCallable$1 (TurboBuilder.java:218)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.CurrentProjectExecution.doWithCurrentProject (CurrentProjectExecution.java:36)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.TurboBuilder.lambda$createBuildCallable$2 (TurboBuilder.java:217)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.SignalingExecutorCompletionService.lambda$submit$1 (SignalingExecutorCompletionService.java:62)
[2025-09-20T03:47:00.595Z] at com.github.seregamorph.maven.turbo.OrderedCallable.call (OrderedCallable.java:25)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.FutureTask.run (FutureTask.java:317)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
[2025-09-20T03:47:00.595Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
[2025-09-20T03:47:00.595Z] at java.lang.Thread.run (Thread.java:1570)
[2025-09-20T03:47:00.595Z] [ERROR]
[2025-09-20T03:47:00.595Z] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[2025-09-20T03:47:00.708Z] [WS-CLEANUP] Deleting project workspace...
[2025-09-20T03:47:00.708Z] [WS-CLEANUP] Skipped based on build state FAILURE
@delanym could you try to run this in TRACE? As there are many diag messages present in there... https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/synccontext/named/NamedLockFactoryAdapter.java#L198
Also, see diagnostics here https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
I'll see if I can squeeze this in somewhere. I'll probably start with - on a windows machine:
- create a lock file for an artifact in my build
- hold it open with a text editor
- try run a build
- ensure 3 attempts are made