aws-sdk-java-v2 icon indicating copy to clipboard operation
aws-sdk-java-v2 copied to clipboard
verified publisher icon aws

ConcurrentModificationException in MetaTableSchemaCache

Open musketyr opened this issue 11 months ago • 4 comments

Describe the bug

When MetaTableSchemaCache is accessed concurrently, it leads to MetaTableSchemaCache ConcurrentModificationException .

java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1229)
	at software.amazon.awssdk.enhanced.dynamodb.internal.mapper.MetaTableSchemaCache.getOrCreate(MetaTableSchemaCache.java:33)
	at com.agorapulse.micronaut.amazon.awssdk.dynamodb.schema.BeanIntrospectionTableSchema.create(BeanIntrospectionTableSchema.java:167)
	at com.agorapulse.micronaut.amazon.awssdk.dynamodb.DefaultTableSchemaCreator.create(DefaultTableSchemaCreator.java:42)
	at com.agorapulse.micronaut.amazon.awssdk.dynamodb.DefaultDynamoDBServiceProvider.lambda$findOrCreate$0(DefaultDynamoDBServiceProvider.java:72)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at com.agorapulse.micronaut.amazon.awssdk.dynamodb.DefaultDynamoDBServiceProvider.findOrCreate(DefaultDynamoDBServiceProvider.java:71)
	at com.agorapulse.micronaut.amazon.awssdk.dynamodb.DynamoDBServiceProvider.findOrCreate(DynamoDBServiceProvider.java:32)
	at agorapulse.report.worker.job.ttl.ResaveJobTest.testResaveInstagramPostInsights(ResaveJobTest.java:55)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:767)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at io.micronaut.test.extensions.junit5.MicronautJunit5Extension$2.proceed(MicronautJunit5Extension.java:142)
	at io.micronaut.test.extensions.AbstractMicronautExtension.interceptEach(AbstractMicronautExtension.java:162)
	at io.micronaut.test.extensions.AbstractMicronautExtension$3.proceed(AbstractMicronautExtension.java:174)
	at io.micronaut.test.context.TestMethodInterceptor.interceptTest(TestMethodInterceptor.java:46)
	at io.micronaut.transaction.test.DefaultTestTransactionExecutionListener.lambda$interceptTest$0(DefaultTestTransactionExecutionListener.java:93)
	at io.micronaut.transaction.support.AbstractPropagatedStatusTransactionOperations.lambda$execute$2(AbstractPropagatedStatusTransactionOperations.java:68)
	at io.micronaut.transaction.TransactionCallback.apply(TransactionCallback.java:37)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeTransactional(AbstractTransactionOperations.java:333)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeWithNewTransaction(AbstractTransactionOperations.java:318)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeNew(AbstractTransactionOperations.java:235)
	at io.micronaut.transaction.support.AbstractTransactionOperations.doExecute(AbstractTransactionOperations.java:137)
	at io.micronaut.transaction.support.AbstractTransactionOperations.lambda$doExecute$0(AbstractTransactionOperations.java:122)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.executeWithNewConnection(AbstractConnectionOperations.java:143)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.execute(AbstractConnectionOperations.java:90)
	at io.micronaut.transaction.support.AbstractTransactionOperations.doExecute(AbstractTransactionOperations.java:120)
	at io.micronaut.transaction.support.AbstractPropagatedStatusTransactionOperations.execute(AbstractPropagatedStatusTransactionOperations.java:65)
	at io.micronaut.transaction.test.DefaultTestTransactionExecutionListener.interceptTest(DefaultTestTransactionExecutionListener.java:91)
	at io.micronaut.test.extensions.AbstractMicronautExtension.interceptEach(AbstractMicronautExtension.java:166)
	at io.micronaut.test.extensions.AbstractMicronautExtension$3.proceed(AbstractMicronautExtension.java:174)
	at io.micronaut.test.context.TestMethodInterceptor.interceptTest(TestMethodInterceptor.java:46)
	at io.micronaut.transaction.test.DefaultTestTransactionExecutionListener.lambda$interceptTest$0(DefaultTestTransactionExecutionListener.java:93)
	at io.micronaut.transaction.support.AbstractPropagatedStatusTransactionOperations.lambda$execute$2(AbstractPropagatedStatusTransactionOperations.java:68)
	at io.micronaut.transaction.TransactionCallback.apply(TransactionCallback.java:37)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeTransactional(AbstractTransactionOperations.java:333)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeWithNewTransaction(AbstractTransactionOperations.java:318)
	at io.micronaut.transaction.support.AbstractTransactionOperations.executeNew(AbstractTransactionOperations.java:235)
	at io.micronaut.transaction.support.AbstractTransactionOperations.doExecute(AbstractTransactionOperations.java:137)
	at io.micronaut.transaction.support.AbstractTransactionOperations.lambda$doExecute$0(AbstractTransactionOperations.java:122)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.executeWithNewConnection(AbstractConnectionOperations.java:143)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.execute(AbstractConnectionOperations.java:90)
	at io.micronaut.transaction.support.AbstractTransactionOperations.doExecute(AbstractTransactionOperations.java:120)
	at io.micronaut.transaction.support.AbstractPropagatedStatusTransactionOperations.execute(AbstractPropagatedStatusTransactionOperations.java:65)
	at io.micronaut.transaction.test.DefaultTestTransactionExecutionListener.interceptTest(DefaultTestTransactionExecutionListener.java:91)
	at io.micronaut.test.extensions.AbstractMicronautExtension.interceptEach(AbstractMicronautExtension.java:166)
	at io.micronaut.test.extensions.AbstractMicronautExtension.interceptTest(AbstractMicronautExtension.java:119)
	at io.micronaut.test.extensions.junit5.MicronautJunit5Extension.interceptTestMethod(MicronautJunit5Extension.java:129)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$8(TestMethodTestDescriptor.java:217)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:156)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:146)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:144)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:100)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:160)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:146)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:144)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:100)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:160)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:146)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:144)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:100)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:124)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:94)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:63)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
	at jdk.proxy2/jdk.proxy2.$Proxy6.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:200)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:121)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

Regression Issue

  • [ ] Select this option if this issue appears to be a regression.

Expected Behavior

MetaTableSchemaCache is thread safe.

Current Behavior

MetaTableSchemaCache is not thread safe and may cause ConcurrentModificationException.

Reproduction Steps

The issue is random and difficult to reproduce but the solution is obvious (see below).

Possible Solution

Replace

private final Map<Class<?>, MetaTableSchema<?>> cacheMap = new HashMap<>();

with

private final Map<Class<?>, MetaTableSchema<?>> cacheMap = new ConcurrentHashMap<>();

Additional Information/Context

No response

AWS Java SDK version used

2.29.7

JDK version used

Azul Zulu OpenJDK 64-Bit Server VM 21.0.6+7-LTS (mixed mode, emulated-client, sharing)

Operating System and version

macOS 15.0.1 (aarch64)

musketyr avatar Mar 13 '25 15:03 musketyr

@musketyr I understand it's hard to reproduce, but can you share a code example showing how you're using the Enhanced Client? We want to have a better picture of the use case.

debora-ito avatar May 05 '25 17:05 debora-ito

@debora-ito you can get better picture by checking https://github.com/agorapulse/micronaut-aws-sdk repository.

this particular issue happens when the test is trying to parallelize loading of the CSV files into DynamoDB.

https://github.com/agorapulse/micronaut-aws-sdk/blob/master/subprojects/micronaut-amazon-awssdk-dynamodb-loader/src/main/java/com/agorapulse/amazon/awssdk/dynamodb/loader/CsvDynamoDbLoader.java#L67-L73

the test within the repository is rather simple

https://github.com/agorapulse/micronaut-aws-sdk/blob/master/subprojects/micronaut-amazon-awssdk-dynamodb-loader/src/test/java/com/agorapulse/amazon/awssdk/dynamodb/loader/DynamoDbLoaderTest.java#L47

but in our codebase we are loading multiple CSV files in parallel at once to minimize the integration test start up time. This triggers transitively parallel lookup for the database service for given entity type and parallel calls to MetaTableSchemaCache for the same entity type.

musketyr avatar May 07 '25 11:05 musketyr

@musketyr can i work on this issue?

Tjs002 avatar May 24 '25 09:05 Tjs002

It's not really up to me to decide 😀 but I hope that PRs are always welcomed.

musketyr avatar May 25 '25 03:05 musketyr