awslabs
Caught exception while sync'ing Kinesis shards and leases
I have an access to AWS services via new user which my customer created for me. I don't know what permission I have. I created new stream and trying to start consumer. Got next Exception
2019-04-12 07:43:46,350 [multi-lang-daemon-0000] ERROR s.a.kinesis.leases.ShardSyncTask [NONE] - Caught exception while sync'ing Kinesis shards and leases
software.amazon.kinesis.leases.exceptions.DependencyException: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: One or more parameter values were invalid: Missing the key Test in the item (Service: DynamoDb, Status Code: 400, Request ID: R2FPISMK0J3IK5R7PLAK0G48AJVSDF5AEMVJF66Q9ASUAAJG)
at software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher.convertAndRethrowExceptions(DynamoDBLeaseRefresher.java:632)
at software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher.createLeaseIfNotExists(DynamoDBLeaseRefresher.java:337)
at software.amazon.kinesis.leases.HierarchicalShardSyncer.checkAndCreateLeaseForNewShards(HierarchicalShardSyncer.java:101)
at software.amazon.kinesis.leases.ShardSyncTask.call(ShardSyncTask.java:67)
at software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:53)
at software.amazon.kinesis.coordinator.Scheduler.initialize(Scheduler.java:247)
at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:213)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:95)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:86)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Can it be that trouble is in permissions of my account?
OK this is my situation but may not apply to you. I created the stream long time (~4months) ago but there is no data coming from upstream until today. And the KCL lease DDB table just died: I increased the kinesis shard count and did deployment for the EC2 hosts connected to them, but the table could not be updated. This is causing the shards & leases not synced so the RecordProcessor could not be initialized.
I basically had to delete the KCL lease table and re-deploy/restart the server(which would recreate that lease table). Luckily this fixed the issue but may not apply to you as this could cause some records being lost.
Another thing I noticed is that ever since I create the stream, there is no checkpoint activity so the server don't have a damn idea where it should pick up processing the records in the shard. If this is not the case for you, I DON'T RECOMMEND to use my approach
Getting the similar issue. Any idea how to correct this? I also have a KCL 1.x Python worker script pulling from 1 shard. That is working fine but not this one where I am using KCL 2.x, the sample script i.e. amazon_kclpy_helper.py --print_command --java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java --properties sample.properties
2020-04-06 12:51:13,600 [multi-lang-daemon-0000] ERROR s.a.kinesis.leases.ShardSyncTask [NONE] - Caught exception while sync'ing Kinesis shards and leases
java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException
at software.amazon.kinesis.retrieval.AWSExceptionManager.apply(AWSExceptionManager.java:65)
at software.amazon.kinesis.leases.KinesisShardDetector.listShards(KinesisShardDetector.java:172)
at software.amazon.kinesis.leases.KinesisShardDetector.listShards(KinesisShardDetector.java:132)
at software.amazon.kinesis.leases.HierarchicalShardSyncer.getShardList(HierarchicalShardSyncer.java:248)
at software.amazon.kinesis.leases.HierarchicalShardSyncer.checkAndCreateLeaseForNewShards(HierarchicalShardSyncer.java:81)
at software.amazon.kinesis.leases.ShardSyncTask.call(ShardSyncTask.java:67)
at software.amazon.kinesis.metrics.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:53)
at software.amazon.kinesis.coordinator.Scheduler.initialize(Scheduler.java:247)
at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:213)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:95)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:86)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:118)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:104)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:236)
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:228)
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:126)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112)
at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.lambda$acquire0$1(HttpOrHttp2ChannelPool.java:84)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:33)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:435)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
... 1 common frames omitted
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: kinesis.ap-southeast-2.amazonaws.com/52.119.211.107:443
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
... 5 common frames omitted
Nope.....
вт, 10 нояб. 2020 г. в 19:08, Nicholas Ball [email protected]:
is there any update to the above? I am in a similar situation
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/awslabs/amazon-kinesis-client-python/issues/97#issuecomment-724801192, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBWJT4QRBQLQTECZTC2LHDSPFQPFANCNFSM4HFNKULA .