aws
Deadlock when using multiple instances of S3CrtClient
Describe the bug
When using multiple instances of S3CrtClient (one per worker thread) then:
- some of them error out with the
GetObject()outcome error message "Client is not initialized or already terminated" - some of them get stuck completely in the S3CrtClient object destructor, so that the program never terminates.
There seems to be some (presumably unintentional) sharing of state between the different S3CrtClient instances.
Regression Issue
- [ ] Select this option if this issue appears to be a regression.
Expected Behavior
Multiple S3CrtClient instances should be able to co-exist so that they can be used as a drop-in replacement for the classic S3Client.
Current Behavior
Multiple S3CrtClient instances are conflicting with each other, leading to error messages "Client is not initialized or already terminated" (from outcome of GetObject()) for some of the threads and to deadlock in S3CrtClient destructor for other threads.
Reproduction Steps
Here is a minimal code example to reproduce. The application is intentionally using no valid S3 credentials and non-existing buckets, so if all goes as it should then the result of GetObject would report an access denied error and the application terminates.
I'm setting export AWS_EC2_METADATA_DISABLED=true because otherwise all threads seem to get serialized and with the serialization the application terminates after several minutes.
Also, the application terminates successfully with a small number of threads, so I'm using a high number of threads here intentionally.
#include <aws/core/Aws.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/GetObjectRequest.h>
#include <iostream>
#include <vector>
#include <thread>
#include <sstream>
const Aws::String BUCKET_NAME = "your-bucket-name";
const Aws::String OBJECT_KEY = "your-object-key";
const Aws::String REGION = "us-east-1";
void DownloadTask() {
Aws::S3Crt::ClientConfiguration config;
config.region = REGION;
Aws::S3Crt::S3CrtClient s3Client(config);
Aws::S3Crt::Model::GetObjectRequest request;
request.SetBucket(BUCKET_NAME);
request.SetKey(OBJECT_KEY);
auto outcome = s3Client.GetObject(request);
if (outcome.IsSuccess()) {
std::stringstream ss;
ss << outcome.GetResult().GetBody().rdbuf();
std::string data = ss.str();
} else {
std::cerr << "Error: " << outcome.GetError().GetMessage() << std::endl;
}
}
int main() {
Aws::SDKOptions options;
Aws::InitAPI(options);
{
std::vector<std::thread> threads;
threads.reserve(64);
for (int i = 0; i < 64; ++i) {
threads.emplace_back(DownloadTask);
}
for (auto& t : threads) {
if (t.joinable()) {
t.join();
}
}
}
Aws::ShutdownAPI(options);
return 0;
}
The output looks like this:
Error: Client is not initialized or already terminated
Error: Client is not initialized or already terminated
Error: Client is not initialized or already terminated
Error: Client is not initialized or already terminated
Error: Client is not initialized or already terminated
Error: Client is not initialized or already terminated
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
Error: The AWS Access Key Id you provided does not exist in our records.
<DEADLOCK>
In the deadlock situation, the backtrace of the application threads looks like this, where they are stuck on waiting for a semaphore inside the S3CrtClient destructor:
(gdb) thread 88
[Switching to thread 88 (Thread 0x7ffe7affd6c0 (LWP 18360))]
#0 0x00007ffff7898d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffe68260460) at ./nptl/futex-internal.c:57
57 in ./nptl/futex-internal.c
(gdb) bt
#0 0x00007ffff7898d71 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffe68260460) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffe68260460) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffe68260460, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
at ./nptl/futex-internal.c:139
#3 0x00007ffff789b7ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffe68260410, cond=0x7ffe68260438) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x7ffe68260438, mutex=0x7ffe68260410) at ./nptl/pthread_cond_wait.c:627
#5 0x0000555555797cdb in Aws::Utils::Threading::Semaphore::WaitOne() ()
#6 0x0000555555655297 in Aws::S3Crt::S3CrtClient::~S3CrtClient() ()
#7 0x00005555555dd423 in DownloadTask () at test-s3crt-2.cpp:32
#8 0x00005555555e57b1 in std::__invoke_impl<void, void (*)()> (__f=@0x555556173678: 0x5555555dd1e9 <DownloadTask()>) at /usr/include/c++/13/bits/invoke.h:61
#9 0x00005555555e575d in std::__invoke<void (*)()> (__fn=@0x555556173678: 0x5555555dd1e9 <DownloadTask()>) at /usr/include/c++/13/bits/invoke.h:96
#10 0x00005555555e56fe in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x555556173678) at /usr/include/c++/13/bits/std_thread.h:292
#11 0x00005555555e56ce in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x555556173678) at /usr/include/c++/13/bits/std_thread.h:299
#12 0x00005555555e56ae in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x555556173670) at /usr/include/c++/13/bits/std_thread.h:244
#13 0x00007ffff7cecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ffff789caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#15 0x00007ffff7929c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Possible Solution
There seems to be some (presumably unintentional) sharing of state and race conditions between the different S3CrtClient instances, so making the instances independent of each other is probably the solution.
Additional Information/Context
I understand that the design of the S3CrtClient with its separate event loop threads does not really encourage multiple instances of the S3CrtClient.
However, I think it would be good to support this to enable the S3CrtClient to be a drop-in replacement for the classic S3Client, where this problem does not exist.
In my specific case at hand, I'm trying to connect to a subset of individual S3 servers for performance and failure testing. Given that S3Client and S3CrtClient have no way of specifying a list of IP addresses to connect to, using multiple instances of the clients has been the simple solution to this so far with the classic S3Client, which I also wanted to use with the S3CrtClient.
AWS CPP SDK version used
1.11.710
Compiler and Version used
gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Operating System and version
Ubuntu 24.04.3 LTS
I've set up a small cmake project called with your code example. Building and running this in docker, I'm correctly seeing the Access Denied output for all threads. See attached.
If you're able to replicate this in docker I'll be happy to look further into it, but I am not seeing the behavior you're describing.
Command to build and run:
docker build -t IMAGE_NAME . && docker run --rm IMAGE_NAME
Thanks for looking into this, @sbaluja .
The missing piece in your reproducer might be the disabling of the EC2 metadata service discovery. With the EC2 metadata service discovery enabled, it is taking several minutes on my host (which is outside of EC2) until all the "Access denied" errors have appeared on the console and the service discovery seems to serialize all the S3CrtClient initializations, so that the race condition does not reproduce and the startup time gets longer with more threads.
When I disable the EC2 metadata service discovery in the main() function of the .cpp from your zip file then the problem reproduces on my host directly on the first attempt.
I did just set the corresponding environment variable in the code here as the first line in main() before the AWS SDK init:
[...]
int main() {
setenv("AWS_EC2_METADATA_DISABLED", "true", 0);
Aws::SDKOptions options;
options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options);
[...]
Thanks @breuner ! I added the setenv("AWS_EC2_METADATA_DISABLED", "true", 0); line to @sbaluja's sample program as you suggested, but I'm still seeing the "Access Denied" errors instead of the deadlock you're observing.
Could you try running the reproducer in a Docker environment to see if it reproduces there? I've attached my updated Dockerfile and code if that helps.
Thanks for looking into this, @kai-ion . Before sending my previous message I did actually reproduce the problem by using the dockerfile from @sbaluja, just with the added setenv(), so identical to what you created in the [issue3653Updated.zip](https://github.com/user-attachments/files/24219621/issue3653Updated.zip) file.
On my little dev machine with only 8 CPU cores, the problem reproduced directly on the first attempt. Today, I also tried the same on a 64 core host and there it was significantly harder to reproduce the hang. I had to increase the number of threads to 1024 to reproduce the problem. Does the host on which you tried this maybe also have a lot of cores?
However, today I also found out something new. The hint for this came from AI, so it might be total non-sense, but at least a quick check in the code seemed indeed to reveal static variables that are globally shared, but I don't have the full overview:
What the AI suggests is that when no config.clientBootstrap is given, then the multiple S3CrtClient instances share the same global instances behind the default elements of bootstrap. And as soon as the first S3CrtClient runs its ~S3CrtClient destructor, it stops/destroys/invalidates these global instances, so that the remaining S3CrtClient instances (subject to race conditions) have a problem.
Does that sound plausible to you or is it rather AI non-sense?
For the sake of completeness, I'm attaching the updated main.cpp file with increased threads to 1024 (instead of previous 64 threads) and the console output of my successful attempt to reproduce the hang with the increased number of threads on a 64 core host, after printing a lot of Client is not initialized or already terminated messages.