microsoft
HTTP Listener Becomes Unresponsive Under Load and Unable to Recover
Issue:
Under high load conditions, the HTTP listener in cpprestsdk becomes unresponsive. It stops accepting new GET/POST requests, only completing the in-flight ones. The listener cannot be restarted; it requires a full application restart to recover.
Steps to Reproduce:
- Set up an HTTP listener using cpprestsdk.
- Simulate high concurrency by sending a large number of GET/POST requests.
- After reaching a certain load threshold, observe that the listener stops accepting new requests.
- Only in-flight requests are completed, and no new requests are processed.
- Attempting to restart the listener does not resolve the issue; a full application restart is required.
Expected Behavior:
- The HTTP listener should continue accepting and processing requests even under high concurrency.
- The listener should handle high-load scenarios without requiring a full restart of the application or any other recovery method.
Analysis:
-
Thread Stuck: All threads are observed stuck on
.wait(), indicating possible thread exhaustion or a deadlock condition. - Resource Limits: It is likely that the system is hitting resource limits, such as threads or sockets, under high concurrency.
-
Non-cpprest Features Unaffected: Other features (e.g.,
healthStatus) remain unaffected and continue to function as expected under load. - Lack of Async Handling or Throttling: cpprestsdk does not appear to have built-in asynchronous handling or throttling mechanisms to manage load effectively under high concurrency.
Environment:
- cpprestsdk version: 2.10.18-1build2
- Operating System: Ubuntu 22.04.5 LTS
- Compiler: g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
- C++ Standard: C++14
- Network Configuration: Default
Additional Context:
This issue has been observed in a production environment where the application serves high-traffic endpoints. The HTTP listener, built using cpprestsdk, handles a mix of lightweight GET requests and heavier POST requests. During periods of high load (e.g., when many simultaneous requests are being made), the listener becomes unresponsive after a certain threshold of concurrent connections is reached.
Request for Guidance:
If anyone has encountered a similar issue or knows of a potential recovery strategy or workaround, any insights would be appreciated. Is there a way to recover from this deadlock or improve concurrency handling in cpprestsdk without needing a full restart of the application?
