apify
chore: Use `shared` mode `ApifyStorageClient` for `test_concurrent_processing_simulation`
Use ApifyStorageClient(request_queue_access='shared') for test_concurrent_processing_simulation. This is the expected usage for concurrent access. And fix the error in the test
Issues
Closes: #529
Could you stress test it a little locally since the CI is now not running integration tests on fork PRs?
Out of about 70 runs, this test failed once due to an actor timeout (execution time exceeding 10 minutes).
But there really is a problem. Normally, it takes less than 10 seconds to execute, but in some runs, it takes up to 5 minutes.
UPD: I will try to find the reasons for this.
Could you stress test it a little locally since the CI is now not running integration tests on fork PRs?
Out of about 70 runs, this test failed once due to an actor timeout (execution time exceeding 10 minutes).
But there really is a problem. Normally, it takes less than 10 seconds to execute, but in some runs, it takes up to 5 minutes.
UPD: I will try to find the reasons for this.
Have you been able to figure this out?
Have you been able to figure this out?
The # Randomly reclaim some requests branch caused the worker to execute reclaim_request and immediately fetch_next_request, looping in this state. This was quite random, as it only reproduced if the worker took on 8 requests (a total of 20 requests for 3 workers).
The worker would only exit this state if, at some point in time, another worker managed to take on this request.