RequestFutureManager holds request body even after request has been completed

[heliheli]

I had a case, where with a large request body and a large timeout, the application crashed with an OutOfMemoryError. After analyzing the heap dump, I found the reason in RequestFutureManager:

    public TarantoolRequestMetadata submitRequest(TarantoolRequest request, int requestTimeout) {
        CompletableFuture<Value> requestFuture = new CompletableFuture<>();
        TarantoolRequestMetadata requestMetadata = new TarantoolRequestMetadata(request, requestFuture);
        long requestId = requestMetadata.getRequestId();
        requestFuture.whenComplete((r, e) -> requestFutures.remove(requestId));
        requestFutures.put(requestId, requestMetadata);
        timeoutScheduler.schedule(() -> {
            if (!requestFuture.isDone()) {
                requestFuture.completeExceptionally(new TimeoutException(String.format(
                    "Failed to get response for %s within %d ms", requestMetadata, requestTimeout)));
            }
        }, requestTimeout, TimeUnit.MILLISECONDS);
        return requestMetadata;
    }

timeoutScheduler holds requestMetadata for logging even after request has been completed. requestMetadata contains TarantoolRequest with TarantoolRequestBody. In my example, request body was 250 MB and requestTimeout was 600 seconds, and we send request every 5 seconds, so in 100 seconds we crashed with OutOfMemoryError on 26 GB xmx.

I can assume, that the problem should be solved in such a way that TarantoolRequestMetadata does not contain the request body.

[akudiyar]

Hi, hm, this is really unexpected that you have request bodies of such size. Normally the bodies are very small even if they include some records for insertion. I would assume that such a large request has a high chance of failing with a network read timeout on the server.

However, the ticket seems reasonable, what is really needed to be kept in the request manager list in most of the cases is only the signature of the request. But in the case of using retry strategy we will still have to keep the full request body somewhere.