Backlog of unfinished operations grows larger than configured value

[scottoaks17]

Context

We have configured our test to use this rate-control: rateControl: - type: fixed-feedback-rate opts: tps: 300 unfinished_per_client: 100

Using a local monitor every second, we typically see 10-20 unfinished transactions each reporting interval, which seems good. However, after certain errors (e.g. a network glitch), we see that number rise well past 100 per client. In fact, this can overwhelm the client: we end up with so many unfinished transactions that the ones later in the queue timeout even if they completed successfully.

Expected Behavior

Unfinished should never go above 100 (assuming one client in this config)

Actual Behavior

Unfinished can grow without bounds

Possible Fix

There are probably better ways, but we changed applyRateControl to sleep in 10 ms increments until the actual number of unfinished transactions is less than the target value (looping rather than the current estimator). This means that the txUpdateTime must be a suitably small value (we also used 10ms there).

Steps to Reproduce

1. In a test with this rateController, introduce a network delay to cause unfinished to grow past the desired bounds

Existing issues

None found

Context

Your Environment

Version used: caliper-cli 0.2.0, fabric 1.4.0, grpc 1.14.2 Environment name and version (e.g. Chrome 39, node.js 5.4): node 10.18.1 Operating System and version (desktop or mobile): OEL 7.7

[nklincoln]

Have you tried the same using the fixed_backlog rate controller?