workload: handle ambiguous errors

[renatolabs]

Clients are expected to handle ambiguous errors; for instance, see our own documentation:

• cockroachlabs.com/docs/stable/transaction-retry-error-reference
• cockroachlabs.com/docs/stable/common-errors#result-is-ambiguous

Despite that, our very own workload does not handle these errors, leading to occasional roachtest failures that should not have happened.

We want workload to be able to handle these errors whenever possible (retrying when safe to do so). It might not be possible to completely eliminate these errors from bubbling up to the caller every time; each workload should be updated independently.

Jira issue: CRDB-30113

[blathers-crl[bot]]

cc @cockroachdb/test-eng

[srosenberg]

Since this has come up in the context of performance benchmarks, we should be careful not to retry excessively. In some sense, a benchmark becomes tainted since retries could yield to performance degradation.

[andrewbaptist]

At least one type of "incorrect ambiguous" error is described here: https://github.com/cockroachdb/cockroach/issues/129427. The short term fix for that issue is to tolerate errors, but this isn't a great general fix.