Timeouts are not working with speculative execution

[sirainen]

We have a test for Dovecot's Cassandra behavior which does:

• Connects to Cassandra and does a lookup
• Drops all network traffic to Cassandra: iptables -I OUTPUT -p tcp --destination-port $CASSANDRAPORT -d $CASSANDRAHOST -j DROP
• Does another lookup

We have timeouts configured, e.g.:

cass_cluster_set_connect_timeout(db->cluster, 1000);
cass_cluster_set_request_timeout(db->cluster, 1000);

This works as expected. But when adding:

cass_cluster_set_constant_speculative_execution_policy(db->cluster, 500, 1);

The driver seems to hang when doing a lookup. The lookup won't finish until the whole cluster is destroyed (at least not for the first 130 seconds). This happens with the latest Scylladb cpp-driver master (and Datastax cpp-driver too). Attached trace logs for the test.

trace.log