Clarify that follower reads are not currently a guarantee of low latency and discuss some more factors that can affect them

[jseldess]

Jesse Seldess (jseldess) commented:

In this PR on topology patterns, @bdarnell called into questions some explicit statements we're making about follower reads, namely that follower reads are available for any read operation at least 48 seconds in the past, and that the experimental_follower_read_timestamp() function is the best way to get a follower read:

Note that this function does not guarantee that a follower read will be performed. It picks a timestamp that should usually result in a follower read, but may perform a remote leaseholder read if things are falling behind.

I'm wary of documenting this function while it's experimental; I think it might be better to have the user hard-code the amount of staleness they can tolerate in the query.

It does pick a timestamp a minute in the past, but that doesn't guarantee a follower read. If things are falling behind, the threshold for performing follower reads increases and this function is not currently aware of that (hard coding the interval would have the same problem). We do not currently provide any tools that would offer a stronger guarantee of being able to perform a follower read.

Jira Issue: DOC-288

[bdarnell]

Note that even though experimental_follower_read_timestamp doesn't guarantee that a follower read will be performed, it's not any weaker than other things we're relying on. For example, in the aftermath of failures, leaseholder preferences may not be honored and you may have to talk to a remote replica. I don't think we need to voice more skepticism about this function than we do for other performance-related things.

The reason to be cautious about this function is not that it doesn't guarantee a follower read, but that it may take a different form by the time it is not experimental.

[jseldess]

@awoods187, @rmloveland, do you have any updated insights here?

[awoods187]

Good news: we understand what needs to be done to lower the time here Bad news: it won't happen by default in 19.2 but we should explain how to lower the default because we expect it to be safe to do so in 19.2 Better news: we will lower the default in 20.1 as soon as the branch is cut

cc @ajwerner in case i missed anything

[rmloveland]

See also: #6001

[rmloveland]

A while back, we made some updates to the Follower Reads doc where we said things like:

Any SELECT statement with an AS OF SYSTEM TIME value at least 4.8 seconds in the past can be a follower read (i.e., served by any replica).

and also added a section called Verify that follower reads are happening

Both of these things clarify (I hope?) that you are not guaranteed to get a follower read at the timestamp output by the function, but that you could get one - and you should check to make sure you are.

Therefore, I think this issue can be closed.

Ben, do you agree with my assessment above? Or do you still think more clarification is needed?

[bdarnell]

I think we should do more to say that follower reads are not currently a guarantee of low latency and discuss some more factors that can affect them. For example, running at high priority is not enough if the resolved timestamps fall behind due to a large write transaction (or schema change).

[rmloveland]

That makes sense - thanks Ben! Will keep this one on the TODO list

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB docs!