Backport #41588 to 22.3: Fix possible pipeline stuck exception for queries with OFFSET

[robot-ch-test-poll2]

Original pull-request #41588 Cherry-pick pull-request #41626

This pull-request is a last step of an automated backporting. Treat it as a standard pull-request: look at the checks and resolve conflicts. Merge it only if you intend to backport changes to the target branch, otherwise just close it.