ctlog: test recover behavior with errors on overwrite

[jellevandenhooff]

Test recovery behavior when using conditional requests to upload immutable files (using "If-Match" on Tigris).

A sequencing failure after uploading a tile but before updating the lock can cause the log to get stuck:

• Sequencing starts.
• The sequencer writer tile file "A".
• The sequences fails to update the lock.
• Sequencing restarts.
• The sequencer tries to write tile file "A".
• The sequencer fails because tile file "A" already exists.

In the log this looks as follows:

testlog_test.go:79: time=2024-03-16T17:44:35.325-07:00 level=DEBUG source=ctlog.go:751 msg="uploading partial data tile" tree_size=1 tile="{H:8 L:-1 N:0 W:1}" size=25
testlog_test.go:79: time=2024-03-16T17:44:35.325-07:00 level=DEBUG source=ctlog.go:771 msg="uploading tree tile" old_tree_size=0 tree_size=1 tile="{H:8 L:0 N:0 W:1}" size=32
testlog_test.go:79: time=2024-03-16T17:44:35.325-07:00 level=DEBUG source=ctlog.go:795 msg="uploading checkpoint" size=209
testlog_test.go:79: time=2024-03-16T17:44:35.325-07:00 level=ERROR source=ctlog.go:671 msg="pool sequencing failed" old_tree_size=0 entries=1 err="fatal sequencing error\nfailing commit checkpoint for test"
...
testlog_test.go:79: time=2024-03-16T17:44:35.326-07:00 level=DEBUG source=ctlog.go:751 msg="uploading partial data tile" tree_size=1 tile="{H:8 L:-1 N:0 W:1}" size=32
testlog_test.go:79: time=2024-03-16T17:44:35.326-07:00 level=DEBUG source=ctlog.go:771 msg="uploading tree tile" old_tree_size=0 tree_size=1 tile="{H:8 L:0 N:0 W:1}" size=32
testlog_test.go:79: time=2024-03-16T17:44:35.326-07:00 level=ERROR source=ctlog.go:671 msg="pool sequencing failed" old_tree_size=0 entries=1 err="couldn't upload a tile: key \"tile/8/0/000.p/1\" already exists"