Mark Walker

Thank you for the quick turnaround @daviesrob. We have been using `set -o pipefail` in all our scripts. We will give your branch a try and get back to you....

Hello @daviesrob, thank you for adding the verbosity feature. I've tested your branch out and it does seem to be catching errors now, but we will need a bit more...

The bed contained 3504 regions: [region.merged.bed.txt](https://github.com/samtools/htslib/files/4314278/region.merged.bed.txt) I can't see what else besides `tabix` could have emitted the message because `gs://talkowski-sv-gnomad-output/1KGP/matrix/1KGP_2504.PCRMINUS.batch_0_0.SR.txt.gz` does not appear anywhere else in the script: ``` set...

The errors are non-deterministic, and we usually see records dropped in contiguous chunks. This seems consistent with some kind of packet loss (but not sure if that's possible here). If...

Thank you! This is what I'm seeing in the logs now: ``` ... * Closing connection 678 * Connection #679 to host talkowski-sv-gnomad-output.storage-download.googleapis.com left intact [E::bgzf_read_block] Invalid BGZF header at...

Here is the full log: [CountSRUnder5kb-5.censored.log](https://github.com/samtools/htslib/files/4339057/CountSRUnder5kb-5.censored.log) I noticed that the content range is much smaller than the previous chunks: ``` < Content-Range: bytes 1626705469-10964074170/10964074171 < Content-Length: 19 ``` And there...

Hexdump: ``` $ curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" -s -r 1626705469- 'https://talkowski-sv-gnomad-output.storage-download.googleapis.com/1KGP/matrix/1KGP_2504.PCRMINUS.batch_0_0.SR.txt.gz' | hexdump -C | head -n 2 00000000 1f 8b 08 04 00 00 00...

Hi @daviesrob, just checking in to see if you have progressed on implementing the block caching feature. I do agree that would probably help. Another idea is to add automatic...

Thank you for the update, @daviesrob. This update may be reducing the number of failures (2 this run compared to 6 last time). Neither threw the HTTP-404 this time, but...

That seems to have cleared up the libcurl error, and I am still seeing the `Invalid BGZF header` error. [CountPEUnder5kb-27.censored.log](https://github.com/samtools/htslib/files/4423614/CountPEUnder5kb-27.censored.log) It seems to me we may have gotten as far...