Tom

Results 206 comments of Tom

cache_dir behavior changed from 0.2.88

Ah, what changed is that conversion of urls to cached names is handled by a function. The default function strips off all directory names, but you can override that, for...

Incorrect Documentation for default_collation_fn

Yes, please submit a PR.

ResourceWarning

Closing this issue as it has been addressed in commit a7674a9 (2025-05-06): - Added proper exception handling to __del__ method in Pipe class - Prevented potential resource leaks during garbage...

webdataset.github.io cache / streaming info incorrect

Thanks, I'll update that.

contract of expand_urls changed

Well, development and avoidance of reversions/incompatibilities are driven by test cases, and they are incomplete. It's best to submit a test case.

compatibility with grain or jax-dataloader

I'm not exactly sure what that would be; there is nothing in Webdataset that cares at all about what the training library is. Are you using the PyTorch Dataloader with...