datacomp

datacomp copied to clipboard

Hi all thank you for the great package. I'm currently running into issues downloading [Commonpool X-Large with Pyspark](https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md). I've downloaded the metadata files and uploaded them to S3, however my...

zanussbaum

We're trying to produce the filtered subset of your `large` pool with the method "Intersection of image-based and CLIP score filtering." I know there is a script provided in the...

88cf6a5ff

Connection error while half-downloading metadata

3

Hello! I'm running the command: `python download_upstream.py --scale medium --data_dir medium --skip_shards` After downloading some files it interrupts with the error: ``` File "/home/oleg/.local/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 94, in thread_map return _executor_map(ThreadPoolExecutor,...

sedol1339

About update metadata with the corresponding image sample in shards

2

Hi all, thanks for your great work!! I want to ask some relevant technical questions: I have used dataset2metadata to generate the metadata set from the 'shards' and store some...

ypwang61

Remove CSAM, if present

5

A recent report definitively found CSAM in LAION-5B, and that dataset has been taken down until the problem can be solved. The DataComp dataset is much larger. Please let us...

ahundt

Pretraining dataset

1

Thank you for your excellent work. I'm currently training my own CLIP model and have a question. If I use LAION-2B, COYO-700M, and Datacomp datasets simultaneously for training, will it...

mactavish91

sagadre

Expose img2dataset distributor

3

* Needed for https://github.com/mlfoundations/datacomp/issues/59 * Closes https://github.com/mlfoundations/datacomp/issues/66 with https://github.com/mlfoundations/datacomp/pull/58#discussion_r1343359420

0x2b3bfa0

Usage with AWS S3 and Ray

5

* Inspired by https://github.com/rom1504/img2dataset/pull/272 * Depends on https://github.com/mlfoundations/datacomp/pull/58 * Depends on https://github.com/mlfoundations/datacomp/pull/60 # Usage ## Cluster creation ```bash ray up --yes cluster.yml ``` ```bash ray dashboard cluster.yml ``` ## Job...

0x2b3bfa0

14% of SHA256 hashes not matching

32

# Introduction We downloaded the [Datacomp 1B set](https://huggingface.co/datasets/mlfoundations/datacomp_1b). For verification, we only kept an image if its SHA256 checksum of the bytes matches with the corresponding entry in the metadata...

pfischer-nvidia