Practicinginhell

Invalid argument when running cc_net

2

Hi everyone, I try to run the cc net using this command `python -m cc_net --dump 2023-06 --task_parallelism 20 --num_shards 5000 -l en --mine_num_processes 20 --hash_in_mem 1`. But the invalid...

Exact deduplication

3

First of all, thank you for providing such an excellent repository. I would like to inquire if the repository supports exact deduplication. Thank you in advance.