Manel ALOUI comments

Repositories
Issues
Comments

Results 4 comments of


                                            Manel ALOUI

Request for text deduplication feature

Hi see this new release from hf [datatrove](https://github.com/huggingface/datatrove) DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt...

Can datasets remove duplicated rows?

Hi @StephennFernandes I'm also trying to pretrain an llm, and need to do deduplication for my dataset, which method you applied please?

Can datasets remove duplicated rows?

Thanks a lot Sure I will check it @StephennFernandes

AttributeError: module 'openllm_client' has no attribute 'GrpcClient'

Hi, where can I find in the documentation a guide to integrate grpc with openllm?