reader icon indicating copy to clipboard operation
reader copied to clipboard
verified publisher icon jina-ai

Pile in reader format

Open thepowerfuldeez opened this issue 1 year ago • 1 comments

Hi! This looks interesting. I wonder if you could convert the Pile dataset taken from respective urls in the jina reader format to experiment with LLM pre-training?

thepowerfuldeez avatar Apr 14 '24 15:04 thepowerfuldeez

Not in the plan. Reader mainly focuses on HTML-type URL.

hanxiao avatar Apr 16 '24 18:04 hanxiao