Yinggan Xu

Hi! Have you work it out? It seems to be the problem of zfilter.

Same Issue on Win11 Use require("electron").clipboard instead of require("clipboard") Object. - [C:\Users\47088\AppData\Local\atom\app-1.60.0\resources\app.asar\exports\clipboard.js:4:6](file:///C:/Users/47088/AppData/Local/atom/app-1.60.0/resources/app.asar/exports/clipboard.js:4:6) Object. - [C:\Users\47088\AppData\Local\atom\app-1.60.0\resources\app.asar\exports\clipboard.js:11:3](file:///C:/Users/47088/AppData/Local/atom/app-1.60.0/resources/app.asar/exports/clipboard.js:11:3) Module._compile - [:11:149913](https://github.com/lukemurray/data-atom/issues/%3Cembedded%3E:11:149913) Object..js - [:11:153507](https://github.com/lukemurray/data-atom/issues/%3Cembedded%3E:11:153507) Module.load - [internal/modules/cjs/loader.js:815:32](https://github.com/lukemurray/data-atom/issues/internal/modules/cjs/loader.js:815:32) Module._load - [internal/modules/cjs/loader.js:727:14](https://github.com/lukemurray/data-atom/issues/internal/modules/cjs/loader.js:727:14)

Issue, at least for me, is caused by Markdown Picture Paste plugin. After it being disabled, the issue fixed

Hi @kellyguo11! Is there any recommendations on whether to use Preview 4 or this repo?

> Hey @dibbla ! > > I just tried your script and it didn't trigger any downloading process, the only process was `loading checkpoint shards`. Can you make sure that...

> Hmm, yes, I have the same files under the directory. It is weird, can you verify that the loading goes through this path and ensures that the passed `model_id`...

> Hey, I'm not sure if this is something we plan to support at the moment. > > Only `pretraining_datasets: ` support processing on the fly. > > For regular...

> Thanks, let us know how it goes Hi @NanoCode012 I kinda found a better way to do this: I can utilize the pre-tokenized datasets (I only have minor loss...

> uhm, we don't really differentiate tokenized datasets in our sampler/collator. I suppose you need to first: > > 1. Adjust the code to not pre-tokenize , use `stream=True` ....