litdata

litdata copied to clipboard

Before submitting - [ ] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements) - [ ] Did you read the [contributor guideline](https://github.com/Lightning-AI/lit-data/blob/main/.github/CONTRIBUTING.md), Pull Request...

tchaton

Training slowed down as time progress with litdata streaming dataset

3

## 🐛 Bug Training slowed down as time progress. ### To Reproduce Unfortunately, I don't know what a good way to reproduce this is. This happens to certain datasets. The...

ouj

Time per sample grows as processed samples grows

13

## 🐛 Bug Time per sample grows as processed samples grows ### To Reproduce Steps to reproduce the behavior: Follow the example provided using optimize. Increase the number of samples....

scritter

StreamingDataset incompatibility with PyTorch Lightning

7

## 🐛 Bug It is a known issue with PyTorch's IterableDataset that issues can occur when the dataset defines `len()`. PyTorch Lightning even raises a warning to make the user...

enrico-stauss

Make `optimize` continue from last checkpoint after crash

2

When running `optimize`, my process somehow crashed after 4h (was estimated to take 10h). Now I have to restart it from scratch. Could you add a checkpointing feature such that...

cgebbe

DataChunkRecipe is not working when used in litgpt's TinyLlama pretraining example

5

DataChunkRecipe is not working when used in litgpt's TinyLlama pretraining example error: AttributeError: 'SlimPajamaDataRecipe' object has no attribute 'is_generator' the type of SlimPajamaDataRecipe is DataChunkRecipe, and i find DataChunkRecipe object...

wen020

## 🚀 Feature `StreamingDataset` has enabled fast data reading, which is amazing when we have a large dataset. However, currently, it does not support reading just a fraction of data,...

yhl48

Adding breakpoint in `random_images` function crashes pdb

4

## 🐛 Bug I'd like to debug the `random_images` function taken from the first example. However, when adding a `breakpoint()` line in that function, python crashes. ### Environment - PyTorch...

cgebbe

Stream selected channels

1

## 🚀 Feature Streaming subsets of channels ### Motivation My geotiff data is typically multispectral and I do experiments using subsets of the channels. I would like to stream only...

robmarkcole

Slow Dataset Preprocessing due to CPU affinity (?) issues

6

## 🐛 Bug I'm attempting to train a model using litgpt, and the openwebtext dataset. I launch the run as normal following their examples, and the dataset preprocessing starts: However,...

mgolub2