ionutmodo

icon location Institute of Science and Technology Austria icon location PhD student at Institute of Science and Technology Austria (ISTA) working in optimization for deep learning

Results 11 comments of ionutmodo

[Feature]: Ability to merge old runs with a new run

is there any update on this? I would also need to merge the runs in two groups, that would be awesome!

Adafactor fails to run on a custom (rfs) resnet12 (with MAML)

are there any updates on this? The issue is still present

Adafactor fails to run on a custom (rfs) resnet12 (with MAML)

I had a look at this error which I also faced when training a ResNet-50 model. I got a similar error as @brando90, except that the dimensions of my tensors...

question for building wheel for transformer-engine

any updates on this? I am facing more or less the same issue. CMAKE fails with an error saying `Could NOT find CUDNN (missing: CUDNN_INCLUDE_DIR CUDNN_LIBRARY)`. However, these variables are...

Allow renaming groups of completed runs

is there any update on this?

Add torch.compile to optimizer.step()

I would like to add a related issue to torch compile: when using `compile=True` and learning rate warmup, the code breaks because the scheduler expects to find the `__func__` attribute,...

[Bug] Exception: cublasLt ran into an error! during fine-tuning LLM in 8bit mode

do you guys have some solution for this?

adding support for LR schedule for full distributed finetune

@felipemello1 I am reviving this thread with an error I encountered. I am trying to do full-finetuning-distributed with cosine learning rate with warmup using the default AdamW optimizer from pytorch...

adding support for LR schedule for full distributed finetune

the error message says `step_fn` doesn't have attribute `__func__`. Below you can find the config: ``` # Config for multi-device full finetuning in full_finetune_distributed.py # using a Llama3.1 8B Instruct...

adding support for LR schedule for full distributed finetune

@felipemello1 I started working on this again and I figured out what the issue is: when using `compile=True`, torchtune compiles the model, the loss, the optimizer step and scale grads...