torchtune

torchtune copied to clipboard

Vision/Multimodal

22

With all the growing activity and focus on multimodal models is this library restricted to tune text only LLM? Do we plan to have Vision or more in general multimodal...

bhack

MPS support

21

#### Context - For testing purposes it can be useful to run directly on a local Mac computer. #### Changelog - Checks support for BF16 on MPS device. - Added...

maximegmd

Gemma lora

2

#### Context - Create lora fine-tune for Gemma model. #### Changelog - ... #### Test plan - .... It can work with `apply_lora_to_mlp = True, apply_lora_to_output = False`, but not...

solitude-alive

Organise steps logic

8

The idea is showing: - A progress bar with the actual total count - Having the same steps logged and reported on the progress bar - Count a training step...

tcapelle

Adding support for Llama2 70B LoRA finetuning

2

#### Context As per title #### Changelog - Builder function + config #### Test plan - Trained for one epoch with the following loss   - Training Speed   ...

kartikayk

Add Selective Activation Checkpointing

1

#### Context This PR updates activation checkpointing (ac) to support selective layer and selective op activation checkpointing. It preserves the previous options enabled of full or None. This is controlled...

lessw2020

Update build_docs.yaml

1

Get torchtune version during the build and add so that it appear in the resulting HTML dropdown

svekars

Speed up model loading for generate

5

**This has not been extensively tested (only mistral 7b) and more of a proposal!** This change does the follow: - Create the model on the meta device - Load the...

albanD

The API enforces that the wrapping policy just be a set of modules, which is sufficient for a few use cases but the underlying API offers more generality in terms...

rohan-varma

[RFC] Single Device Full Fine-tune for Llama7B in < 16GB

12

## Context On a single device, our current Llama7B full fine-tune recipe either OOMs with the ```AdamW``` optimizer, or takes > 55GB with ```SGD```. Given the importance of single device...

kartikayk