Fast-LLM

Fast-LLM copied to clipboard

[Prototype] LoRA

14

# ✨ Description Fix: #149 LoRA support * [x] Basic LoRA wrapper * [x] Basic LoRA Config * [x] Add LoRA support in attention * [x] Add LoRA support in...

jlamypoirier

[Prototype] Option to configure layers independently

2

# ✨ Description Fixes: #154, #155. This PR proposes a simple way to obtain layer-dependent configuration by leveraging Fast-LLM's existing config update mechanism. It works by providing a "default" layer...

jlamypoirier

[bug] 16 unit tests fail on main with custom install

6

# 🐞 Describe the Bug The following tests fail on main branch ``` FAILED tests/data/test_sampling.py::test_gpt_sample[full-0] - AssertionError: 3 != 6 FAILED tests/data/test_sampling.py::test_gpt_sample[full-32] - AssertionError: 1 != 6 FAILED tests/data/test_sampling.py::test_gpt_sample[full-88] -...

bigximik

# ✨ Description Migrated from #248; this PR allows a dataset with prompt and completion specifically and in general any pair of text columns (eg: question and answer) to be...

nitsanluke

# ✨ Description This PR creates a common interface for all `GPTHuggingfaceDatasetConfig` input columns via the new `source_schema` variable. Beyond the variable `filed` we require additional keys to preprocess and...

nitsanluke

# 🎯 **Goal (What & Why)** Support chat template during dataset preparation to make it easier for SFT, DPO and other instruction finetuning methods. This takes away from the user...

sohamparikh

[feat] FP8 training

4

# 🧐 Problem Description FP8 training can significantly improve training throughput by reducing memory requirements and improving computational efficiency. However, challenges remain in integrating FP8 across all components of the...

tscholak

# 🎯 **Goal (What & Why)** Add support for training [Nemotron-H models](https://research.nvidia.com/labs/adlr/nemotronh/). Nemotron-H is a family of hybrid SSM-Transformer models (8B, 47B, 56B) trained by NVIDIA in FP8 on 20T...

tscholak

[feat] Generalize dynamic config classes

5

# 🧐 Problem Description #104 introduced a mechanism tor selecting config classes dynamically. This can be made useful elsewhere, especially for user-made plugins. # 💡 Proposed Solution * Generalize the...

jlamypoirier

Simplify config validation

2

# 🎯 **Goal (What & Why)** See discussion in #211. The config validation scheme currently makes little distinction between validation, mutation, and derivation, which can make things difficult to follow....

jlamypoirier