Denis Kochetkov

# ✨ Description Creates **Evaluator** abstraction so additional evaluators beyond **Loss** can be added. Adds an `evaluate` command that accepts the same training config and enables evaluation on the last...

# 🐞 Describe the Bug The generated tokens from Fast-LLM occasionally differ completely from the Hugging Face (HF) counterpart. HF consistently generates the same output, so the issue likely lies...

Sandbox for Implementation of generate and integration of lm_eval (evaluation harness)

7

# ✨ Description This PR draft will be split in 3 PRs ## 🔍 Type of change Select all that apply: - [ ] 🐛 **Bug fix** (non-breaking change that...

Add data cleaning in fast-llm prepare, concept

3

# ✨ Description part of #112 Closes # ## 🔍 Type of change Select all that apply: - [ ] 🐛 **Bug fix** (non-breaking change that addresses a specific issue)...

[bug] 16 unit tests fail on main with custom install

6

# 🐞 Describe the Bug The following tests fail on main branch ``` FAILED tests/data/test_sampling.py::test_gpt_sample[full-0] - AssertionError: 3 != 6 FAILED tests/data/test_sampling.py::test_gpt_sample[full-32] - AssertionError: 1 != 6 FAILED tests/data/test_sampling.py::test_gpt_sample[full-88] -...

Run lm-eval-harness benchmarks during validation

4

# 🎯 **Goal (What & Why)** Enable Fast-LLM to run structured evaluations using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness). This allows benchmarking Fast-LLM models across many standard tasks using the in-memory model during validation, leveraging...

## 🎯 **Goal (What & Why)** Currently, if we restart an experiment (e.g., due to a job being preempted), the iteration count in WandB will be higher than the actual...

Discussion about dataset preparation speed

2

# 🎯 **Description** With 1,250 bin files, dataset preparation takes around 6.5 minutes on 100 cores, 1TB RAM, and 8 GPUs. However, if I simply iterate through these files, read...

Move Config Validations (e.g., Dataset Usage vs. Definitions) to `_validate` for Dry Run Checks

1

# 🎯 **Goal (What & Why)** Most config validations are performed in `_validate`, but not all. Some checks, such as ensuring all used datasets are defined, are currently in `data`...

Online dataset mixing based on validation metrics

1

# 🎯 **Goal (What & Why)** Create a Blended Dataset which can use validation metrics to re-arrange its sampling probabilities of different subsets. This will allow to implement on the...