Julien Launay

Plot scaling laws of our baseline models

2

For our three baselines on different datasets (OSCAR, C4, The Pile), we would like to plot scaling laws and retrieve their coefficients. Specifically, we are looking to reproduce Figure 1...

Implement the ML Flow experiment tracker

9

**Motivation**. As @sashavor suggested, the carbon footprint working group needs an experiment tracker to properly follow all runs being done. An experiment tracker could also be more broadly interesting to...

Setup zero-shot evaluation with EAI harness

1

## Description For zero-shot evaluation, we would ideally want to use the [EAI evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness). Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or...

## Description To optimize zero-shot performance, we are taking our MLM models through LM adaptation (see #5). For now, we are considering doing this for ~10% of the pre-training steps...