architecture-objective
architecture-objective copied to clipboard
Setup zero-shot evaluation with EAI harness
Description
For zero-shot evaluation, we would ideally want to use the EAI evaluation harness. Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or (2) connect the EAI harness to T5X.
We are currently thinking of going with (1), as it seems like an easier choice.
Action items
- [ ] Write a converter from T5X checkpoints to HF Transformers;
- [ ] Reproduce evaluation results obtained with SeqIO (see #3).
@DanielHesslow will take this up.