architecture-objective icon indicating copy to clipboard operation
architecture-objective copied to clipboard

Setup zero-shot evaluation with EAI harness

Open slippylolo opened this issue 4 years ago • 1 comments

Description

For zero-shot evaluation, we would ideally want to use the EAI evaluation harness. Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or (2) connect the EAI harness to T5X.

We are currently thinking of going with (1), as it seems like an easier choice.

Action items

  • [ ] Write a converter from T5X checkpoints to HF Transformers;
  • [ ] Reproduce evaluation results obtained with SeqIO (see #3).

slippylolo avatar Nov 04 '21 16:11 slippylolo

@DanielHesslow will take this up.

thomasw21 avatar Nov 16 '21 13:11 thomasw21