Add support for T-Few

[lewtun]

T-Few is a PEFT method for few-shot learning that is currently the SOTA on many NLP benchmarks. It uses a nifty technique called (IA)^3 to update a small number of parameters during training and would be an impactful method to include IMO.

Although research code exists, it is tightly bound to the paper and doesn't run easily on hardware that isn't an 80GB A100. The peft library could help make this work more accessible to industry practitioners (where few-shot is actually valuable)

cc @craffel

Paper: https://arxiv.org/abs/2205.05638 GitHub: https://github.com/r-three/t-few

[sayakpaul]

(IA)^3 is already in the backlog: https://github.com/huggingface/peft#backlog

[savasy]

IA^3 is already implemented here https://github.com/adapter-hub/adapter-transformers

I will give it a try

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[patricebechard]

Any updates on this? Is there a timeline about when this would be supported or has the priorities of the team changed?

[BenjaminBossan]

Quick update, the PR #578 implements this and is quite advanced, will probably land soon.