Matej Sirovatka

Parallelism config

1

Currently: useless Eventually: becomes a centralized place for configuring smaller types of parallelisms (tp, cp, dp, ep, pp)

WIP: Compose TP + FSDP2

8

Reopening this: managed working TP + FSDP2 @ [2a13375](https://github.com/huggingface/accelerate/pull/3498/commits/2a13375c577c309fa1ca0f4f37bc2e76033e5261). Run `accelerate launch examples/fsdp2/fsdp2_tp.py --apply-tp --apply-fsdp` on 8 GPUs (runs TP2 FSDP4) and can be compared with `CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch examples/fsdp2/fsdp2_tp.py...

[FSDP2] Context parallel

1

Building on top of #3585, this PR enables context parallelism together with FSDP2 (branch stems from that PR, that's why so many changes). ~~ Possibly, this will be moved to...

Fix: bypass validation

1

WIP

[WIP] Async checkpointing

11

Very much WIP, overrides bunch of stuff I'm not sure that is stable to do. TODO: discuss if we want to do a bit different approach (and more easily maintainable)