Dongxu

Currently use default parameters of AdamW: https://github.com/salesforce/LAVIS/blob/main/lavis/runners/runner_base.py#L103. This has to be made more flexible. Also, it would be good to have a standalone optimizer class, and related registry etc.

Currently, when using iteration-based runner, checkpointing always happens after each inner epoch, thus coupling with the number of iterations per inner epoch. Ideally, the frequency to checkpoint/logging/ lr scheduling should...

For partial model training, e.g. BLIP2, it is not necessary to checkpoint the entire model. Saving partial weights is sufficient. This is currently not supported by the runner.

ViT (if not finetuned) and LLM weights are not stored in the current BLIP2 checkpoints. The default load_checkpoint() in BaseModel raises warning on missing keys, which causes confusion. Proposed solution...