ci: re-build dockers with each commit to `main`

[Borda]

Before submitting

• [x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Did you make sure to update the docs?
• [x] Did you write any new necessary tests?

What does this PR do?

this would enable some quick updates

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[t-vi]

Uh, OK, what does that mean for the CI load?

[Borda]

Uh, OK, what does that mean for the CI load?

with #36 we are going back to update only two images, but yes each commit to main would be followed by 1h compilation/install time split into two machines

[Borda]

To resolve the actual blocker, we need to:

1. land following:
   • https://github.com/Lightning-AI/pytorch-lightning/pull/19688
   • https://github.com/Lightning-AI/pytorch-lightning/pull/19687
2. make new nightly releases of lightning - Sunday
3. update litGPT -> https://github.com/Lightning-AI/litgpt/pull/1185
4. update pin here -> https://github.com/Lightning-AI/lightning-thunder/pull/66

[t-vi]

a 1-hour job after each commit sounds like it would put quite a strain on the CI.

[Borda]

a 1-hour job after each commit sounds like it would put quite a strain on the CI.

I think it is fine, regarding you will gain a way to rebuild dockers out of 4 4-hour cycle... it is also not 1h job but two jobs 20+40min :)