tchittesh

Here's a minimal reproduction of the deadlock (and the fix for it). Some notes: - It's important to instantiate `CustomCallback` in `LightningModule.configure_callbacks()` instead of directly passing it to the trainer....

I don't understand. Isn't this just hiding the bug in `TQDMProgressBar`? If someone later comes in and disables the `ModelCheckpoint` callback, won't they hit the same deadlock? Could you provide...

Agreed that we should add a test here. However, I strongly believe that reordering callbacks is not a good fix. Using some pseudo-code for simplicity, there's a ``` if global_rank...

Hi @SkafteNicki ! I have a local branch that adds targeted tests and fixes this issue. How can I push my branch to this repo and open a PR? ```...

awesome! I opened a PR above^. I'm not sure if the CI failures are related to my changes. AFAICT, those tests have `enable_progress_bar=False`

recently rebased, and the tests pass! @SkafteNicki if you could take a look when you get the chance, that would be great, thank you! https://github.com/Lightning-AI/pytorch-lightning/pull/21322