Tqdm print multi lines with refresh

[name-used]

Bug description

while running validation process, my console will be filled with printed content from tqdm I find that the problem only appears at validation process. other processes are all ok I checked this problem and finally found this in your code:

there is an special variable "has_main_bar" only appears at "init_validation_tqdm" function

other functions dont have this variable

so why? what does this variable mean?

What version are you seeing the problem on?

v2.5

Reproduced in studio

No response

How to reproduce the bug

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):

More info

No response

[bhimrazy]

Hi @name-used! Thanks for reporting. Would you mind adding a screenshot of the console logs and a minimal reproducible script, if possible? That would really help in debugging the issue.

[name-used]

the console log seems like that Im trying to make a pure demo code from the project —— it too heavy to run the full project, it highly dependents on data and configuration

[name-used]

Hi @name-used! Thanks for reporting. Would you mind adding a screenshot of the console logs and a minimal reproducible script, if possible? That would really help in debugging the issue.

This code can replay this tqdm problem:

import torch.nn
import pytorch_lightning as pl


def main():
    ckpt_path = './test_model.ckpt'
    model = DemoModel()
    # torch.save(model, ckpt_path)

    train_dataloader = torch.utils.data.DataLoader(DemoDataset())
    val_dataloader = torch.utils.data.DataLoader(DemoDataset())
    test_dataloader = torch.utils.data.DataLoader(DemoDataset())

    trainer = pl.Trainer(
        max_epochs=1, callbacks=[], logger=None,
        accelerator="gpu", precision='16-mixed', devices=1,
    )
    trainer.fit(model, train_dataloader, val_dataloader)
    trainer.save_checkpoint(ckpt_path, weights_only=True)
    trainer.test(model, test_dataloader, ckpt_path=ckpt_path, verbose=True)


class DemoModel(pl.LightningModule):
    def __init__(self):
        super(DemoModel, self).__init__()

    def training_step(self, batch, batch_idx):
        pass

    def predict_step(self, batch, batch_idx):
        pass

    def evaluation_step(self, batch, batch_idx, prefix):
        pass

    def validation_step(self, batch, batch_idx):
        pass

    def test_step(self, batch, batch_idx):
        pass

    def configure_optimizers(self):
        pass


class DemoDataset(torch.utils.data.Dataset):
    def __init__(self):
        pass

    def __getitem__(self, item):
        return item

    def __len__(self):
        return 1000


if __name__ == '__main__':
    main()

[JeisonPham]

@name-used Are you using PyCharm? Was able to resolve the issue by opening up the configuration, going to modify options -> Python -> Emulate terminal in output console. Though I will say the Validation Dataloader bar disappears after it is done.

[name-used]

@name-used Are you using PyCharm? Was able to resolve the issue by opening up the configuration, going to modify options -> Python -> Emulate terminal in output console. Though I will say the Validation Dataloader bar disappears after it is done.

Thank you! Yes, I use Pycharm. I have tried to solve this problem by modifying the origin code in pytorch_lightning. Now it preforms ok, but this solution is a bit clumsy. It can only solve my own problem. My method are shown as following:

1. In `/python3.9/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py`  
   Around line 360, insert an empty `print()` inside the `if should_check_val:` block:

       if should_check_val:
           print()
           # This needs to be set so the correct `trainer._active_loop` is picked
           self.trainer.validating = True

2. In `/python3.9/lib/python3.9/site-packages/pytorch_lightning/callbacks/progress/tqdm_progress.py`  
   Around line 224, remove the `has_main_bar` condition and update the function to the following:

       def init_validation_tqdm(self) -> Tqdm:
           return Tqdm(
               desc=self.validation_description,
               position=(2 * self.process_position),
               disable=self.is_disabled,
               leave=True,
               dynamic_ncols=True,
               file=sys.stdout,
               bar_format=self.BAR_FORMAT,
           )

This suppresses the excessive validation progress bar flickering. A proper fix still depends on an official update from the PyTorch Lightning team.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

[SkafteNicki]

I looked through the issue and tried debugging. In the end this is not really a lightning problem, more how tqdm works. This happens in general when tqdm does not detect a TTY (interactive terminal). Some good examples of this is VS Code’s "Python Interactive", Jupyter notebooks, or PyCharm’s run window (which is being used in this issue).

The particular reason for having leave=not has_main_bar and not just leave=True which would solve this for some of the above cases, can be tracked back to this discussion when the line was added: https://github.com/Lightning-AI/pytorch-lightning/pull/11657#discussion_r796441209

So changing the main code is not possible. For these cases there is really not a lot we can do. I would suggest either using another terminal to show the progress bar or as an alternative create a custom progress bar with the proposed changes:

class CustomTQDMProgressBar(TQDMProgressBar):
       def init_validation_tqdm(self) -> Tqdm:
           return Tqdm(
               desc=self.validation_description,
               position=(2 * self.process_position),
               disable=self.is_disabled,
               leave=True,
               dynamic_ncols=True,
               file=sys.stdout,
               bar_format=self.BAR_FORMAT,
           )

Closing issue, but feel free to ping me and reopen if necessary.