How to prevent "collapse" during training?

[imagio87]

Hi guys, I have read your paper and really like the approach of Fixmatch. I am trying to implement this method for a different domain than image classification. But I have some issues with training dynamics.

Over training time the number of unlabeled examples that have pseudo labels with confidence above threshold converges to zero. So in the end no unlabeled data is used in the loss function (mask rate is approaching 0%). And actually this makes sense from an optimization point of view.

The model could learn to predict class distributions that are below the threshold value for the unlabeled data and therby making the unsupervised loss term zero. It then can overfit the few labeled examples and loss is down to zero.

So I have two questions here. 1.) What will prevent the model from this kind of training collapse? Or am I missing something here? 2.) Why are you using the weakly augmented labeled examples for training and not the strongly augmented data like you are doing in the fully supervised setting? I have the feeling that this even increases the above problem since it is more easy to overfit the weakly augmented labeled data.

I would appreciate your help. Best regards.