ostris
Broken training with the new polarity match guidance
The new noise polarity match changes for flowmatch are breaking the training.
Everything works decent until it reaches a certain step number, depending on the number of images in the dataset. Sometimes it's 1600, other times it's 2000.
The image breaks as following
It tries to remediate in the next iteration and the ones after, but at this point it's pretty much done.
It's as if it breaks everything down to 0 and then it would try to re-do the training. For the meantime, i've reverted the lines of code responsible for causing this, to their prior functional state.
Here's a link to the code responsible for this https://github.com/ostris/ai-toolkit/commit/0355662e8e0e03938b2e275ece61a9806b526a18?diff=split&w=0#r145596747
currently experimenting with some different changes, will have results in the next few hours