huggingface
Added get_velocity function to EulerDiscreteScheduler.
What does this PR do?
This PR adds the get_velocity function to EulerDiscreteScheduler, which is needed for training/fine-tuning with this scheduler if the prediction_type is v_prediction.
Fixes #6359
Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline?
- [x] Did you read our philosophy doc (important for complex PRs)?
- [ ] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
- [ ] Did you write any new necessary tests?
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@yiyixuxu
@RuiningLi I don't understand how this is related to https://github.com/huggingface/diffusers/pull/6395 ?
Fixes https://github.com/huggingface/diffusers/pull/6395
@RuiningLi I don't understand how this is related to #6395 ?
Fixes #6395
Sorry, I meant #6359 .
thanks!
I would like to understand the use case a little bit more
Why do you need to use EulerDiscreteScheduler for training?
thanks! I would like to understand the use case a little bit more Why do you need to use
EulerDiscreteSchedulerfor training?
Stable Video Diffusion by default uses EulerDiscreteScheduler, so it would be nice to keep it consistent if we fine-tune the model.
@RuiningLi we normally use DDPM for training regardless of which scheduler it uses for inference by default
I don't think we have a training script for SVD, do we?
@yiyixuxu Not yet. Also, SVD itself uses EulerDiscreteScheduler for fine-tuning, and it also uses a continuous sigma sampler rather than a discrete one.
EulerDiscreteScheduler is used by internal training done by HuggingFace, for example, the SDXL ControlNet models were trained with --use_euler, but this means you can't train v-prediction models with that.. because the scheduler.get_velocity() is used if prediction_type == "v_prediction"
EulerDiscreteScheduler is used by internal training done by HuggingFace, for example, the SDXL ControlNet models were trained with
--use_euler, but this means you can't train v-prediction models with that.. because thescheduler.get_velocity()is used ifprediction_type == "v_prediction"
What do you mean exactly? If Huggingface internally use this scheduler to train, why does it prevent from having the get_velocity function?
that's not the point i was making, they use it internally on epsilon prediction models. it's just that training on Euler isn't unordinary and it's even done inside HF itself. so supporting it in trainers doesn't seem unusual, either. it will unify the internal and example scripts. but this is required to merge before that can be done.
that's not the point i was making, they use it internally on epsilon prediction models. it's just that training on Euler isn't unordinary and it's even done inside HF itself. so supporting it in trainers doesn't seem unusual, either. it will unify the internal and example scripts. but this is required to merge before that can be done.
So do you mean that we need to first get this PR merged before we can have example training code for Euler v_prediction? If that is what you meant, then I agree with you!
I'm ok with adding this if there is a use case in training for it @sayakpaul what do you think?
Yes, indeed. This is an important thing to support as we move more and more to Karras-style training. IMO, we should support it.
Good for me to merge once the test cases pass. For the quality tests, I think the error messages on the CI should be helpful.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
