refactor SD3, speedup tests

[a-r-r-o-w]

What does this PR do?

• Refactor some of the SD3 modeling implementation. Accessing configuration parameters in __init__ via the config attribute is not necessary or clean. Can use the parameters directly
• Improve docs
• Speedup tests

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

@DN6 LMK your thoughts here. Will update the expected test slices after approval

[a-r-r-o-w]

Will update test slices which are needed due to making models smaller

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

@a-r-r-o-w is this still relevant?

[a-r-r-o-w]

@sayakpaul This is still relevant, yes. We were waiting for SD3.5 to be in when this went stale, so I will revive this soon as it's in now.