[tests] Speed up AnimateDiff test

[a-r-r-o-w]

What does this PR do?

Part of #7677

Before:

pytest -s tests/pipelines/animatediff/test_animatediff.py  213.80s user 10.60s system 182% cpu 2:02.86 total

After:

pytest -s tests/pipelines/animatediff/test_animatediff.py  40.48s user 4.53s system 88% cpu 51.084 total

Errors before these changes:

FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_float16_inference - AssertionError: nan not less than 0.05 : The outputs of the fp16 and fp32 pipelines are too different.
FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_from_pipe_consistent_forward_pass_cpu_offload - AssertionError: 0.011593044 not less than 0.001 : The outputs of the pipelines created with `from_pipe` and `__init__` are different.
FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_save_load_float16 - AssertionError: nan not less than 0.01 : The output of the fp16 pipeline changed after saving and loading.
FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_vae_tiling - TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Errors after these changes (need help fixing):

FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_from_pipe_consistent_config - RuntimeError: Error(s) in loading state_dict for ModuleList:
FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_from_pipe_consistent_forward_pass_cpu_offload - AssertionError: 0.0140529275 not less than 0.001 : The outputs of the pipelines created with `from_pipe` and `__init__` are different.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@sayakpaul

[a-r-r-o-w]

Hi @sayakpaul @DN6, some updates on the failing tests.

FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_from_pipe_consistent_forward_pass_cpu_offload

This fails in both old and new version of tests and I believe it is somewhat device dependent. I tested on different diffusers checkouts but seems like it always fails locally for me but not on CI.

FAILED tests/pipelines/animatediff/test_animatediff.py::AnimateDiffPipelineFastTests::test_from_pipe_consistent_config - RuntimeError: Error(s) in loading state_dict for ModuleList:

This test might need some rework because it assumes that the underlying model is the same as hf-internal-testing/tiny-stable-diffusion-pipe but since we've modified UNet to make the tests faster, it no longer can load the state dict correctly.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.