custom_pipeline not being cached

[John0x]

Describe the bug

Components of custom pipelines (defined via custom_pipeline parameter) do not seem to be cached. I have a multistep system where I'm loading all my components and then caching them explicitly using transformers.utils.move_cache. I have an sdxl pipeline with custom_pipeline="lpw_stable_diffusion_xl". The normal pipeline components seem to be cached correctly but not the components of the custom_pipeline.

I'm restoring the cached components by using the Pipeline with local_files_only=True, however diffusers is always trying to reach the internet for the custom_pipeline components, which it cannot, since the env where the restoration happens, doesn't have a connection.

Reproduction

StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16,
custom_pipeline="lpw_stable_diffusion_xl"
)

Logs

No response

System Info

latest diffusers version from main branch

Who can help?

@yiyixuxu @sayakpaul @DN6