Load local safetensors file raised invalid json format

[lamguy]

Describe the bug

I have this Dockerfile to download the checkpoint from Dreamshaper XL Turbo but when I tried to load the checkpoint with AutoPipelineForText2Image.from_pretrained I got the following error:

#12 35.68 Traceback (most recent call last):
#12 35.68   File "/app/src/download_weights.py", line 3, in <module>
#12 35.68     img_gen = ImageGenerator() #Initializing an instance of generator to download all weights
#12 35.68               ^^^^^^^^^^^^^^^^
#12 35.68   File "/app/src/image_generation_logic.py", line 26, in __init__
#12 35.68     self.pipeline = AutoPipelineForText2Image.from_pretrained(
#12 35.68                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 35.68   File "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
#12 35.68     return fn(*args, **kwargs)
#12 35.68            ^^^^^^^^^^^^^^^^^^^
#12 35.68   File "/usr/local/lib/python3.11/dist-packages/diffusers/pipelines/auto_pipeline.py", line 326, in from_pretrained
#12 35.68     config = cls.load_config(pretrained_model_or_path, **load_config_kwargs)
#12 35.68              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 35.68   File "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
#12 35.68     return fn(*args, **kwargs)
#12 35.68            ^^^^^^^^^^^^^^^^^^^
#12 35.68   File "/usr/local/lib/python3.11/dist-packages/diffusers/configuration_utils.py", line 436, in load_config
#12 35.68     raise EnvironmentError(f"It looks like the config file at '{config_file}' is not a valid JSON file.")
#12 35.68 OSError: It looks like the config file at '/app/models/DreamShaperXL_Turbo_V2-SFW.safetensors' is not a valid JSON file.

Reproduction

My Docker file:

RUN mkdir -p /app/models
RUN pip install -U "huggingface_hub[cli]" && huggingface-cli download --help
RUN huggingface-cli download Lykon/dreamshaper-xl-v2-turbo DreamShaperXL_Turbo_V2-SFW.safetensors --local-dir /app/models
RUN ls -al /app/models

Python code:

        self.pipeline = AutoPipelineForText2Image.from_pretrained(
            "/app/models/DreamShaperXL_Turbo_V2-SFW.safetensors",
            torch_dtype=torch.float16,
            use_safetensors=True,
            image_encoder=self.image_encoder
        )

Logs

No response

System Info

diffusers-0.27.2

Who can help?

@yiyixuxu @sayakpaul @DN6

[sayakpaul]

You are supposed to either pass a repo_id on the Hugging Face Hub platform or a path to the directory that contains the checkpoint when calling from_pretrained().

So, doing

pipeline = AutoPipelineForText2Image.from_pretrained("Lykon/dreamshaper-xl-v2-turbo")

should work.

[lamguy]

@sayakpaul yeah, that's what I did to just use that model, but there is a version in the repo I want to use which is the SFW version https://huggingface.co/Lykon/dreamshaper-xl-v2-turbo/blob/main/DreamShaperXL_Turbo_V2-SFW.safetensors. From diffusers docs, it mentions I could use the path of the local but maybe I interpreted it wrong

from_pretrained
<
source
>
( pretrained_model_or_path**kwargs )

Parameters

pretrained_model_name_or_path (str or os.PathLike, optional) — Can be either:
A string, the repo id (for example CompVis/ldm-text2im-large-256) of a pretrained pipeline hosted on the Hub.
A path to a directory (for example ./my_pipeline_directory/) containing pipeline weights saved using [save_pretrained()](https://huggingface.co/docs/diffusers/v0.27.2/en/api/pipelines/overview#diffusers.DiffusionPipeline.save_pretrained).

[sayakpaul]

In that case, you need to use from_single_file() and not from_pretrained().

[lamguy]

Hi @sayakpaul , since AutoPipelineForText2Image does not have that method from_single_file, I tried to do this:

        self.pipeline = StableDiffusionXLPipeline.from_single_file(
            "https://huggingface.co/Lykon/dreamshaper-xl-v2-turbo/blob/main/DreamShaperXL_Turbo_V2-SFW.safetensors",
            torch_dtype=torch.float16,
            use_safetensors=True
        )

But then I got this error

Some weights of the model checkpoint were not used when initializing CLIPTextModelWithProjection: 
['text_model.embeddings.position_ids']
...
mat1 and mat2 shapes cannot be multiplied (16962x1664 and 1280x1280)

I just want to load the DreamShaperXL_Turbo_V2-SFW.safetensors file inside this repo Lykon/dreamshaper-xl-v2-turbo :(

[sayakpaul]

Cc: @DN6

[DN6]

Hi @lamguy can you update to diffusers 0.28.0 and try running

from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_single_file("https://huggingface.co/Lykon/dreamshaper-xl-v2-turbo/blob/main/DreamShaperXL_Turbo_V2-SFW.safetensors")
pipe.enable_model_cpu_offload()
pipe("a painting of a cat", num_inference_steps=20).images[0].save("dreamshaper.png")

[lamguy]

That fixed it! Thank you!