Updated stable_diffusion_tensorrt_txt2img.py

[Bhavay-2001]

What does this PR do?

Fixes #6984

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline?
• [x] Did you read our philosophy doc (important for complex PRs)?
• [x] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [ ] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

[Bhavay-2001]

@standardAI @yiyixuxu

[yiyixuxu]

@standardAI can you do a review? thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[tolgacangoz]

OK, let's continue. So far so good. When I run the current implementation by using the test script from the previous PR page I encounter an attribute error. Could you work on it? I would like to suggest a workflow: Try to work until no error is seen. If you cannot solve an error then ask ChatGPT and/or Gemini. If you don't get a possible solution within 1 hour then could you ask here?

[Bhavay-2001]

Alright. I will work with that test example.

[Bhavay-2001]

Hi @standardAI, I will continue now. I am out of shape. Can you pls let me know how to proceed? Thanks

[Bhavay-2001]

import torch
from diffusers import DDIMScheduler
from diffusers.pipelines import DiffusionPipeline

Use the DDIMScheduler scheduler here instead
scheduler = DDIMScheduler.from_pretrained("stabilityai/stable-diffusion-2-1",
                                            subfolder="scheduler")

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1",
                                                custom_pipeline="/workspace/diffusers/examples/community/stable_diffusion_tensorrt_txt2img.py",
                                                revision='fp16',
                                                torch_dtype=torch.float16,
                                                scheduler=scheduler,)

# re-use cached folder to save ONNX models and TensorRT Engines
pipe.set_cached_folder("stabilityai/stable-diffusion-2-1", revision='fp16',)

pipe = pipe.to("cuda")

prompt = "a beautiful photograph of Mt. Fuji during cherry blossom"
image = pipe(prompt).images[0]
image.save('tensorrt_mt_fuji.png')

Should I try to run this script on google colab with the stable_diffusion_tensorrt_txt2img.py from the changes I have made?

[surajp92]

@Bhavay-2001 let me know if you want to work together to finish this. I was reviewing this PR and I got the below error

ValueError: Pipeline <class 'diffusers_modules.local.stable_diffusion_tensorrt_txt2img.TensorRTStableDiffusionPipeline'> expected {'tokenizer', 'text_encoder', 'vae', 'unet', 'safety_checker', 'feature_extractor', 'scheduler', 'image_encoder'}, but only {'tokenizer', 'text_encoder', 'vae', 'unet', 'scheduler'} were passed.

@standardAI you mentioned that you got an attributeerror while trying to run the test script. Did you set the custom_pipeline path to the new file ( ~/stable_diffusion_tensorrt_txt2img.py)? I got the AttributeError related to tensorrt package while using the stable_diffusion_tensorrt_txt2img pipeline, i.e., the one based on the StableDiffusionPipeline

[tolgacangoz]

@surajp92 Tbh, I don't remember what I did. Now, I am seeing a similar ValueError:

ValueError: Pipeline <class 'diffusers_modules.local.stable_diffusion_tensorrt_txt2img.TensorRTStableDiffusionPipeline'> expected {'tokenizer', 'unet', 'scheduler', 'safety_checker', 'vae', 'feature_extractor', 'text_encoder'}, but only {'tokenizer', 'unet', 'scheduler', 'vae', 'feature_extractor', 'text_encoder'} were passed.

@Bhavay-2001, we can use it for testing. Just replace revision with variant to avoid using a deprecated parameter. If you look around other pipelines' initializer/constructor functions, you will see something about the two additional parameters in the ValueError message. Could you try to take references from them? Isn't diffusers.yml file irrelevant here? Lastly, the corresponding README part needs to be updated.

[631068264]

@Bhavay-2001 Can it use for sdxl ? I'm looking for sth like stable_diffusion_xl_tensorrt_txt2img

[asfiyab-nvidia]

@Bhavay-2001 do you have an update on this? Are you able to run the existing example

import torch
from diffusers import DDIMScheduler
from diffusers.pipelines import DiffusionPipeline

# Use the DDIMScheduler scheduler here instead
scheduler = DDIMScheduler.from_pretrained("stabilityai/stable-diffusion-2-1",
                                            subfolder="scheduler")

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1",
                                                custom_pipeline="/workspace/diffusers/examples/community/stable_diffusion_tensorrt_txt2img.py",
                                                variant='fp16',
                                                torch_dtype=torch.float16,
                                                scheduler=scheduler,)

# re-use cached folder to save ONNX models and TensorRT Engines
pipe.set_cached_folder("stabilityai/stable-diffusion-2-1", revision='fp16',)

pipe = pipe.to("cuda")

prompt = "a beautiful photograph of Mt. Fuji during cherry blossom"
image = pipe(prompt).images[0]
image.save('tensorrt_mt_fuji.png')

[asfiyab-nvidia]

@yiyixuxu @tolgacangoz I've updated the pipeline locally and resolved the errors. There are also some API changes with TRT 10.0 that are included. Can I go ahead and update this PR if @Bhavay-2001 isn't available?

[tolgacangoz]

@Bhavay-2001 Can it use for sdxl ? I'm looking for sth like stable_diffusion_xl_tensorrt_txt2img

@631068264 Maybe this one?

[tolgacangoz]

It seems that @Bhavay-2001 isn't available for this PR.

[Bhavay-2001]

@yiyixuxu @tolgacangoz I've updated the pipeline locally and resolved the errors. There are also some API changes with TRT 10.0 that are included. Can I go ahead and update this PR if @Bhavay-2001 isn't available?

Hi @asfiyab-nvidia, sure you can go ahead woth the PR.

[631068264]

@Bhavay-2001 Can it use for sdxl ? I'm looking for sth like stable_diffusion_xl_tensorrt_txt2img

@631068264 Maybe this one?

No, I haved tried to load model from onnx or OpenVINO only support cpu and too slow. @tolgacangoz I also tried the TensorRT demo for sdxl, but the results weren't satisfactory. It didn't speed up much and it used way too much memory.

[tolgacangoz]

@631068264 Did/Could you try to use this page for ORTStableDiffusionXLPipeline? Also, there are faster SD-XLs with the same license: LCM-LoRA SDXL, SDXL-Lightning. Maybe, we should open a discussion for this.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.