Use prompt tokens instead of prompt for sdxl? for the purpose of interpolation

[prismspecs]

Working from the following code, but would like to use the tokenized prompt so that I can interpolate between 2 prompts and produce a video output. The creation of the video from frames is easy enough, but I’m stuck here at the beginning. Any advice is appreciated. Perhaps this already exists but I can't find the answer anywhere.

from diffusers import AutoPipelineForText2Image
import torch

torch.cuda.empty_cache()

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16"
)
pipe.to("cuda")

for item in pipe.components:
    print(item)

prompt = "A cinematic shot of a baby racoon wearing an intricate italian priest robe."

# tokenizing and encoding the prompt into embeddings.
prompt_tokens = pipe.tokenizer(
    prompt,
    padding="max_length",
    max_length=pipe.tokenizer.model_max_length,
    truncation=True,
    return_tensors="pt",
)

image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images[0]

# save image
image.save("output.png")

[tolgacangoz]

Could you examine this blog post: Images Interpolation with Stable Diffusion

[prismspecs]

Thanks for your reply. I have checked that out and understand it, but my problem is in using SDXL since it uses the StableDiffusionXLPipeline

[asomoza]

Actually, the pipeline class abstracts all the encoding of embeddings, so you don’t really need to implement it manually, don't know why those examples do it that way.

I always wanted to test this and understand how to do it, so I wrote a post in the discussions with an example, hope it helps you.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

@prismspecs does this solve your problem? https://github.com/huggingface/diffusers/discussions/7581 I'm closing this now if you still have questions, let's open a discussion https://github.com/huggingface/diffusers/discussions