automatic
automatic copied to clipboard
vladmandic
[Feature]: allow FaceID together with T2I adapter
Issue Description
When using the Control endpoints in combination with the Face script, an error is produced. It happens both within controlnet and T2I Adapter.
Steps to reproduce:
- Go to "Control" tab
- Update an image for "Control Input"
- Choose T2I and then the adapter OpenPose
- Select the face script
- Select a FaceID Model
- Upload the image for FaceID
- Click Generate
This will result in the error which is included inside the Relevant log.
After this i was switching between controlnet and t2i adapter and got a different error after rerunning again. Not sure if this is pure because of toggling a different pipeline, but i include this error as well just in case:
05:39:50-716723 ERROR Control pipeline failed: type=t2i adapter units=1 error=StableDiffusionAdapterPipeline.__call__() got an unexpected keyword argument 'shortcut'
05:39:50-717775 ERROR Control: TypeError
╭────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮
│ /home/ubuntu/ai_api/automatic_diffusers/modules/control/run.py:611 in control_run │
│ │
│ 610 │ │ │ │ │ │ │
│ ❱ 611 │ │ │ │ │ │ processed = p.scripts.run(p, *p.script_args) │
│ 612 │ │ │ │ │ │ if processed is None: │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/scripts.py:483 in run │
│ │
│ 482 │ │ parsed = p.per_script_args.get(script.title(), args[script.args_from:script.args_to]) │
│ ❱ 483 │ │ processed = script.run(p, *parsed) │
│ 484 │ │ s.record(script.title()) │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/__init__.py:119 in run │
│ │
│ 118 │ │ │ from modules.face.faceid import face_id │
│ ❱ 119 │ │ │ processed_images = face_id(p, app=app, source_images=input_images, model=ip_model, override=ip_override, cache=ip_cache, scale=ip_strength, structure=ip_structure) │
│ 120 │ │ │ processed = processing.Processed(p, images_list=processed_images, seed=p.seed, subseed=p.subseed, index_of_first_image=0) # manually created processed object │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/faceid.py:228 in face_id │
│ │
│ 227 │ │ │ │ debug(f"FaceID: {ip_model_dict}") │
│ ❱ 228 │ │ │ │ res = faceid_model.generate(**ip_model_dict) │
│ 229 │ │ │ │ if isinstance(res, list): │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/ip_adapter/ip_adapter_faceid.py:243 in generate │
│ │
│ 242 │ │ │
│ ❱ 243 │ │ images = self.pipe( │
│ 244 │ │ │ prompt_embeds=prompt_embeds, │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py:115 in decorate_context │
│ │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: StableDiffusionAdapterPipeline.__call__() got an unexpected keyword argument 'shortcut'
Version Platform Description
05:28:27-799123 INFO Starting SD.Next
05:28:27-801990 INFO Logger: file="/home/ubuntu/ai_api/automatic_diffusers/sdnext.log" level=INFO size=1448809 mode=append
05:28:27-802963 INFO Python version=3.10.12 platform=Linux bin="/home/ubuntu/ai_api/automatic_diffusers/venv/bin/python3" venv="/home/ubuntu/ai_api/automatic_diffusers/venv"
05:28:27-813136 INFO Version: app=sd.next updated=2024-06-17 hash=549f9ff7 branch=dev url=https://github.com/vladmandic/automatic/tree/dev ui=dev
05:28:28-092087 INFO Latest published version: a3ffd478e54c1735a1affc8b4760cef81594c293 2024-06-13T18:57:56Z
05:28:28-094240 INFO Platform: arch=x86_64 cpu=x86_64 system=Linux release=6.2.0-37-generic python=3.10.12
05:28:28-095296 INFO HF cache folder: /home/ubuntu/.cache/huggingface/hub
Relevant log output
05:31:12-511403 ERROR Control pipeline failed: type=t2i adapter units=0 error=Input is in incorrect format. Currently, we only support <class 'PIL.Image.Image'>, <class
'numpy.ndarray'>, <class 'torch.Tensor'>
05:31:12-512715 ERROR Control: ValueError
╭────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮
│ /home/ubuntu/ai_api/automatic_diffusers/modules/control/run.py:611 in control_run │
│ │
│ 610 │ │ │ │ │ │ │
│ ❱ 611 │ │ │ │ │ │ processed = p.scripts.run(p, *p.script_args) │
│ 612 │ │ │ │ │ │ if processed is None: │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/scripts.py:483 in run │
│ │
│ 482 │ │ parsed = p.per_script_args.get(script.title(), args[script.args_from:script.args_to]) │
│ ❱ 483 │ │ processed = script.run(p, *parsed) │
│ 484 │ │ s.record(script.title()) │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/__init__.py:119 in run │
│ │
│ 118 │ │ │ from modules.face.faceid import face_id │
│ ❱ 119 │ │ │ processed_images = face_id(p, app=app, source_images=input_images, model=ip_model, override=ip_override, cache=ip_cache, scale=ip_strength, structure=ip_structure) │
│ 120 │ │ │ processed = processing.Processed(p, images_list=processed_images, seed=p.seed, subseed=p.subseed, index_of_first_image=0) # manually created processed object │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/modules/face/faceid.py:228 in face_id │
│ │
│ 227 │ │ │ │ debug(f"FaceID: {ip_model_dict}") │
│ ❱ 228 │ │ │ │ res = faceid_model.generate(**ip_model_dict) │
│ 229 │ │ │ │ if isinstance(res, list): │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/ip_adapter/ip_adapter_faceid.py:243 in generate │
│ │
│ 242 │ │ │
│ ❱ 243 │ │ images = self.pipe( │
│ 244 │ │ │ prompt_embeds=prompt_embeds, │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py:115 in decorate_context │
│ │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py:1036 in __call__ │
│ │
│ 1035 │ │ # 4. Preprocess image │
│ ❱ 1036 │ │ image = self.image_processor.preprocess(image) │
│ 1037 │
│ │
│ /home/ubuntu/ai_api/automatic_diffusers/venv/lib/python3.10/site-packages/diffusers/image_processor.py:536 in preprocess │
│ │
│ 535 │ │ if not is_valid_image_imagelist(image): │
│ ❱ 536 │ │ │ raise ValueError( │
│ 537 │ │ │ │ f"Input is in incorrect format. Currently, we only support {', '.join(str(x) for x in supported_formats)}" │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Input is in incorrect format. Currently, we only support <class 'PIL.Image.Image'>, <class 'numpy.ndarray'>, <class 'torch.Tensor'>
Backend
Diffusers
Branch
Dev
Model
SD 1.5
Acknowledgements
- [X] I have read the above and searched for existing issues
- [X] I confirm that this is classified correctly and its not an extension issue
FaceID is a very specific pipeline, it cannot be combined freely with T2I. that may be looked at feature-request for the future, but even then its a stretch.
i'll have to close this as making faceid compatible with anything but itself is a nightmare - there isn't sufficient demand to invest into fully reimplementing it and that's what it would take.