huggingface
Make chat templates part of ProcessorMixin
Right now, IDEFICS2 supports chat templates, but in class-specific code. In this PR, we'll try to move that support to the generic ProcessorMixin so that we don't have to keep duplicating it as more VLMs (like LLaVA) start to need chat templates.
Draft PR for now until I make sure this doesn't break things!
Hi @Rocketknight1, very nice! FYI, I'm working on standardizing the processors here https://github.com/huggingface/transformers/pull/30511 and I was thinking how to make this compatible with apply_chat_template - thought you might want to take a look at the design here, and I'll keep an eye here as well :)
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
@molbap I didn't see your PR, but we should definitely use your approach! Supporting chat templates should be simple enough - all they require is moving apply_chat_template to ProcessorMixin, and then supporting chat_template as an attribute for any processor that wants to use them. Do you want me to pause my PR until yours is merged?
I don't think you need to pause yours! I'll try and get mine merged soon - it's already approved, just need to make sure nothing breaks
Cool! I'll probably wait until yours is merged, in that case, and then adding chat templates should be clean and straightforward.
This should be ready - cc @amyeroberts for core maintainer review! I'm speeding things up because the lack of this feature is causing some user issues, and we have a lot of VLMs coming.