huggingface
[textual inversion] Load learned_embeddings.bin if it exists in the target folder
Here is the next part of my improvements to the textual inversion script.
Quite a lot changed in the meantime, so it may need some changes (e.g. where save_path should be defined). The code for loading embeddings is based on https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.
Hey @allo-,
Could you explain the use case here?
Is this function relevant when continuing training from a saved checkpoint?
This feature is essentially identical to "resume from checkpoint loading". Let's try to solve this cleanly, the same way across examples: https://github.com/huggingface/diffusers/issues/732
It loads checkpoints, but from the saved embedding and not the saved model. This is important when using the new only save embeds option.
I think dreambooth must always save the full model? So it might be a different usecase as one would in the best case never need to store a changed model for embeddings, once loading embeds is implemented in all relevant scripts. This function currently is the exact counterpart to save embeds.
But it may be a good idea to allow loading embeds when loading the model, e.g., by having a parameter in fron-pretrained that gets a list of embeds tensors.
#1687? Looks like it does checking pointing for the script now. But I think that are checkpoints including trainer state and so on and possibly not directly usable in scripts?
What about a --initial_embedding option for adding a showcase how to load an embedding (and not training checkpoint)?
I am still thinking if the Pipelines should be able to load embeddings. Maybe some .add_pretrained_embedding method:
def add_pretrained_embedding(self, filename, token_name=None, replace_existing_token=False):
embed = self.load_embed(filename)
name = token_name or embed.token_name
if not replace_existing_token:
assert name not in self.tokenizer
tokenizer[name] = embed.embedding
Hi @allo-! Yes, #1687 implements checkpointing for training scripts, and those checkpoints can be used for inference using a process like this: https://huggingface.co/docs/diffusers/main/en/training/dreambooth#performing-inference-using-a-saved-checkpoint
I'm sorry, but I still don't fully understand your proposal here. Are you suggesting to load the embeddings in the training script, or making it easier to load them at inference time? If it's for training, the checkpointing mechanism should work.
I have a by now quite old fork of the script and try to merge the features one by one. This one served mostly the purpose to exploit the regularly written embeddings as checkpoints. The function itself is also an example for how to load embeddings, e.g, for inference.
For the inference part, the implantation via the pipeline would probably be useful, as it makes things easier for people implementing inference scripts.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hey @allo-,
Doesn't #1687 already solve the problem? What exactly is needed additionally ?
As far as I understand it, these are training checkpoints and not the learned embeddings.
What I am trying to get in here, and maybe later even in the pipeline (possibly even initialization) is loading the embeddings from a file, that only stores the embeddings.
Currently I use this code in my fork of the training script and in some inference script (to be released when I have the time to clean it up) that loads a list of embedding files from a given folder, and I think in general it is useful to load the embeddings after loading a full stable diffusion checkpoint, or loading the checkpoint and a list of embeddings in the initialization at the same time.
I only suspect, that a PR for changing pipelines without having the example in here before may take even more time to be merged.
Hey @allo-,
Thanks for clarifying, yes I agree we should only save the trained embeddings and not the whole model during training. Let's make sure to solve this with #1985 (also cc @patil-suraj )
Yes, #1985 looks good. This is the whole thing I try to get to with the PR and #781 before. Saving a full model with one additional embedding has only the advantage of inference scripts not needing to know about embeddings, but the disadvantage of being large and making it hard to merge two embeddings into the same model.
So when #1985 gets implemented I would suggest:
- Train textual inversion by loading (using the new API) and storing the embedding (only).
- Optionally store the full model (or keep the opt-out using
--only_save_embedsor only save the full model in the last step instead of for each checkpoint) - Save snapshots of the embedding and the optimizer parameters in the training
- Provide an example for loading (several) embeddings into a vanilla model
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Should I just put the script with the other features I would make PRs for somewhere instead of waiting for each PR separately to merge one after another?
Actually, I just noticed that we already have loading logic / resume logic for textual inversion here: https://github.com/huggingface/diffusers/blob/801484840a3fa71285a7e096cc07f93b1ae681b7/examples/textual_inversion/textual_inversion.py#L705 , so I'd be in favor of not going forward with this. I don't see the benefit for now.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.