A certain probability of outputting a generated image with a lot of noise remaining.

[ShotaroIshiguro]

Thank you for making the code public. I ran DiffusionRig in my environment as well and was amazed at the innovation of the model. However, I have one question regarding the execution results. As the title says, most of the time it produces a clean edit result, but sometimes there is a lot of noise in the inference.py output result. What can I do to solve this problem? Perhaps it is unavoidable because we are using DDIM as the diffusion model? Thank you in advance for your answer. Attached below is the error text for the output.

Could not import EmoSwinModule. SWIN models will not be available. Make sure you pull the repository with submodules to enable SWIN.
SWIN not found, will not be able to use SWIN models
creating model and diffusion...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jisaku_training/trump_aligned/
jisaku_training/obama_aligned/obama_20.png
output_dir_FFHQ/target_obama_20/trump/trainDECA_resnet50_targetEMOCA_sourceDECA/
exp
EMOCA
DECA
log_FFHQ/stage2/trump/deca_resnet50/model010000.pt
scripts/inference50.py:227: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  ckpt = th.load(args.model_path)
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
creating the FLAME Decoder
trained model found. load /home/diffusion-rig-Emo/DiffusionRig_main/data/deca_model.tar
/home/diffusion-rig-Emo/DiffusionRig_main/decalib/deca.py:103: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(model_path)
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/pytorch3d/io/obj_io.py:550: UserWarning: Mtl file does not exist: /home/diffusion-rig-Emo/DiffusionRig_main/data/template.mtl
  warnings.warn(f"Mtl file does not exist: {f}")
Taking config of stage 'detail'
dict_keys(['coarse', 'detail'])
Looking for checkpoint in '/home/diffusion-rig-Emo/DiffusionRig_main/assets/EMOCA/models/EMOCA_v2_lr_mse_20/detail/checkpoints'
Found 1 checkpoints
 - /home/diffusion-rig-Emo/DiffusionRig_main/assets/EMOCA/models/EMOCA_v2_lr_mse_20/detail/checkpoints/deca-epoch=10-val_loss/dataloader_idx_0=3.25521111.ckpt
Selecting checkpoint '/home/diffusion-rig-Emo/DiffusionRig_main/assets/EMOCA/models/EMOCA_v2_lr_mse_20/detail/checkpoints/deca-epoch=10-val_loss/dataloader_idx_0=3.25521111.ckpt'
Loading checkpoint '/home/diffusion-rig-Emo/DiffusionRig_main/assets/EMOCA/models/EMOCA_v2_lr_mse_20/detail/checkpoints/deca-epoch=10-val_loss/dataloader_idx_0=3.25521111.ckpt'
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py:33: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(f, map_location=map_location)
Creating classic detail generator.
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG19_Weights.IMAGENET1K_V1`. You can also use `weights=VGG19_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
fc.weight  not available in reconstructed resnet
fc.bias  not available in reconstructed resnet
copy resnet state dict finished!
creating the FLAME Decoder
/home/diffusion-rig-Emo/DiffusionRig_main/gdl/models/DecaFLAME.py:93: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  torch.tensor(lmk_embeddings['dynamic_lmk_faces_idx'], dtype=torch.long))
/home/diffusion-rig-Emo/DiffusionRig_main/gdl/models/DecaFLAME.py:95: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  torch.tensor(lmk_embeddings['dynamic_lmk_bary_coords'], dtype=self.dtype))
fc.weight  not available in reconstructed resnet
fc.bias  not available in reconstructed resnet
copy resnet state dict finished!
fc.weight  not available in reconstructed resnet
fc.bias  not available in reconstructed resnet
copy resnet state dict finished!
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/pytorch3d/io/obj_io.py:550: UserWarning: Mtl file does not exist: /home/diffusion-rig-Emo/DiffusionRig_main/assets/FLAME/geometry/template.mtl
  warnings.warn(f"Mtl file does not exist: {f}")
use meanshape:  personalLMDB/personal_deca_trump.lmdb/mean_shape.pkl
/home/diffusion-rig-Emo/DiffusionRig_main/gdl/utils/DecaUtils.py:211: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at /opt/conda/conda-bld/pytorch_1724789115405/work/aten/src/ATen/native/Cross.cpp:62.)
  torch.cross(vertices_faces[:, 2] - vertices_faces[:, 1],
/home/anaconda3/envs/DiffRigWithEmoca/lib/python3.8/site-packages/torch/nn/functional.py:4373: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.