OneTrainer icon indicating copy to clipboard operation
OneTrainer copied to clipboard

[Bug]: Resuming from backup results in TypeError: argument of type 'NoneType' is not iterable under cetain conditions.

Open allenbenz opened this issue 11 months ago • 0 comments

What happened?

Tried resuming from backup of an sdxl lora training session, got an exception. I have some random changes made to the codebase that may be a precondition for this so it might not be happening to anyone else, but the conditionals listed below are bugged irrespective of that.

Offending line is if i in include_parameter_indices or include_parameter_indices is None:

Swapping the conditional to if include_parameter_indices is None or i in include_parameter_indices: resolved the issue and allowed training to resume.

There's three instances of it in torch_util.py. https://github.com/Nerogar/OneTrainer/blob/11c99878c225c9f8395fb567cc56e1222a8d9153/modules/util/torch_util.py#L33 https://github.com/Nerogar/OneTrainer/blob/11c99878c225c9f8395fb567cc56e1222a8d9153/modules/util/torch_util.py#L61 https://github.com/Nerogar/OneTrainer/blob/11c99878c225c9f8395fb567cc56e1222a8d9153/modules/util/torch_util.py#L84

What did you expect would happen?

Backup to load and training to resume

Relevant log output

Traceback (most recent call last):
  File ".\modules\ui\TrainUI.py", line 569, in __training_thread_function
    trainer.start()
  File ".\modules\trainer\GenericTrainer.py", line 127, in start
    self.model_setup.setup_model(self.model, self.config)
  File ".\modules\modelSetup\StableDiffusionXLLoRASetup.py", line 172, in setup_model
    init_model_parameters(model, self.create_parameters(model, config), self.train_device)
  File ".\modules\util\optimizer_util.py", line 59, in init_model_parameters
    optimizer_to_device_(model.optimizer, train_device)
  File ".\modules\util\torch_util.py", line 72, in optimizer_to_device_
    tensors_to_device_(state, device)
  File ".\modules\util\torch_util.py", line 65, in tensors_to_device_
    tensor_transferred |= tensors_to_device_(elem, device, non_blocking=non_blocking, allocator=allocator)
  File ".\modules\util\torch_util.py", line 61, in tensors_to_device_
    if i in include_parameter_indices or include_parameter_indices is None:
TypeError: argument of type 'NoneType' is not iterable

Output of pip freeze

No response

allenbenz avatar Feb 11 '25 19:02 allenbenz