drimeF0

Hello, I am currently using hivemind for my project to distribute MoE models with huggingface transformers to peer-to-peer network participants for their inference and training. At the moment I'm stuck...

 the command I used to run the script: ```bash %cd /content/sd-scripts !source /content/venv/bin/activate;accelerate launch "sdxl_train_network.py" --in_json "/content/dataset_lat.json" --console_log_simple --pretrained_model_name_or_path "/content/model.safetensors" --dataset_repeats "1" --train_data_dir "/content/dataset" --output_dir "/content/drive/MyDrive/out" --network_alpha "32" --network_dim...

training loop freezes after first step on TPU

2

https://colab.research.google.com/drive/1dxNpliW4JZIt6aut340dbUG7I9xLVpWI?usp=sharing accelerate version: 0.31.0 diffusers version: 0.29.2 torch version: 2.3.0 torch_xla[tpu] version: 2.3.0  At the moment, the code did not continue even after another 30 minutes, then the duration...