[ERROR] Read-only file system: 'vocab.json' when I created a training job.

[eloelo-techadmin]

Training Code:

training = replicate.trainings.create(
        destination=f"{REPLICATE_USERNAME}/{model_name}",
        version=BASE_MODEL,
        input={
            "steps": 1000,
            "lora_rank": 16,
            "optimizer": "adamw8bit",
            "batch_size": 1,
            "resolution": "512,768,1024",
            "autocaption": True,
            "input_images": "https://api.replicate.com/v1/files/MWEzOTFjZDktODRiOC00NGU3LTk2NDgtNmU3OTFiYzI4ODAx/download?expiry=1728649027&owner=eloelo-techadmin&signature=Jax4T4zzjb%252BKcQqog45X1DkXRU%252BP2EQJhDx9UDU1REI%253D",
            "trigger_word": model_name,
            "learning_rate": 0.0004,
            # "wandb_project": "flux_train_replicate",
            # "wandb_save_interval": 100,
            "caption_dropout_rate": 0.05,
            "cache_latents_to_disk": False,
            "wandb_sample_interval": 100
        }
    )

ERROR:

Traceback (most recent call last):
File "/root/.pyenv/versions/3.10.15/lib/python3.10/site-packages/cog/server/worker.py", line 349, in _predict
result = predict(**payload)
File "/src/train.py", line 207, in train
clean_up()
File "/src/train.py", line 478, in clean_up
shutil.rmtree(WEIGHTS_PATH)
File "/root/.pyenv/versions/3.10.15/lib/python3.10/shutil.py", line 725, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/root/.pyenv/versions/3.10.15/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/root/.pyenv/versions/3.10.15/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/root/.pyenv/versions/3.10.15/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 30] Read-only file system: 'vocab.json'

Can someone please help us? We are stuck on this. cc @mattt