speechbrain
speechbrain copied to clipboard
speechbrain
Recipe CommonVoice/ASR/transformer fails with error 'WhisperTokenizerFast' object has no attribute 'normalize'
Describe the bug
Since speechbrain==1.0.3, I face an error when launching recipe CommonVoice/ASR/transformer/:
AttributeError: 'WhisperTokenizerFast' object has no attribute 'normalize'
Expected behaviour
Recipe works fine ;-)
To Reproduce
Host-0:~/speechbrain/recipes/CommonVoice/ASR/transformer$ torchrun train_with_whisper.py hparams/train_hf_whisper.yaml --language=en --skip_prep=True --save_folder=/tmp/speechbrain/ --whisper_hub=openai/whisper-large-v2 --data_folder=/tmp/datasets --train_csv=/tmp/datasets/dataset_train.csv --valid_csv=/tmp/datasets/dataset_val.csv --test_csv=/tmp/datasets/dataset_test.csv --output_folder=/tmp/outputs/
Environment Details
speechbrain==1.0.3
Relevant Log Output
/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
speechbrain.lobes.models.huggingface_transformers.whisper - whisper encoder is frozen.
speechbrain.utils.quirks - Applied quirks (see `speechbrain.utils.quirks`): [allow_tf32, disable_jit_profiling]
speechbrain.utils.quirks - Excluded quirks specified by the `SB_DISABLE_QUIRKS` environment (comma-separated list): []
speechbrain.core - Beginning experiment!
speechbrain.core - Experiment folder: /tmp/outputs
speechbrain.core - Info: precision arg from hparam file is used
speechbrain.core - Info: eval_precision arg from hparam file is used
speechbrain.core - Info: max_grad_norm arg from hparam file is used
speechbrain.core - Info: ckpt_interval_minutes arg from hparam file is used
speechbrain.core - Info: grad_accumulation_factor arg from hparam file is used
speechbrain.core - Gradscaler enabled: `True`
speechbrain.core - Using training precision: `--precision=fp16`
speechbrain.core - Using evaluation precision: `--eval_precision=fp16`
speechbrain.core - Exception:
Traceback (most recent call last):
File "train_with_whisper.py", line 296, in <module>
asr_brain = ASR(
File "/usr/local/lib/python3.8/dist-packages/speechbrain/core.py", line 770, in __init__
self.training_ctx = TorchAutocast(
File "/usr/local/lib/python3.8/dist-packages/speechbrain/utils/autocast.py", line 79, in __init__
self.context = torch.autocast(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/amp/autocast_mode.py", line 241, in __init__
raise RuntimeError(
RuntimeError: User specified an unsupported autocast device_type 'cuda:0'
[2025-04-11 10:10:58,397] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 15988) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 812, in main
run(args)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 135, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train_with_whisper.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-04-11_10:10:58
host : Host-0
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 15988)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Additional Context
It worked fine with speechbrain <=1.0.2
Hey, Sorry for this issue!
Can you please try to replace this block of code : transformer/train_with_whisper.py#L231-L232 as well as transformer/train_with_whisper.py#L231-L232
To be: https://github.com/speechbrain/speechbrain/blob/develop/recipes/LibriSpeech/ASR/transformer/train_with_whisper.py#L111-L115 ?
I think we didn't modified the CommonVoice recipe compared to the LibriSpeech one.
Alternatively, you can just pass --normalized_transcripts=False to remove the call to the whisper text normalizer.
Thanks
@Adel-Moumen ,
I updated speechbrain/recipes/CommonVoice/ASR/transformer/train_with_whisper.py with the following code, and it worked for me:
#!/usr/bin/env python3
"""Recipe for training a whisper-based ASR system with CommonVoice.
The system employs whisper from OpenAI (https://cdn.openai.com/papers/whisper.pdf).
This recipe take the whisper encoder-decoder to fine-tune on.
To run this recipe, do the following:
> python train_with_whisper.py hparams/train_hf_whisper.yaml
Authors
* Pooneh Mousavi 2022
* Adel Moumen 2024
"""
import sys
import torch
import torchaudio
from hyperpyyaml import load_hyperpyyaml
import speechbrain as sb
from speechbrain.utils.data_utils import undo_padding
from speechbrain.utils.distributed import if_main_process, run_on_main
from speechbrain.utils.logger import get_logger
logger = get_logger(__name__)
# Define training procedure
class ASR(sb.Brain):
def compute_forward(self, batch, stage):
"""Forward computations from the waveform batches to the output probabilities."""
batch = batch.to(self.device)
wavs, wav_lens = batch.sig
bos_tokens, bos_tokens_lens = batch.tokens_bos
# Add waveform augmentation if specified.
if stage == sb.Stage.TRAIN and hasattr(self.hparams, "wav_augment"):
wavs, wav_lens = self.hparams.wav_augment(wavs, wav_lens)
bos_tokens = self.hparams.wav_augment.replicate_labels(bos_tokens)
bos_tokens_lens = self.hparams.wav_augment.replicate_labels(
bos_tokens_lens
)
# We compute the padding mask and replace the values with the pad_token_id
# that the Whisper decoder expect to see.
abs_tokens_lens = (bos_tokens_lens * bos_tokens.shape[1]).long()
pad_mask = (
torch.arange(abs_tokens_lens.max(), device=self.device)[None, :]
< abs_tokens_lens[:, None]
)
bos_tokens[~pad_mask] = self.tokenizer.pad_token_id
# Forward encoder + decoder
enc_out, logits, _ = self.modules.whisper(wavs, bos_tokens)
log_probs = self.hparams.log_softmax(logits)
hyps = None
if stage == sb.Stage.VALID:
hyps, _, _, _ = self.hparams.valid_search(
enc_out.detach(), wav_lens
)
elif stage == sb.Stage.TEST:
hyps, _, _, _ = self.hparams.test_search(enc_out.detach(), wav_lens)
return log_probs, hyps, wav_lens
def compute_objectives(self, predictions, batch, stage):
"""Computes the loss NLL given predictions and targets."""
(log_probs, hyps, wav_lens) = predictions
batch = batch.to(self.device)
ids = batch.id
tokens_eos, tokens_eos_lens = batch.tokens_eos
# Augment Labels
if stage == sb.Stage.TRAIN and hasattr(self.hparams, "wav_augment"):
tokens_eos = self.hparams.wav_augment.replicate_labels(tokens_eos)
tokens_eos_lens = self.hparams.wav_augment.replicate_labels(
tokens_eos_lens
)
loss = self.hparams.nll_loss(
log_probs, tokens_eos, length=tokens_eos_lens
)
if stage != sb.Stage.TRAIN:
tokens, tokens_lens = batch.tokens
# Decode token terms to words
predicted_words = [
self.tokenizer.decode(t, skip_special_tokens=True).strip()
for t in hyps
]
# Convert indices to words
target_words = undo_padding(tokens, tokens_lens)
target_words = self.tokenizer.batch_decode(
target_words, skip_special_tokens=True
)
if hasattr(self.hparams, "normalized_transcripts"):
if hasattr(self.tokenizer, "normalize"):
normalized_fn = self.tokenizer.normalize
else:
normalized_fn = self.tokenizer._normalize
predicted_words = [
normalized_fn(text).split(" ")
for text in predicted_words
]
target_words = [
normalized_fn(text).split(" ")
for text in target_words
]
else:
predicted_words = [text.split(" ") for text in predicted_words]
target_words = [text.split(" ") for text in target_words]
self.wer_metric.append(ids, predicted_words, target_words)
self.cer_metric.append(ids, predicted_words, target_words)
return loss
def on_stage_start(self, stage, epoch):
"""Gets called at the beginning of each epoch"""
if stage != sb.Stage.TRAIN:
self.cer_metric = self.hparams.cer_computer()
self.wer_metric = self.hparams.error_rate_computer()
def on_stage_end(self, stage, stage_loss, epoch):
"""Gets called at the end of an epoch."""
# Compute/store important stats
stage_stats = {"loss": stage_loss}
if stage == sb.Stage.TRAIN:
self.train_stats = stage_stats
else:
stage_stats["CER"] = self.cer_metric.summarize("error_rate")
stage_stats["WER"] = self.wer_metric.summarize("error_rate")
# Perform end-of-iteration things, like annealing, logging, etc.
if stage == sb.Stage.VALID:
lr = self.hparams.lr_annealing_whisper.current_lr
self.hparams.train_logger.log_stats(
stats_meta={"epoch": epoch, "lr": lr},
train_stats=self.train_stats,
valid_stats=stage_stats,
)
self.checkpointer.save_and_keep_only(
meta={"WER": stage_stats["WER"]},
min_keys=["WER"],
)
elif stage == sb.Stage.TEST:
self.hparams.train_logger.log_stats(
stats_meta={"Epoch loaded": self.hparams.epoch_counter.current},
test_stats=stage_stats,
)
if if_main_process():
with open(
self.hparams.test_wer_file, "w", encoding="utf-8"
) as w:
self.wer_metric.write_stats(w)
def dataio_prepare(hparams, tokenizer):
"""This function prepares the datasets to be used in the brain class.
It also defines the data processing pipeline through user-defined functions.
"""
data_folder = hparams["data_folder"]
train_data = sb.dataio.dataset.DynamicItemDataset.from_csv(
csv_path=hparams["train_csv"],
replacements={"data_root": data_folder},
)
if hparams["sorting"] == "ascending":
# we sort training data to speed up training and get better results.
train_data = train_data.filtered_sorted(
sort_key="duration",
key_max_value={"duration": hparams["avoid_if_longer_than"]},
)
# when sorting do not shuffle in dataloader ! otherwise is pointless
hparams["train_loader_kwargs"]["shuffle"] = False
elif hparams["sorting"] == "descending":
train_data = train_data.filtered_sorted(
sort_key="duration",
reverse=True,
key_max_value={"duration": hparams["avoid_if_longer_than"]},
)
# when sorting do not shuffle in dataloader ! otherwise is pointless
hparams["train_loader_kwargs"]["shuffle"] = False
elif hparams["sorting"] == "random":
pass
else:
raise NotImplementedError(
"sorting must be random, ascending or descending"
)
valid_data = sb.dataio.dataset.DynamicItemDataset.from_csv(
csv_path=hparams["valid_csv"],
replacements={"data_root": data_folder},
)
valid_data = valid_data.filtered_sorted(sort_key="duration")
# test is separate
test_data = sb.dataio.dataset.DynamicItemDataset.from_csv(
csv_path=hparams["test_csv"],
replacements={"data_root": data_folder},
)
datasets = [train_data, valid_data, test_data]
# 2. Define audio pipeline:
@sb.utils.data_pipeline.takes("wav")
@sb.utils.data_pipeline.provides("sig")
def audio_pipeline(wav):
info = torchaudio.info(wav)
sig = sb.dataio.dataio.read_audio(wav)
if info.sample_rate != hparams["sample_rate"]:
sig = torchaudio.transforms.Resample(
info.sample_rate, hparams["sample_rate"]
)(sig)
return sig
sb.dataio.dataset.add_dynamic_item(datasets, audio_pipeline)
# 3. Define text pipeline:
@sb.utils.data_pipeline.takes("wrd")
@sb.utils.data_pipeline.provides(
"wrd", "tokens_list", "tokens_bos", "tokens_eos", "tokens"
)
def text_pipeline(wrd):
if hasattr(hparams, "normalized_transcripts"):
if hasattr(tokenizer, "normalize"):
normalized_fn = tokenizer.normalize
else:
normalized_fn = tokenizer._normalize
wrd = normalized_fn(wrd)
yield wrd
tokens_list = tokenizer.encode(wrd, add_special_tokens=False)
yield tokens_list
tokens_list = tokenizer.build_inputs_with_special_tokens(tokens_list)
tokens_bos = torch.LongTensor(tokens_list[:-1])
yield tokens_bos
tokens_eos = torch.LongTensor(tokens_list[1:])
yield tokens_eos
tokens = torch.LongTensor(tokens_list)
yield tokens
sb.dataio.dataset.add_dynamic_item(datasets, text_pipeline)
# 4. Set output:
sb.dataio.dataset.set_output_keys(
datasets,
["id", "sig", "tokens_list", "tokens_bos", "tokens_eos", "tokens"],
)
return train_data, valid_data, test_data
if __name__ == "__main__":
# CLI:
hparams_file, run_opts, overrides = sb.parse_arguments(sys.argv[1:])
# create ddp_group with the right communication protocol
sb.utils.distributed.ddp_init_group(run_opts)
with open(hparams_file, encoding="utf-8") as fin:
hparams = load_hyperpyyaml(fin, overrides)
# Create experiment directory
sb.create_experiment_directory(
experiment_directory=hparams["output_folder"],
hyperparams_to_save=hparams_file,
overrides=overrides,
)
# Dataset prep (parsing Librispeech)
from common_voice_prepare import prepare_common_voice # noqa
# multi-gpu (ddp) save data preparation
run_on_main(
prepare_common_voice,
kwargs={
"data_folder": hparams["data_folder"],
"save_folder": hparams["save_folder"],
"train_tsv_file": hparams["train_tsv_file"],
"dev_tsv_file": hparams["dev_tsv_file"],
"test_tsv_file": hparams["test_tsv_file"],
"accented_letters": hparams["accented_letters"],
"language": hparams["language"],
"skip_prep": hparams["skip_prep"],
},
)
# Defining tokenizer and loading it
tokenizer = hparams["whisper"].tokenizer
# here we create the datasets objects as well as tokenization and encoding
train_data, valid_data, test_data = dataio_prepare(hparams, tokenizer)
# Trainer initialization
asr_brain = ASR(
modules=hparams["modules"],
hparams=hparams,
run_opts=run_opts,
checkpointer=hparams["checkpointer"],
opt_class=hparams["whisper_opt_class"],
)
# We load the pretrained whisper model
if "pretrainer" in hparams.keys():
hparams["pretrainer"].collect_files()
hparams["pretrainer"].load_collected(asr_brain.device)
# We dynamically add the tokenizer to our brain class.
# NB: This tokenizer corresponds to the one used for Whisper.
asr_brain.tokenizer = tokenizer
# Training
asr_brain.fit(
asr_brain.hparams.epoch_counter,
train_data,
valid_data,
train_loader_kwargs=hparams["train_loader_kwargs"],
valid_loader_kwargs=hparams["valid_loader_kwargs"],
)
# Testing
asr_brain.hparams.test_wer_file = hparams["test_wer_file"]
asr_brain.evaluate(
test_data,
min_key="WER",
test_loader_kwargs=hparams["test_loader_kwargs"],
)
asr_brain.hparams.test_wer_file = hparams["valid_wer_file"]
asr_brain.evaluate(
valid_data,
min_key="WER",
test_loader_kwargs=hparams["test_loader_kwargs"],
)