bilstm_crf_pytorch代码中shape mismatch错误可能出现在哪里

[ghost]

File "/home/XXX/CLUENER2020/bilstm_crf_pytorch/crf.py", line 166, in _calculate_loss_old gold_score = self._score_sentence(features, tags, lengths) File "/home/XXX/CLUENER2020/bilstm_crf_pytorch/crf.py", line 149, in score_sentence ) + torch.sum(feats[i, r, tags[i, : lens[i]]]) IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [589], [563]

你好，感谢提供开源代码，报错信息如上，请问问题可能出现在哪里？

[haosiqing]

我也是这个问题==请问您解决了吗

[in-the-wind]

在原数据集没有遇到这个问题，但是换上自己的数据集后也遇到了这个问题，请问现在解决了吗. 同时在偶然修改batch_size时，发现有的batch_size不会出现这个问题，有的会，有点奇怪

[SnailDM]

这个问题出现在data_loader.py的preprocess这里，tokens的长度需要和tag_ids的长度保持一致，但是它这里文本处理逻辑没有考虑全面，它按空格来切分字，一旦你的文本不止一个空格就会报错误，所以我改了下处理逻辑，我们的文本有多个空格的， text_a = d['context']后面添加一行 text_a = text_a.replace(" ", " ")。具体还是要看你自己的文本，问题定位可以在这里找。

[zhouyongxiang]

请问您解决了吗，这个问题

[YueCongPKU]

您好！ 您的邮件我已收到，我会尽快查看的。谢谢哈！

[zhouyongxiang]

请问您解决了吗，这个问题

[zhouyongxiang]

您可以帮我解决这个问题吗？我用自己数据集也是出现这个错误，Traceback (most recent call last): File "run_lstm_crf.py", line 245, in main() File "run_lstm_crf.py", line 236, in main train(args,model,processor) File "run_lstm_crf.py", line 42, in train features, loss = model.forward_loss(input_ids, input_mask, input_lens, input_tags) File "C:\Users\AI501\PycharmProjects\bilstm_crf_pytorch\model.py", line 43, in forward_loss return features, self.crf.calculate_loss(features, tag_list=input_tags, lengths=input_lens) File "C:\Users\AI501\PycharmProjects\bilstm_crf_pytorch\crf.py", line 164, in calculate_loss return self._calculate_loss_old(scores, lengths, tag_list) File "C:\Users\AI501\PycharmProjects\bilstm_crf_pytorch\crf.py", line 168, in _calculate_loss_old gold_score = self._score_sentence(features, tags, lengths) File "C:\Users\AI501\PycharmProjects\bilstm_crf_pytorch\crf.py", line 150, in score_sentence ) + torch.sum(feats[i, r, tags[i, :lens[i]]]) IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [1885], [1659]