evaluate icon indicating copy to clipboard operation
evaluate copied to clipboard

BLEU: add COCO/PTB tokenization mode (tokenizer_name) for parity with pycocoevalcap

Open bavi404 opened this issue 6 months ago • 0 comments

fixes #693

Summary

  • Add tokenizer_name option to bleu metric.
  • New modes: "coco"/"ptb" (COCO PTBTokenizer; requires pycocoevalcap) and "whitespace".
  • Default behavior unchanged (tokenizer_13a).

Motivation

  • Address discrepancy with pycocoevalcap BLEU caused by PTB vs 13a tokenization differences, esp. around commas/periods.

Changes

  • metrics/bleu/tokenizer_13a.py: add CocoPTBTokenizer, WhitespaceTokenizer.
  • metrics/bleu/bleu.py: new tokenizer_name param; accepts "coco"|"ptb"|"whitespace".
  • metrics/bleu/README.md: document new option and example.
  • tests/test_bleu_coco_tokenization.py: focused tests; skipped if pycocoevalcap not installed.

Backwards compatibility

  • Default tokenizer unchanged; no breaking API changes.

Notes

  • COCO/PTB mode matches the issue’s reported scores.

bavi404 avatar Aug 13 '25 08:08 bavi404