evaluate
evaluate copied to clipboard
BLEU: add COCO/PTB tokenization mode (tokenizer_name) for parity with pycocoevalcap
fixes #693
Summary
- Add
tokenizer_nameoption tobleumetric. - New modes:
"coco"/"ptb"(COCO PTBTokenizer; requirespycocoevalcap) and"whitespace". - Default behavior unchanged (
tokenizer_13a).
Motivation
- Address discrepancy with
pycocoevalcapBLEU caused by PTB vs 13a tokenization differences, esp. around commas/periods.
Changes
- metrics/bleu/tokenizer_13a.py: add
CocoPTBTokenizer,WhitespaceTokenizer. - metrics/bleu/bleu.py: new
tokenizer_nameparam; accepts"coco"|"ptb"|"whitespace". - metrics/bleu/README.md: document new option and example.
- tests/test_bleu_coco_tokenization.py: focused tests; skipped if
pycocoevalcapnot installed.
Backwards compatibility
- Default tokenizer unchanged; no breaking API changes.
Notes
- COCO/PTB mode matches the issue’s reported scores.