Helsinki-NLP
opus_read fails to extract CCMatrix
I tried to extract the aligned sentence pairs from CCMatrix, previously downloaded using opus_express. The command I used was
opus_read --source en --target fi --directory CCMatrix --preprocess xml --leave_non_alignments_out --write_mode moses --write CCMatrix.raw.en CCMatrix.raw.fi --write_ids CCMatrix.raw.ids
The command runs for several days at 100% CPU, without producing any output. Perhaps expat is choking on some error in the data. To rule out package corruption after download, I allowed opus_read to download it again, with the same hanging result.
Traceback when killed:
File "/home/stiggronroos/venvs/opustools/bin/opus_read", line 135, in <module>
OpusRead(**vars(args)).printPairs()
File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/opus_read.py", line 214, in printPairs
self.alignmentParser.collect_links()
File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/alignment_parser.py", line 107, in collect_links
blocks = self.bp.get_complete_blocks()
File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/block_parser.py", line 98, in get_complete_blocks
self.parse_line(line)
File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/block_parser.py", line 82, in parse_line
self.p.Parse(line)
KeyboardInterrupt
Workaround: (re)download the corpus directly in moses format from https://opus.nlpl.eu/CCMatrix.php
I guess it's because CCMatrix is so big that reading only the sentence IDs and links to be retrieved from the monolingual corpora takes too much memory to run efficiently. We don't have a good solution for this at the moment but should add some robustness to the tools to also run on bigger data sets. The workaround with moses files is the only solution I can recommend at this moment ....
Why can't the "preprocess" parameter be set to "Moses" directly ? I mean, "xml"、"raw" and "parsed" are all time-conmusing relatively.
There is now --chunk_size parameter to control memory consumption, although the current implementation is still slow for corpora with huge documents.
Regarding moses, it also possible to download moses files with the opus_get script. For example:
to list available files:
opus_get -s en -t fi -d CCMatrix -p moses -l
to download the files:
opus_get -s en -t fi -d CCMatrix -p moses
I'm still leaving this issue open until we find a better solution for processing huge documents.
