CodeGen icon indicating copy to clipboard operation
CodeGen copied to clipboard

`fastBPE` fix path

Open nmd2k opened this issue 3 years ago • 0 comments

I against assertion AssertionError: failed to learn bpe on /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb, command: /home/dungnm/CodeGen/fastBPE/fast learnbpe 50000 /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb > /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.codes

It turn out the command itself was not right. The fastBPE path will be located at "codegen_sources/model/tools/fastBPE/fast" according to install_env.sh instead of "fastBPE/fast"

Suggest file codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py change

FAST = str(Path(__file__).parents[3].joinpath("fastBPE/fast"))

to

FAST = str(Path(__file__).parents[3].joinpath("codegen_sources/model/tools/fastBPE/fast"))

nmd2k avatar Nov 01 '22 03:11 nmd2k