lashmore

I'm also unable to run setup_all.sh. Despite trying a variety of different versions for tensorflow and tensorflow-gpu as specified in the requirements.txt, I get the same error posted above by...

Interestingly, when I installed tensorflow 1.12.0 and I QUIT OUT OF PyCharm and/or Terminal session and reopened and reran setup_all.sh, it worked.

I've been trying to follow the instructions in this thread dutifully, but I'm still struggling to get a model trained on RAMs 1.0. In particular the `scripts/cache_bert_hf.py` script which generates...

Excellent - I'll check it out and let you know how it goes!

Nice!! I got the cache_bert_hf.py to work with the aid of minimize_json.py - **thank you**! The training partition hdf5 file is ~12 GB, and the dev and test partitions is...

Is anyone getting a KeyError: "_"? ``` def add_byte_decoder(): byte_decoder = {} alphabet = pre_tokenizers.ByteLevel(False, False).alphabet() known_vals = set([]) for j in range(256): for k in range(256): for l in...

@am-bean - I confirm your fix works for Llama3! Any pointers on where to look in the Guidance repo to understand how to customize a similar hack that you curated...

@am-bean please let us know if you do end up doing that! I'm looking at the tokenizers package right now. I see Mistral, Starling and LLama3 all use SentencePieceBPETokenizer tokenization...