Mitchell Gordon

Minimal example: ``` task foo { x=a case $x in a) echo "hi" ;; b) echo "hello" ;; esac } ``` Gives ``` bash-4.2$ ducttape tmp.tape ducttape 0.3 by Jonathan...

Various Scale Things

8

Hey Lucid, I've been working on scaling the DB up to contain the whole Pile in my free time. En route to this, I've made a few changes that you...

Double [CLS] token in the first doc chunk

1

I noticed when we tokenize, we set `add_special_tokens` to True here: https://github.com/lucidrains/RETRO-pytorch/blob/main/retro_pytorch/retrieval.py#L72 which adds a [CLS] token to the beginning of the doc tokens. But when we embed the chunks...

Memory Decreases! But Latency Increases....

12

Things seem to be working as intended! I went from using GPT-J-6B with ```python model = AutoModelForCausalLM.from_pretrained("/mnt/models",torch_dtype=torch.float16,low_cpu_mem_usage=True).to(torch.device("cuda",0)) ``` to ```python model = AutoModelForCausalLM.from_pretrained("/mnt/models",device_map="auto",load_in_8bit=True) ``` With nvidia-smi reporting a decrease in...

- Add requirements.txt for easy pip install - Fix calls to OpenCV findContours - Make compatible with Python3

GPT-J Preprocessing Incorrectly Tokenizes `<|endoftext|>`

8

### Description Expected behavior: ```shell >>> from transformers import AutoTokenizer >>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B") >>> tokenizer.encode('') [50256] ``` ### Reproduced Steps Actual behavior: ```shell $ cd all_models/gptj/preprocessing/1 $ python >>>...