Lily Erickson

JAX is already a library that is optimized for GPU training, and the NeoX repo itself already requires significant GPU resources that could benefit from offloading.

This should be compatible with Deepspeed, would it be possible to add a basic training pipeline?

train_on_inputs clarification

1

When you say train_on_inputs = False, I presume you mean to mask out the prompt, and train the loss only on the response that the model is supposed to produce....

Newest peft update breaks load_from_checkpoint

1

https://github.com/huggingface/peft/issues/285 Still trying to sort out how to proceed on my own branch, but I figured I'd bring it up here too, since it'll invalidate the codebase, and maybe more...

Issues with set_peft_model_state_dict, dictionary not properly updating.

13

So in the previous peft version, before the recent adalora changes, set_peft_model_state_dict returned a wrapped model object. Now, it appears to function as a mutator (returns None). So I changed...

Fixed NoneType attribute crash in tokenization_utils_base.py

4

The attribute self.model_max_length is not universally set in all tokenizers. If these cases, the tokenizer will crash the program without the listed change. I noticed it specifically when loading tokenizers...