Colm Evans

Has any thought been given to using LoRA to increase the number of experts (100x) with minimal memory?

8

As I understand the current MoeLayer, a gate calculates the weight to be applied to the output of each expert, the top k are selected and run on the data,...

Added a basic example (examples/pufferl_lstm_wrapper.py) showing the use of LSTMWrapper with the Default model from models.