Keltin Grimes

keltin13.github.io

Software Engineering Institute Pittsburgh, PA ML Researcher @ SEI | CMU Stats/CS Alum

Results 2 comments of


                                            Keltin Grimes

why some prompt doesn't work, the hidden_states will be nan after GemmaModel.forward

Try loading the model with `dtype=torch.bfloat16` instead of float16.

After deplyed google/gemma-7b-it, there always is error response.

Try loading the model with dtype=torch.bfloat16 instead of float16.