panml icon indicating copy to clipboard operation
panml copied to clipboard
verified publisher icon Pan-ML

Model size and memory needed to run

Open vsraptor opened this issue 2 years ago • 2 comments

Is there a way to figure in advance how much memory :

  1. It will require to LOAD the model
  2. It will require to RUN the model

f.e. when I try to load google/flan5-large it seems to initially consume ~6GB of RAM and then settles down to ~3GB which is the ~file size.

  • Is this normal behavior ? i.e. requiring double amount of RAM ?
  • Some models seem to be multiple files ! How do you figure the needed RAM ?

f.e. https://huggingface.co/stabilityai/stablelm-base-alpha-3b/tree/main

vsraptor avatar May 31 '23 20:05 vsraptor

Yeah I'm thinking about this too. Definitely a good one to put in. I think we can have a lookup and also pull this info when user asks

wanoz avatar May 31 '23 22:05 wanoz

Maybe this is the reason it doubles the CPU RAM https://discuss.huggingface.co/t/how-much-memory-required-to-load-t0pp/10904

emekaokoli19 avatar Jul 03 '23 23:07 emekaokoli19