pinchedsquare comments

Repositories
Issues
Comments

Results 2 comments of


                                            pinchedsquare

'str' object has no attribute 'device' and other issues when using quantized model

@SunMarc Just curious fundamentally why is `load_and_quantize_model()` using more memory than `AutoModelForCausalLM.from_pretrained(self.model_name, load_in_8bit=True, device_map="auto")` ? In the following 2 examples, first one succeeds and 2nd one fails with OOM on...

'str' object has no attribute 'device' and other issues when using quantized model

Will await your response. Thanks.