candle icon indicating copy to clipboard operation
candle copied to clipboard

Using the `--model` parameter with the quantized-qwen3 example fails

Open azw413 opened this issue 11 months ago • 2 comments

Using the --model parameter with the quantized-qwen3 example results in an error e.g.

cargo run --example quantized-qwen3 --release --features cuda -- --model 4b --prompt "A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?"
Finished `release` profile [optimized] target(s) in 0.42s
     Running `target/release/examples/quantized-qwen3 --model 4b --prompt 'A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?'`
avx: false, neon: true, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: No such file or directory (os error 2)

It works fine with no --model parameter using the default 0.6b model.

Running on ARM Linux on an nVidia Jetson Orin AGX.

azw413 avatar May 10 '25 19:05 azw413

It seems the argument is called --which 4b and not --model as indicated in the doc.

azw413 avatar May 10 '25 19:05 azw413

yup, I also received error message:

avx: true, neon: false, simd128: false, f16c: true
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: No such file or directory (os error 2)

which works with below:

cargo run --example quantized-qwen3 --release -- --which 4b   --prompt "A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?"

I'll create a PR to fix the docs. Same problem occurs in quantized-qwen2-instruct though...

maximizemaxwell avatar May 15 '25 02:05 maximizemaxwell