candle
candle copied to clipboard
Using the `--model` parameter with the quantized-qwen3 example fails
Using the --model parameter with the quantized-qwen3 example results in an error e.g.
cargo run --example quantized-qwen3 --release --features cuda -- --model 4b --prompt "A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?"
Finished `release` profile [optimized] target(s) in 0.42s
Running `target/release/examples/quantized-qwen3 --model 4b --prompt 'A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?'`
avx: false, neon: true, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: No such file or directory (os error 2)
It works fine with no --model parameter using the default 0.6b model.
Running on ARM Linux on an nVidia Jetson Orin AGX.
It seems the argument is called --which 4b and not --model as indicated in the doc.
yup, I also received error message:
avx: true, neon: false, simd128: false, f16c: true
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Error: No such file or directory (os error 2)
which works with below:
cargo run --example quantized-qwen3 --release -- --which 4b --prompt "A train is travelling at 120mph, how far does it travel in 3 minutes 30 seconds?"
I'll create a PR to fix the docs. Same problem occurs in quantized-qwen2-instruct though...