Lei comments

Repositories
Issues
Comments

Results 2 comments of

Lei

How to analyze large models like llama 3 70B that requires model parallelism?

I was able to use perf_analyzer to instrument llama 3 70b on 4*A100 (trtllm backend) via a launched triton server as below ```shell python3 scripts/launch_triton_server.py --world_size 4 --model_repo=llama_ifb/ perf_analyzer -m...

Bug Report API Error: 400 `tool_use` ids were found without `tool_result` blocks immediately after...

1.0.48 hit the same issue today.