Qwen

/

Qwen2-7B-Instruct-GPTQ-Int8

like 16

I tried vllm and without vllm, "RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 during inference with transformer" still exist!