|
--- |
|
license: apache-2.0 |
|
tags: |
|
- text-generation-inference |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
<div align="center"> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64ccdc322e592905f922a06e/VhwQtaklohkUXFWkjA-3M.png" width="450"/> |
|
|
|
English | [简体ä¸æ–‡](README_zh-CN.md) |
|
|
|
</div> |
|
|
|
<p align="center"> |
|
👋 join us on <a href="https://twitter.com/intern_lm" target="_blank">Twitter</a>, <a href="https://discord.gg/xa29JuW87d" target="_blank">Discord</a> and <a href="https://r.vansin.top/?r=internwx" target="_blank">WeChat</a> |
|
</p> |
|
|
|
|
|
# W4A16 LLM Model Deployment |
|
|
|
LMDeploy supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80. |
|
|
|
Before proceeding with the inference, please ensure that lmdeploy(>=v0.0.4) is installed. |
|
|
|
```shell |
|
pip install lmdeploy |
|
``` |
|
|
|
## 4-bit LLM model Inference |
|
|
|
You can download the pre-quantized 4-bit weight models from LMDeploy's [model zoo](https://huggingface.co/lmdeploy) and conduct inference using the following command. |
|
|
|
Alternatively, you can quantize 16-bit weights to 4-bit weights following the ["4-bit Weight Quantization"](#4-bit-weight-quantization) section, and then perform inference as per the below instructions. |
|
|
|
|
|
```shell |
|
git-lfs install |
|
git clone https://huggingface.co/lmdeploy/internlm-chat-7b-w4 |
|
``` |
|
|
|
As demonstrated in the command below, first convert the model's layout using `turbomind.deploy`, and then you can interact with the AI assistant in the terminal |
|
|
|
```shell |
|
|
|
## Convert the model's layout and store it in the default path, ./workspace. |
|
python3 -m lmdeploy.serve.turbomind.deploy \ |
|
--model-name internlm \ |
|
--model-path ./internlm-chat-7b-w4 \ |
|
--model-format awq \ |
|
--group-size 128 |
|
|
|
## inference |
|
python3 -m lmdeploy.turbomind.chat ./workspace |
|
``` |
|
|
|
## Serve with gradio |
|
|
|
If you wish to interact with the model via web ui, please initiate the gradio server as indicated below: |
|
|
|
```shell |
|
python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} ----server_port {port} |
|
``` |
|
|
|
Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model |
|
|
|
## Inference Performance |
|
|
|
We benchmarked the Llama 2 7B and 13B with 4-bit quantization on NVIDIA GeForce RTX 4090 using [profile_generation.py](https://github.com/InternLM/lmdeploy/blob/main/benchmark/profile_generation.py). And we measure the token generation throughput (tokens/s) by setting a single prompt token and generating 512 tokens. All the results are measured for single batch inference. |
|
|
|
| model | llm-awq | mlc-llm | turbomind | |
|
| ----------- | ------- | ------- | --------- | |
|
| Llama 2 7B | 112.9 | 159.4 | 206.4 | |
|
| Llama 2 13B | N/A | 90.7 | 115.8 | |
|
|
|
```shell |
|
python benchmark/profile_generation.py \ |
|
./workspace \ |
|
--concurrency 1 --input_seqlen 1 --output_seqlen 512 |
|
``` |
|
|
|
## 4-bit Weight Quantization |
|
|
|
It includes two steps: |
|
|
|
- generate quantization parameter |
|
- quantize model according to the parameter |
|
|
|
### Step 1: Generate Quantization Parameter |
|
|
|
```shell |
|
python3 -m lmdeploy.lite.apis.calibrate \ |
|
--model $HF_MODEL \ |
|
--calib_dataset 'c4' \ # Calibration dataset, supports c4, ptb, wikitext2, pileval |
|
--calib_samples 128 \ # Number of samples in the calibration set, if memory is insufficient, you can appropriately reduce this |
|
--calib_seqlen 2048 \ # Length of a single piece of text, if memory is insufficient, you can appropriately reduce this |
|
--work_dir $WORK_DIR \ # Folder storing Pytorch format quantization statistics parameters and post-quantization weight |
|
``` |
|
|
|
### Step2: Quantize Weights |
|
|
|
LMDeploy employs AWQ algorithm for model weight quantization. |
|
|
|
```shell |
|
python3 -m lmdeploy.lite.apis.auto_awq \ |
|
--model $HF_MODEL \ |
|
--w_bits 4 \ # Bit number for weight quantization |
|
--w_sym False \ # Whether to use symmetric quantization for weights |
|
--w_group_size 128 \ # Group size for weight quantization statistics |
|
--work_dir $WORK_DIR \ # Directory saving quantization parameters from Step 1 |
|
``` |
|
|
|
After the quantization is complete, the quantized model is saved to `$WORK_DIR`. Then you can proceed with model inference according to the instructions in the ["4-Bit Weight Model Inference"](#4-bit-llm-model-inference) section. |
|
|
