👋 join us on Discord and WeChat
## Introduction TODO ## InternLM2.5-1.8B-Chat ### Performance Evaluation TODO - The evaluation results were obtained from [OpenCompass](https://github.com/internLM/OpenCompass/) (some data marked with *, which means come from the original papers), and evaluation configuration can be found in the configuration files provided by [OpenCompass](https://github.com/internLM/OpenCompass/). - The evaluation data may have numerical differences due to the version iteration of [OpenCompass](https://github.com/internLM/OpenCompass/), so please refer to the latest evaluation results of [OpenCompass](https://github.com/internLM/OpenCompass/). **Limitations:** Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information. ### Import from Transformers To load the InternLM2.5 1.8B Chat model using Transformers, use the following code: ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2_5-1_8b-chat", trust_remote_code=True) # Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error. model = AutoModelForCausalLM.from_pretrained("internlm/internlm2_5-1_8b-chat", torch_dtype=torch.float16, trust_remote_code=True).cuda() model = model.eval() response, history = model.chat(tokenizer, "hello", history=[]) print(response) # Hello! How can I help you today? response, history = model.chat(tokenizer, "please provide three suggestions about time management", history=history) print(response) ``` The responses can be streamed using `stream_chat`: ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_path = "internlm/internlm2_5-1_8b-chat" model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16, trust_remote_code=True).cuda() tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) model = model.eval() length = 0 for response, history in model.stream_chat(tokenizer, "Hello", history=[]): print(response[length:], flush=True, end="") length = len(response) ``` ## Deployment ### LMDeploy LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. ```bash pip install lmdeploy ``` You can run batch inference locally with the following python code: ```python import lmdeploy pipe = lmdeploy.pipeline("internlm/internlm2_5-1_8b-chat") response = pipe(["Hi, pls intro yourself", "Shanghai is"]) print(response) ``` Or you can launch an OpenAI compatible server with the following command: ```bash lmdeploy serve api_server internlm/internlm2_5-1_8b-chat --model-name internlm2_5-1_8b-chat --server-port 23333 ``` Then you can send a chat request to the server: ```bash curl http://localhost:23333/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "internlm2_5-1_8b-chat", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Introduce deep learning to me."} ] }' ``` Find more details in the [LMDeploy documentation](https://lmdeploy.readthedocs.io/en/latest/) ### vLLM Launch OpenAI compatible server with `vLLM>=0.3.2`: ```bash pip install vllm ``` ```bash python -m vllm.entrypoints.openai.api_server --model internlm/internlm2_5-1_8b-chat --served-model-name internlm2_5-1_8b-chat --trust-remote-code ``` Then you can send a chat request to the server: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "internlm2_5-1_8b-chat", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Introduce deep learning to me."} ] }' ``` Find more details in the [vLLM documentation](https://docs.vllm.ai/en/latest/index.html) ## Open Source License The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. To apply for a commercial license, please fill in the [application form (English)](https://wj.qq.com/s2/12727483/5dba/)/[申请表（中文）](https://wj.qq.com/s2/12725412/f7c1/). For other questions or collaborations, please contact