README.md

metadata

license: apache-2.0

Huggingface RWKV EagleX 7B v2 Model

! Important Note !

The following is the HF transformers implementation of the EagleX 7B 1.7T model. This is meant to be used with the huggingface transformers For the full model weights on its own, to use with other RWKV libraries, refer to (here)[https://huggingface.co/RWKV/v5-EagleX-v2-7B-pth]

This is not an instruct tune model! (soon...)

Evaluation

The following shows the progression of the model from 1.1T trained to 2.25T trained.

Model	Eagle-7B-HF	EagleX-7B-HF-v1	EagleX-7B-HF-v2
Param Count	7.52 B	7.52 B	7.52 B
Tokens Trained	1.1 T	1.7 T	2.25 T
avg_acc	0.4822	0.5391	0.5495
glue (acc)	0.5752	0.7463	0.7439
anli (acc)	0.3594	0.4847	0.5097
mnli (acc)	0.3802	0.7928	0.7884
mnli_mismatch (acc)	0.3687	0.7985	0.784
swag (acc)	0.568	0.5814	0.5905
lambada_standard (acc)	0.685	0.686	0.7004
lambada_openai (acc)	0.7425	0.7522	0.7502
mmlu (acc)	0.3321	0.4014	0.438
winogrande (acc)	0.674	0.7206	0.7332
wnli (acc)	0.4225	0.4648	0.493
truthfulqa (acc)	0.3303	0.3268	0.3401
logiqa (acc)	0.2458	0.2458	0.2458
logiqa2 (acc)	0.2494	0.2595	0.2621
sciq (acc)	0.955	0.96	0.93
piqa (acc)	0.7704	0.7758	0.7764
arc_easy (acc)	0.7382	0.7555	0.7445
arc_challenge (acc)	0.3951	0.4087	0.4155
hellaswag (acc)	0.5264	0.5411	0.56
openbookqa (acc)	0.302	0.296	0.304
mathqa (acc)	0.26	0.26	0.2593
arithmetic (acc)	0.245	0.0634	0.1703

Compared against other top performing models in the same weight class.

Model	allenai/OLMo-7B	tiiuae/falcon-7b	Llama-2-7b-hf	EagleX-7B-HF-v2	Mistral-7B-v0.1
Param Count	6.89 B	6.92 B	6.74 B	7.52 B	7.24 B
Tokens Trained	2.5 T	1.5 T	2 T	2.25 T	2 - 7 T?
avg_acc	0.4578	0.4775	0.5045	0.5495	0.5676
glue (acc)	0.474	0.4578	0.4289	0.7439	0.515
anli (acc)	0.3478	0.3541	0.3697	0.5097	0.3803
mnli (acc)	0.3294	0.3893	0.4269	0.7884	0.4542
mnli_mismatch (acc)	0.3348	0.404	0.4395	0.784	0.4632
swag (acc)	0.5512	0.5685	0.5658	0.5905	0.5756
lambada_standard (acc)	0.6396	0.6868	0.6808	0.7004	0.6944
lambada_openai (acc)	0.6872	0.746	0.7353	0.7502	0.7553
mmlu (acc)	0.2812	0.2512	0.4077	0.438	0.5964
winogrande (acc)	0.6725	0.6709	0.6914	0.7332	0.7364
wnli (acc)	0.5775	0.4789	0.4648	0.493	0.5775
truthfulqa (acc)	0.3015	0.2826	0.3205	0.3401	0.3537
logiqa (acc)	0.2335	0.2151	0.2535	0.2458	0.2427
logiqa2 (acc)	0.2506	0.2252	0.2564	0.2621	0.3022
sciq (acc)	0.927	0.944	0.939	0.93	0.959
piqa (acc)	0.7878	0.7949	0.7807	0.7764	0.8052
arc_easy (acc)	0.7353	0.7479	0.7643	0.7445	0.8081
arc_challenge (acc)	0.3677	0.4027	0.4309	0.4155	0.5009
hellaswag (acc)	0.5572	0.5772	0.5713	0.56	0.6131
openbookqa (acc)	0.292	0.306	0.316	0.304	0.33
mathqa (acc)	0.26	0.2884	0.2801	0.2593	0.3554
arithmetic (acc)	0.0069	0.2367	0.4703	0.1703	0.9004

See the following, for the full details on this model: https://blog.rwkv.com/p/336f47bf-d8e9-4174-ac1d-02c6c8a99bc0

Links

• HF Demo
• Our wiki
• Full eval data

Running on CPU via HF transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def generate_prompt(instruction, input=""):
    instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
    input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
    if input:
        return f"""Instruction: {instruction}

Input: {input}

Response:"""
    else:
        return f"""User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: {instruction}

Assistant:"""


model = AutoModelForCausalLM.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True).to(torch.float32)
tokenizer = AutoTokenizer.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True)

text = "请介绍北京的旅游景点"
prompt = generate_prompt(text)

inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(inputs["input_ids"], max_new_tokens=333, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

output:

User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: 请介绍北京的旅游景点

Assistant: 北京是中国的首都，拥有众多的旅游景点，以下是其中一些著名的景点：
1. 故宫：位于北京市中心，是明清两代的皇宫，内有大量的文物和艺术品。
2. 天安门广场：是中国最著名的广场之一，是中国人民政治协商会议的旧址，也是中国人民政治协商会议的中心。
3. 颐和园：是中国古代皇家园林之一，有着悠久的历史和丰富的文化内涵。
4. 长城：是中国古代的一道长城，全长约万里，是中国最著名的旅游景点之一。
5. 北京大学：是中国著名的高等教育机构之一，有着悠久的历史和丰富的文化内涵。
6. 北京动物园：是中国最大的动物园之一，有着丰富的动物资源和丰富的文化内涵。
7. 故宫博物院：是中国最著名的博物馆之一，收藏了大量的文物和艺术品，是中国最重要的文化遗产之一。
8. 天坛：是中国古代皇家

Running on GPU via HF transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def generate_prompt(instruction, input=""):
    instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
    input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
    if input:
        return f"""Instruction: {instruction}

Input: {input}

Response:"""
    else:
        return f"""User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: {instruction}

Assistant:"""


model = AutoModelForCausalLM.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True, torch_dtype=torch.float16).to(0)
tokenizer = AutoTokenizer.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True)

text = "介绍一下大熊猫"
prompt = generate_prompt(text)

inputs = tokenizer(prompt, return_tensors="pt").to(0)
output = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

output:

User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: 介绍一下大熊猫

Assistant: 大熊猫是一种中国特有的哺乳动物，也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体，有着黑色的毛发和白色的耳朵。大熊猫的食物主要是竹子，它们会在竹林中寻找竹子，并且会将竹子放在竹笼中进行储存。大熊猫的寿命约为20至30年，但由于栖息地的丧失和人类活动的

Batch Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def generate_prompt(instruction, input=""):
    instruction = instruction.strip().replace('\r\n', '\n').replace('\n\n', '\n')
    input = input.strip().replace('\r\n', '\n').replace('\n\n', '\n')
    if input:
        return f"""Instruction: {instruction}

Input: {input}

Response:"""
    else:
        return f"""User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: {instruction}

Assistant:"""

model = AutoModelForCausalLM.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True).to(torch.float32)
tokenizer = AutoTokenizer.from_pretrained("RWKV/v5-Eagle-7B-HF", trust_remote_code=True)

texts = ["请介绍北京的旅游景点", "介绍一下大熊猫", "乌兰察布"]
prompts = [generate_prompt(text) for text in texts]

inputs = tokenizer(prompts, return_tensors="pt", padding=True)
outputs = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )

for output in outputs:
    print(tokenizer.decode(output.tolist(), skip_special_tokens=True))

output:

User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: 请介绍北京的旅游景点

Assistant: 北京是中国的首都，拥有丰富的旅游资源和历史文化遗产。以下是一些北京的旅游景点：
1. 故宫：位于北京市中心，是明清两代的皇宫，是中国最大的古代宫殿建筑群之一。
2. 天安门广场：位于北京市中心，是中国最著名的城市广场之一，也是中国最大的城市广场。
3. 颐和
User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: 介绍一下大熊猫

Assistant: 大熊猫是一种生活在中国中部地区的哺乳动物，也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体，有着黑色的毛发和圆圆的眼睛。大熊猫是一种濒危物种，目前只有在野外的几个保护区才能看到它们的身影。大熊猫的食物主要是竹子，它们会在竹子上寻找食物，并且可以通
User: hi

Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.

User: 乌兰察布

Assistant: 乌兰察布是中国新疆维吾尔自治区的一个县级市，位于新疆维吾尔自治区中部，是新疆的第二大城市。乌兰察布市是新疆的第一大城市，也是新疆的重要城市之一。乌兰察布市是新疆的经济中心，也是新疆的重要交通枢纽之一。乌兰察布市的人口约为2.5万人，其中汉族占绝大多数。乌

Acknowledgement

We are grateful for the help and support from the following key groups:

• Recursal.ai team for financing the GPU resources, and managing the training of this foundation model - you can run the Eagle line of RWKV models on their cloud / on-premise platform today.
• EleutherAI for their support, especially in the v5/v6 Eagle/Finch paper
• Linux Foundation AI & Data group for supporting and hosting the RWKV project