Mistral-7B-OpenOrca-lora-merged

This is a test.

This is a regenerated model that combines the base model Mistral-7B-v0.1 with the LoRA model Mistral-7B-OpenOrca-lora.

This LoRA model is extracted from the efficient parameter fine-tuned model (Mistral-7B-OpenOra), and now it needs to be verified whether this LoRA model can achieve comparable performance with the original model.

The final goal is to create a toolkit that can simultaneously load multiple LoRA modules, and automatically switch to the appropriate combination of LoRA modules based on user queries to generate the best answer.

The source code is here

Mistral-7B-OpenOrca

• Extract lora model Mistral-7B-OpenOrca-lora from Mistral-7B-OpenOrca;

• Merge the base model Mistral-7B-v0.1 with lora model to Mistral-7B-OpenOrca-lora-merged

• LLM Evaluation ...

Local Test

	ARC_acc_norm (25-shot)	HellaSwag_acc_norm (10-shot)	MMLU_acc (5-shot)	TruthfulQA_mc2 (0-shot)	GSM8K_acc (8-shot)	Open LLM Score
Mistral-7B-OpenOrca	71	83	61.42	45	40	65.11
r=256	68	84	64.28	46.953	41	65.81
r=64	67	84	64.26	47.32	41	65.65
r=16	65	83	62.84	46.95	38	64.45

Open LLM Leaderboard

	ARC_acc_norm (25-shot)	HellaSwag_acc_norm (10-shot)	MMLU_acc (5-shot)	TruthfulQA_mc2 (0-shot)	Open LLM Score
Mistral-7B-SlimOrca	62.54	83.86	62.77	54.23	65.85
Mistral-7B-OpenOrca	64.08	83.99	62.24	53.05	65.84

lm-evaluation-harness

Open LLM Leaderboard

Metric	Mistral-7B-OpenOrca	Mistral-7B-OpenOrca-lora	Mistral-7B-OpenOrca-lora-merged
ARC	64.08
HellaSwag	83.99
MMLU	62.24
TruthfulQA	53.05
Average	65.84

HumanEval

Metric	Mistral-7B-OpenOrca	Mistral-7B-OpenOrca-lora	Mistral-7B-OpenOrca-lora-merged
humaneval-python	35.976

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16

Framework versions

• PEFT 0.5.0