--- language: - en license: apache-2.0 model-index: - name: Solar-10.7B-SLERP results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 70.73 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 87.87 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 65.77 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 65.72 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 82.48 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 61.26 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=jan-ai/Solar-10.7B-SLERP name: Open LLM Leaderboard --- # WARNING This is a model file only for evaluation. Please use the model here: - Model: [Solar-10.7B-SLERP](https://huggingface.co/janhq/Solar-10.7B-SLERP) - GGUF: [Solar-10.7B-SLERP-GGUF](https://huggingface.co/janhq/Solar-10.7B-SLERP-GGUF)
Jan banner

Jan - Discord

# Model Description This model uses the `Slerp` merge method from the best models on 14th Dec on the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard): 1. [upstage/SOLAR-10.7B-Instruct-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0) 2. [janhq/Pandora-v1-10.7B](https://huggingface.co/janhq/Pandora-v1-10.7B) - base model: [upstage/SOLAR-10.7B-Instruct-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0) The yaml config file for this model is here: ```yaml slices: - sources: - model: upstage/SOLAR-10.7B-Instruct-v1.0 layer_range: [0, 48] - model: janhq/Pandora-v1-10.7B layer_range: [0, 48] merge_method: slerp base_model: upstage/SOLAR-10.7B-Instruct-v1.0 parameters: t: - filter: self_attn value: [0, 0.5, 0.3, 0.7, 1] - filter: mlp value: [1, 0.5, 0.7, 0.3, 0] - value: 0.5 dtype: bfloat16 ``` # Prompt template - **ChatML** ``` <|im_start|>system {system_message}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` # Run this model You can run this model using [Jan](https://jan.ai/) on Mac, Windows, or Linux. **Jan is an open source, ChatGPT alternative that is:** 💻 **100% offline on your machine**: Your conversations remain confidential, and visible only to you. 🗂️ **An Open File Format**: Conversations and model settings stay on your computer and can be exported or deleted at any time. 🌐 **OpenAI Compatible**: Local server on port ` 1337` with OpenAI compatible endpoints 🌍 **Open Source & Free**: We build in public; check out our [Github](https://github.com/janhq ) - Please use the [Solar-10.7B-SLERP-GGUF](https://huggingface.co/janhq/Solar-10.7B-SLERP-GGUF) when using on Jan. ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65713d70f56f9538679e5a56/r7VmEBLGXpPLTu2MImM7S.png) # About Jan Jan believes in the need for an open-source AI ecosystem and is building the infra and tooling to allow open-source AIs to compete on a level playing field with proprietary ones. Jan's long-term vision is to build a cognitive framework for future robots, who are practical, useful assistants for humans and businesses in everyday life. # Jan Model Merger This is a test project for merging models. # Open LLM Leaderboard Evaluation Results Detailed results can be found here. | Metric | Value | |-----------------------|---------------------------| | Avg. | ?| | ARC (25-shot) | ? | | HellaSwag (10-shot) | ? | | MMLU (5-shot) | ?| | TruthfulQA (0-shot) | ? | | Winogrande (5-shot) | ? | | GSM8K (5-shot) | ? | # Acknowlegement - [mergekit](https://github.com/cg123/mergekit) - [DARE](https://github.com/yule-BUAA/MergeLM/blob/main/README.md) - [SLERP](https://github.com/Digitous/LLM-SLERP-Merge) - [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_jan-ai__Solar-10.7B-SLERP) | Metric |Value| |---------------------------------|----:| |Avg. |72.31| |AI2 Reasoning Challenge (25-Shot)|70.73| |HellaSwag (10-Shot) |87.87| |MMLU (5-Shot) |65.77| |TruthfulQA (0-shot) |65.72| |Winogrande (5-shot) |82.48| |GSM8k (5-shot) |61.26|