Description

This repo contains GGUF format model files for Llama3.1-8B-Fireplace2.

Files Provided

Name	Quant	Bits	File Size	Remark
llama3.1-8b-fireplace2.Q2_K.gguf	Q2_K	2	3.18 GB	2.96G, +3.5199 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q3_K.gguf	Q3_K	3	4.02 GB	3.74G, +0.6569 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q4_0.gguf	Q4_0	4	4.66 GB	4.34G, +0.4685 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q4_K.gguf	Q4_K	4	4.92 GB	4.58G, +0.1754 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q5_K.gguf	Q5_K	5	5.73 GB	5.33G, +0.0569 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q6_K.gguf	Q6_K	6	6.60 GB	6.14G, +0.0217 ppl @ Llama-3-8B
llama3.1-8b-fireplace2.Q8_0.gguf	Q8_0	8	8.54 GB	7.96G, +0.0026 ppl @ Llama-3-8B

Parameters

path	type	architecture	rope_theta	sliding_win	max_pos_embed
ValiantLabs/Llama3.1-8B-Fireplace2	llama	LlamaForCausalLM	500000.0	null	131072

Original Model Card

Fireplace 2 is a chat model, adding helpful structured outputs to Llama 3.1 8b Instruct.

an expansion pack of supplementary outputs - request them at will within your chat:
- Inline function calls
- SQL queries
- JSON objects
- Data visualization with matplotlib
Mix normal chat and structured outputs within the same conversation.
Fireplace 2 supplements the existing strengths of Llama 3.1, providing inline capabilities within the Llama 3 Instruct format.

Version

This is the 2024-07-23 release of Fireplace 2 for Llama 3.1 8b.

We're excited to bring further upgrades and releases to Fireplace 2 in the future.

Help us and recommend Fireplace 2 to your friends!

Prompting Guide

Fireplace uses the Llama 3.1 Instruct prompt format. The example script below can be used as a starting point for general chat with Llama 3.1 and also includes the different special tokens used for Fireplace 2's added features:

import transformers import torch

model_id = "ValiantLabs/Llama3.1-8B-Fireplace2"

pipeline = transformers.pipeline( "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto", )

messages = [ {"role": "system", "content": "You are Fireplace, an expert technical assistant."}, {"role": "user", "content": "Hi, can you explain local area networking to me?"}, #general Llama 3.1 chat #{"role": "user", "content": "I have the following SQL table: employees (job_id VARCHAR, salary INTEGER)\n\nCan you find all employees with a salary above $75000?<|request_sql|>"}, #for SQL query #{"role": "user", "content": "{""name"": ""get_news_headlines"",""description"": ""Get the latest news headlines"",""parameters"": {""type"": ""object"",""properties"": {""country"": {""type"": ""string"",""description"": ""The country for which news headlines are to be retrieved""}},""required"": [""country""]}}\n\nHi, can you get me the latest news headlines for the United States?<|request_function_call|>"}, # for function call #{"role": "user", "content": "Show me an example of a histogram with a fixed bin size. Use attractive colors.<|request_matplotlib|>"}, #for data visualization #{"role": "user", "content": "Can you define the word 'presence' for me, thanks!<|request_json|>"}, #for JSON output ]

outputs = pipeline( messages, max_new_tokens=512, ) print(outputs[0]["generated_text"][-1])

While Fireplace 2 is trained to minimize incorrect structured outputs, they can still occur occasionally. Production uses of Fireplace 2 should verify the structure of all model outputs and remove any unneeded components of the output.

For handling of function call responses, use the Llama 3.1 Instruct tool response style.

Special Tokens

Fireplace 2 utilizes special tokens applied to the Llama 3.1 tokenizer:

<|request_json|>
<|start_json|>
<|end_json|>
<|request_sql|>
<|start_sql|>
<|end_sql|>
<|request_matplotlib|>
<|start_matplotlib|>
<|end_matplotlib|>
<|request_function_call|>
<|start_function_call|>
<|end_function_call|>

These are supplemental to the existing special tokens used by Llama 3.1, such as <|python_tag|> and <|start_header_id|>. Fireplace 2 has been trained using the Llama 3.1 Instruct chat structure, with new special tokens added within the conversation.

The 'request' tokens are used by the user to request a specific type of structured output. They should be appended to the end of the user's message and can be alternated with normal chat responses throughout the conversation.

The Model

Fireplace 2 is built on top of Llama 3.1 8b Instruct.

This version of Fireplace 2 uses data from the following datasets:

Additional capabilities will be added to future releases.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	18.31
IFEval (0-Shot)	54.83
BBH (3-Shot)	24.07
MATH Lvl 5 (4-Shot)	5.82
GPQA (0-shot)	5.15
MuSR (0-shot)	4.38
MMLU-PRO (5-shot)	15.63

Fireplace 2 is created by Valiant Labs.

Check out our HuggingFace page for Shining Valiant 2 and our other models!

We care about open source. For everyone to use.

We encourage others to finetune further from our models.

koesn
/

Llama3.1-8B-Fireplace2-GGUF