Model Overview:
Training Data: This model was trained on a dataset with columns for initial reasoning, step-by-step thinking, verifications after each step, and final answers based on full context. Is it better than the original base model? Hard to say without proper evaluations, and I donβt have the resources to run them manually.
Context Handling: The model benefits from larger contexts (minimum 4k up to 16k tokens, though it was trained on 32k tokens). It tends to "overthink," so providing a longer context helps it perform better.
Performance: Based on my very few manual tests, the model seems to excel in conversational settingsβespecially for mental health, creative tasks and explaining stuff. However, I encourage you to try it out yourself using this Colab Notebook.
Dataset Note: The publicly available dataset is only a partial version. The full dataset was originally designed for a custom Mixture of Experts (MoE) architecture, but I couldn't afford to run the full experiment.
Acknowledgment: Special thanks to KingNish for reigniting my passion to revisit this project. I almost abandoned it after my first attempt a month ago. Enjoy this experimental model!
Inference Code:
- Feel free to make the steps and verifications collapsable and the initial reasoning too, you can show only the final answer to get an o1 feel(i don't know)
- Note: A feature we have here is the ability to control how many steps and verifications you want.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Lyte/Llama-3.2-3B-Overthinker"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
def generate_response(prompt, max_tokens=16384, temperature=0.8, top_p=0.95, repeat_penalty=1.1, num_steps=3):
messages = [{"role": "user", "content": prompt}]
# Generate reasoning
reasoning_template = tokenizer.apply_chat_template(messages, tokenize=False, add_reasoning_prompt=True)
reasoning_inputs = tokenizer(reasoning_template, return_tensors="pt").to(model.device)
reasoning_ids = model.generate(
**reasoning_inputs,
max_new_tokens=max_tokens // 3,
temperature=temperature,
top_p=top_p,
repetition_penalty=repeat_penalty
)
reasoning_output = tokenizer.decode(reasoning_ids[0, reasoning_inputs.input_ids.shape[1]:], skip_special_tokens=True)
# Generate thinking (step-by-step and verifications)
messages.append({"role": "reasoning", "content": reasoning_output})
thinking_template = tokenizer.apply_chat_template(messages, tokenize=False, add_thinking_prompt=True, num_steps=num_steps)
thinking_inputs = tokenizer(thinking_template, return_tensors="pt").to(model.device)
thinking_ids = model.generate(
**thinking_inputs,
max_new_tokens=max_tokens // 3,
temperature=temperature,
top_p=top_p,
repetition_penalty=repeat_penalty
)
thinking_output = tokenizer.decode(thinking_ids[0, thinking_inputs.input_ids.shape[1]:], skip_special_tokens=True)
# Generate final answer
messages.append({"role": "thinking", "content": thinking_output})
answer_template = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
answer_inputs = tokenizer(answer_template, return_tensors="pt").to(model.device)
answer_ids = model.generate(
**answer_inputs,
max_new_tokens=max_tokens // 3,
temperature=temperature,
top_p=top_p,
repetition_penalty=repeat_penalty
)
answer_output = tokenizer.decode(answer_ids[0, answer_inputs.input_ids.shape[1]:], skip_special_tokens=True)
return reasoning_output, thinking_output, answer_output
# Example usage:
prompt = "Explain the process of photosynthesis."
response = generate_response(prompt, num_steps=5)
print("Response:", response)
Uploaded model
- Developed by: Lyte
- License: apache-2.0
- Finetuned from model : unsloth/llama-3.2-3b-instruct-bnb-4bit
This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.
Notice:
- The problem with runnning evals is that they won't make use of the correct template and it won't be a true eval then would it? so these barely test the model.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 19.00 |
IFEval (0-Shot) | 64.08 |
BBH (3-Shot) | 20.10 |
MATH Lvl 5 (4-Shot) | 2.64 |
GPQA (0-shot) | 1.23 |
MuSR (0-shot) | 3.90 |
MMLU-PRO (5-shot) | 22.06 |
- Downloads last month
- 412
Model tree for Lyte/Llama-3.2-3B-Overthinker
Dataset used to train Lyte/Llama-3.2-3B-Overthinker
Spaces using Lyte/Llama-3.2-3B-Overthinker 2
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard64.080
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard20.100
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard2.640
- acc_norm on GPQA (0-shot)Open LLM Leaderboard1.230
- acc_norm on MuSR (0-shot)Open LLM Leaderboard3.900
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard22.060