Edit model card

Model Overview:

  • Training Data: This model was trained on a dataset with columns for initial reasoning, step-by-step thinking, verifications after each step, and final answers based on full context. Is it better than the original base model? Hard to say without proper evaluations, and I don’t have the resources to run them manually.

  • Context Handling: The model benefits from larger contexts (minimum 4k up to 16k tokens, though it was trained on 32k tokens). It tends to "overthink," so providing a longer context helps it perform better.

  • Performance: Based on my very few manual tests, the model seems to excel in conversational settingsβ€”especially for mental health, creative tasks and explaining stuff. However, I encourage you to try it out yourself using this Colab Notebook.

  • Dataset Note: The publicly available dataset is only a partial version. The full dataset was originally designed for a custom Mixture of Experts (MoE) architecture, but I couldn't afford to run the full experiment.

  • Acknowledgment: Special thanks to KingNish for reigniting my passion to revisit this project. I almost abandoned it after my first attempt a month ago. Enjoy this experimental model!

Inference Code:

  • Feel free to make the steps and verifications collapsable and the initial reasoning too, you can show only the final answer to get an o1 feel(i don't know)
  • Note: A feature we have here is the ability to control how many steps and verifications you want.
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Lyte/Llama-3.2-3B-Overthinker"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

def generate_response(prompt, max_tokens=16384, temperature=0.8, top_p=0.95, repeat_penalty=1.1, num_steps=3):
    messages = [{"role": "user", "content": prompt}]

    # Generate reasoning
    reasoning_template = tokenizer.apply_chat_template(messages, tokenize=False, add_reasoning_prompt=True)
    reasoning_inputs = tokenizer(reasoning_template, return_tensors="pt").to(model.device)

    reasoning_ids = model.generate(
        **reasoning_inputs,
        max_new_tokens=max_tokens // 3,
        temperature=temperature,
        top_p=top_p,
        repetition_penalty=repeat_penalty
    )
    reasoning_output = tokenizer.decode(reasoning_ids[0, reasoning_inputs.input_ids.shape[1]:], skip_special_tokens=True)
    
    # Generate thinking (step-by-step and verifications)
    messages.append({"role": "reasoning", "content": reasoning_output})
    thinking_template = tokenizer.apply_chat_template(messages, tokenize=False, add_thinking_prompt=True, num_steps=num_steps)
    thinking_inputs = tokenizer(thinking_template, return_tensors="pt").to(model.device)

    thinking_ids = model.generate(
        **thinking_inputs,
        max_new_tokens=max_tokens // 3,
        temperature=temperature,
        top_p=top_p,
        repetition_penalty=repeat_penalty
    )
    thinking_output = tokenizer.decode(thinking_ids[0, thinking_inputs.input_ids.shape[1]:], skip_special_tokens=True)
    
    # Generate final answer
    messages.append({"role": "thinking", "content": thinking_output})
    answer_template = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    answer_inputs = tokenizer(answer_template, return_tensors="pt").to(model.device)

    answer_ids = model.generate(
        **answer_inputs,
        max_new_tokens=max_tokens // 3,
        temperature=temperature,
        top_p=top_p,
        repetition_penalty=repeat_penalty
    )
    answer_output = tokenizer.decode(answer_ids[0, answer_inputs.input_ids.shape[1]:], skip_special_tokens=True)
    return reasoning_output, thinking_output, answer_output

# Example usage:
prompt = "Explain the process of photosynthesis."
response = generate_response(prompt, num_steps=5)

print("Response:", response)

Uploaded model

  • Developed by: Lyte
  • License: apache-2.0
  • Finetuned from model : unsloth/llama-3.2-3b-instruct-bnb-4bit

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

Notice:

  • The problem with runnning evals is that they won't make use of the correct template and it won't be a true eval then would it? so these barely test the model.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 19.00
IFEval (0-Shot) 64.08
BBH (3-Shot) 20.10
MATH Lvl 5 (4-Shot) 2.64
GPQA (0-shot) 1.23
MuSR (0-shot) 3.90
MMLU-PRO (5-shot) 22.06
Downloads last month
412
Safetensors
Model size
3.21B params
Tensor type
FP16
Β·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for Lyte/Llama-3.2-3B-Overthinker

Merges
4 models
Quantizations
5 models

Dataset used to train Lyte/Llama-3.2-3B-Overthinker

Spaces using Lyte/Llama-3.2-3B-Overthinker 2

Evaluation results