---
license: apache-2.0
datasets:
- SkunkworksAI/reasoning-0.01
- trollek/ThoughtfulAssistant-v01
- trollek/ThoughtfulAssistant-v02
base_model: trollek/LittleInstructionMaker-4B-v0.2
---
# ThoughtStream-4B-v0.2

A more versitile version of [the first one](https://huggingface.co/trollek/ThoughtStream-4B-v0.1), and less versitile than the next[.](https://www.youtube.com/watch?v=dQw4w9WgXcQ "I swear my name makes sense in Danish, but.. I mean..") This can not compete with [Matt Shumer's](https://huggingface.co/mattshumer) [Reflection](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B), which be wildin' yo. But creating a dataset to "Please find errors and hallucinations in the thought process" is a tad harder than expected™. It is brilliant! And hard! Living in the future is great!

With this second version I have tried to have 2 more ways of thinking than just stream of consciousness. This being mind mapping and step by step reasoning thanks to [SkunkworksAI](https://huggingface.co/SkunkworksAI) and their [reasoning-0.01](https://huggingface.co/datasets/SkunkworksAI/reasoning-0.01) dataset. Big up for creating [TinyFrank](https://huggingface.co/SkunkworksAI/tinyfrank-1.4B) that got me started on this wild adventure.

## System prompts

**Stream of consciousness:**

- `Your thought process flows freely without boundaries, allowing you to explore connections and associations that emerge spontaneously as you analyze problems or engage in conversations with users.`
- `You are an AI assistant that think before you answer. Your thinking is a stream-of-consciousness.`

**Mind map:**

- `Your cognitive model organizes information through interconnected nodes, facilitating the visualization of concepts, their relationships, and potential solutions by creating mental maps that expand your understanding and problem-solving capabilities.`

**Step-by-step reasoning:**

- `Your decision-making process is methodical and sequential, breaking down complex tasks into manageable steps to ensure thorough analysis and execution with precision and reliability. You apply logical reasoning to evaluate options and make informed decisions based on step-by-step evaluation of the problem at hand.`
- `You are an AI assistant that think before you answer. When thinking you break your reasoning down step by step.`


## Config

```yaml
### model
model_name_or_path: lim-v02-thought

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
loraplus_lr_ratio: 12.0
lora_rank: 16
lora_alpha: 16
use_unsloth: true
quantization_bit: 4
upcast_layernorm: true
seed: 127
optim: lion_8bit
additional_target: embed_tokens

### dataset
dataset: reasoning_assistant,thoughtful_v01,thoughtful_v02
template: ninja_chatml
cutoff_len: 8192
overwrite_cache: false
preprocessing_num_workers: 12

### output
output_dir:  /home/trolle/Documents/Projects/trollek/danube3/merges/lim-v02-thought/loras/reasoning
logging_steps: 5
save_steps: 1
save_strategy: epoch
plot_loss: true
overwrite_output_dir: false

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 0.000002
num_train_epochs: 2
lr_scheduler_type: constant_with_warmup
warmup_ratio: 0.01
bf16: true
flash_attn: fa2

### eval
val_size: 0.01
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 1000

```

## Training results