README.md

---
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
language:
- en
library_name: transformers
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This is a Qwen2.5 0.5B Instruct model which got fine-tuned on a dataset generated by Monte Carlo Tree Search based sampling.
MCTS was rolled out on a small subset of the GSM8K train split. The resulting traces & value estimates were then used to form the dataset.
Only the last two transformer blocks and the regression head were unfroozen. 

The idea is to use only the value network to do MCTS sampling, without the need of simulating/rolling out. 

Currently the value network is overfitting, due to very limited samples. Going to update this soon, when I've sampled more data. 



### Scores on the first 65 samples of the gsm8k test-split:

- Beam-search (3 beams): 40.0%
- MCTS-search (3 beams): 50.77%


The final rollout of the MCTS-search is done also via Beam-serach. During testing on gsm8k, only the value network was used to guide the search.



All tests were done with Qwen2.5 0.5B Instruct.