--- license: apache-2.0 language: - en datasets: - appvoid/no-prompt-15k pipeline_tag: text-generation --- ![palmer](https://huggingface.co/appvoid/no-prompt-1.3b/resolve/main/_ccd1a5dd-2ddc-4d5a-8163-fd6d1b39f5f4.jpeg?download=true) # no-prompt ### a sheared-llama-1.3b fine-tuning This model uses an 1.3 billion parameters model as base to be further fine-tuned on the same data as palmer. It works pretty good and even surpasses sota model on `hellaswag`. ### evaluation |Model| ARC_C| HellaSwag| PIQA| Winogrande| |------|-----|-----------|------|-------------| |tinyllama-2t| 0.2807| 0.5463| 0.7067| 0.5683| |palmer-001 | 0.2807| 0.5524| 0.7106| 0.5896| |sheared-1.3b| 0.2910| 0.5935| 0.7339| 0.5809| |no-prompt-1.3b| 0.3157| **0.6022**| 0.7334| 0.5864| |falcon-rw-1b-instruct-openorca (sota) | **0.3362**| 0.5997| **0.7394**| **0.6148**| This model was trained on less than 25% of the dataset yet achieves competitive performance to current sota on open llm leaderboard. ### training Training took ~5 P100 gpu hours. It was trained on 15,000 gpt-4 shuffled samples. palmer was fine-tuned using lower learning rates ensuring it keeps as much general knowledge as possible. ### prompt ``` no prompt ``` ### limitations Hallucinations are frequent, just as any transformer model this size.