README.md

---
license: llama2
language:
- hu
- en
tags:
- puli
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- finetuned
base_model: NYTK/PULI-LlumiX-32K
datasets:
- boapps/szurkemarha
pipeline_tag: text-generation
---

# PULI LlumiX 32K instruct (6.74B billion parameter)

Intruct finetuned version of NYTK/PULI-LlumiX-32K.

## Training platform
[Lightning AI Studio](https://lightning.ai/studios) L4 GPU

## Hyper parameters

- Epoch: 3
- LoRA rank (r): 16
- LoRA alpha: 16
- Lr: 2e-4
- Lr scheduler: cosine
- Optimizer: adamw_8bit
- Weight decay: 0.01

## Dataset

boapps/szurkemarha

In total ~30k instructions were selected.

## Prompt template: ChatML
```
<|im_start|>system
Az alábbiakban egy feladatot leíró utasítás található. Írjál olyan választ, amely megfelelően teljesíti a kérést.<|im_end|>
<|im_start|>user
Ki a legerősebb szuperhős?<|im_end|>
<|im_start|>assistant
A legerősebb szuperhős a Marvel univerzumában Hulk.<|im_end|>
```

## Base model

- Trained with OpenChatKit [github](https://github.com/togethercomputer/OpenChatKit)
- The [LLaMA-2-7B-32K](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K) model were continuously pretrained on Hungarian dataset
- The model has been extended to a context length of 32K with position interpolation
- Checkpoint: 100 000 steps

## Dataset for continued pretraining

- Hungarian: 7.9 billion words, documents (763K) that exceed 5000 words in length
- English: Long Context QA (2 billion words), BookSum (78 million words)

## Limitations

- max_seq_length = 32 768
- float16
- vocab size: 32 000