File size: 3,535 Bytes
70339c5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
# Planck-OpenLAiNN 🤗
Hey there fellow researchers, developers, and AI enthusiasts! Today I'm releasing a new family of Models, Planck LAiNN, These are probably some of the smallest LLMs that are on HF. They aren't super useful but it was a fun expierment!~
## Models Overview
- **Panck-OpenLAiNN-10M**: A Truely Tiny model with just 10 Million parameters, this is probably boarderline useless, but it *IS* functional.
- **Panck-OpenLAiNN-25M**: The second smallest model, 25 million parameters, it's not that much better.
- **Panck-OpenLAiNN-50M**: Surprisingly smart, it's 50 Million parameters and could potentially maybe, Possibly even be useful ;)
- **Panck-OpenLAiNN-75M**: The current *""heavy""* weight of the Plank-OpenLAiNN Models.
## Pretraining Details
Plank-OpenLAiNN was trained on 32B tokens of the Fineweb dataset, it's the same one that was used for the Pico-LAiNN family of models. The model was pretrained with a context length of 1024 tokens.
## Other information:
- **Compatibility**: Built to be compatible with existing projects that use LLAMA 2's tokenizer and architecture.
- **Ease of Use**: No need to reinvent the wheel. These models are ready to be plugged into your applications.
- **Open Source**: Fully open source, so you can tweak, tune, and twist them to your heart's content.
## Getting Started
To start using these models, you can simply load them via the Hugging Face `transformers` library:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = "UUFO-Aigis/Panck-OpenLAiNN-25M"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)
def generate_text(prompt, model, tokenizer, max_length=512, temperature=1, top_k=50, top_p=0.95):
inputs = tokenizer.encode(prompt, return_tensors="pt")
outputs = model.generate(
inputs,
max_length=max_length,
temperature=temperature,
top_k=top_k,
top_p=top_p,
do_sample=True
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return generated_text
def main():
# Define your prompt
prompt = "According to all known laws of aviation, there is no way a bee should be able to fly."
generated_text = generate_text(prompt, model, tokenizer)
print(generated_text)
if __name__ == "__main__":
main()
```
# Benchy
| Tasks | Value | |Stderr|
|--------------|------:|---|-----:|
|arc_challenge | 0.1817|± |0.0113|
|arc_easy | 0.3291|± |0.0096|
|boolq | 0.6138|± |0.0085|
|hellaswag | 0.2700|± |0.0044|
|lambada_openai| 0.1104|± |0.0044|
|piqa | 0.5740|± |0.0115|
|winogrande | 0.5170|± |0.0140|
## Future Plans
- **More Models**: I'm currenetly training the bigger siblings of Pico-OpenLAiNN, including a 1B parameter version and beyond. 2-4 Billion parameter versions are planned. These will be Released as OpenLAiNN.
- **New architecture**: This is still up in the air and I'm still developing it, things are going well and I'll post updates.
- **Paper**: A detailed paper or training data will be posted at some point.
## Credit Where Credit's Due
If you find these models useful and decide to use these models, a link to this repository would be highly appreciated. I am a one man show running this, Thanks 🤗
## Contact
If you have questions, Please reach out to me at urlsys32dll@gmail.com
<p align="center">
<img src="UUFO.png" alt="U.U.F.O Research Logo" width="250"/>
</p>
|