| # Planck-OpenLAiNN-25M 🤗 | |
| Hey there fellow researchers, developers, and AI enthusiasts! Today I'm releasing a new family of Models, Planck LAiNN, These are probably some of the smallest LLMs that are on HF. They aren't super useful but it was a fun expierment!~ | |
| These are the GGUF quants of the models. For the original models, you can find them [here](https://huggingface.co/UUFO-Aigis/Planck-OpenLAiNN-25M-gguf). | |
| ## Models Overview | |
| - **Panck-OpenLAiNN-10M**: A Truely Tiny model with just 10 Million parameters, this is probably boarderline useless, but it *IS* functional. | |
| - **Panck-OpenLAiNN-25M**: The second smallest model, 25 million parameters, it's not that much better. | |
| - **Panck-OpenLAiNN-50M**: Surprisingly smart, it's 50 Million parameters and could potentially maybe, Possibly even be useful ;) | |
| - **Panck-OpenLAiNN-75M**: The current *""heavy""* weight of the Plank-OpenLAiNN Models. | |
| ## Pretraining Details | |
| Plank-OpenLAiNN was trained on 32B tokens of the Fineweb dataset, it's the same one that was used for the Pico-LAiNN family of models. The model was pretrained with a context length of 1024 tokens. | |
| ## Other information: | |
| - **Compatibility**: Built to be compatible with existing projects that use LLAMA 2's tokenizer and architecture. | |
| - **Ease of Use**: No need to reinvent the wheel. These models are ready to be plugged into your applications. | |
| - **Open Source**: Fully open source, so you can tweak, tune, and twist them to your heart's content. | |
| ## Getting Started | |
| To start using these models, you can simply load them via the Hugging Face `transformers` library: | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| MODEL_NAME = "UUFO-Aigis/Panck-OpenLAiNN-25M" | |
| tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) | |
| model = AutoModelForCausalLM.from_pretrained(MODEL_NAME) | |
| def generate_text(prompt, model, tokenizer, max_length=512, temperature=1, top_k=50, top_p=0.95): | |
| inputs = tokenizer.encode(prompt, return_tensors="pt") | |
| outputs = model.generate( | |
| inputs, | |
| max_length=max_length, | |
| temperature=temperature, | |
| top_k=top_k, | |
| top_p=top_p, | |
| do_sample=True | |
| ) | |
| generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) | |
| return generated_text | |
| def main(): | |
| # Define your prompt | |
| prompt = "According to all known laws of aviation, there is no way a bee should be able to fly." | |
| generated_text = generate_text(prompt, model, tokenizer) | |
| print(generated_text) | |
| if __name__ == "__main__": | |
| main() | |
| ``` | |
| # Benchy | |
| | Tasks | Value | |Stderr| | |
| |--------------|------:|---|-----:| | |
| |arc_challenge | 0.1817|± |0.0113| | |
| |arc_easy | 0.3291|± |0.0096| | |
| |boolq | 0.6138|± |0.0085| | |
| |hellaswag | 0.2700|± |0.0044| | |
| |lambada_openai| 0.1104|± |0.0044| | |
| |piqa | 0.5740|± |0.0115| | |
| |winogrande | 0.5170|± |0.0140| | |
| ## Future Plans | |
| - **More Models**: I'm currenetly training the bigger siblings of Pico-OpenLAiNN, including a 1B parameter version and beyond. 2-4 Billion parameter versions are planned. These will be Released as OpenLAiNN. | |
| - **New architecture**: This is still up in the air and I'm still developing it, things are going well and I'll post updates. | |
| - **Paper**: A detailed paper or training data will be posted at some point. | |
| ## Credit Where Credit's Due | |
| If you find these models useful and decide to use these models, a link to this repository would be highly appreciated. I am a one man show running this, Thanks 🤗 | |
| ## Contact | |
| If you have questions, Please reach out to me at urlsys32dll@gmail.com | |
| <p align="center"> | |
| <img src="UUFO.png" alt="U.U.F.O Research Logo" width="250"/> | |
| </p> | |