File size: 2,060 Bytes
316218e 07a82d7 316218e 07a82d7 316218e 2602fea 07a82d7 316218e 07a82d7 316218e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
---
license: apache-2.0
language:
- nl
library_name: transformers
---
[Pieter Delobelle](https://pieter.ai), [François Remy](https://fremycompany.com), [Miryam de Lhoneux](https://people.cs.kuleuven.be/~miryam.delhoneux/), [Thomas Demeester](https://tdmeeste.github.io)
<p align="center">
<img src="https://huggingface.co/DTAI-KULeuven/tweety-7b-dutch/resolve/main/tweety-7b-dutch.png?download=true" alt="Tweety-7b-dutch: A Dutch Large Language Model" width="20%">
</p>
# Model Card for tweety-7b-dutch
tweety-7b-dutch is a foundation model with a focus on the Dutch language, incorporating a [Dutch tokenizer](https://huggingface.co/yhavinga/gpt-neo-1.3B-dutch) for better understanding and generation of Dutch text. It's built on the mistral architecture, employing flash attention for efficient processing within a context window of 8192 tokens. Tweety-7b-dutch is trained on the [cleaned Dutch mC4 dataset](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned), without instruction finetuning.
## Model Details
### Model Description
Our tweety-7b-dutch model has an Apache 2.0 license, encouraging applications in research, content creation, and language analysis.
- **Tokenizer:** Dutch, 50k tokens ([yhavinga/gpt-neo-1.3B-dutch](https://huggingface.co/yhavinga/gpt-neo-1.3B-dutch))
- **Pre-training data:** Scraped Dutch ([yhavinga/mc4_nl_cleaned](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned))
- **Context window**: 8196 tokens
- **Developed by:** KU Leuven and UGent
- **Funded by:** KU Leuven BOF, VSC (Flemish Supercomputer Center), [Vlaams AI-onderzoeksprogramma](https://www.flandersairesearch.be/nl)
- **Model type:** Foundation model
- **License:** Apache 2.0
## Uses
As a base model, tweety-7b-dutch is primed for direct applications across text generation and understanding within the Dutch language.
## Technical Specifications
### Compute Infrastructure
#### Hardware
Training utilized Nvidia H100 and A100 GPUs. Inference is accessible on lower-end GPUs, basically any GPU capable of running mistral models. |