Triangle104/aya-expanse-8b-Q6_K-GGUF

This model was converted to GGUF format from CohereForAI/aya-expanse-8b using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Model details:

Aya Expanse is an open-weight research release of a model with highly advanced multilingual capabilities. It focuses on pairing a highly performant pre-trained Command family of models with the result of a year’s dedicated research from Cohere For AI, including data arbitrage, multilingual preference training, safety tuning, and model merging. The result is a powerful multilingual large language model serving 23 languages.

We cover 23 languages: Arabic, Chinese (simplified & traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese

This model card corresponds to the 8-billion version of the Aya Expanse model. We also released an 32-billion version which you can find here.

Developed by: Cohere For AI
Point of Contact: Cohere For AI: cohere.for.ai
License: CC-BY-NC, requires also adhering to C4AI's Acceptable Use Policy
Model: Aya Expanse 8B
Model Size: 8 billion parameters

Try Aya Expanse

Before downloading the weights, you can try out Aya Expanse in our hosted Hugging Face Space.

Usage

Please install transformers from the source repository.

pip install 'git+https://github.com/huggingface/transformers.git'

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/aya-expanse-8b" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id)

Format the message with the chat template

messages = [{"role": "user", "content": "Anneme onu ne kadar sevdiğimi anlatan bir mektup yaz"}] input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")

<|START_OF_TURN_TOKEN|><|USER_TOKEN|>Anneme onu ne kadar sevdiğimi anlatan bir mektup yaz<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

gen_tokens = model.generate( input_ids, max_new_tokens=100, do_sample=True, temperature=0.3, )

gen_text = tokenizer.decode(gen_tokens[0]) print(gen_text)

Example Notebooks

Fine-Tuning:

This notebook showcases a detailed use of fine-tuning Aya Expanse on more languages.

Example Use cases:

The following notebooks contributed by Cohere For AI Community members show how Aya Expanse can be used for different use cases:

Mulitlingual Writing Assistant
AyaMCooking
Multilingual Question-Answering System

Model Details

Input: Models input text only.

Output: Models generate text only.

Model Architecture: Aya Expanse 8B is an auto-regressive language model that uses an optimized transformer architecture. Post-training includes supervised finetuning, preference training, and model merging.

Languages covered: The model is particularly optimized for multilinguality and supports the following languages: Arabic, Chinese (simplified & traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese

Context length: 8K

Model Card Contact

For errors or additional questions about details in this model card, contact info@for.ai.

Terms of Use

We hope that the release of this model will make community-based research efforts more accessible, by releasing the weights of a highly performant multilingual model to researchers all over the world. This model is governed by a CC-BY-NC License with an acceptable use addendum, and also requires adhering to C4AI's Acceptable Use Policy. Try the model today

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/aya-expanse-8b-Q6_K-GGUF --hf-file aya-expanse-8b-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/aya-expanse-8b-Q6_K-GGUF --hf-file aya-expanse-8b-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/aya-expanse-8b-Q6_K-GGUF --hf-file aya-expanse-8b-q6_k.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/aya-expanse-8b-Q6_K-GGUF --hf-file aya-expanse-8b-q6_k.gguf -c 2048