SandLogicTechnologies/LLama3-Gaja-Hindi-8B-GGUF

SandLogic Technologies - Quantized LLama3-Gaja-Hindi-8B Models

Model Description

We have quantized the LLama3-Gaja-Hindi-8B model into two variants:

1. Q5_KM
2. Q4_KM

These quantized models offer improved efficiency while maintaining performance.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

• Name: LLama3-Gaja-Hindi-8B-v0.1
• Developer: Cognitivelab.in
• Base Model: Llama3-8B
• Model Type: Bilingual English/Hindi language model
• Parameters: 8 billion
• Specialization: Natural language understanding, particularly for instructional pairs

Model Capabilities

LLama3-Gaja-Hindi-8B is designed for bilingual (English/Hindi) natural language understanding tasks, with a focus on:

• Responding appropriately in either English or Hindi based on user prompts
• Understanding and generating instructional content in both languages
• Handling a variety of natural language processing tasks across both languages

Training Approach

The model underwent supervised fine-tuning with low-rank adaptation, focusing on bilingual instruct fine-tuning. The training data consisted of a curated dataset of translated instructional pairs.

Use Cases

1. Bilingual Conversational AI: Chatbots and virtual assistants with English/Hindi capabilities
2. Language Learning Tools: Interactive platforms for English and Hindi learners
3. Content Translation: Assistance in translating between English and Hindi, especially for instructional materials
4. Cross-lingual Information Retrieval: Enabling queries in one language with responses in either English or Hindi
5. Cultural Context Understanding: Helping users grasp cultural nuances in both languages
6. Multilingual Customer Support: Powering customer service applications for diverse user bases

Model Variants

We offer two quantized versions of the LLama3-Gaja-Hindi-8B model:

1. Q5_KM: 5-bit quantization using the KM method
2. Q4_KM: 4-bit quantization using the KM method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Input and Output

• Input: Text prompts or instructions in either English or Hindi
• Output: Generated text responses in the same language as the input, or as specified in the prompt

Usage

pip install llama-cpp-python 

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/7B/LLama3-Gaja-Hindi-8B-v0.1.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm.create_chat_completion(
    messages =[
    {
        "role": "system",
        "content": """ You are an AI assistant trained on top of Llama 3 Large language model (LLM), proficient in English and Hindi. You can respond in both languages based on the user's request."""
            
        ,
    },
    {"role": "user", "content": "Write an poem in hindi"},
]
)

print(output["choices"][0]['message']['content'])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/LLama3-Gaja-Hindi-8B-GGUF",
    filename="*llama3-gaja-hindi-8b-v0.1.Q5_K_M.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

Ethical Considerations

Users should be aware of potential biases in the model's outputs, especially when dealing with cultural or linguistic nuances. Always review and validate the model's responses for sensitive applications.

Acknowledgements

We thank Cognitivelab.in for developing the original LLama3-Gaja-Hindi-8B model and the creators of Llama3 for their foundational work.Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at support@sandlogic.com or visit our support page.