metadata

language:
  - en
pipeline_tag: text-generation
tags:
  - Pytorch
  - Llama3
  - M42
  - Health
  - HealthCare
base_model: m42-health/Llama3-Med42-8B

SandLogic Technologies - Quantized Llama3-Med42-8B Models

Model Description

We have quantized the Llama3-Med42-8B model into two variants:

Q5_KM
Q4_KM

These quantized models offer improved efficiency while maintaining performance.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

Name: Llama3-Med42-8B
Developer: M42 Health AI Team
Base Model: Llama3-8B-Instruct
Model Type: Clinical large language model (LLM)
Parameters: 8 billion
Context Length: 8k tokens
Input: Text only
Output: Text only
License: Llama 3 Community License Agreement

Model Capabilities

Llama3-Med42-8B is designed for medical and healthcare-related tasks, including:

Medical question answering
Patient record summarization
Aiding medical diagnosis
General health Q&A

Training Data

The model was instruction-tuned using a dataset of approximately 1 billion tokens compiled from various open-access and high-quality sources, including:

Medical flashcards
Exam questions
Open-domain dialogues

Important Limitations and Safe Use

DISCLAIMER: This model is not yet ready for clinical use without further testing and validation. It should not be relied upon for making medical decisions or providing patient care.

Potential for generating incorrect or harmful information
Risk of perpetuating biases in training data
Requires extensive human evaluation to ensure safety

Use Cases

While not ready for real clinical use, potential applications include:

Medical Education: Assist in studying and understanding medical concepts
Research Support: Aid in literature review and hypothesis generation
Health Information: Provide general health information (with appropriate disclaimers)
Clinical Decision Support: (Future potential) Enhance clinical decision-making processes

Model Variants

We offer two quantized versions of the Llama3-Med42-8B model:

Q5_KM: 5-bit quantization using the KM method
Q4_KM: 4-bit quantization using the KM method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Usage

pip install llama-cpp-python

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/7B/Llama3-Med42-8B.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm.create_chat_completion(
    messages =[
    {
        "role": "system",
        "content": (
            "You are a helpful, respectful and honest medical assistant. You are a second version of Med42 developed by the AI team at M42, UAE. "
            "Always answer as helpfully as possible, while being safe. "
            "Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. "
            "Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. "
            "If you don’t know the answer to a question, please don’t share false information."
        ),
    },
    {"role": "user", "content": "What are the symptoms of diabetes?"},
]
)

print(output["choices"][0]['message']['content'])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Llama3-Med42-8B-GGUF",
    filename="*Llama3-Med42-8B-Q5_K_M.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

Ethical Considerations

Users must be aware of the model's limitations and potential biases. It should not be used for direct medical advice or decision-making without proper validation and human oversight.

Acknowledgements

We thank the M42 Health AI Team and the creators of Llama3 for their contributions to the field of medical AI.Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at support@sandlogic.com or visit our support page.