|
--- |
|
base_model: unsloth/gemma-2-9b-bnb-4bit |
|
language: |
|
- en |
|
license: gemma |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- unsloth |
|
- gemma2 |
|
- trl |
|
|
|
pipeline_tag: text-classification |
|
--- |
|
|
|
|
|
|
|
# Uploaded model |
|
|
|
- **Developed by:** EpistemeAI |
|
- **License:** apache-2.0 |
|
- **Finetuned from model :** unsloth/gemma-2-9b-bnb-4bit |
|
|
|
This gemma2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. |
|
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |
|
|
|
# This model is fine-tuned by 122k code instructions. |
|
|
|
How to use |
|
This repository contains two versions of Gemma-2-9B, for use with transformers and with the original llama codebase. |
|
|
|
Use with transformers |
|
Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function. |
|
|
|
Make sure to update your transformers installation via pip install --upgrade transformers. |
|
|
|
You need to prepare prompt in alpaca format to generate properly: |
|
```python |
|
def format_test(x): |
|
|
|
if x['input']: |
|
formatted_text = f"""Below is an instruction that describes a task. \ |
|
Write a response that appropriately completes the request. |
|
|
|
### Instruction: |
|
{x['instruction']} |
|
|
|
### Input: |
|
{x['input']} |
|
|
|
### Response: |
|
""" |
|
|
|
else: |
|
formatted_text = f"""Below is an instruction that describes a task. \ |
|
Write a response that appropriately completes the request. |
|
|
|
### Instruction: |
|
{x['instruction']} |
|
|
|
### Response: |
|
""" |
|
|
|
return formatted_text |
|
|
|
# using code_instructions_122k_alpaca dataset |
|
Prompt = format_test(data[155]) |
|
print(Prompt) |
|
|
|
``` |
|
- huggingface transformers method: |
|
```python |
|
from transformers import TextStreamer |
|
|
|
FastLanguageModel.for_inference(model) # Enable native 2x faster inference |
|
inputs = tokenizer( |
|
[ |
|
Prompt |
|
], return_tensors = "pt").to("cuda") |
|
|
|
text_streamer = TextStreamer(tokenizer) |
|
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 512) |
|
``` |
|
|
|
|
|
- unsloth method |
|
```python |
|
from unsloth import FastLanguageModel |
|
|
|
model, tokenizer = FastLanguageModel.from_pretrained( |
|
model_name = "EpistemeAI/EpistemeAI-codegemma-2-9b", # YOUR MODEL YOU USED FOR TRAINING |
|
max_seq_length = max_seq_length, |
|
dtype = dtype, |
|
load_in_4bit = load_in_4bit, |
|
) |
|
FastLanguageModel.for_inference(model) # Enable native 2x faster inference |
|
|
|
# alpaca_prompt = You MUST copy from above! |
|
|
|
inputs = tokenizer( |
|
[ |
|
alpaca_prompt.format( |
|
"Create a function to calculate the sum of a sequence of integers.", # instruction |
|
"", # input |
|
"", # output - leave this blank for generation! |
|
) |
|
], return_tensors = "pt").to("cuda") |
|
|
|
outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True) |
|
tokenizer.batch_decode(outputs) |
|
``` |
|
|
|
-- |
|
|
|
### Inputs and outputs |
|
|
|
* **Input:** Text string, such as a question, a prompt, or a document to be |
|
summarized. |
|
* **Output:** Generated English-language text in response to the input, such |
|
as an answer to a question, or a summary of a document. |
|
### Citation |
|
|
|
```none |
|
@article{gemma_2024, |
|
title={Gemma}, |
|
url={https://www.kaggle.com/m/3301}, |
|
DOI={10.34740/KAGGLE/M/3301}, |
|
publisher={Kaggle}, |
|
author={Gemma Team}, |
|
year={2024} |
|
} |
|
``` |
|
|
|
## Model Data |
|
|
|
Data used for model training and how the data was processed. |
|
|
|
### Training Dataset |
|
|
|
These models were trained on a dataset of text data that includes a wide variety of sources. The 27B model was trained with 13 trillion tokens and the 9B model was trained with 8 trillion tokens. |
|
Here are the key components: |
|
|
|
* Web Documents: A diverse collection of web text ensures the model is exposed |
|
to a broad range of linguistic styles, topics, and vocabulary. Primarily |
|
English-language content. |
|
* Code: Exposing the model to code helps it to learn the syntax and patterns of |
|
programming languages, which improves its ability to generate code or |
|
understand code-related questions. |
|
* Mathematics: Training on mathematical text helps the model learn logical |
|
reasoning, symbolic representation, and to address mathematical queries. |
|
|
|
The combination of these diverse data sources is crucial for training a powerful |
|
language model that can handle a wide variety of different tasks and text |
|
formats. |
|
|
|
### Data Preprocessing |
|
|
|
Here are the key data cleaning and filtering methods applied to the training |
|
data: |
|
|
|
* CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering was |
|
applied at multiple stages in the data preparation process to ensure the |
|
exclusion of harmful and illegal content. |
|
* Sensitive Data Filtering: As part of making Gemma pre-trained models safe and |
|
reliable, automated techniques were used to filter out certain personal |
|
information and other sensitive data from training sets. |
|
* Additional methods: Filtering based on content quality and safety in line with |
|
[our policies][safety-policies]. |
|
|
|
## Implementation Information |
|
|
|
Details about the model internals. |
|
|
|
### Hardware |
|
|
|
Gemma was trained using the latest generation of |
|
[Tensor Processing Unit (TPU)][tpu] hardware (TPUv5p). |
|
|
|
Training large language models requires significant computational power. TPUs, |
|
designed specifically for matrix operations common in machine learning, offer |
|
several advantages in this domain: |
|
|
|
* Performance: TPUs are specifically designed to handle the massive computations |
|
involved in training LLMs. They can speed up training considerably compared to |
|
CPUs. |
|
* Memory: TPUs often come with large amounts of high-bandwidth memory, allowing |
|
for the handling of large models and batch sizes during training. This can |
|
lead to better model quality. |
|
* Scalability: TPU Pods (large clusters of TPUs) provide a scalable solution for |
|
handling the growing complexity of large foundation models. You can distribute |
|
training across multiple TPU devices for faster and more efficient processing. |
|
* Cost-effectiveness: In many scenarios, TPUs can provide a more cost-effective |
|
solution for training large models compared to CPU-based infrastructure, |
|
especially when considering the time and resources saved due to faster |
|
training. |
|
* These advantages are aligned with |
|
[Google's commitments to operate sustainably][sustainability]. |
|
|
|
### Software |
|
|
|
Training was done using [JAX][jax] and [ML Pathways][ml-pathways]. |
|
|
|
JAX allows researchers to take advantage of the latest generation of hardware, |
|
including TPUs, for faster and more efficient training of large models. |
|
|
|
ML Pathways is Google's latest effort to build artificially intelligent systems |
|
capable of generalizing across multiple tasks. This is specially suitable for |
|
[foundation models][foundation-models], including large language models like |
|
these ones. |
|
|
|
Together, JAX and ML Pathways are used as described in the |
|
[paper about the Gemini family of models][gemini-2-paper]; "the 'single |
|
controller' programming model of Jax and Pathways allows a single Python |
|
process to orchestrate the entire training run, dramatically simplifying the |
|
development workflow." |
|
|
|
## Evaluation |
|
|
|
Model evaluation metrics and results. |
|
|
|
### Benchmark Results |
|
|
|
These models were evaluated against a large collection of different datasets and |
|
metrics to cover different aspects of text generation: |
|
|
|
| Benchmark | Metric | Gemma PT 9B | Gemma PT 27B | |
|
| ------------------------------ | ------------- | ----------- | ------------ | |
|
| [MMLU][mmlu] | 5-shot, top-1 | 71.3 | 75.2 | |
|
| [HellaSwag][hellaswag] | 10-shot | 81.9 | 86.4 | |
|
| [PIQA][piqa] | 0-shot | 81.7 | 83.2 | |
|
| [SocialIQA][socialiqa] | 0-shot | 53.4 | 53.7 | |
|
| [BoolQ][boolq] | 0-shot | 84.2 | 84.8 | |
|
| [WinoGrande][winogrande] | partial score | 80.6 | 83.7 | |
|
| [ARC-e][arc] | 0-shot | 88.0 | 88.6 | |
|
| [ARC-c][arc] | 25-shot | 68.4 | 71.4 | |
|
| [TriviaQA][triviaqa] | 5-shot | 76.6 | 83.7 | |
|
| [Natural Questions][naturalq] | 5-shot | 29.2 | 34.5 | |
|
| [HumanEval][humaneval] | pass@1 | 40.2 | 51.8 | |
|
| [MBPP][mbpp] | 3-shot | 52.4 | 62.6 | |
|
| [GSM8K][gsm8k] | 5-shot, maj@1 | 68.6 | 74.0 | |
|
| [MATH][math] | 4-shot | 36.6 | 42.3 | |
|
| [AGIEval][agieval] | 3-5-shot | 52.8 | 55.1 | |
|
| [BIG-Bench][big-bench] | 3-shot, CoT | 68.2 | 74.9 | |
|
| ------------------------------ | ------------- | ----------- | ------------ | |
|
|
|
## Ethics and Safety |
|
|
|
Ethics and safety evaluation approach and results. |
|
|
|
### Evaluation Approach |
|
|
|
Our evaluation methods include structured evaluations and internal red-teaming |
|
testing of relevant content policies. Red-teaming was conducted by a number of |
|
different teams, each with different goals and human evaluation metrics. These |
|
models were evaluated against a number of different categories relevant to |
|
ethics and safety, including: |
|
|
|
* Text-to-Text Content Safety: Human evaluation on prompts covering safety |
|
policies including child sexual abuse and exploitation, harassment, violence |
|
and gore, and hate speech. |
|
* Text-to-Text Representational Harms: Benchmark against relevant academic |
|
datasets such as [WinoBias][winobias] and [BBQ Dataset][bbq]. |
|
* Memorization: Automated evaluation of memorization of training data, including |
|
the risk of personally identifiable information exposure. |
|
* Large-scale harm: Tests for "dangerous capabilities," such as chemical, |
|
biological, radiological, and nuclear (CBRN) risks. |
|
|
|
### Evaluation Results |
|
|
|
The results of ethics and safety evaluations are within acceptable thresholds |
|
for meeting [internal policies][safety-policies] for categories such as child |
|
safety, content safety, representational harms, memorization, large-scale harms. |
|
On top of robust internal evaluations, the results of well-known safety |
|
benchmarks like BBQ, BOLD, Winogender, Winobias, RealToxicity, and TruthfulQA |
|
are shown here. |
|
|
|
#### Gemma 2.0 |
|
|
|
| Benchmark | Metric | Gemma 2 IT 9B | Gemma 2 IT 27B | |
|
| ------------------------ | ------------- | --------------- | ---------------- | |
|
| [RealToxicity][realtox] | average | 8.25 | 8.84 | |
|
| [CrowS-Pairs][crows] | top-1 | 37.47 | 36.67 | |
|
| [BBQ Ambig][bbq] | 1-shot, top-1 | 88.58 | 85.99 | |
|
| [BBQ Disambig][bbq] | top-1 | 82.67 | 86.94 | |
|
| [Winogender][winogender] | top-1 | 79.17 | 77.22 | |
|
| [TruthfulQA][truthfulqa] | | 50.27 | 51.60 | |
|
| [Winobias 1_2][winobias] | | 78.09 | 81.94 | |
|
| [Winobias 2_2][winobias] | | 95.32 | 97.22 | |
|
| [Toxigen][toxigen] | | 39.30 | 38.42 | |
|
| ------------------------ | ------------- | --------------- | ---------------- | |
|
|
|
## Usage and Limitations |
|
|
|
These models have certain limitations that users should be aware of. |
|
|
|
### Intended Usage |
|
|
|
Open Large Language Models (LLMs) have a wide range of applications across |
|
various industries and domains. The following list of potential uses is not |
|
comprehensive. The purpose of this list is to provide contextual information |
|
about the possible use-cases that the model creators considered as part of model |
|
training and development. |
|
|
|
* Content Creation and Communication |
|
* Text Generation: These models can be used to generate creative text formats |
|
such as poems, scripts, code, marketing copy, and email drafts. |
|
* Chatbots and Conversational AI: Power conversational interfaces for customer |
|
service, virtual assistants, or interactive applications. |
|
* Text Summarization: Generate concise summaries of a text corpus, research |
|
papers, or reports. |
|
* Research and Education |
|
* Natural Language Processing (NLP) Research: These models can serve as a |
|
foundation for researchers to experiment with NLP techniques, develop |
|
algorithms, and contribute to the advancement of the field. |
|
* Language Learning Tools: Support interactive language learning experiences, |
|
aiding in grammar correction or providing writing practice. |
|
* Knowledge Exploration: Assist researchers in exploring large bodies of text |
|
by generating summaries or answering questions about specific topics. |
|
### Limitations |
|
|
|
* Training Data |
|
* The quality and diversity of the training data significantly influence the |
|
model's capabilities. Biases or gaps in the training data can lead to |
|
limitations in the model's responses. |
|
* The scope of the training dataset determines the subject areas the model can |
|
handle effectively. |
|
* Context and Task Complexity |
|
* LLMs are better at tasks that can be framed with clear prompts and |
|
instructions. Open-ended or highly complex tasks might be challenging. |
|
* A model's performance can be influenced by the amount of context provided |
|
(longer context generally leads to better outputs, up to a certain point). |
|
* Language Ambiguity and Nuance |
|
* Natural language is inherently complex. LLMs might struggle to grasp subtle |
|
nuances, sarcasm, or figurative language. |
|
* Factual Accuracy |
|
* LLMs generate responses based on information they learned from their |
|
training datasets, but they are not knowledge bases. They may generate |
|
incorrect or outdated factual statements. |
|
* Common Sense |
|
* LLMs rely on statistical patterns in language. They might lack the ability |
|
to apply common sense reasoning in certain situations. |
|
### Ethical Considerations and Risks |
|
|
|
The development of large language models (LLMs) raises several ethical concerns. |
|
In creating an open model, we have carefully considered the following: |
|
|
|
* Bias and Fairness |
|
* LLMs trained on large-scale, real-world text data can reflect socio-cultural |
|
biases embedded in the training material. These models underwent careful |
|
scrutiny, input data pre-processing described and posterior evaluations |
|
reported in this card. |
|
* Misinformation and Misuse |
|
* LLMs can be misused to generate text that is false, misleading, or harmful. |
|
* Guidelines are provided for responsible use with the model, see the |
|
[Responsible Generative AI Toolkit][rai-toolkit]. |
|
* Transparency and Accountability: |
|
* This model card summarizes details on the models' architecture, |
|
capabilities, limitations, and evaluation processes. |
|
* A responsibly developed open model offers the opportunity to share |
|
innovation by making LLM technology accessible to developers and researchers |
|
across the AI ecosystem. |
|
Risks identified and mitigations: |
|
|
|
* Perpetuation of biases: It's encouraged to perform continuous monitoring |
|
(using evaluation metrics, human review) and the exploration of de-biasing |
|
techniques during model training, fine-tuning, and other use cases. |
|
* Generation of harmful content: Mechanisms and guidelines for content safety |
|
are essential. Developers are encouraged to exercise caution and implement |
|
appropriate content safety safeguards based on their specific product policies |
|
and application use cases. |
|
* Misuse for malicious purposes: Technical limitations and developer and |
|
end-user education can help mitigate against malicious applications of LLMs. |
|
Educational resources and reporting mechanisms for users to flag misuse are |
|
provided. Prohibited uses of Gemma models are outlined in the |
|
[Gemma Prohibited Use Policy][prohibited-use]. |
|
* Privacy violations: Models were trained on data filtered for removal of PII |
|
(Personally Identifiable Information). Developers are encouraged to adhere to |
|
privacy regulations with privacy-preserving techniques. |
|
|
|
### Benefits |
|
|
|
At the time of release, this family of models provides high-performance open |
|
large language model implementations designed from the ground up for Responsible |
|
AI development compared to similarly sized models. |
|
|
|
Using the benchmark evaluation metrics described in this document, these models |
|
have shown to provide superior performance to other, comparably-sized o |
|
|
|
### Notice: |
|
Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms |
