GemmaLM-for-Cannabis

This repository contains a fine-tuned version of the Gemma 2B model, specifically adapted for cannabis-related queries using Low Rank Adaptation (LoRA).

Model Details

• Base Model: Gemma 2B
• Fine-tuning Method: Low Rank Adaptation (LoRA)
• LoRA Rank: 4
• Training Data: Custom dataset derived from cannabis strain information
• Task: Causal Language Modeling for cannabis-related queries

Fine-tuning Process

The model was fine-tuned using a custom dataset created from cannabis strain information. The dataset includes details about various cannabis strains, their effects, flavors, and descriptions. The fine-tuning process involved:

1. Preprocessing the cannabis dataset into a prompt-response format
2. Implementing LoRA with a rank of 4 to efficiently adapt the model
3. Training for a limited number of epochs with a small subset of data for demonstration purposes

Usage

This model can be used to generate responses to cannabis-related queries. Example usage:

import keras
import keras_nlp

# Load the model
model = keras.models.load_model("gemma_lm_model.keras")

# Set up the sampler
sampler = keras_nlp.samplers.TopKSampler(k=5, seed=2)
model.compile(sampler=sampler)

# Generate a response
prompt = "Instruction:\nWhat does OG Kush feel like\nResponse:\n"
response = model.generate(prompt, max_length=256)
print(response)

Limitations

• The model was fine-tuned on a limited dataset for demonstration purposes. For production use, consider training on a larger dataset for more epochs.
• The current LoRA rank is set to 4, which may limit the model's adaptability. Experimenting with higher ranks could potentially improve performance.

Future Improvements

To enhance the model's performance, consider:

1. Increasing the size of the fine-tuning dataset
2. Training for more epochs
3. Experimenting with higher LoRA rank values
4. Adjusting hyperparameters such as learning rate and weight decay

License

Please refer to the Gemma model's original license for usage terms and conditions.

Acknowledgements

This project uses the Gemma model developed by Google. We acknowledge the Keras and KerasNLP teams for providing the tools and frameworks used in this project.