File size: 2,915 Bytes
44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 44d7432 ba78638 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
---
license: llama2
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
- Dolly
- ipex
- Max Series GPU
base_model: meta-llama/Llama-2-7b-hf
datasets:
- databricks/databricks-dolly-15k
model-index:
- name: My_AGI_llama_2_7B
results: []
language:
- en
metrics:
- accuracy
- bertscore
- bleu
pipeline_tag: question-answering
---
# My_AGI_llama_2_7B
**Model Type:** Fine-Tuned
**Model Base:** [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
**Datasets Used:** [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
**Author:** [Yuri Achermann](https://huggingface.co/yuriachermann)
**Date:** June 03, 2024
-------------------------
## Training procedure
### Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.05
- training_steps: 593
### Framework versions
- PEFT==0.11.1
- Transformers==4.41.2
- Pytorch==2.1.0.post0+cxx11.abi
- Datasets==2.19.2
- Tokenizers==0.19.1
-------------------------
## Intended uses & limitations
**Primary Use Case:** The model is intended for generating human-like responses in conversational applications, like chatbots or virtual assistants.
**Limitations:** The model may generate inaccurate or biased content as it reflects the data it was trained on. It is essential to evaluate the generated responses in context and use the model responsibly.
-------------------------
## Evaluation
The evaluation platform consists of Gaudi Accelerators and Xeon CPUs running benchmarks from the [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
| Average | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande |
|:-------:|:-----:|:---------:|:-----:|:----------:|:----------:|
| 54.904 | 45.65 | 76.8 | 42.02 | 40.2 | 69.85 |
-------------------------
## Ethical Considerations
The model may inherit biases present in the training data. It is crucial to use the model in a way that promotes fairness and mitigates potential biases.
-------------------------
## Acknowledgments
This fine-tuning effort was made possible by the support of Intel, that provided the computing resources, and [Eduardo Alvarez](https://huggingface.co/eduardo-alvarez).
Additional shout-out to the creators of the Llama-2-7b-hf model and the contributors to the databricks-dolly-15k dataset.
-------------------------
## Contact Information
For questions or feedback about this model, please contact **[Yuri Achermann](mailto:yuri.achermann@gmail.com)**.
-------------------------
## License
This model is distributed under **Apache 2.0 License**. |