Update README.md
Browse files
README.md
CHANGED
@@ -71,21 +71,15 @@ if __name__ == "__main__":
|
|
71 |
```
|
72 |
|
73 |
# Training Procedure
|
74 |
-
The
|
75 |
|
76 |
-
|
77 |
-
This model, WizardCoder-Guanaco-15B-V1.0, is simply building on the efforts of two great teams to evaluate the performance of a combined model with the strengths of the [WizardCoder base model](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and the [openassistant-guanaco dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
|
78 |
-
|
79 |
-
A sincere appreciation goes out to the developers and the community involved in the creation and refinement of these models. Their commitment to providing open source tools and datasets have been instrumental in making this project a reality.
|
80 |
-
|
81 |
-
Moreover, a special note of thanks to the [Hugging Face](https://huggingface.co/) team, whose transformative library has not only streamlined the process of model creation and adaptation, but also democratized the access to state-of-the-art machine learning technologies. Their impact on the development of this project cannot be overstated.
|
82 |
|
83 |
-
|
84 |
-
The base WizardCoder model was finetuned on the openassistant-guanaco dataset using QLORA, which was trimmed to within 2 standard deviations of token size for question sets and randomized. All non-English data was also removed from this finetuning dataset.
|
85 |
|
86 |
## Acknowledgements
|
87 |
This model, WizardCoder-Guanaco-15B-V1.0, is simply building on the efforts of two great teams to evaluate the performance of a combined model with the strengths of the [WizardCoder base model](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and the [openassistant-guanaco dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
|
88 |
|
89 |
A sincere appreciation goes out to the developers and the community involved in the creation and refinement of these models. Their commitment to providing open source tools and datasets have been instrumental in making this project a reality.
|
90 |
|
91 |
-
Moreover, a special note of thanks to the [Hugging Face](https://huggingface.co/) team, whose transformative library has not only streamlined the process of model creation and adaptation, but also democratized the access to state-of-the-art machine learning technologies. Their impact on the development of this project cannot be overstated.
|
|
|
71 |
```
|
72 |
|
73 |
# Training Procedure
|
74 |
+
The WizardCoder model, serving as the base, was fine-tuned on a modified version of the openassistant-guanaco dataset. This dataset underwent a significant revision, replacing every single answer with responses generated by the AI model GPT-4. It was then expanded by approximately 50%, emphasizing high school and abstract algebra-related questions, using a mix of GPT-4 and GPT-3.5-Turbo for answer generation.
|
75 |
|
76 |
+
The selected dataset was standardized to fall within two standard deviations of token size for the question sets, ensuring consistency in data handling. The order of the questions was also randomized to mitigate any potential biases during the training phase.
|
|
|
|
|
|
|
|
|
|
|
77 |
|
78 |
+
In the interest of optimizing memory usage during the training process, the dataset was streamlined to only include English language content. As a result, all non-English data was systematically expunged from this fine-tuning dataset. It's worth noting that this modification limits the model's performance in language translation tasks, but it significantly boosts its efficiency and effectiveness when dealing with English language questions and answers.
|
|
|
79 |
|
80 |
## Acknowledgements
|
81 |
This model, WizardCoder-Guanaco-15B-V1.0, is simply building on the efforts of two great teams to evaluate the performance of a combined model with the strengths of the [WizardCoder base model](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and the [openassistant-guanaco dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
|
82 |
|
83 |
A sincere appreciation goes out to the developers and the community involved in the creation and refinement of these models. Their commitment to providing open source tools and datasets have been instrumental in making this project a reality.
|
84 |
|
85 |
+
Moreover, a special note of thanks to the [Hugging Face](https://huggingface.co/) team, whose transformative library has not only streamlined the process of model creation and adaptation, but also democratized the access to state-of-the-art machine learning technologies. Their impact on the development of this project cannot be overstated.
|