Update README.md (#2)

- Update README.md (91c39340f3377857bbbee0e40edfd5c2150c7396)


Co-authored-by: Mikhail Arkhipov <mshny@users.noreply.huggingface.co>

README.md

@@ -1,3 +1,92 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- JetBrains/KStack-clean
+base_model: meta-llama/CodeLlama-7b-hf
+results:
+- task:
+    type: text-generation
+  dataset:
+    name: MultiPL-HumanEval (Kotlin)
+    type: openai_humaneval
+  metrics:
+  - name: pass@1
+    type: pass@1
+    value: 37.89
+tags:
+- code
+---
+
+# Model description
+
+CodeLlama-7B-KStack-clean model is a fine-tuned open-source generative text model fine-tuned on [JetBrains/KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset. 
+This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
+
+# Model use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load pre-trained model and tokenizer
+model_name = 'JetBrains/CodeLlama-7B-KStack-clean'
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
+
+# Create and encode input
+input_text = """\
+This function takes an integer n and returns factorial of a number:
+fun factorial(n: Int): Int {\
+"""
+input_ids = tokenizer.encode(
+    input_text, return_tensors='pt'
+).to('cuda')
+
+# Generate
+output = model.generate(
+    input_ids, max_length=60, num_return_sequences=1, 
+    pad_token_id=tokenizer.eos_token_id
+)
+
+# Decode output
+generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+As with the base model, we can use FIM. To do this, the following format must be used: 
+```
+'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
+```
+
+# Training setup
+
+The model was trained on one A100 GPU with following hyperparameters:
+
+|         **Hyperparameter**           |             **Value**              |
+|:---------------------------:|:----------------------------------------:|
+|        `warmup`            |           100 steps            |
+|        `max_lr`        |          5e-5          |
+|        `scheduler`        |          linear          |
+|        `total_batch_size`        |        32 (~30K tokens per step)          |
+|        `num_epochs`        |          2          |
+
+More details about finetuning can be found in the technical report
+
+# Fine-tuning data
+
+For this model we used 25K exmaples of [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) selected according to educational value for learning algorithms. In total dataset contains about 23M tokens. 
+For more information about the dataset follow the link.
+
+# Evaluation 
+
+To evaluate we used Kotlin Humaneval ([more infromation here](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval))
+
+Fine-tuned model:
+
+|         **Model name**           |             **Kotlin HumanEval Pass Rate**              |
+|:---------------------------:|:----------------------------------------:|
+|           `base model`            |           26.89            |
+|        `fine-tuned model`        |          37.89        |
+
+# Ethical Considerations and Limitations
+
+CodeLlama-7B-KStack-clean and its variants are a new technology that carries risks with use. The testing conducted to date could not cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.