Model save

README.md

@@ -0,0 +1,75 @@
+---
+library_name: transformers
+license: mit
+base_model: openai-community/gpt2
+tags:
+- trl
+- orpo
+- generated_from_trainer
+datasets:
+- piqa
+model-index:
+- name: orpo-model-output
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# orpo-model-output
+
+This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on the piqa dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.2999
+- Rewards/chosen: -0.3182
+- Rewards/rejected: -0.3369
+- Rewards/accuracies: 0.6693
+- Rewards/margins: 0.0187
+- Logps/rejected: -3.3688
+- Logps/chosen: -3.1821
+- Logits/rejected: -24.5739
+- Logits/chosen: -24.6603
+- Nll Loss: 3.2330
+- Log Odds Ratio: -0.6695
+- Log Odds Chosen: 0.2013
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 3
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen |
+|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:--------------:|:---------------:|
+| 3.1799        | 1.0   | 3223 | 3.1575          | -0.2992        | -0.3103          | 0.6472             | 0.0111          | -3.1030        | -2.9917      | -16.2781        | -16.3180      | 3.0900   | -0.6758        | 0.1201          |
+| 2.7077        | 2.0   | 6446 | 3.1544          | -0.3005        | -0.3160          | 0.6652             | 0.0154          | -3.1595        | -3.0051      | -21.7517        | -21.8387      | 3.0878   | -0.6671        | 0.1676          |
+| 2.2691        | 3.0   | 9669 | 3.2999          | -0.3182        | -0.3369          | 0.6693             | 0.0187          | -3.3688        | -3.1821      | -24.5739        | -24.6603      | 3.2330   | -0.6695        | 0.2013          |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.0.0
+- Datasets 2.16.1
+- Tokenizers 0.19.1

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.2"
+}

logs/events.out.tfevents.1725050627.algo-1.65.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bef0168cae6bdce8bc993dc6bf7d012d07e68b7d868d664abb900f91c592e46c
-size 23649
+oid sha256:2bbd6d7eebe278afd1a52851689ccfb6a954a5f4130078fc96095df10e9401a0
+size 24912

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:123c25eaaa4c048566a1284fd3bef017bc909490be0ee8f8ea9c3123a7028d53
+oid sha256:9c30cf80d2319bd95da14aa7b8fec9165945ccef07088aabedb3738bf0f5655c
 size 497774208