|
|
|
--- |
|
license: apache-2.0 |
|
|
|
language: |
|
- ru |
|
tags: |
|
- PyTorch |
|
- Transformers |
|
--- |
|
|
|
# BERT base model for pair ranking (reward model for RLHF) in Russian language. |
|
For training i use the next [pair-ranking-loss](https://pytorch.org/docs/stable/generated/torch.nn.MarginRankingLoss.html) |
|
|
|
Datasets have been translated with google-translate-api for reward training: |
|
|
|
- [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) |
|
- [Dahoas/synthetic-instruct-gptj-pairwise](https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise) |
|
- [openai/webgpt_comparisons](https://huggingface.co/datasets/openai/webgpt_comparisons) |
|
|
|
For better quality, use mean token embeddings. |
|
## Usage (HuggingFace Models Repository) |
|
You can use the model directly from the model repository to compute score: |
|
```python |
|
|
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
#Create model object and inits pretrain weights: |
|
reward_name = "Andrilko/ruBert-base-reward" |
|
rank_model = AutoModelForSequenceClassification.from_pretrained(reward_name) |
|
tokenizer = AutoTokenizer.from_pretrained(reward_name) |
|
|
|
|
|
#Sentences that we want to score: |
|
sentences = ['Человек: Что такое QR-код?','Ассистент: QR-код - это тип матричного штрих-кода.'] |
|
|
|
#Compute token embeddings |
|
inputs = tokenizer(sentences[0], sentences[1], return_tensors='pt') |
|
score = rank_model(**inputs).logits[0].cpu().detach() |
|
print(score) |
|
``` |
|
|
|
# Authors |
|
+ Aleksandr Abramov: [Github](https://github.com/Ab1992ao), [Kaggle Competitions Master](https://www.kaggle.com/andrilko); |
