|
--- |
|
tags: |
|
- mteb |
|
- qihoo360 |
|
- 奇虎360 |
|
- RAG-reranking |
|
model-index: |
|
- name: 360Zhinao-1_8B-reranking |
|
results: |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: None |
|
name: MTEB CMedQAv1 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 86.75017961853382 |
|
- type: mrr |
|
value: 89.15436507936508 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: None |
|
name: MTEB CMedQAv2 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 87.91572151930174 |
|
- type: mrr |
|
value: 89.98869047619048 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: None |
|
name: MTEB MMarcoReranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 37.28779203409935 |
|
- type: mrr |
|
value: 36.23730158730159 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: None |
|
name: MTEB T2Reranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 68.55153559405632 |
|
- type: mrr |
|
value: 79.62773774596725 |
|
license: apache-2.0 |
|
library_name: transformers |
|
--- |
|
|
|
<div align="center"> |
|
<h1> |
|
360智脑 |
|
</h1> |
|
</div> |
|
<div align="center"> |
|
<a href="https://huggingface.co/qihoo360">Hugging Face</a>   |   n |
|
<a href="https://www.modelscope.cn/profile/qihoo360">ModelScope</a>   ｜   n |
|
</div> |
|
<br> |
|
<p align="center"> |
|
Welcome to 360 Zhinao<a href="https://ai.360.com"> https://ai.360.com </a> |
|
</p> |
|
|
|
<br> |
|
|
|
# MTEB Leaderboard Chinese Reranking Results |
|
We have validated the performance of our model on the [mteb-chinese-reranking leaderboard](https://huggingface.co/spaces/mteb/leaderboard). Currently, the open-source models on this leaderboard are primarily bidirectional discriminative models (BERT-like models). The only unidirectional generative model (GPT-like model) is gte-Qwen1.5-7B-instruct, which has an average score of 66.38, ranking 25th, with less than ideal results. Our self-developed unidirectional generative model, zhinao_1-8b_reranking, achieved an average score of 70.13, currently ranking first overall and first among open-source models, opening up new possibilities for generative models to undertake discriminative tasks. |
|
|
|
| Model | T2Reranking | MMarcoReranking | CMedQAv1 | CMedQAv2 | Avg | |
|
|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:| |
|
| **360Zhinao-1_8B-reranking** | 68.55 | 37.29 | 86.75 | 87.92 | 70.13 | |
|
| piccolo-large-zh-v2 | 67.15 | 33.39 | 90.14 | 89.31 | 70 | |
|
| Baichuan-text-embedding | 67.85 | 34.3 | 88.46 | 88.06 | 69.67 | |
|
| stella-mrl-large-zh-v3.5-1792d | 66.43 | 28.85 | 89.18 | 89.33 | 68.45 | |
|
| PEG | 69.43 | 33.55 | 86.56 | 84.09 | 68.41 | |
|
| bge-reranker-base | 67.28 | 35.46 | 81.27 | 84.1 | 67.03 | |
|
| bge-reranker-large | 67.6 | 37.17 | 82.14 | 84.19 | 67.78 | |
|
|
|
|
|
# Requirements |
|
|
|
```bash |
|
pip install -r requirements.txt |
|
``` |
|
|
|
If your GPU supports fp16 or bf16 precision, we also recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**now with support for flash attention 2**) to improve your runtime efficiency and reduce memory usage. (**flash-attention is optional and not required for running this project**) |
|
|
|
```bash |
|
git clone https://github.com/Dao-AILab/flash-attention |
|
cd flash-attention && pip install . |
|
# The installation below is optional and might be slow. |
|
# pip install csrc/layer_norm |
|
# No need to install the following if the flash-attn version is above 2.1.1. |
|
# pip install csrc/rotary |
|
``` |
|
|
|
# Model Introduction |
|
|
|
The zhinao_1-8b_reranking model utilizes the self-developed zhinao_1-8b_base model as its foundation. Through iterative discovery and resolution of the following technical issues, it continuously stimulates the world knowledge inherent in the large model during the pre-training phase, better bridging the gap between generative models and discriminative tasks. |
|
|
|
## Data Processing |
|
|
|
The model training did not utilize world knowledge, meaning it neither continued pre-training with domain-specific data nor fine-tuned datasets outside of the four datasets on the leaderboard. It only used the four datasets within the leaderboard, carefully iterating through data perception, and targeting different datasets for data cleaning and mining to ensure that the ranking in individual tasks could reach the top three level. |
|
|
|
## Resolving Task Conflicts |
|
|
|
When merging four tasks, due to different data domain distributions, answer patterns, training data volumes, convergence steps, and even sequence lengths, conflicts exist between different tasks. Deeply resolving these conflict issues is crucial to obtaining a universal model with the best comprehensive indicators across different tasks. |
|
|
|
## Resolving Training Instability |
|
|
|
Unlike generative tasks that produce multiple characters, using generative models for discriminative tasks requires the model to output a continuous value. Therefore, there is an oscillation problem during the training process. Deeply analyzing and resolving training instability can result in a model with better generalization and robustness. |
|
|
|
|
|
# Inference Script |
|
|
|
```python |
|
from typing import cast, List, Union, Tuple, Dict, Optional |
|
|
|
import numpy as np |
|
import torch |
|
from tqdm import tqdm |
|
from transformers import AutoModel, AutoTokenizer, AutoModelForSequenceClassification |
|
import transformers |
|
from transformers.trainer_pt_utils import LabelSmoother |
|
IGNORE_TOKEN_ID = LabelSmoother.ignore_index |
|
|
|
def preprocess( |
|
sources, |
|
tokenizer: transformers.PreTrainedTokenizer, |
|
max_len: int = 1024, |
|
system_message: str = "", |
|
#system_message: str = "You are a helpful assistant.", |
|
device = None, |
|
) -> Dict: |
|
roles = {"user": "<|im_start|>user", "assistant": "<|im_start|>assistant"} |
|
answer_len = 64 |
|
|
|
im_start = tokenizer.im_start_id |
|
im_end = tokenizer.im_end_id |
|
nl_tokens = tokenizer('\n').input_ids |
|
_system = tokenizer('system').input_ids + nl_tokens |
|
_user = tokenizer('user').input_ids + nl_tokens |
|
_assistant = tokenizer('assistant').input_ids + nl_tokens |
|
|
|
# Apply prompt templates |
|
input_ids, targets = [], [] |
|
for i, source in enumerate(sources): |
|
## system_message |
|
input_id, target = [], [] |
|
system = [im_start] + _system + tokenizer(system_message, max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens |
|
input_id += system |
|
target += [im_start] + [IGNORE_TOKEN_ID] * (len(system)-3) + [im_end] + nl_tokens |
|
assert len(input_id) == len(target) |
|
|
|
## query ans |
|
source = "\n\n".join(source) |
|
role = "<|im_start|>user" |
|
_input_id = tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids + nl_tokens + \ |
|
tokenizer(source, max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens |
|
input_id += _input_id |
|
if role == '<|im_start|>user': |
|
_target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens |
|
elif role == '<|im_start|>assistant': |
|
_target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids) + \ |
|
_input_id[len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids)+1:-2] + [im_end] + nl_tokens |
|
else: |
|
raise NotImplementedError |
|
target += _target |
|
|
|
## label use placeholder 0; It will be masked later in the modeling_zhinao.py |
|
role = "<|im_start|>assistant" |
|
_input_id = tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids + nl_tokens + \ |
|
tokenizer("0", max_length=max_len-answer_len, truncation=True).input_ids + [im_end] + nl_tokens |
|
input_id += _input_id |
|
if role == '<|im_start|>user': |
|
_target = [im_start] + [IGNORE_TOKEN_ID] * (len(_input_id)-3) + [im_end] + nl_tokens |
|
elif role == '<|im_start|>assistant': |
|
_target = [im_start] + [IGNORE_TOKEN_ID] * len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids) + \ |
|
_input_id[len(tokenizer(role, max_length=max_len-answer_len, truncation=True).input_ids)+1:-2] + [im_end] + nl_tokens |
|
else: |
|
raise NotImplementedError |
|
target += _target |
|
|
|
assert len(input_id) == len(target) |
|
input_id += [tokenizer.pad_token_id] * (max_len - len(input_id)) |
|
target += [IGNORE_TOKEN_ID] * (max_len - len(target)) |
|
if len(input_id) > max_len: |
|
print("max_len_error") |
|
print(tokenizer.decode(input_id)) |
|
|
|
input_ids.append(input_id[:max_len]) |
|
targets.append(target[:max_len]) |
|
input_ids = torch.tensor(input_ids, dtype=torch.int) |
|
targets = torch.tensor(targets, dtype=torch.int) |
|
#print(f"input_ids {input_ids.shape}") |
|
#print(f"targets {targets.shape}") |
|
|
|
return dict( |
|
input_ids=input_ids.to(device), |
|
labels=targets.to(device), |
|
attention_mask=input_ids.ne(tokenizer.pad_token_id).to(device), |
|
) |
|
|
|
class FlagRerankerCustom: |
|
def __init__( |
|
self, |
|
model_name_or_path: str = None, |
|
use_fp16: bool = False |
|
) -> None: |
|
self.tokenizer = transformers.AutoTokenizer.from_pretrained( |
|
model_name_or_path, |
|
model_max_length=1024, |
|
padding_side="right", |
|
use_fast=False, |
|
trust_remote_code=True |
|
) |
|
self.tokenizer.pad_token_id = self.tokenizer.eod_id |
|
config = transformers.AutoConfig.from_pretrained( |
|
model_name_or_path, |
|
trust_remote_code=True, |
|
bf16=True, |
|
) |
|
config.use_cache = False |
|
self.model = transformers.AutoModelForCausalLM.from_pretrained( |
|
model_name_or_path, |
|
config=config, |
|
trust_remote_code=True, |
|
) |
|
self.model.linear.bfloat16() |
|
|
|
if torch.cuda.is_available(): |
|
self.device = torch.device('cuda') |
|
elif torch.backends.mps.is_available(): |
|
self.device = torch.device('mps') |
|
else: |
|
self.device = torch.device('cpu') |
|
use_fp16 = False |
|
if use_fp16: |
|
self.model.half() |
|
|
|
self.model = self.model.to(self.device) |
|
|
|
self.model.eval() |
|
|
|
self.num_gpus = torch.cuda.device_count() |
|
if self.num_gpus > 1: |
|
print(f"----------using {self.num_gpus}*GPUs----------") |
|
self.model = torch.nn.DataParallel(self.model) |
|
|
|
@torch.no_grad() |
|
def compute_score(self, sentence_pairs: Union[List[Tuple[str, str]], Tuple[str, str]], batch_size: int =128, |
|
max_length: int = 1024) -> List[float]: |
|
if self.num_gpus > 0: |
|
batch_size = batch_size * self.num_gpus |
|
|
|
assert isinstance(sentence_pairs, list) |
|
if isinstance(sentence_pairs[0], str): |
|
sentence_pairs = [sentence_pairs] |
|
|
|
all_scores = [] |
|
for start_index in tqdm(range(0, len(sentence_pairs), batch_size), desc="Compute Scores", |
|
disable=len(sentence_pairs) < 128): |
|
sentences_batch = sentence_pairs[start_index:start_index + batch_size] # [[q,ans],[q, ans]...] |
|
inputs = preprocess(sources=sentences_batch, tokenizer=self.tokenizer,max_len=1024,device=self.device) |
|
scores = self.model(**inputs, return_dict=True).logits.view(-1, ).float() |
|
all_scores.extend(scores.cpu().numpy().tolist()) |
|
|
|
if len(all_scores) == 1: |
|
return all_scores[0] |
|
return all_scores |
|
|
|
|
|
if __name__ == "__main__": |
|
model_name_or_path = "360Zhinao-1_8B-reranking" |
|
model = FlagRerankerCustom(model_name_or_path, use_fp16=False) |
|
inputs=[["What Color Is the Sky","Blue"], ["What Color Is the Sky","Pink"],] |
|
ret = model.compute_score(inputs) |
|
print(ret) |
|
|
|
``` |
|
|
|
## License |
|
The source code of this repository follows the open-source license Apache 2.0. |
|
360Zhinao open-source models support commercial use. If you wish to use these models or continue training them for commercial purposes, please contact us via email (g-zhinao-opensource@360.cn) to apply. For the specific license agreement, please see <<360 Zhinao Open-Source Model License>>. |
|
|
|
|
|
|
