mistral-finetuned.md

  # Fine-tuned Mistral Model for Multi-Document Summarization  
  This model a fine-tuned model based on [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on 
   [multi_x_science_sum](https://huggingface.co/datasets/multi_x_science_sum) dataset.

  ## Model description

Mistral-7B-multixscience-finetuned is finetuned on multi_x_science_sum
dataset in order to extend the capabilities of the original 
Mistral model in multi-document summarization tasks. 
The fine-tuned model leverages the power of Mistral model fundation, 
adapting it to synthesize and summarize information from 
multiple documents efficiently.

## Training and evaluation dataset

Multi_x_science_sum is a large-scale multi-document 
summarization dataset created from scientific articles. 
Multi-XScience introduces a challenging multi-document 
summarization task: writing the related-work section of a 
paper based on its abstract and the articles it references.

* [paper](https://arxiv.org/pdf/2010.14235.pdf)
* [Source](https://huggingface.co/datasets/multi_x_science_sum)

The training and evaluation datasets were uniquely generated 
to facilitate the fine-tuning of the model for
multi-document summarization, particularly focusing on 
generating related work sections for scientific papers. 
Using a custom-designed prompt-generation process, the dataset
is created to simulate the task of synthesizing related work 
sections based on a given paper's abstract and the abstracts 
of its referenced papers.

### Dataset Generation process

The process involves generating prompts that instruct the 
model to use the abstract of the current paper along with 
the abstracts of cited papers to generate a new related work 
section. This approach aims to mimic the real-world scenario
where a researcher synthesizes information from multiple 
sources to draft the related work section of a paper.

* **Prompt Structure:** Each data point consists of an instructional prompt that includes:
  
  * The abstract of the current paper.
  * Abstracts from cited papers, labeled with unique identifiers.
  * An expected model response in the form of a generated related work section.

### Prompt generation Code

```
def generate_related_work_prompt(data):
    prompt = "[INST] <<SYS>>\n"
    prompt += "Use the abstract of the current paper and the abstracts of the cited papers to generate new related work.\n"
    prompt += "<</SYS>>\n\n"
    prompt += "Input:\nCurrent Paper's Abstract:\n- {}\n\n".format(data['abstract'])
    prompt += "Cited Papers' Abstracts:\n"
    for cite_id, cite_abstract in zip(data['ref_abstract']['cite_N'], data['ref_abstract']['abstract']):
        prompt += "- {}: {}\n".format(cite_id, cite_abstract)
    prompt += "\n[/INST]\n\nGenerated Related Work:\n{}\n".format(data['related_work'])
    return {"text": prompt}
```
The dataset generated through this process was used to train 
and evaluate the finetuned model, ensuring that it learns to 
accurately synthesize information from multiple sources into 
cohesive summaries.

## Training hyperparameters

The following hyperparameters were used during training:
```
learning_rate: 2e-5
train_batch_size: 4
eval_batch_size: 4
seed: 42
optimizer: adamw_8bit
num_epochs: 5
```
## Usage

```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftConfig, PeftModel

base_model = "mistralai/Mistral-7B-v0.1"
adapter = "OctaSpace/Mistral7B-fintuned"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    base_model,
    add_bos_token=True,
    trust_remote_code=True,
    padding_side='left'
)

# Create peft model using base_model and finetuned adapter
config = PeftConfig.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path,
                                             load_in_4bit=True,
                                             device_map='auto',
                                             torch_dtype='auto')
model = PeftModel.from_pretrained(model, adapter)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

# Prompt content:
messages = [] # Put here your related work generation instruction

input_ids = tokenizer.apply_chat_template(conversation=messages,
                                          tokenize=True,
                                          add_generation_prompt=True,
                                          return_tensors='pt').to(device)
summary_ids = model.generate(input_ids=input_ids, max_new_tokens=512, do_sample=True, pad_token_id=2)
summaries = tokenizer.batch_decode(summary_ids.detach().cpu().numpy(), skip_special_tokens = True)

# Model response: 
print(summaries[0])

```