Update README.md

README.md

@@ -1,28 +1,66 @@
-# Training Data
-**Autochart:** Zhu, J., Ran, J., Lee, R. K. W., Choo, K., & Li, Z. (2021). AutoChart: A Dataset for Chart-to-Text Generation Task. arXiv preprint arXiv:2108.06897.
+# Tackling Hallucinations in Neural Chart Summarization
 
-**Gitlab Link for the data**: https://gitlab.com/bottle_shop/snlg/chart/autochart
+## Introduction
 
-Train split for this model: Train 23336, Validation 1297, Test 1296
+The trained model for investigations and state-of-the-art (SOTA) improvements are detailed in the paper: [Tackling Hallucinations in Neural Chart Summarization](https://aclanthology.org/2023.inlg-main.30/).
 
-# Example use: 
-Append ```C2T: ``` before every input to the model
+### Abstract
 
+Hallucinations in text generation occur when the system produces text that is not grounded in the input. In this work, we address the problem of hallucinations in neural chart summarization. Our analysis reveals that the target side of chart summarization training datasets often contains additional information, leading to hallucinations. We propose a natural language inference (NLI) based method to preprocess the training data and demonstrate through human evaluation that our approach significantly reduces hallucinations. Additionally, we found that shortening long-distance dependencies in the input sequence and adding chart-related information such as titles and legends enhances overall performance.
 
-```
-tokenizer = AutoTokenizer.from_pretrained(saadob12/t5_C2T_autochart)
-model =   AutoModelForSeq2SeqLM.from_pretrained(saadob12/t5_C2T_autochart)
+## Main Findings from the Paper
+
+- **Enhanced Context Provision:** Emphasizing the importance of providing more context and reducing long-distance dependencies in the input format.
+- **NLI Cleaning Step:** Introducing an NLI-based cleaning step to eliminate ungrounded information in the training data.
+- **Reduction of Intrinsic Hallucinations:** Demonstrating that reducing long-distance dependencies and adding more context leads to fewer intrinsic hallucinations.
+- **Cause of Extrinsic Hallucinations:** Identifying that extrinsic hallucinations are caused by ungrounded information in training summaries.
+- **Human Evaluation Results:** Showing that using NLI to filter training summaries significantly reduces hallucinations.
+
+## Training Data
+
+### GitHub Link for the Original AutoChart Data
+
+[AutoChart](https://gitlab.com/bottle_shop/snlg/chart/autochart)
+
+### Training Details
+
+The model was trained using optimized prompts based on findings from the paper [Tackling Hallucinations in Neural Chart Summarization](https://aclanthology.org/2023.inlg-main.30/).
+
+- **Optimized Prompt Dataset:** [Hallucinations-C2T Data](https://github.com/WorldHellow/Hallucinations-C2T/tree/main/data)
+
+## Example Use
+
+### Input Prompt Template
+
+The input prompt template consists of the `title, x-y labels, and x-y values`.
 
-data = 'Trade statistics of Qatar with developing economies in North Africa  bar_chart Year-Trade with economies of Middle East & North Africa(%)(Merchandise             exports,Merchandise imports) x-y1-y2 values 2000 0.591869968616745 3.59339030672154 , 2001 0.53415012207203 3.25371165779341 , 2002 3.07769793440318 1.672796364224 , 2003 0.6932513078579471 1.62522475477827 , 2004 1.17635914189321 1.80540331396412'
+**Appending C2T Prefix:**
+Before every input to the model, append `C2T:`.
+
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("saadob12/t5_C2T_big")
+model = AutoModelForSeq2SeqLM.from_pretrained("saadob12/t5_C2T_big")
+
+data = 'Breakdown of coronavirus (COVID-19) deaths in South Korea as of March 16, 2020\n' \
+    'by chronic disease x-y labels\n' \
+    'Response - Share of cases, x-y values Circulatory system disease* 62.7%, ' \
+    'Endocrine and metabolic diseases** 46.7%, Mental illness*** 25.3%, ' \
+    'Respiratory diseases*** 24%, Urinary and genital diseases 14.7%, Cancer 13.3%, ' \
+    'Nervous system diseases 4%, Digestive system diseases 2.7%, Blood and hematopoietic diseases 1.3%' \
 
 prefix = 'C2T: '
-tokens = tokenizer.encode(prefix + data,  truncation=True, padding='max_length', return_tensors='pt')
+tokens = tokenizer.encode(prefix + data, truncation=True, padding='max_length', return_tensors='pt')
 generated = model.generate(tokens, num_beams=4, max_length=256)
 tgt_text = tokenizer.decode(generated[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
-summary = str(tgt_text).strip('[]""')
-#Summary: This barchart shows the number of trade statistics of qatar with developing economies in north africa from 2000 through 2004. The unit of measurement in this graph is Trade with economies of Middle East & North Africa(%) as shown on the y-axis. The first group data denotes the change of Merchandise exports. There is a go up and down trend of the number. The peak of the number is found in 2002 and the lowest number is found in 2001. The changes in the number may be related to the conuntry's national policies. The second group data denotes the change of Merchandise imports. There is a go up and down trend of the number. The number in 2000 being the peak, and the lowest number is found in 2003. The changes in the number may be related to the conuntry's national policies. 
+summary = tgt_text.strip('[]""')
+# Summary:
+#As of March 16, 2020, around 62.7% of COVID-19 deaths in South Korea were related to circulatory system diseases.
+#Other chronic diseases include endocrine and metabolic diseases, mental illness, and cancer.
+#South Korea confirmed 30,017 infection cases, including 501 deaths.
 ```
-# Limitations
+# Intended Use and Limitations
 You can use the model to generate summaries of data files.
 Works well for general statistics like the following: 
 
@@ -39,21 +77,29 @@ May or may not generate an **okay** summary at best for the following kind of da
 
 | Model  | BLEU score  | BLEURT|
 |:---:|:---:|:---:|
-| t5-small  | 25.4  | -0.11 | 
-| t5-base  | 28.2 | 0.12 |
-| t5-large  |  35.4 | 0.34 |
-
+| bert-small  | 25.4  | -0.11 | 
+| bert-base  | 28.2 | 0.12 |
+| bert-large  |  35.4 | 0.34 |
 
 
 # Citation
-
 Kindly cite my work. Thank you.
 ``` 
-  @misc{obaid ul islam_2022, 
-      title={saadob12/t5_C2T_autochart Hugging Face}, 
-      url={https://huggingface.co/saadob12/t5_C2T_autochart}, 
-      journal={Huggingface.co}, 
-      author={Obaid ul Islam, Saad}, 
-      year={2022} 
-  }
-```
+@inproceedings{obaid-ul-islam-etal-2023-tackling,
+    title = {Tackling Hallucinations in Neural Chart Summarization},
+    author = {Obaid ul Islam, Saad and Škrjanec, Iza and Dusek, Ondrej and Demberg, Vera},
+    booktitle = {Proceedings of the 16th International Natural Language Generation Conference},
+    month = sep,
+    year = {2023},
+    address = {Prague, Czechia},
+    publisher = {Association for Computational Linguistics},
+    url = {https://aclanthology.org/2023.inlg-main.30},
+    doi = {10.18653/v1/2023.inlg-main.30},
+    pages = {414--423},
+    abstract = {Hallucinations in text generation occur when the system produces text that is not grounded in the input. In this work, we tackle the problem of hallucinations in neural chart summarization. Our analysis shows that the target side of chart summarization training datasets often contains additional information, leading to hallucinations. We propose a natural language inference (NLI) based method to preprocess the training data and show through human evaluation that our method significantly reduces hallucinations. We also found that shortening long-distance dependencies in the input sequence and adding chart-related information like title and legends improves the overall performance.}
+}
+
+```
+
+
+