phanerozoic
commited on
Commit
•
ef324db
1
Parent(s):
35e42b9
Update README.md
Browse files
README.md
CHANGED
@@ -81,65 +81,58 @@ The training process resulted in the following performance metrics:
|
|
81 |
- **ROUGE-2**: 21.06
|
82 |
- **ROUGE-L**: 30.65
|
83 |
|
84 |
-
## Comparing Performance to Base
|
85 |
|
86 |
-
The performance of BART-Large-CNN-scratch is compared against Facebook's base BART-large-cnn model
|
87 |
|
88 |
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|
89 |
|--------------------------------|---------|---------|---------|
|
90 |
| Facebook BART-large-cnn | 42.949 | 20.815 | 30.619 |
|
91 |
-
| Enhanced BART-large-cnn | 45.370 | 22.000 | 31.170 |
|
92 |
| BART-Large-CNN-scratch | 44.070 | 21.060 | 30.650 |
|
93 |
|
94 |
-
### Analysis of
|
95 |
|
96 |
-
####
|
97 |
-
- **Facebook BART-large-cnn**: 42.949
|
98 |
-
- **Enhanced BART-large-cnn**: 45.370
|
99 |
-
- **BART-Large-CNN-scratch**: 44.070
|
100 |
|
101 |
-
|
|
|
102 |
|
103 |
-
|
104 |
-
-
|
105 |
-
- **Enhanced BART-large-cnn**: 22.000
|
106 |
-
- **BART-Large-CNN-scratch**: 21.060
|
107 |
|
108 |
-
|
|
|
|
|
|
|
109 |
|
110 |
-
####
|
111 |
-
- **Facebook BART-large-cnn**: 30.619
|
112 |
-
- **Enhanced BART-large-cnn**: 31.170
|
113 |
-
- **BART-Large-CNN-scratch**: 30.650
|
114 |
|
115 |
-
|
|
|
116 |
|
117 |
-
|
118 |
-
|
119 |
-
1. **Reproducibility**:
|
120 |
-
- The BART-Large-CNN-scratch model successfully reproduced the performance of the Facebook BART-large-cnn model. This is evidenced by the close match in ROUGE scores and identical summaries generated for the same input text. This confirms the robustness and reliability of the BART architecture and the training methodology when applied to the CNN/DailyMail dataset.
|
121 |
-
|
122 |
-
2. **Enhanced Model Comparison**:
|
123 |
-
- The enhanced BART-large-cnn model, which was fine-tuned for an additional epoch, shows slightly better ROUGE scores compared to both the Facebook BART-large-cnn and BART-Large-CNN-scratch models. This indicates that additional fine-tuning can further improve the model's performance in capturing relevant information and generating coherent summaries.
|
124 |
|
125 |
-
|
126 |
-
|
|
|
|
|
127 |
|
128 |
-
|
129 |
-
- The BART-Large-CNN-scratch model is highly effective for text summarization tasks in English, particularly for news articles. It can be applied in various domains such as news aggregation, content summarization, and information retrieval where concise and accurate summaries are essential.
|
130 |
-
|
131 |
-
### Overall Appraisal
|
132 |
|
133 |
-
|
|
|
134 |
|
135 |
-
|
|
|
136 |
|
137 |
-
|
|
|
138 |
|
139 |
-
|
140 |
|
141 |
-
|
142 |
|
143 |
## Acknowledgments
|
144 |
|
145 |
-
Special thanks to the developers of the BART architecture and the Hugging Face team. Their tools and frameworks were instrumental in the development and fine-tuning of this model. The NVIDIA RTX 6000 Ada Lovelace hardware provided the necessary computational power to achieve these results.
|
|
|
81 |
- **ROUGE-2**: 21.06
|
82 |
- **ROUGE-L**: 30.65
|
83 |
|
84 |
+
## Comparing Performance to Base Model
|
85 |
|
86 |
+
The performance of BART-Large-CNN-scratch is compared against Facebook's base BART-large-cnn model:
|
87 |
|
88 |
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|
89 |
|--------------------------------|---------|---------|---------|
|
90 |
| Facebook BART-large-cnn | 42.949 | 20.815 | 30.619 |
|
|
|
91 |
| BART-Large-CNN-scratch | 44.070 | 21.060 | 30.650 |
|
92 |
|
93 |
+
### Analysis of Summaries
|
94 |
|
95 |
+
#### Eiffel Tower Article Summary Comparison
|
|
|
|
|
|
|
96 |
|
97 |
+
##### Facebook BART-Large-CNN Summary:
|
98 |
+
"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world."
|
99 |
|
100 |
+
##### BART-Large-CNN-scratch Summary:
|
101 |
+
"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. Its base is square, measuring 125 metres (410 ft) on each side. It is the second tallest free-standing structure in France after the Millau Viaduct."
|
|
|
|
|
102 |
|
103 |
+
- **Comparison**:
|
104 |
+
- Both summaries start with identical descriptions of the Eiffel Tower's height and base dimensions.
|
105 |
+
- The Facebook summary mentions the historical significance of the Eiffel Tower surpassing the Washington Monument.
|
106 |
+
- The scratch summary includes the detail of the Eiffel Tower being the second tallest free-standing structure in France, providing a different historical context.
|
107 |
|
108 |
+
#### Paper Clip Article Summary Comparison
|
|
|
|
|
|
|
109 |
|
110 |
+
##### Facebook BART-Large-CNN Summary:
|
111 |
+
"The earliest form of the paper clip dates back to the 13th century. The most widely recognized design is attributed to the Norwegian inventor Johan Vaaler. The design of paper clips has continued to evolve, with various shapes and sizes available on the market. During World War II, paper clips became a symbol of resistance in Norway."
|
112 |
|
113 |
+
##### BART-Large-CNN-scratch Summary:
|
114 |
+
"The paper clip dates back to the 13th century, when a device made of a bent metal wire was used to hold sheets of paper together. The most widely recognized design is attributed to the Norwegian inventor Johan Vaaler, who received a patent for his paper clip design in 1899. During World War II, the paper clip became a symbol of resistance in Norway."
|
|
|
|
|
|
|
|
|
|
|
115 |
|
116 |
+
- **Comparison**:
|
117 |
+
- Both summaries start with descriptions of the origins of the paper clip and Johan Vaaler's contributions.
|
118 |
+
- The Facebook summary briefly mentions the evolution of paper clip designs and their availability in various shapes and sizes.
|
119 |
+
- The scratch summary includes additional historical details about the use of bent metal wires in the 13th century and Vaaler's patent, providing a richer historical context.
|
120 |
|
121 |
+
### Implications
|
|
|
|
|
|
|
122 |
|
123 |
+
1. **Reproducibility**:
|
124 |
+
- The BART-Large-CNN-scratch model closely reproduces the performance of the Facebook BART-large-cnn model, capturing key historical points and providing concise summaries. However, it shows some differences in detail prioritization, indicating that while the reproduction is effective, it is not exact.
|
125 |
|
126 |
+
2. **Model Training from Scratch**:
|
127 |
+
- Training from scratch has proven to be effective, with the BART-Large-CNN-scratch model achieving competitive ROUGE scores. However, the summaries differ in detail compared to the Facebook model, suggesting areas for further fine-tuning.
|
128 |
|
129 |
+
3. **Practical Applications**:
|
130 |
+
- Both models are effective for summarizing historical and technical articles. The BART-Large-CNN-scratch model is excellent for concise overviews, while the Facebook model provides more comprehensive context.
|
131 |
|
132 |
+
### Conclusion
|
133 |
|
134 |
+
The BART-Large-CNN-scratch model demonstrates strong performance, capturing essential historical points and providing concise summaries. While it does not exactly reproduce the Facebook model's summaries, it achieves similar quality and even exceeds in ROUGE scores. This makes it a robust tool for text summarization applications.
|
135 |
|
136 |
## Acknowledgments
|
137 |
|
138 |
+
Special thanks to the developers of the BART architecture and the Hugging Face team. Their tools and frameworks were instrumental in the development and fine-tuning of this model. The NVIDIA RTX 6000 Ada Lovelace hardware provided the necessary computational power to achieve these results.
|