updating with new datasets

README.md

@@ -45,23 +45,27 @@ print(result.text) # This sentence has bad grammar and is comrpessed.
 ## Model Description
 The `t5-small-spoken-typo` model is specifically designed to tackle the challenges of text correction within user-generated content, particularly in short, conversation-like sentences. It corrects for missing spaces, removes unnecessary punctuation, introduces and then corrects typos, and normalizes text by replacing informal contractions and abbreviations with their full forms.
 It has been training on
-- BNC 2014 Spoken
+- [BNC 2014 Spoken](http://cass.lancs.ac.uk/cass-projects/spoken-bnc2014/)
 - [Daily Dialog](https://huggingface.co/datasets/daily_dialog)
+- [Comm2 - AAC Text](https://www.aactext.org/comm2/)
+- [C4-200M - 25K Subset](https://huggingface.co/datasets/leslyarun/c4_200m_gec_train100k_test25k)
+- [JFLEG](https://huggingface.co/datasets/jhu-clsp/jfleg)
+
 
 Then injecting  typos from a range of places
+- **Using NLPAUG** We've made some typos in Comm2 by usiing this library https://github.com/makcedward/nlpaug
 - **Typo lists, Birkbeck, etc.**: These datasets contain lists of commonly misspelled words, making them invaluable for training models to recognize and correct spelling errors.
   - Find these resources [here](https://www.dcs.bbk.ac.uk/~ROGER/corpora.html).
 - **TOEFL Spell** A dataset of Spelling Annotations for English language learner essays written for TOEFL exams.
   - Find this [here](https://github.com/EducationalTestingService/TOEFL-Spell/tree/master) 
+- **Homonyms** We replace words in BNC and Dialy Dialog occasionally with homonyms from this list https://github.com/pimentel/homophones/
 
-And then compressing versions of the sentences (i.e. removing spaces)- both correct and typod
-
-We have also provided the C4-200M-250K subset data and the JFLEG dataset for base grammar correction
+And then compressing versions of the sentences (i.e. removing spaces)- both correct and typod we add to our dataset. (This is to solve a problem where some people write without spaces)
+Note we use a ``grammar: `` prefix for each sentence in training. 
 
 Full script to build the [dataset is here](https://colab.research.google.com/drive/1VkKU9KKIWkWQZ-pPzdDFLeRnwFxdWUtq?usp=sharing)
 
 
-
 ## Developed by:
 - **Name**: Will Wade
 - **Affiliation**: Research & Innovation Manager, Occupational Therapist, Ace Centre, UK
@@ -108,24 +112,71 @@ The model was trained on a curated subset of the DailyDialog and BNC corpora (20
 Sentences were stripped of apostrophes and commas, spaces were removed, and typos were introduced programmatically to simulate common errors in user-generated content.
 
 ### Speeds, Sizes, Times
-- Training was conducted on Google Colab, taking approximately 11 hrs to complete.
+- Training was conducted on LlambdaLabs, taking approximately 4 hrs to complete.
 
 # Evaluation
 
 ## Testing Data, Factors & Metrics
 
 ### Testing Data
-Evaluation was performed on a held-out test set derived from the same corpora and similar sentences, ensuring a diverse range of sentence structures and error types were represented.
+The evaluation was performed on a held-out test set derived from the same corpora and similar sentences, ensuring a diverse range of sentence structures and error types were represented.
 
-### Metrics
-Performance was measured using the accuracy of space insertion and typo correction alongside qualitative assessments of text normalisation.
 
 ## Results 
 The model demonstrates high efficacy in correcting short, erroneous sentences, with particular strength in handling real-world, conversational text.
 
-# Environmental Impact
+It performs nearly on par with GPTTurbo16k at around 93% sentence similarity. But there are gaps. 
+
+Take for example this output. 
+
+  Original: Didyoucatchthegamelastnight?
+  Corrected: Did you catch the game last night?
+  
+  Original: Wannagrabcoffeetomorrow?
+  Corrected: Wanna grab coffee tomorrow?
+  
+  Original: ImdyingsomeonecancellsoIcandogsitter!
+  Corrected: I'm dying someone cancell so I can dogsitter!
+  
+  Original: Hahahahahahahathats hilarious!
+  Corrected: Haha ha ha ha that's hilarious!
+  
+  Original: OMGyouneedtoseethelatestmeme!
+  Corrected: OMG you need to see the latest meme!
+  
+  Original: Seriouslythisweatherissocrazy!
+  Corrected: Seriously this weather is so crazy!
+  
+  Original: Whatchauptomefriend?
+  Corrected: What's his friend?
+  
+  Original: Feelingburntoutaftettodayhelp!
+  Corrected: Feeling burnt out today help!
+  
+  Original: Guesswhosingleagain!
+  Corrected: Guess who single again!
+  
+  Original: Youwontyoubelievewhatjusthappened!
+  Corrected: You want you believe what just happened!
+  
+  Original: Moviemarathonatmyplacethisweekend?
+  Corrected: Movie Marathon at my place this weekend?
+  
+  Original: Needstudymotivationanyideas?
+  Corrected: Need study motivation any ideas?
+  
+  Original: Sostressedaboutthispresentation!
+  Corrected: So stressed about this presentation!
+  
+  Original: Finallyfinishedthatbookyourecommended!
+  Corrected: Finally finished that book you're recommended!
+  
+  Original: Anygoodshowsbingeablelately?
+  Corrected: Any good shows biteable recently?
+
+
+
 
-The training was conducted with an emphasis on efficiency and minimising carbon emissions. Users leveraging cloud compute resources are encouraged to consider the environmental impact of large-scale model training and inference.
 
 # Technical Specifications