Upload tokenizer

README.md

@@ -1,50 +1,43 @@
 ---
-license: apache-2.0
 language:
 - en
+license: apache-2.0
+tags:
+- NLP
 pipeline_tag: summarization
 widget:
-- text: >-
-    Maria: Alright, let's start dinner. How did everyone's day go?
-    Steven: It wasn't too bad. Had a couple of meetings but I did manage to finalize the design for the new project I've been working on.
-    Lilly: That's great, Dad! Finally!
-    Ryan: Cool, does that mean we get to see it now?
-    Steven: Well, I'll have to keep it under wraps for a bit longer, but soon.
-    Maria: Congrats, Steven. Lilly, how about you? Anything interesting at school today?
-    Lilly: Yeah, we had an astronaut from NASA come in and talk about the space missions he's been on. It was quite cool.
-    Steven: That sounds really interesting. Anything in particular that stood out?
-    Lilly: Well, he said he was about my age when he first started training. It’s really amazing how much they have to learn and experience.
-    Ryan: Did he talk about aliens?
-    Lilly: He did Ryan, but mostly about how we're not sure if they exist yet. It was thought-provoking.
-    Maria: Those experiences are definitely something to think about. Ryan, what about your day? Anything exciting in school?
-    Ryan: We did a geography quiz today. I got the highest marks!
-    Steven: That's my boy! Did you enjoy it?
-    Ryan: Yeah, it was fun! I want to try more at home.
-    Maria: Well, we can definitely help you with that. What do you think about setting up some geography puzzles this weekend?
-    Ryan: Yes, let's do it! Lilly, you're helping me.
-    Lilly: Absolutely, I'd love to help, Ryan. We can make it a fun game.
-    Steven: Sounds like a plan. Maria, remember we need to go grocery shopping this weekend.
-    Maria: Yes, thanks for reminding me. We’ll do that on Saturday afternoon.
-    Lilly: Can I go to the library after that? I need to pick up some books for my project.
-    Maria: Sure, just make sure to finish your homework before that.
-    Lilly: Deal.
-    Ryan: Can we go to the park on Saturday?
-    Steven: If it's not raining, sure. We'll all go together.
-    Maria: Sounds like we've got a fun weekend planned. Let's make sure to finish all the chores tomorrow so we have time.
-    Lilly: Agreed.
-    Ryan: Can I get extra dessert tonight?
-    Maria: Only if you finish your vegetables, Ryan.
-    Ryan: Deal, but only because you make the best roasted broccoli.
-    Steven: Good job, Ryan. And Maria, thanks for the dinner. It was perfect.
-    Maria: You're welcome, everyone. Enjoy your meal.
-    Lilly: Is it time for some dessert now?
-    Ryan: Yes, dessert time!
-    Maria: Alright, dessert it is. But remember, tomorrow is chore day, okay?
-    Everyone: Deal!
-
+- text: 'Maria: Alright, let''s start dinner. How did everyone''s day go? Steven:
+    It wasn''t too bad. Had a couple of meetings but I did manage to finalize the
+    design for the new project I''ve been working on. Lilly: That''s great, Dad! Finally!
+    Ryan: Cool, does that mean we get to see it now? Steven: Well, I''ll have to keep
+    it under wraps for a bit longer, but soon. Maria: Congrats, Steven. Lilly, how
+    about you? Anything interesting at school today? Lilly: Yeah, we had an astronaut
+    from NASA come in and talk about the space missions he''s been on. It was quite
+    cool. Steven: That sounds really interesting. Anything in particular that stood
+    out? Lilly: Well, he said he was about my age when he first started training.
+    It’s really amazing how much they have to learn and experience. Ryan: Did he talk
+    about aliens? Lilly: He did Ryan, but mostly about how we''re not sure if they
+    exist yet. It was thought-provoking. Maria: Those experiences are definitely something
+    to think about. Ryan, what about your day? Anything exciting in school? Ryan:
+    We did a geography quiz today. I got the highest marks! Steven: That''s my boy!
+    Did you enjoy it? Ryan: Yeah, it was fun! I want to try more at home. Maria: Well,
+    we can definitely help you with that. What do you think about setting up some
+    geography puzzles this weekend? Ryan: Yes, let''s do it! Lilly, you''re helping
+    me. Lilly: Absolutely, I''d love to help, Ryan. We can make it a fun game. Steven:
+    Sounds like a plan. Maria, remember we need to go grocery shopping this weekend.
+    Maria: Yes, thanks for reminding me. We’ll do that on Saturday afternoon. Lilly:
+    Can I go to the library after that? I need to pick up some books for my project.
+    Maria: Sure, just make sure to finish your homework before that. Lilly: Deal.
+    Ryan: Can we go to the park on Saturday? Steven: If it''s not raining, sure. We''ll
+    all go together. Maria: Sounds like we''ve got a fun weekend planned. Let''s make
+    sure to finish all the chores tomorrow so we have time. Lilly: Agreed. Ryan: Can
+    I get extra dessert tonight? Maria: Only if you finish your vegetables, Ryan.
+    Ryan: Deal, but only because you make the best roasted broccoli. Steven: Good
+    job, Ryan. And Maria, thanks for the dinner. It was perfect. Maria: You''re welcome,
+    everyone. Enjoy your meal. Lilly: Is it time for some dessert now? Ryan: Yes,
+    dessert time! Maria: Alright, dessert it is. But remember, tomorrow is chore day,
+    okay? Everyone: Deal!'
 example_title: Conversation Arc Example 1
-tags:
-- NLP
 ---
 # Model Card for Conversation Arc PredictorArc of the Conversation Model
 ## Model Details

tokenizer.json

@@ -1,6 +1,11 @@
 {
   "version": "1.0",
-  "truncation": null,
+  "truncation": {
+    "direction": "Right",
+    "max_length": 1024,
+    "strategy": "LongestFirst",
+    "stride": 0
+  },
   "padding": null,
   "added_tokens": [
     {