Commit
•
0aecafc
1
Parent(s):
d139d98
remove added tokens
Browse files
tokenizer/added_tokens.json
DELETED
|
@@ -1,4 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"<|endoftext|>": 49407,
|
| 3 |
-
"<|startoftext|>": 49406
|
| 4 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tokenizer_2/added_tokens.json
DELETED
|
@@ -1,5 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"!": 0,
|
| 3 |
-
"<|endoftext|>": 49407,
|
| 4 |
-
"<|startoftext|>": 49406
|
| 5 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|