Corianas

/

llama-tiny-reactor

like 0

Hi. Why is your "pad_token": "", the same as the "eos_token": {
"content": "",

Why the "pad_token": definition does not include a "content" key? Do these anomalies cancel each other out?

What did you actually use the "pad_token" for in developing your model?

{
"bos_token": {
"content": "",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "",
"unk_token": {
"content": "",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}