Special Tokens
#4
by
strangelex42
- opened
Hello, thanks for you work!
I see that model contain special tokens:
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
These tokens trained? If yes, can you give more details if not, how that tokens initialized? Thank you!
no, we did not train that tokens, it's just inherited from Qwen tokenizer, but I think it's used only in Qwen vision models
heylimon
changed discussion status to
closed