Spaces:

Xenova
/

the-tokenizer-playground

Running

Could I get a simple workflow to add custom tokenizer?

by sh0416 - opened Jan 22

sh0416

Jan 22

I want to see the tokenized result of phi tokenizer.
How to modify the code to add that tokenizer?
Thank you.

Xenova

Owner Jan 22

The source code for this demo can be found here, and you can modify it to add support for new tokenizers.

Xenova changed discussion status to closed Mar 7

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment