max lenght of the model

#70
by eqemen - opened

Incase you need to know the max lenght of the model is 5104 characters.

Has it changed recently? It was 1024 tokens and not 5104 tokens.
I had to create chunk and do recursive summarization to process long text.

Don't know. when I try directly from text it has some issues with generating.

Some of them solved when I reduce the character size to 5104. Other problems solve when I completely discard all the punctuations.

But these all resolved completely when I use tokenizer instead of pipeline.
With importing Tokenizier method, I will response all my text files and no need to remove any punctuation.

This comment has been hidden
eqemen changed discussion status to closed

god why I cannot delete my comments? ☹️

eqemen changed discussion status to open

what is the min_length of the model?

Sign up or log in to comment