chunked/content_aware_chunking/_philosophy/chunk_21.txt

A tokenizer stores the vocabulary for each model and provide methods for encoding and decoding strings in a list of token embedding indices to be fed to a model.