Sync with data tooling repo, using edugp/kenlm models, updating viz to use quantiles for coloring and ad-hoc viz for the registry dataset

Run tokenizer before computing perplexity and format

Replicate default cc_net preprocessing at inference time on KenlmModel.get_perplexity

Format file

Add distiluse-base-multilingual-cased-v2

Add tests and fix issue when splitting into sentences, to grab the minimum number between total sentences and sample size, rather than total original documents and sample size

Update README

Check if file exists before attempting to remove

Remove also the '{language}.sp.model' file on failure

Add CLI and refactor

Remove corrupt KenLM model files.

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.

Rename text input description

Use latest embedding-lenses

Upgrade dependencies to match embedding-lenses

Upgrade streamlit

Install embedding-lenses from wheel

Fix issue with csv files

Do not lock the version of embedding-lenses

Update requirements.txt to install kenlm

Inital commit for perplexity lenses

initial commit