metadata

license: apache-2.0
datasets:
  - enjalot/fineweb-edu-sample-10BT-chunked-500-nomic-text-v1.5
language:
  - en

Latent SAE

A series of SAEs trained on embeddings from nomic-embed-text-v1.5

The SAEs were trained on the 100BT sample of Fineweb-EDU, see an example of the 10BT sample of Fineweb-Edu.

Run the models or train your own with Latent SAE which is heavily borrowing from https://github.com/EleutherAI/sae

Training

The models were trained using Modal Labs infrastructure with the command:

modal run train_modal.py --batch-size 512 --grad-acc-steps 4 --k 64 --expansion-factor 32

Error and dead latents charts can be seen here: