File size: 955 Bytes
4063bc6 529624e 5ff2505 529624e 44cc84a 529624e 5ff2505 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
---
license: apache-2.0
datasets:
- enjalot/fineweb-edu-sample-10BT-chunked-500-nomic-text-v1.5
language:
- en
---
# Latent SAE
A series of SAEs trained on embeddings from [nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5)
The SAEs were trained on the 100BT sample of Fineweb-EDU, see an example of the [10BT sample of Fineweb-Edu](https://huggingface.co/datasets/enjalot/fineweb-edu-sample-10BT-chunked-500).
Run the models or train your own with [Latent SAE](https://github.com/enjalot/latent-sae) which is heavily borrowing from https://github.com/EleutherAI/sae
# Training
The models were trained using Modal Labs infrastructure with the command:
```bash
modal run train_modal.py --batch-size 512 --grad-acc-steps 4 --k 64 --expansion-factor 32
```
Error and dead latents charts can be seen here:
|