A bit about me.
Growing up, I have cared about computers, but I was fascinated by language. In college I studied philosophy and math. I cared about the structure of reasoning, the limits of knowledge, and the nature of language.
ChatGPT took the world by storm my senior year. Before 2022, I thought philosophy was the best subject for studying the nature of language and reason. But philosophy hasn't seen much progress over the past 40 years. And transformer-based architectures are taking the world by storm.
The world of tech is trying to teach computers how to reason. I want to contribute to that process.
This is my first contribution, a small langauge model trained from scratch. It's about 3 million parameters. It was pretrained on the entire Western Canon--or, at least, 70 million tokens of the Western Canon. I finetuned it on the essays I wrote in college.
I will add the dataset to Hugging Face. It is still a work in progress, but I believe it represents the best open source repository of its kind.
To set up the environment, ensure you have Conda installed. Then, run the following command to create the environment with all necessary dependencies:
conda env create -f environment.yml
To test the model, just run sample.py. The weights are in test_safetensors/finetuned_plato_v1_step_880.safetensors. For the GPT module, go to gpt_layer.py. To see the code I used to run the model, go to train_on_canon.py.
You can skip trimmingcanon.py. It is a collection of scripts I used to remove noise from the datset.
The dataset for this model is on Hugging Face.
https://huggingface.co/datasets/wordgrammer/The_Entire_Western_Canon
This model is not very good. A 3 million parameter model is not large enough to understand philosophy. In fact, most of the frontier models struggle to understand philosophy. I made this primarily as a way to understand the inner workings of a transformer from first principles. I would only recommend building a language model from scratch for this reason. It is much more cost-effective to finetune an open source model. But as an educational experience, training a model from scratch is invaluable.
I will not update this repository further.
Many thanks to the following resources:
- The math behind attention. https://x-dev.pages.jsc.fz-juelich.de/2022/07/13/transformers-matmul.html
- Useful diagram for the parts of a Transformer beyond attention: https://en.m.wikipedia.org/wiki/Generative_pre-trained_transformer
- The Scaling Laws paper Kaplan et al: https://arxiv.org/pdf/2001.08361
- The Chinchilla paper to calculate model hyperparameters: https://arxiv.org/pdf/2203.15556
- Andrej Karpathy's nanoGPT repository, plus his various other tutorials. https://github.com/karpathy/nanoGPT
- The MLX team.
- Open Source ebook providers that wish to remain unnamed.
- I didn’t use this paper, but I wish I had it when I started this project: https://arxiv.org/pdf/2207.09238