Add README

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- nlp
+- llm
+---
+# Amber
+
+<center><img src="amber_logo.png" alt="amber logo" width="300"/></center>
+
+We present Amber, the first model in the LLM360 family. Amber is an
+7B English language model with the LLaMA architecture. 
+
+## About LLM360
+LLM360 is an initiative for comprehensive and fully open-sourced LLMs, 
+where all training details, model checkpoints, intermediate results, and 
+additional analyses are made available to the community. Our goal is to advance 
+the field by inviting the community to deepen the understanding of LLMs 
+together. As the first step of the project LLM360, we release all intermediate 
+model checkpoints, our fully-prepared pre-training dataset, all source code and
+configurations, and training details. We are
+committed to continually pushing the boundaries of LLMs through this open-source 
+effort.
+
+Get access now at [LLM360 site](https://www.llm360.ai/)
+
+## Model Description
+
+- **Model type:** Language model with the same architecture as LLaMA-7B
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Resources for more information:**
+  - [Training Code](https://github.com/LLM360/amber-train)
+  - [Data Preparation](https://github.com/LLM360/amber-data-prep)
+  - [Metrics](https://github.com/LLM360/Analysis360)
+  - [Fully processed Amber pretraining data](https://huggingface.co/datasets/LLM360/AmberDatasets)
+
+
+# Loading Amber 
+
+To load a specific checkpoint, simply set the `CHECKPOINT_NUM` to a value between `0` and `359`. By default, checkpoints will be cached and not re-downloaded for future runs of the script. 
+
+```python
+from huggingface_hub import snapshot_download
+from transformers import LlamaTokenizer, LlamaForCausalLM
+
+CHECKPOINT_NUM = 359
+
+model_path = snapshot_download(
+    repo_id="LLM360/Amber",
+    repo_type="model",
+    allow_patterns=[f"ckpt_{CHECKPOINT_NUM:03}/*"],
+)
+
+tokenizer = LlamaTokenizer.from_pretrained(f"{model_path}/ckpt_{CHECKPOINT_NUM:03}")
+model = LlamaForCausalLM.from_pretrained(f"{model_path}/ckpt_{CHECKPOINT_NUM:03}")
+
+input_text = "translate English to German: How old are you?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids
+
+outputs = model.generate(input_ids)
+print(tokenizer.decode(outputs[0]))
+
+```
+
+# Amber Training Details
+
+## DataMix
+| Subset      | Tokens (Billion) |
+| ----------- | ----------- |
+| Arxiv      | 30.00       |
+| Book   | 28.86        |
+| C4   | 197.67        |
+| Refined-Web   | 665.01        |
+| StarCoder   | 291.92        |
+| StackExchange   | 21.75        |
+| Wikipedia   | 23.90        |
+| Total | 1259.13 |
+
+## Hyperparameters
+| Hyperparameter      | Value |
+| ----------- | ----------- |
+| Total Parameters      | 6.7B       |
+| Hidden Size   | 4096        |
+| Intermediate Size (MLPs)   | 11008        |
+| Number of Attention Heads   | 32        |
+| Number of Hidden Lyaers  | 32        |
+| RMSNorm ɛ  | 1e^-6        |
+| Max Seq Length   | 2048        |
+| Vocab Size | 32000 |
+
+| Training Loss                                              |
+|------------------------------------------------------------|
+| <img src="loss_curve.png" alt="loss curve" width="400"/> |
+
+
+# Evaluation
+
+Please refer to our [W&B project page](https://wandb.ai/llm360/CrystalCoder) for complete training logs and evaluation results.
+
+| ARC                                                 | HellSwag                                                  | 
+|------------------------------------------------------|------------------------------------------------------------|
+| <img src="amber-arc-curve.png" alt="arc" width="400"/> | <img src="amber-hellaswag-curve.png" alt="hellaswag" width="400"/> | 
+
+|MMLU                                                 | TruthfulQA                                                 |
+|-----------------------------------------------------|-----------------------------------------------------------|
+|<img src="amber-mmlu-curve.png" alt="mmlu" width="400"/> | <img src="amber-truthfulqa-curve.png" alt="truthfulqa" width="400"/> |
+
+# Citation
+
+Coming soon...