leo-pekelis-gradient
commited on
Commit
•
126be6a
1
Parent(s):
32295b6
Update README.md
Browse files
README.md
CHANGED
@@ -9,13 +9,13 @@ license: llama3
|
|
9 |
---
|
10 |
<a href="https://www.gradient.ai" target="_blank"><img src="https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/TSa3V8YpoVagnTYgxiLaO.png" width="200"/></a>
|
11 |
|
12 |
-
# Llama-3 70B Gradient Instruct
|
13 |
|
14 |
Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. If you're looking to build custom AI models or agents, email us a message contact@gradient.ai.
|
15 |
|
16 |
For more info see our [End-to-end development service for custom LLMs and AI systems](https://gradient.ai/development-lab)
|
17 |
|
18 |
-
This model extends LLama-3 70B's context length from 8k to >
|
19 |
|
20 |
**[ADD EVAL PLOT HERE]**
|
21 |
|
|
|
9 |
---
|
10 |
<a href="https://www.gradient.ai" target="_blank"><img src="https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/TSa3V8YpoVagnTYgxiLaO.png" width="200"/></a>
|
11 |
|
12 |
+
# Llama-3 70B Gradient Instruct 1048K
|
13 |
|
14 |
Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. If you're looking to build custom AI models or agents, email us a message contact@gradient.ai.
|
15 |
|
16 |
For more info see our [End-to-end development service for custom LLMs and AI systems](https://gradient.ai/development-lab)
|
17 |
|
18 |
+
This model extends LLama-3 70B's context length from 8k to > 1048K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 34M tokens for this stage, and ~430M tokens total for all stages, which is < 0.003% of Llama-3's original pre-training data.
|
19 |
|
20 |
**[ADD EVAL PLOT HERE]**
|
21 |
|