language:
- en
library_name: nemo
datasets:
- the_pile
tags:
- text generation
- pytorch
- causal-lm
license: cc-by-4.0
Palmyra-20B
Model Description
Model description Palmyra was primarily pretrained with English text, there is still a trace amount of non-English data present within the training corpus that was accessed through CommonCrawl. A causal language modeling (CLM) objective was utilized during the process of the model's pretraining. Similar to GPT-3, Palmyra is a member of the same family of models that only contain a decoder. As a result, it was pretrained utilizing the objective of self-supervised causal language modeling. Palmyra uses the prompts and general experimental setup from GPT-3 in order to conduct its evaluation in accordance with GPT-3. Read the official paper if you want more information about this.
Getting started
Step 1: Install NeMo and dependencies
You will need to install NVIDIA Apex and NeMo.
git clone https://github.com/ericharper/apex.git
cd apex
git checkout nm_v1.11.0
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
pip install nemo_toolkit['nlp']==1.11.0
Step 2: Launch eval server
Note. The example below launches a model variant with Tensor Parallelism (TP) of 4 and Pipeline Parallelism (PP) of 1 on two GPUs.
git clone https://github.com/NVIDIA/NeMo.git
cd NeMo/examples/nlp/language_modeling
git checkout v1.11.0
python megatron_gpt_eval.py gpt_model_file=palmyara_gpt_20b.nemo server=True tensor_model_parallel_size=4 trainer.devices=4
Step 3: Send prompts to your model!
import json
import requests
port_num = 5555
headers = {"Content-Type": "application/json"}
def request_data(data):
resp = requests.put('http://localhost:{}/generate'.format(port_num),
data=json.dumps(data),
headers=headers)
sentences = resp.json()['sentences']
return sentences
data = {
"sentences": ["Tell me an interesting fact about space travel."]*1,
"tokens_to_generate": 50,
"temperature": 1.0,
"add_BOS": True,
"top_k": 0,
"top_p": 0.9,
"greedy": False,
"all_probs": False,
"repetition_penalty": 1.2,
"min_tokens_to_generate": 2,
}
sentences = request_data(data)
print(sentences)
Training Data
| part | MassiveText (sampling) | tokens (B) | url | sampling ratio |
|---|---|---|---|---|
| mc4 filtered | MassiveWeb (48%) | 1331 | gs://mc4/final/web | 58% |
| TrustedWeb | - | - | gs://mc4/final/trusted_web | - |
| realnews | News (10%) | 21 | gs://mc4/final/news | 10% |
| c4 | c4 (10%) | - | gs://mc4/final/c4 | - |
| wikipedia-40B | wikipedia (2%) | 2 | gs://mc4/final/wikipedia | 5% |
| github | github (3%) | - | gs://mc4/final/github | - |
| books | books (27%) | 24 | gs://mc4/final/books | 27% |
| youtube | - | - | gs://mc4/final/youtube | - |
Evaluation results
Zero-shot performance. Evaluated using LM Evaluation Test Suite from AI21
| ARC-Challenge | ARC-Easy | RACE-middle | RACE-high | Winogrande | RTE | BoolQA | HellaSwag | PiQA |
|---|---|---|---|---|---|---|---|---|
| 0.3976 | 0.5566 | 0.5007 | 0.4171 | 0.6133 | 0.5812 | 0.6356 | 0.6298 | 0.7492 |
Limitations
The model was trained on the data originally crawled from the Internet. This data contains toxic language and societal biases. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts.
References
[1] Improving Language Understanding by Generative Pre-Training
[2] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
[4] The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Licence
License to use this model is covered by the CC-BY-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-4.0 license.