---
library_name: transformers
---
## How to run it

### setup enviroment

```
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

pip install fbgemm-gpu==0.8.0rc4

Download the enablement fork, https://huggingface.co/sllhf/transformers_enablement_fork/tree/main

unzip the file

cd transformers

# add changes from this PR https://github.com/huggingface/transformers/pull/32047

git fetch origin pull/32047/head:new-quant-method

git merge new-quant-method


pip install -e .

git clone https://github.com/huggingface/accelerate.git

cd accelerate

pip install -e .

# Next install vLLM PR https://github.com/vllm-project/vllm/pull/6559

git clone https://github.com/vllm-project/vllm.git

cd vllm

git fetch origin pull/6559/head:fbgemm-checkpoints

git checkout fbgemm-checkpoints

pip install -e .



```

### Load back the HF model 

```
from transformers import FbgemmFp8Config, AutoModelForCausalLM, AutoTokenizer

model_name = "sllhf/Meta-Llama-3.1-405B-Instruct-FP8"

quantization_config = FbgemmFp8Config()
quantized_model = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

# make sure to set up your own params, temperature, top_p etc.

output = quantized_model.generate(**input_ids, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### How to run in it with vLLM

```
from vllm import LLM
model = LLM("sllhf/Meta-Llama-3.1-405B-Instruct-FP8")

```