Hugging Face's logo

amd
/
Llama-3.1-8B-Instruct-FP8-KV

Safetensors
llama
fp8
Model card Files Community
14
Llama-3.1-8B-Instruct-FP8-KV / README.md
luow-amd's picture
luow-amd
Update Readme.md (#3)
f7aa269 verified
preview code
|
raw
history blame
2.51 kB
metadata
license: llama3.1

Meta-Llama-3.1-8B-Instruct-FP8-KV

This model was created by applying Quark with calibration samples from Pile dataset.

  • Quantization Stragegy

    • Quantized Layers：All linear layers excluding "lm_head"
    • Weight：FP8 symmetric per-tensor
    • Activation: FP8 symmetric per-tensor
    • KV Cache: FP8 symmetric per-tensor

  • Quick Start

  1. Download and install Quark
  2. Run the quantization script in the example folder using the following command line:
export MODEL_DIR = [local model checkpoint folder] or meta-llama/Meta-Llama-3.1-8B-Instruct 
# single GPU
python3 quantize_quark.py \ 
        --model_dir $MODEL_DIR \
        --output_dir llama31_8b_amd \                           
        --quant_scheme w_fp8_a_fp8 \
        --kv_cache_dtype fp8 \
        --num_calib_data 128  \
        --model_export quark_safetensors

# If model size is too large for single GPU, please use multi GPU instead.
python3 quantize_quark.py \ 
        --model_dir $MODEL_DIR \
        --output_dir llama31_8b_amd \                           
        --quant_scheme w_fp8_a_fp8 \
        --kv_cache_dtype fp8 \
        --num_calib_data 128  \
        --multi_gpu \
        --model_export quark_safetensors

Evaluation

Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.

Evaluation scores

Benchmark Meta-Llama-3.1-8B-Instruct Meta-Llama-3.1-8B-Instruct-FP8-KV(this model)
Perplexity-wikitext2 7.2169 7.2752

License

Copyright (c) 2018-2024 Advanced Micro Devices, Inc. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.