EricB's picture
EricB HF staff
Create README.md
b7dc7ff verified
|
raw
history blame
1.58 kB
metadata
tags:
  - uqff
  - mistral.rs
base_model: google/gemma-1.1-7b-it
base_model_relation: quantized

google/gemma-1.1-7b-it, UQFF quantization

Run with mistral.rs. Documentation: UQFF docs.

  1. Flexible ๐ŸŒ€: Multiple quantization formats in one file format with one framework to run them all.
  2. Reliable ๐Ÿ”’: Compatibility ensured with embedded and checked semantic versioning information from day 1.
  3. Easy ๐Ÿค—: Download UQFF models easily and quickly from Hugging Face, or use a local file.
  4. Customizable ๐Ÿ› ๏ธ: Make and publish your own UQFF files in minutes.

Files

Quantization type(s) Example
FP8 ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-f8e4m3.uqff
HQQ4 ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-hqq4.uqff
HQQ8 ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-hqq8.uqff
Q3K ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-q3k.uqff
Q4K ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-q4k.uqff
Q5K ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-q5k.uqff
Q8_0 ./mistralrs-server -i plain -m EricB/gemma-1.1-7b-it-UQFF --from-uqff gemma1.1-7b-instruct-q8_0.uqff