Edit model card

Llama-3-8B-Instruct-Gradient-4194k-GGUF

Fixing prompt format issues

  • Use iMatrix for Llama 3 prompt format on Q4 and below, or try Q4_K_M fixed
  • Use ChatML for Q6 and below
  • Use Llama 3, see issues

Issues

  • Context length is not defined correctly in quant, not sure if this is a llama.cpp issue Use RoPE settings
  • Output ends with or other EOS tokens, might be an issue with their training data

This model was converted to GGUF format from gradientai/Llama-3-8B-Instruct-Gradient-4194k using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Filename Quant Size Description
llama-3-8b-instruct-gradient-4194k.f16.gguf f16 14.9GB Lossless
llama-3-8b-instruct-gradient-4194k.Q8_0.gguf Q8_0 8.54GB Extremely high quality
llama-3-8b-instruct-gradient-4194k.Q6_K.gguf Q6_K 6.60GB Very high quality, near perfect, recommended.
llama-3-8b-instruct-gradient-4194k.Q5_K_M.gguf Q5_K_M 5.73GB High quality
llama-3-8b-instruct-gradient-4194k.Q5_K_S.gguf Q5_K_S 5.60GB Even higher quality
llama-3-8b-instruct-gradient-4194k.Q4_K_M.gguf Q4_K_M 4.92GB Recommended, medium-high quality
llama-3-8b-instruct-gradient-4194k.Q4_K_M.fixed.gguf Q4_K_M (fixed) 4.92GB Fixed version (requanted)
llama-3-8b-instruct-gradient-4194k.Q4_K_S.gguf Q4_K_S 4.69GB Recommended, medium quality
llama-3-8b-instruct-gradient-4194k.Q4_0.gguf Q4_0 4.66GB Usable, better than Q3 but worse than Q4
llama-3-8b-instruct-gradient-4194k.Q3_K_L.gguf Q3_K_L 4.32GB Usable
llama-3-8b-instruct-gradient-4194k.Q3_K_M.gguf Q3_K_M 4.02GB Bad quality, use Q4
llama-3-8b-instruct-gradient-4194k.Q3_K_S.gguf Q3_K_S 3.66GB Not recommended
llama-3-8b-instruct-gradient-4194k.Q2_K.gguf Q2_K 2.95GB Very low quality, would not use on 8b models

Use with llama.cpp

Install llama.cpp through brew.

brew install ggerganov/ggerganov/llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo leafspark/llama-3-8b-instruct-gradient-4194k.Q8_0-GGUF --model llama-3-8b-instruct-gradient-4194k.Q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo leafspark/llama-3-8b-instruct-gradient-4194k.Q8_0-GGUF --model llama-3-8b-instruct-gradient-4194k.Q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

git clone https://github.com/ggerganov/llama.cpp &&             cd llama.cpp &&             make &&             ./main -m llama-3-8b-instruct-gradient-4194k.Q8_0.gguf -n 128
Downloads last month
1,055
GGUF
Model size
8.03B params
Architecture
llama

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Collection including leafspark/Llama-3-8B-Instruct-Gradient-4194k-GGUF