Unexpected error: Model file is malformed.
this is strange, the problem occurs when loading gguf versions of the models, maybe some bug? Have its been tested? I tested Imatrix same thing, q6, q8. I checked the variant of another user (q8, (maybe there the same file I don't know) same thing.
@happy5 I just tested this model myself and it works without any issues on latest llama.cpp:
root@StormPeak:~/llama.cpp# ./llama-cli -m Llama-3.1-8B-Stheno-v3.4.Q4_K_S.gguf -p "I believe the meaning of life is" -n 128 -c 7000
Log start
main: build = 3606 (90db8146)
main: built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu
main: seed = 1724145622
llama_model_loader: loaded meta data with 39 key-value pairs and 292 tensors from Llama-3.1-8B-Stheno-v3.4.Q4_K_S.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Llama 3.1 8B Stheno v3.4
llama_model_loader: - kv 3: general.version str = v3.4
llama_model_loader: - kv 4: general.organization str = Sao10K
llama_model_loader: - kv 5: general.finetune str = Stheno
llama_model_loader: - kv 6: general.basename str = Llama-3.1
llama_model_loader: - kv 7: general.size_label str = 8B
llama_model_loader: - kv 8: general.license str = cc-by-nc-4.0
llama_model_loader: - kv 9: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 10: general.datasets arr[str,2] = ["Setiaku/Stheno-v3.4-Instruct", "Set...
llama_model_loader: - kv 11: llama.block_count u32 = 32
llama_model_loader: - kv 12: llama.context_length u32 = 131072
llama_model_loader: - kv 13: llama.embedding_length u32 = 4096
llama_model_loader: - kv 14: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 15: llama.attention.head_count u32 = 32
llama_model_loader: - kv 16: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 17: llama.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 18: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 19: general.file_type u32 = 14
llama_model_loader: - kv 20: llama.vocab_size u32 = 128256
llama_model_loader: - kv 21: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 22: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 23: tokenizer.ggml.pre str = llama-bpe
llama_model_loader: - kv 24: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 25: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 26: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 27: tokenizer.ggml.bos_token_id u32 = 128000
llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 128009
llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 128004
llama_model_loader: - kv 30: tokenizer.chat_template str = {{ '<|begin_of_text|>' }}{% if messag...
llama_model_loader: - kv 31: general.quantization_version u32 = 2
llama_model_loader: - kv 32: general.url str = https://huggingface.co/mradermacher/L...
llama_model_loader: - kv 33: mradermacher.quantize_version str = 2
llama_model_loader: - kv 34: mradermacher.quantized_by str = mradermacher
llama_model_loader: - kv 35: mradermacher.quantized_at str = 2024-08-20T04:58:21+02:00
llama_model_loader: - kv 36: mradermacher.quantized_on str = db3
llama_model_loader: - kv 37: general.source.url str = https://huggingface.co/Sao10K/Llama-3...
llama_model_loader: - kv 38: mradermacher.convert_type str = hf
llama_model_loader: - type f32: 66 tensors
llama_model_loader: - type q4_K: 217 tensors
llama_model_loader: - type q5_K: 8 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 128256
llm_load_print_meta: n_merges = 280147
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 8B
llm_load_print_meta: model ftype = Q4_K - Small
llm_load_print_meta: model params = 8.03 B
llm_load_print_meta: model size = 4.36 GiB (4.67 BPW)
llm_load_print_meta: general.name = Llama 3.1 8B Stheno v3.4
llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
llm_load_print_meta: PAD token = 128004 '<|finetune_right_pad_id|>'
llm_load_print_meta: LF token = 128 'Ä'
llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size = 0.14 MiB
llm_load_tensors: CPU buffer size = 4467.80 MiB
.......................................................................................
llama_new_context_with_model: n_ctx = 7008
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 876.00 MiB
llama_new_context_with_model: KV self size = 876.00 MiB, K (f16): 438.00 MiB, V (f16): 438.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.49 MiB
llama_new_context_with_model: CPU compute buffer size = 483.69 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 1
system_info: n_threads = 32 / 64 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
sampling:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 7008, n_batch = 2048, n_predict = 128, n_keep = 1
I believe the meaning of life is to help others. The more we can reach out and touch the lives of those around us with kindness and love, the more meaningful our own existence becomes.
We can make a difference. We can help people become healthy, happy, and whole. This is our calling.
Let's connect with each other. Let's share our wisdom, our experiences, and our ideas. Let's support each other.
I believe that the true purpose of life is to spread love. When we love others, we show that we care about them. When we show that we care, we inspire them to love and care for others in return.
So, let
llama_print_timings: load time = 586.26 ms
llama_print_timings: sample time = 6.12 ms / 128 runs ( 0.05 ms per token, 20911.62 tokens per second)
llama_print_timings: prompt eval time = 69.39 ms / 8 tokens ( 8.67 ms per token, 115.29 tokens per second)
llama_print_timings: eval time = 4199.70 ms / 127 runs ( 33.07 ms per token, 30.24 tokens per second)
llama_print_timings: total time = 4290.47 ms / 135 tokens
Log end
Thanks for the reply, then I guess I should look for a problem in my software.
Thanks for the reply, then I guess I should look for a problem in my software.
I assume your software is most likely outdated and not using the latest llama.cpp. Try upgrading it to the latest version available or use llama.cpp directly. For above test I built the latest llama.cpp master branch from source using make -j without any GPU acceleration enabled because why would I need GPU acceleration if I'm getting 30.24 tokens/second without it for such a small model.
for prompt processing, nico, for prompt processing.
for prompt processing, nico, for prompt processing.
I just did a quick test. Here the results:
Compiled only for CPU (with AVX-512):
llama_print_timings: sample time = 6.12 ms / 128 runs ( 0.05 ms per token, 20911.62 tokens per second)
llama_print_timings: prompt eval time = 69.39 ms / 8 tokens ( 8.67 ms per token, 115.29 tokens per second)
llama_print_timings: eval time = 4199.70 ms / 127 runs ( 33.07 ms per token, 30.24 tokens per second)
llama_print_timings: total time = 4290.47 ms / 135 token
Compiled with Cuda but without any layers on GPU (-ngl 0)
llama_print_timings: sample time = 6.77 ms / 128 runs ( 0.05 ms per token, 18895.78 tokens per second)
llama_print_timings: prompt eval time = 60.13 ms / 8 tokens ( 7.52 ms per token, 133.04 tokens per second)
llama_print_timings: eval time = 4155.65 ms / 127 runs ( 32.72 ms per token, 30.56 tokens per second)
llama_print_timings: total time = 4239.78 ms / 135 tokens
Compiled with Cuda and all layers on the GPU (-ngl 33)
llama_print_timings: sample time = 4.68 ms / 128 runs ( 0.04 ms per token, 27344.58 tokens per second)
llama_print_timings: prompt eval time = 11.84 ms / 8 tokens ( 1.48 ms per token, 675.90 tokens per second)
llama_print_timings: eval time = 953.15 ms / 127 runs ( 7.51 ms per token, 133.24 tokens per second)
llama_print_timings: total time = 980.58 ms / 135 tokens
Conclusion:
- prompt eval time went from 115.29 tokens per second to 133.04 tokens per second when using the GPU which I wouldn't consider a significant change. It however went to 675.90 tokens per second when all layers where on the GPU.
- eval time was unaffected by adding a GPU but increased by 4.4 times when all layers where on the GPU
Conclusion: "I have very expensive and fast hardware that practically nobody else has." Most people have 4 cores and often one dimm. If it's not faster on your monster hardware, what is the result for typical users? Until a year ago, my workstation was a 4 core haswell with 16GB of RAM.
(Also, eval time decreased, but that is what you meant anyway)
As a concrete counter example, with koboldcpp (shouldn't make a difference, but it is more representative of what 8b-users use), on my TV (don't ask) with an RTX 4060 and llama-3.1 at Q8_0 and the 4 performance cores of an overclocked i7-13700k and 2 x ddr-5-5600, I get:
CPU: 15.75T/s 3.98T/s (prompt tps + inference tps)
GPU: 291.84T/s 3.67T/s (--gpulayers 0)
GPU: 396.06T/s 6.36T/s (--gpulayers 18 of 33, the 4060 has generous 8GB VRAM)
Yes, that is more than 16 minutes of waiting time (CPU) vs. one (GPU) for the 16k test prompt that koboldpcp uses for its benchmark.
And a 13700k + rtx 4060 are hardly very low-end specs, if you are just, say, a gamer, and not a llm enthusiast. Although it is a bit mismatched, which is why I limited the cpu to 4 cores (but still 5.3GHz per core, measured). Also, llama has bugs that prevent it from using all cores properly in big.LITTLE-type CPUs. OTOH, the box wasn't completely idle (I was watching a movie, which is, however, mostly taxing the GPU), but it did have the performance cores for its exclusive use.
Lastly, you only measured 8 tokens for prompt processing, or am I misinterpreting the output? If true, that's hardly representative of anything, as batching is typically done in 512 token chunks.
without any GPU acceleration enabled because why would I need GPU acceleration if I'm getting 30.24 tokens/second without it for such a small model.
Indeed, why would you... but for us vramlets it's signifcant.