Hugging Face's logo

legraphista
/
Phi-3-mini-4k-instruct-IMat-GGUF

Text Generation
GGUF
English
quantized
GGUF
imatrix
quantization
imat
static
conversational
Model card Files Community
legraphista commited on
Commit
27892ee
•
1 Parent(s): 4be0971

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +166 -0
imatrix.log ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ main: build = 2998 (9588f196)
2
+ main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
3
+ main: seed = 1716735959
4
+ llama_model_loader: loaded meta data with 26 key-value pairs and 195 tensors from Phi-3-mini-4k-instruct-IMat-GGUF/Phi-3-mini-4k-instruct.gguf (version GGUF V3 (latest))
5
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
6
+ llama_model_loader: - kv 0: general.architecture str = phi3
7
+ llama_model_loader: - kv 1: general.name str = Phi3
8
+ llama_model_loader: - kv 2: phi3.context_length u32 = 4096
9
+ llama_model_loader: - kv 3: phi3.rope.scaling.original_context_length u32 = 4096
10
+ llama_model_loader: - kv 4: phi3.embedding_length u32 = 3072
11
+ llama_model_loader: - kv 5: phi3.feed_forward_length u32 = 8192
12
+ llama_model_loader: - kv 6: phi3.block_count u32 = 32
13
+ llama_model_loader: - kv 7: phi3.attention.head_count u32 = 32
14
+ llama_model_loader: - kv 8: phi3.attention.head_count_kv u32 = 32
15
+ llama_model_loader: - kv 9: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010
16
+ llama_model_loader: - kv 10: phi3.rope.dimension_count u32 = 96
17
+ llama_model_loader: - kv 11: phi3.rope.freq_base f32 = 10000.000000
18
+ llama_model_loader: - kv 12: general.file_type u32 = 0
19
+ llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
20
+ llama_model_loader: - kv 14: tokenizer.ggml.pre str = default
21
+ llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
22
+ llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00...
23
+ llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
24
+ llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1
25
+ llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 32000
26
+ llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0
27
+ llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32000
28
+ llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
29
+ llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
30
+ llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
31
+ llama_model_loader: - kv 25: general.quantization_version u32 = 2
32
+ llama_model_loader: - type f32: 195 tensors
33
+ llm_load_vocab: special tokens definition check successful ( 323/32064 ).
34
+ llm_load_print_meta: format = GGUF V3 (latest)
35
+ llm_load_print_meta: arch = phi3
36
+ llm_load_print_meta: vocab type = SPM
37
+ llm_load_print_meta: n_vocab = 32064
38
+ llm_load_print_meta: n_merges = 0
39
+ llm_load_print_meta: n_ctx_train = 4096
40
+ llm_load_print_meta: n_embd = 3072
41
+ llm_load_print_meta: n_head = 32
42
+ llm_load_print_meta: n_head_kv = 32
43
+ llm_load_print_meta: n_layer = 32
44
+ llm_load_print_meta: n_rot = 96
45
+ llm_load_print_meta: n_embd_head_k = 96
46
+ llm_load_print_meta: n_embd_head_v = 96
47
+ llm_load_print_meta: n_gqa = 1
48
+ llm_load_print_meta: n_embd_k_gqa = 3072
49
+ llm_load_print_meta: n_embd_v_gqa = 3072
50
+ llm_load_print_meta: f_norm_eps = 0.0e+00
51
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-05
52
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
53
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
54
+ llm_load_print_meta: f_logit_scale = 0.0e+00
55
+ llm_load_print_meta: n_ff = 8192
56
+ llm_load_print_meta: n_expert = 0
57
+ llm_load_print_meta: n_expert_used = 0
58
+ llm_load_print_meta: causal attn = 1
59
+ llm_load_print_meta: pooling type = 0
60
+ llm_load_print_meta: rope type = 2
61
+ llm_load_print_meta: rope scaling = linear
62
+ llm_load_print_meta: freq_base_train = 10000.0
63
+ llm_load_print_meta: freq_scale_train = 1
64
+ llm_load_print_meta: n_yarn_orig_ctx = 4096
65
+ llm_load_print_meta: rope_finetuned = unknown
66
+ llm_load_print_meta: ssm_d_conv = 0
67
+ llm_load_print_meta: ssm_d_inner = 0
68
+ llm_load_print_meta: ssm_d_state = 0
69
+ llm_load_print_meta: ssm_dt_rank = 0
70
+ llm_load_print_meta: model type = 3B
71
+ llm_load_print_meta: model ftype = all F32
72
+ llm_load_print_meta: model params = 3.82 B
73
+ llm_load_print_meta: model size = 14.23 GiB (32.00 BPW)
74
+ llm_load_print_meta: general.name = Phi3
75
+ llm_load_print_meta: BOS token = 1 '<s>'
76
+ llm_load_print_meta: EOS token = 32000 '<|endoftext|>'
77
+ llm_load_print_meta: UNK token = 0 '<unk>'
78
+ llm_load_print_meta: PAD token = 32000 '<|endoftext|>'
79
+ llm_load_print_meta: LF token = 13 '<0x0A>'
80
+ llm_load_print_meta: EOT token = 32007 '<|end|>'
81
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
82
+ ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
83
+ ggml_cuda_init: found 1 CUDA devices:
84
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
85
+ llm_load_tensors: ggml ctx size = 0.22 MiB
86
+ llm_load_tensors: offloading 32 repeating layers to GPU
87
+ llm_load_tensors: offloading non-repeating layers to GPU
88
+ llm_load_tensors: offloaded 33/33 layers to GPU
89
+ llm_load_tensors: CPU buffer size = 375.75 MiB
90
+ llm_load_tensors: CUDA0 buffer size = 14200.51 MiB
91
+ ....................................................................................
92
+ llama_new_context_with_model: n_ctx = 512
93
+ llama_new_context_with_model: n_batch = 512
94
+ llama_new_context_with_model: n_ubatch = 512
95
+ llama_new_context_with_model: flash_attn = 0
96
+ llama_new_context_with_model: freq_base = 10000.0
97
+ llama_new_context_with_model: freq_scale = 1
98
+ llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB
99
+ llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
100
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB
101
+ llama_new_context_with_model: CUDA0 compute buffer size = 83.00 MiB
102
+ llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB
103
+ llama_new_context_with_model: graph nodes = 1286
104
+ llama_new_context_with_model: graph splits = 2
105
+
106
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
107
+ compute_imatrix: tokenizing the input ..
108
+ compute_imatrix: tokenization took 137.725 ms
109
+ compute_imatrix: computing over 234 chunks with batch_size 512
110
+ compute_imatrix: 0.30 seconds per pass - ETA 1.17 minutes
111
+ [1]6.4163,[2]4.6267,[3]4.5595,[4]5.1081,[5]5.4605,[6]5.6088,[7]5.0117,[8]5.4421,[9]5.7096,
112
+ save_imatrix: stored collected data after 10 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
113
+ [10]6.0191,[11]5.9802,[12]5.5105,[13]5.6421,[14]5.5106,[15]5.9354,[16]6.0264,[17]6.3397,[18]6.4955,[19]6.6964,
114
+ save_imatrix: stored collected data after 20 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
115
+ [20]6.8681,[21]6.9361,[22]7.1511,[23]6.8709,[24]6.6723,[25]6.6847,[26]6.3294,[27]6.0572,[28]5.7524,[29]5.7223,
116
+ save_imatrix: stored collected data after 30 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
117
+ [30]5.8253,[31]5.8999,[32]5.9491,[33]5.9243,[34]5.9786,[35]5.9800,[36]5.7571,[37]5.6174,[38]5.5555,[39]5.5252,
118
+ save_imatrix: stored collected data after 40 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
119
+ [40]5.5062,[41]5.4366,[42]5.4770,[43]5.5190,[44]5.5656,[45]5.6379,[46]5.7200,[47]5.7966,[48]5.9288,[49]6.0379,
120
+ save_imatrix: stored collected data after 50 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
121
+ [50]6.1566,[51]6.2585,[52]6.3527,[53]6.3179,[54]6.2309,[55]6.1629,[56]6.2619,[57]6.3117,[58]6.3224,[59]6.3814,
122
+ save_imatrix: stored collected data after 60 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
123
+ [60]6.4569,[61]6.4848,[62]6.5566,[63]6.6002,[64]6.6757,[65]6.7114,[66]6.7519,[67]6.7979,[68]6.8413,[69]6.9015,
124
+ save_imatrix: stored collected data after 70 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
125
+ [70]6.9406,[71]6.9868,[72]7.0199,[73]6.9781,[74]6.9301,[75]6.8699,[76]6.8068,[77]6.7955,[78]6.7426,[79]6.6893,
126
+ save_imatrix: stored collected data after 80 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
127
+ [80]6.6263,[81]6.6002,[82]6.5518,[83]6.5114,[84]6.5246,[85]6.5523,[86]6.5657,[87]6.6032,[88]6.6182,[89]6.6034,
128
+ save_imatrix: stored collected data after 90 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
129
+ [90]6.5683,[91]6.5913,[92]6.5972,[93]6.6130,[94]6.6263,[95]6.6374,[96]6.6660,[97]6.6881,[98]6.6607,[99]6.6185,
130
+ save_imatrix: stored collected data after 100 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
131
+ [100]6.6271,[101]6.6484,[102]6.6377,[103]6.6032,[104]6.5465,[105]6.5318,[106]6.5372,[107]6.5445,[108]6.5248,[109]6.5123,
132
+ save_imatrix: stored collected data after 110 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
133
+ [110]6.4918,[111]6.4988,[112]6.5087,[113]6.5075,[114]6.5182,[115]6.5142,[116]6.5163,[117]6.5127,[118]6.5195,[119]6.4967,
134
+ save_imatrix: stored collected data after 120 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
135
+ [120]6.5001,[121]6.4848,[122]6.4622,[123]6.4794,[124]6.4723,[125]6.4746,[126]6.4611,[127]6.4623,[128]6.4705,[129]6.4538,
136
+ save_imatrix: stored collected data after 130 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
137
+ [130]6.4285,[131]6.4159,[132]6.4164,[133]6.3683,[134]6.3750,[135]6.3563,[136]6.3404,[137]6.3186,[138]6.3002,[139]6.2780,
138
+ save_imatrix: stored collected data after 140 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
139
+ [140]6.2542,[141]6.2435,[142]6.2221,[143]6.2269,[144]6.2263,[145]6.2099,[146]6.1863,[147]6.1874,[148]6.1752,[149]6.1666,
140
+ save_imatrix: stored collected data after 150 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
141
+ [150]6.1604,[151]6.1472,[152]6.1466,[153]6.1380,[154]6.1257,[155]6.1513,[156]6.1265,[157]6.1221,[158]6.1391,[159]6.1336,
142
+ save_imatrix: stored collected data after 160 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
143
+ [160]6.1388,[161]6.1515,[162]6.1562,[163]6.1759,[164]6.1878,[165]6.2100,[166]6.2205,[167]6.2173,[168]6.2186,[169]6.2248,
144
+ save_imatrix: stored collected data after 170 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
145
+ [170]6.2362,[171]6.2251,[172]6.2265,[173]6.2429,[174]6.2455,[175]6.2640,[176]6.2731,[177]6.2840,[178]6.2901,[179]6.3206,
146
+ save_imatrix: stored collected data after 180 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
147
+ [180]6.3296,[181]6.3803,[182]6.3970,[183]6.4255,[184]6.4316,[185]6.4381,[186]6.4458,[187]6.4516,[188]6.4422,[189]6.4465,
148
+ save_imatrix: stored collected data after 190 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
149
+ [190]6.4541,[191]6.4686,[192]6.4725,[193]6.5001,[194]6.4898,[195]6.4599,[196]6.5008,[197]6.5382,[198]6.5683,[199]6.6173,
150
+ save_imatrix: stored collected data after 200 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
151
+ [200]6.6652,[201]6.6736,[202]6.6782,[203]6.6368,[204]6.6336,[205]6.6392,[206]6.6598,[207]6.6555,[208]6.6580,[209]6.6579,
152
+ save_imatrix: stored collected data after 210 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
153
+ [210]6.6662,[211]6.6791,[212]6.6773,[213]6.6725,[214]6.6790,[215]6.6973,[216]6.7151,[217]6.7180,[218]6.7177,[219]6.7118,
154
+ save_imatrix: stored collected data after 220 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
155
+ [220]6.7025,[221]6.6999,[222]6.6970,[223]6.7114,[224]6.6944,[225]6.7004,[226]6.6857,[227]6.7212,[228]6.7598,[229]6.8028,
156
+ save_imatrix: stored collected data after 230 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
157
+ [230]6.8418,[231]6.8619,[232]6.8401,[233]6.8200,[234]6.7933,
158
+ save_imatrix: stored collected data after 234 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat
159
+
160
+ llama_print_timings: load time = 2408.01 ms
161
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
162
+ llama_print_timings: prompt eval time = 52520.09 ms / 119808 tokens ( 0.44 ms per token, 2281.18 tokens per second)
163
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
164
+ llama_print_timings: total time = 55220.72 ms / 119809 tokens
165
+
166
+ Final estimate: PPL = 6.7933 +/- 0.07020