main: build = 2998 (9588f196) main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu main: seed = 1716735959 llama_model_loader: loaded meta data with 26 key-value pairs and 195 tensors from Phi-3-mini-4k-instruct-IMat-GGUF/Phi-3-mini-4k-instruct.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = phi3 llama_model_loader: - kv 1: general.name str = Phi3 llama_model_loader: - kv 2: phi3.context_length u32 = 4096 llama_model_loader: - kv 3: phi3.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 4: phi3.embedding_length u32 = 3072 llama_model_loader: - kv 5: phi3.feed_forward_length u32 = 8192 llama_model_loader: - kv 6: phi3.block_count u32 = 32 llama_model_loader: - kv 7: phi3.attention.head_count u32 = 32 llama_model_loader: - kv 8: phi3.attention.head_count_kv u32 = 32 llama_model_loader: - kv 9: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: phi3.rope.dimension_count u32 = 96 llama_model_loader: - kv 11: phi3.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 12: general.file_type u32 = 0 llama_model_loader: - kv 13: tokenizer.ggml.model str = llama llama_model_loader: - kv 14: tokenizer.ggml.pre str = default llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32064] = ["", "~~", "~~", "<0x00>", "<... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32000 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... llama_model_loader: - kv 25: general.quantization_version u32 = 2 llama_model_loader: - type f32: 195 tensors llm_load_vocab: special tokens definition check successful ( 323/32064 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = phi3 llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32064 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 4096 llm_load_print_meta: n_embd = 3072 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 96 llm_load_print_meta: n_embd_head_k = 96 llm_load_print_meta: n_embd_head_v = 96 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 3072 llm_load_print_meta: n_embd_v_gqa = 3072 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 8192 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 3B llm_load_print_meta: model ftype = all F32 llm_load_print_meta: model params = 3.82 B llm_load_print_meta: model size = 14.23 GiB (32.00 BPW) llm_load_print_meta: general.name = Phi3 llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 32000 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: PAD token = 32000 '<|endoftext|>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: EOT token = 32007 '<|end|>' ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.22 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 375.75 MiB llm_load_tensors: CUDA0 buffer size = 14200.51 MiB .................................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB llama_new_context_with_model: CUDA0 compute buffer size = 83.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB llama_new_context_with_model: graph nodes = 1286 llama_new_context_with_model: graph splits = 2 system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | compute_imatrix: tokenizing the input .. compute_imatrix: tokenization took 137.725 ms compute_imatrix: computing over 234 chunks with batch_size 512 compute_imatrix: 0.30 seconds per pass - ETA 1.17 minutes [1]6.4163,[2]4.6267,[3]4.5595,[4]5.1081,[5]5.4605,[6]5.6088,[7]5.0117,[8]5.4421,[9]5.7096, save_imatrix: stored collected data after 10 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [10]6.0191,[11]5.9802,[12]5.5105,[13]5.6421,[14]5.5106,[15]5.9354,[16]6.0264,[17]6.3397,[18]6.4955,[19]6.6964, save_imatrix: stored collected data after 20 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [20]6.8681,[21]6.9361,[22]7.1511,[23]6.8709,[24]6.6723,[25]6.6847,[26]6.3294,[27]6.0572,[28]5.7524,[29]5.7223, save_imatrix: stored collected data after 30 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [30]5.8253,[31]5.8999,[32]5.9491,[33]5.9243,[34]5.9786,[35]5.9800,[36]5.7571,[37]5.6174,[38]5.5555,[39]5.5252, save_imatrix: stored collected data after 40 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [40]5.5062,[41]5.4366,[42]5.4770,[43]5.5190,[44]5.5656,[45]5.6379,[46]5.7200,[47]5.7966,[48]5.9288,[49]6.0379, save_imatrix: stored collected data after 50 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [50]6.1566,[51]6.2585,[52]6.3527,[53]6.3179,[54]6.2309,[55]6.1629,[56]6.2619,[57]6.3117,[58]6.3224,[59]6.3814, save_imatrix: stored collected data after 60 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [60]6.4569,[61]6.4848,[62]6.5566,[63]6.6002,[64]6.6757,[65]6.7114,[66]6.7519,[67]6.7979,[68]6.8413,[69]6.9015, save_imatrix: stored collected data after 70 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [70]6.9406,[71]6.9868,[72]7.0199,[73]6.9781,[74]6.9301,[75]6.8699,[76]6.8068,[77]6.7955,[78]6.7426,[79]6.6893, save_imatrix: stored collected data after 80 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [80]6.6263,[81]6.6002,[82]6.5518,[83]6.5114,[84]6.5246,[85]6.5523,[86]6.5657,[87]6.6032,[88]6.6182,[89]6.6034, save_imatrix: stored collected data after 90 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [90]6.5683,[91]6.5913,[92]6.5972,[93]6.6130,[94]6.6263,[95]6.6374,[96]6.6660,[97]6.6881,[98]6.6607,[99]6.6185, save_imatrix: stored collected data after 100 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [100]6.6271,[101]6.6484,[102]6.6377,[103]6.6032,[104]6.5465,[105]6.5318,[106]6.5372,[107]6.5445,[108]6.5248,[109]6.5123, save_imatrix: stored collected data after 110 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [110]6.4918,[111]6.4988,[112]6.5087,[113]6.5075,[114]6.5182,[115]6.5142,[116]6.5163,[117]6.5127,[118]6.5195,[119]6.4967, save_imatrix: stored collected data after 120 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [120]6.5001,[121]6.4848,[122]6.4622,[123]6.4794,[124]6.4723,[125]6.4746,[126]6.4611,[127]6.4623,[128]6.4705,[129]6.4538, save_imatrix: stored collected data after 130 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [130]6.4285,[131]6.4159,[132]6.4164,[133]6.3683,[134]6.3750,[135]6.3563,[136]6.3404,[137]6.3186,[138]6.3002,[139]6.2780, save_imatrix: stored collected data after 140 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [140]6.2542,[141]6.2435,[142]6.2221,[143]6.2269,[144]6.2263,[145]6.2099,[146]6.1863,[147]6.1874,[148]6.1752,[149]6.1666, save_imatrix: stored collected data after 150 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [150]6.1604,[151]6.1472,[152]6.1466,[153]6.1380,[154]6.1257,[155]6.1513,[156]6.1265,[157]6.1221,[158]6.1391,[159]6.1336, save_imatrix: stored collected data after 160 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [160]6.1388,[161]6.1515,[162]6.1562,[163]6.1759,[164]6.1878,[165]6.2100,[166]6.2205,[167]6.2173,[168]6.2186,[169]6.2248, save_imatrix: stored collected data after 170 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [170]6.2362,[171]6.2251,[172]6.2265,[173]6.2429,[174]6.2455,[175]6.2640,[176]6.2731,[177]6.2840,[178]6.2901,[179]6.3206, save_imatrix: stored collected data after 180 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [180]6.3296,[181]6.3803,[182]6.3970,[183]6.4255,[184]6.4316,[185]6.4381,[186]6.4458,[187]6.4516,[188]6.4422,[189]6.4465, save_imatrix: stored collected data after 190 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [190]6.4541,[191]6.4686,[192]6.4725,[193]6.5001,[194]6.4898,[195]6.4599,[196]6.5008,[197]6.5382,[198]6.5683,[199]6.6173, save_imatrix: stored collected data after 200 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [200]6.6652,[201]6.6736,[202]6.6782,[203]6.6368,[204]6.6336,[205]6.6392,[206]6.6598,[207]6.6555,[208]6.6580,[209]6.6579, save_imatrix: stored collected data after 210 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [210]6.6662,[211]6.6791,[212]6.6773,[213]6.6725,[214]6.6790,[215]6.6973,[216]6.7151,[217]6.7180,[218]6.7177,[219]6.7118, save_imatrix: stored collected data after 220 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [220]6.7025,[221]6.6999,[222]6.6970,[223]6.7114,[224]6.6944,[225]6.7004,[226]6.6857,[227]6.7212,[228]6.7598,[229]6.8028, save_imatrix: stored collected data after 230 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat [230]6.8418,[231]6.8619,[232]6.8401,[233]6.8200,[234]6.7933, save_imatrix: stored collected data after 234 chunks in Phi-3-mini-4k-instruct-IMat-GGUF/imatrix.dat llama_print_timings: load time = 2408.01 ms llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: prompt eval time = 52520.09 ms / 119808 tokens ( 0.44 ms per token, 2281.18 tokens per second) llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: total time = 55220.72 ms / 119809 tokens Final estimate: PPL = 6.7933 +/- 0.07020