mychen76/openmixtral-6x7b-v2-GGUF

openmixtral-6x7b-merged_v2

openmixtral-6x7b-merged_v2 is a merge of the following models:

🧩 Configuration

base_model: mlabonne/Marcoro14-7B-slerp
experts:
  - source_model: openchat/openchat-3.5-1210
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
  - source_model: Weyaxi/Einstein-v4-7B
    positive_prompts:
    - "physics"
    - "biology"
    - "chemistry"
    - "science"    
  - source_model: BioMistral/BioMistral-7B
    positive_prompts:
    - "medical"
    - "pubmed"
    - "healthcare"
    - "health"        
  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B
    positive_prompts:
    - "code"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"
  - source_model: maywell/PiVoT-0.1-Starling-LM-RP
    positive_prompts:
    - "storywriting"
    - "write"
    - "scene"
    - "story"
    - "character"
  - source_model: WizardLM/WizardMath-7B-V1.1
    positive_prompts:
    - "reason"
    - "math"
    - "mathematics"
    - "solve"
    - "count"
tokenizer_source: union

💻 Usage

# install llamacpp see here: https://github.com/ggerganov/llama.cpp
# or other GGUF tool like llamacpp-python: https://github.com/abetlen/llama-cpp-python

MODEL_REPO="openmixtral-6x7b-merged_v2-GGUF"
MODEL_NAME="openmixtral-6x7b-merged_v2"
method="Q4_K_M"

prompt="why the sky is blue"

qtype = f"{MODEL_REPO}/{MODEL_NAME.lower()}.{method.upper()}.gguf"
!./llama.cpp/main -m {qtype} -n 128 --color -ngl 0 -p "{prompt}"

Log Result

Log start
main: build = 2382 (621e86b3)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1710306347
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes
llama_model_loader: loaded meta data with 25 key-value pairs and 803 tensors from openmixtral-6x7b-merged_v2-GGUF/openmixtral-6x7b-merged_v2.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = .
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:                         llama.expert_count u32              = 6
llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2
llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  13:                          general.file_type u32              = 15
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  24:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:   32 tensors
llama_model_loader: - type q4_K:  593 tensors
llama_model_loader: - type q6_K:  113 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 6
llm_load_print_meta: n_expert_used    = 2
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 35.43 B
llm_load_print_meta: model size       = 19.96 GiB (4.84 BPW) 
llm_load_print_meta: general.name     = .
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 1 '<s>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.31 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size = 20441.87 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =    64.00 MiB
llama_new_context_with_model: KV self size  =   64.00 MiB, K (f16):   32.00 MiB, V (f16):   32.00 MiB
llama_new_context_with_model:  CUDA_Host input buffer size   =    10.01 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   114.52 MiB
llama_new_context_with_model: graph splits (measure): 1

system_info: n_threads = 12 / 24 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | 
sampling: 
    repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
    top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
    mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order: 
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature 
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 1


 why the sky is blue

# Why The Sky is Blue: Rayleigh Scattering

## Introduction to Rayleigh Scattering

Have you ever wondered why the sky appears blue during a clear day? You might think that it's because there are more blue molecules in the Earth's atmosphere, or maybe that the sun emits blue light. However, neither of these explanations is accurate. The true reason behind the blue color of the sky has to do with a phenomenon called Rayleigh scattering.

Rayleigh scattering is a fundamental process in physics and atmospheric science that occurs when sunlight interacts with mole
llama_print_timings:        load time =    1605.68 ms
llama_print_timings:      sample time =      14.83 ms /   128 runs   (    0.12 ms per token,  8632.32 tokens per second)
llama_print_timings: prompt eval time =     312.86 ms /     6 tokens (   52.14 ms per token,    19.18 tokens per second)
llama_print_timings:        eval time =   21083.26 ms /   127 runs   (  166.01 ms per token,     6.02 tokens per second)
llama_print_timings:       total time =   21439.97 ms /   133 tokens
Log end

Quantized Model Response

can you spot the difference?

#### User Prompt
why the sky is blue

#### Q2_K Response
The sky appears blue because of a phenomenon called Rayleigh scattering. This process occurs when sunlight (which is made up of many different colors or wavelengths of light) interacts with particles in the atmosphere like air molecules, water vapor and dust. The shorter wavelengths of light, such as violet and blue, are more easily scattered than longer wavelengths like red and yellow, due to their smaller size compared to the wavelengths they're interacting with.
When we look up at the sky, most of these scattered shorter-wavelength lights (violet, blue) are

#### Q3_K_M Response
In the late 19th century, an English scientist named Lord Rayleigh and a Scottish scientist named William Thomson (later Lord Kelvin) embarked on a quest to solve one of the most enduring mysteries in science: why is the sky blue? Their findings, which are now widely accepted, can be attributed to a combination of physics principles and molecular behavior.

The color we perceive in the sky is actually the result of how sunlight interacts with various gases and particles in our atmosphere. Sunlight is composed of multiple colors, or wavelengths, ranging from red at one end

#### Q4_K_M Response
The reason why we see the sky as blue is due to a natural phenomenon known as Rayleigh scattering. When sunlight enters the Earth’s atmosphere, it is made up of numerous colors or wavelengths, including blue, green, yellow, orange, and red.

Rayleigh scattering occurs when the shorter-wavelength light, such as blue and violet colors, interacts with the molecules (mainly nitrogen and oxygen) in the atmosphere. These tiny particles scatter these short-wave lengths more effectively than longer ones like red or yellow, which results in the sky appearing blue to our eyes.

#### Q5_K_M Response
the blue color of the sky, a phenomenon that has been observed and admired since ancient times, can be explained through the principles of physics and atmospheric science.

Sunlight, which appears white to us, is actually made up of many colors of light, each with a different wavelength. These colors include red, orange, yellow, green, blue, indigo, and violet; together, they are known as the visible light spectrum.

When sunlight enters Earth's atmosphere, it encounters various gases, such as nitrogen and oxygen, which scatter the shorter wavelengths of light.

#### Q6_K Response
In order to understand why the sky appears blue, we must first understand how light behaves and interacts with air molecules. Visible light is a form of electromagnetic radiation that spans wavelengths between approximately 400 nanometers (violet) and 700 nanometers (red). When sunlight reaches Earth's atmosphere, it encounters various gases and particles that scatter or absorb different wavelengths of light.

The scattering of light is due to the interaction between electromagnetic waves and free-moving electrons within air molecules, particularly nitrogen and oxygen...etc

#### Q8_0 Response
I’m just curious, it seems like a lot of people know this but I can’t recall ever being taught. Why exactly is the sky blue? I know it reflects sunlight and so forth but how does that lead to the blue color?

The reason the sky appears blue has to do with a combination of Rayleigh scattering and Earth's atmosphere. Here's a brief explanation:

1. Sunlight consists of multiple colors (wavelengths) including red, orange, yellow, green, blue, indigo, and violet. These different colors are represented by their respective wavelengths in...etc