This is a very impressive model. Using the 8bit version.
It is the only model I have tested that correctly answers this "tricky" question.
Mary, a woman, has 3 brothers. Her brothers have 2 sisters. How many sisters does Mary have?
It answers 1. Most other models (except for airoboros 70b 8bit) get this wrong.
llm_load_print_meta: model ftype = mostly Q4_K - Small
llm_load_print_meta: model params = 67.43 B
llm_load_print_meta: model size = 35.62 GiB (4.54 BPW)
llm_load_print_meta: general.name = LLaMA v2
llm_load_print_meta: BOS token = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 126 'Ä'
llm_load_tensors: ggml ctx size = 0.31 MiB
llm_load_tensors: using OpenCL for GPU acceleration
llm_load_tensors: mem required = 36471.28 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/96 layers to GPU
llm_load_tensors: VRAM used: 0.00 MiB
...................................................................................................
llama_new_context_with_model: n_ctx = 512
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size = 190.00 MiB
llama_build_graph: non-view tensors processed: 2189/2189
llama_new_context_with_model: compute buffer total size = 219.06 MiB
hello
Hello! How can I assist you today?
Mary, a woman, has 3 brothers. Her brothers have 2 sisters. How many sisters does Mary have?
If Mary has 3 brothers and her brothers have 2 sisters, then Mary is one of the two sisters that her brothers are referring to. Therefore, Mary has only one sister.
============================================================================
./build/bin/main -m ./models/deepseek_coder/deepseek-llm-67b-chat.Q4_K_S.gguf
-ins
--top-p 0.80
--top-k 200
--temp 0.144
-t 18
--multiline-input
--color
--log-disable