Hugging Face's logo

KingNish
/
WizardLM2-2bit

Text Generation
Transformers
Safetensors
GGUF
mixtral
quantized
2-bit
GGUF
mistral
arxiv:2304.12244
arxiv:2306.08568
arxiv:2308.09583
Inference Endpoints
text-generation-inference
conversational
Model card Files Community
3

trial

#3
by KingNish - opened
base: refs/heads/main
←
from: refs/pr/3
Discussion Files changed
+113
-1858
Files changed (40) hide show
  1. README.md +11 -4
  2. WizardLM-2-8x22B.IQ4_XS-00003-of-00005.gguf +3 -0
  3. WizardLM-2-8x22B.IQ4_XS-00004-of-00005.gguf +3 -0
  4. tokenizer.model → WizardLM-2-8x22B.IQ4_XS-00005-of-00005.gguf +2 -2
  5. WizardLM-2-8x22B.Q3_K_L-00001-of-00005.gguf +3 -0
  6. WizardLM-2-8x22B.Q3_K_L-00002-of-00005.gguf +3 -0
  7. WizardLM-2-8x22B.Q3_K_L-00003-of-00005.gguf +3 -0
  8. WizardLM-2-8x22B.Q3_K_L-00004-of-00005.gguf +3 -0
  9. WizardLM-2-8x22B.Q3_K_L-00005-of-00005.gguf +3 -0
  10. WizardLM-2-8x22B.Q3_K_M-00001-of-00005.gguf +3 -0
  11. WizardLM-2-8x22B.Q3_K_M-00002-of-00005.gguf +3 -0
  12. WizardLM-2-8x22B.Q3_K_M-00003-of-00005.gguf +3 -0
  13. WizardLM-2-8x22B.Q3_K_M-00004-of-00005.gguf +3 -0
  14. WizardLM-2-8x22B.Q3_K_M-00005-of-00005.gguf +3 -0
  15. WizardLM-2-8x22B.Q3_K_S-00001-of-00005.gguf +3 -0
  16. WizardLM-2-8x22B.Q3_K_S-00002-of-00005.gguf +3 -0
  17. WizardLM-2-8x22B.Q3_K_S-00003-of-00005.gguf +3 -0
  18. WizardLM-2-8x22B.Q3_K_S-00004-of-00005.gguf +3 -0
  19. WizardLM-2-8x22B.Q3_K_S-00005-of-00005.gguf +3 -0
  20. WizardLM-2-8x22B.Q4_K_M-00001-of-00005.gguf +3 -0
  21. WizardLM-2-8x22B.Q4_K_M-00002-of-00005.gguf +3 -0
  22. WizardLM-2-8x22B.Q4_K_M-00003-of-00005.gguf +3 -0
  23. WizardLM-2-8x22B.Q4_K_M-00004-of-00005.gguf +3 -0
  24. WizardLM-2-8x22B.Q4_K_M-00005-of-00005.gguf +3 -0
  25. WizardLM-2-8x22B.Q4_K_S-00001-of-00005.gguf +3 -0
  26. WizardLM-2-8x22B.Q4_K_S-00002-of-00005.gguf +3 -0
  27. WizardLM-2-8x22B.Q4_K_S-00003-of-00005.gguf +3 -0
  28. WizardLM-2-8x22B.Q4_K_S-00004-of-00005.gguf +3 -0
  29. WizardLM-2-8x22B.Q4_K_S-00005-of-00005.gguf +3 -0
  30. WizardLM-2-8x22B.Q5_K_M-00001-of-00005.gguf +3 -0
  31. WizardLM-2-8x22B.Q5_K_M-00002-of-00005.gguf +3 -0
  32. WizardLM-2-8x22B.Q5_K_M-00003-of-00005.gguf +3 -0
  33. WizardLM-2-8x22B.Q5_K_M-00004-of-00005.gguf +3 -0
  34. WizardLM-2-8x22B.Q5_K_S-00001-of-00005.gguf +3 -0
  35. WizardLM-2-8x22B.Q5_K_S-00004-of-00005.gguf +3 -0
  36. config.json +0 -31
  37. generation_config.json +0 -6
  38. model.safetensors.index.json +0 -0
  39. special_tokens_map.json +0 -24
  40. tokenizer_config.json +0 -44
README.md CHANGED
@@ -2,6 +2,11 @@
2
  tags:
3
  - quantized
4
  - 2-bit
 
 
 
 
 
5
  - GGUF
6
  - transformers
7
  - safetensors
@@ -17,16 +22,18 @@ tags:
17
  - region:us
18
  - text-generation
19
  model_name: WizardLM-2-8x22B-GGUF
20
- inference: true
21
- base_model: mistral-community/Mixtral-8x22B-v0.1
 
22
  pipeline_tag: text-generation
23
  quantized_by: MaziyarPanahi
24
  ---
25
  # [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF)
26
- - Base model: [mistral-community/Mixtral-8x22B-v0.1](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1)
 
27
 
28
  ## Description
29
- [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF) contains GGUF format model files for [mistral-community/Mixtral-8x22B-v0.1](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1).
30
 
31
  ## How to download
32
  You can download only the quants you need instead of cloning the entire repository as follows:
 
2
  tags:
3
  - quantized
4
  - 2-bit
5
+ - 3-bit
6
+ - 4-bit
7
+ - 5-bit
8
+ - 6-bit
9
+ - 8-bit
10
  - GGUF
11
  - transformers
12
  - safetensors
 
22
  - region:us
23
  - text-generation
24
  model_name: WizardLM-2-8x22B-GGUF
25
+ base_model: microsoft/WizardLM-2-8x22B
26
+ inference: false
27
+ model_creator: microsoft
28
  pipeline_tag: text-generation
29
  quantized_by: MaziyarPanahi
30
  ---
31
  # [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF)
32
+ - Model creator: [microsoft](https://huggingface.co/microsoft)
33
+ - Original model: [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B)
34
 
35
  ## Description
36
+ [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF) contains GGUF format model files for [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B).
37
 
38
  ## How to download
39
  You can download only the quants you need instead of cloning the entire repository as follows:
WizardLM-2-8x22B.IQ4_XS-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52ae1b3b1b75dabaf96484ec1e0141f0ce65225f1b98a73d2261a50d6cae3f15
3
+ size 16187875456
WizardLM-2-8x22B.IQ4_XS-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce912e4e6c0948169b3b192268cbdb04af0c1bd8f3790bfcb9cd2fc39d4f0291
3
+ size 17432699040
tokenizer.model → WizardLM-2-8x22B.IQ4_XS-00005-of-00005.gguf RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
3
- size 493443
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:762ac686d247b8b7499ea8076c432df8f409807401819c47f7042be617ad8bce
3
+ size 6869523776
WizardLM-2-8x22B.Q3_K_L-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:677254a8764e9cdc736e638e6634c960c3dbb305c99eb15963f5edb28b4f81f2
3
+ size 16530777024
WizardLM-2-8x22B.Q3_K_L-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3c810fd9c410f7d21effb1d75fb1255a3a06770a696755d8bdadd618f07bf58
3
+ size 17108099264
WizardLM-2-8x22B.Q3_K_L-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5129af244ebaaecbf40f89cb6eede1b2035c4f9a8f1df066ed97413e87fb4eec
3
+ size 15552635008
WizardLM-2-8x22B.Q3_K_L-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ecef6b5d484ae03d0684ec579fb086e3864cf2b50805c85dd3a6dbffb69fd8fe
3
+ size 16775340192
WizardLM-2-8x22B.Q3_K_L-00005-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af2364a5dc9bde78cfccf90063df9b01658410629124ceadb40eef2edb9f628b
3
+ size 6612950336
WizardLM-2-8x22B.Q3_K_M-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b4132b1115f0fa12451c9049c24c0f3d1ab3348f70fa4609a94d193a6d58da1
3
+ size 15650661312
WizardLM-2-8x22B.Q3_K_M-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:001d7eb7125217a694781fecf9ca2cb49c655fb5b0bd124a4025b198dd140bae
3
+ size 15916261568
WizardLM-2-8x22B.Q3_K_M-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:577ea20c1fcc89ee42e5fabdd372dc7974857bf0eaedfab111ca94c6ba0fc607
3
+ size 14480924800
WizardLM-2-8x22B.Q3_K_M-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a96a5ad47fa40eebd46723a2db791e5834430bbb1a25014839028fa64db87bf
3
+ size 15583502496
WizardLM-2-8x22B.Q3_K_M-00005-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f79c807121753e7c4d8312f02330c5315a37e47ac1e1feb6a64f395416fb6af1
3
+ size 6158294336
WizardLM-2-8x22B.Q3_K_S-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02c67f83658f8039c146642741936c7a1ad18d786af0364e7486f616c45508fc
3
+ size 13958259648
WizardLM-2-8x22B.Q3_K_S-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4578f7ef6150c93b7347bdb77c6f2c3f839bec145ec9022bccea8900d8e3caf2
3
+ size 14525849792
WizardLM-2-8x22B.Q3_K_S-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbf516d44516ac5f1478f48dfb37c45ebd91940111b32af93edbcf629441e8a5
3
+ size 13197467776
WizardLM-2-8x22B.Q3_K_S-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5f4433ced22748b00fabffe18ef41b93aa164214029603286ecc55ad7ef6f8e1
3
+ size 14193090720
WizardLM-2-8x22B.Q3_K_S-00005-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13b984ca43dcbf4d0a873d7658400060ab00629cc784716142819dbb045da01c
3
+ size 5623520576
WizardLM-2-8x22B.Q4_K_M-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ff155aaade7d6d9c208e5ad3933cbc02974c30a6742ee3404f7b51e617a4f3e
3
+ size 19982390208
WizardLM-2-8x22B.Q4_K_M-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cda9d26caeeb40eb916e04b4ef5efc0117b90ffe63d5f8fe997262fb04b4b333
3
+ size 19699687616
WizardLM-2-8x22B.Q4_K_M-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2765d9a73e1d558781fcd77b908d3a26ce92772b02a2cf38ac0717f95f1ab58f
3
+ size 17938473088
WizardLM-2-8x22B.Q4_K_M-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee19ae577798150ee11f6cffe220f9257e4658dcfb6eec6b430aaf97aaa42ba6
3
+ size 19675209888
WizardLM-2-8x22B.Q4_K_M-00005-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4bef8ea8f78d69ad3b9b13f59ee6b69234732670e31580b201f56e4091e1419f
3
+ size 8290999616
WizardLM-2-8x22B.Q4_K_S-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:92456ec422ddf695a19c990df3ac4186f0b12cc757504e3970d200d449044e3a
3
+ size 18818470848
WizardLM-2-8x22B.Q4_K_S-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c5c0adcd16f8254086b04498820138236615e71c7a50b01a78523530a2da260
3
+ size 18869215424
WizardLM-2-8x22B.Q4_K_S-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ab35ad5d7f7e4e8cd8cfafb2291b7ff5e8baa595cfcfaca209fa4dfc7c022bc
3
+ size 17108000896
WizardLM-2-8x22B.Q4_K_S-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6e84aa55e44859ffb4da18f320bdef3b705ede320ad5118ed5d3bc3fd8717d39
3
+ size 18429501600
WizardLM-2-8x22B.Q4_K_S-00005-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:966d15e0c80a8305614d44b0337fbad8c4253bd8bf0973161d39ae2695769110
3
+ size 7252909376
WizardLM-2-8x22B.Q5_K_M-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8fa60ba9d5be2dae5875904f22716eb3cf7fdf99a9871304e324e403369cfc58
3
+ size 22987543488
WizardLM-2-8x22B.Q5_K_M-00002-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0acd9445e3681da199b3243ba9467952106a5d599af0226626c63d2431da6129
3
+ size 23384907968
WizardLM-2-8x22B.Q5_K_M-00003-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29febc013ad3af2c7201bb81433f596e25fa75c060821e20edb65cb1eacb708f
3
+ size 21216321664
WizardLM-2-8x22B.Q5_K_M-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:df9a71ae9a4e57454c1812a51171a5729dee28f0ff1f657cf10aaf79ecd094e9
3
+ size 23058440352
WizardLM-2-8x22B.Q5_K_S-00001-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72fa0b4a2b0b09a62e751473c4fa04f0e60fc71ca7b3dbc127cd7dd658375ec0
3
+ size 22024950720
WizardLM-2-8x22B.Q5_K_S-00004-of-00005.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8e1e46cf212b7fdeb2e97dd13d902386b58e47e6041d8136b33b31a67af88fcf
3
+ size 22416711840
config.json DELETED
@@ -1,31 +0,0 @@
1
- {
2
- "_name_or_path": "",
3
- "architectures": [
4
- "MixtralForCausalLM"
5
- ],
6
- "attention_dropout": 0.0,
7
- "bos_token_id": 1,
8
- "eos_token_id": 2,
9
- "hidden_act": "silu",
10
- "hidden_size": 6144,
11
- "initializer_range": 0.02,
12
- "intermediate_size": 16384,
13
- "max_position_embeddings": 65536,
14
- "model_type": "mixtral",
15
- "num_attention_heads": 48,
16
- "num_experts_per_tok": 2,
17
- "num_hidden_layers": 56,
18
- "num_key_value_heads": 8,
19
- "num_local_experts": 8,
20
- "output_router_logits": false,
21
- "rms_norm_eps": 1e-05,
22
- "rope_theta": 1000000,
23
- "router_aux_loss_coef": 0.001,
24
- "router_jitter_noise": 0.0,
25
- "sliding_window": null,
26
- "tie_word_embeddings": false,
27
- "torch_dtype": "bfloat16",
28
- "transformers_version": "4.36.2",
29
- "use_cache": false,
30
- "vocab_size": 32000
31
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
generation_config.json DELETED
@@ -1,6 +0,0 @@
1
- {
2
- "_from_model_config": true,
3
- "bos_token_id": 1,
4
- "eos_token_id": 2,
5
- "transformers_version": "4.36.2"
6
- }
 
 
 
 
 
 
 
model.safetensors.index.json DELETED
The diff for this file is too large to render. See raw diff
 
special_tokens_map.json DELETED
@@ -1,24 +0,0 @@
1
- {
2
- "bos_token": {
3
- "content": "<s>",
4
- "lstrip": false,
5
- "normalized": false,
6
- "rstrip": false,
7
- "single_word": false
8
- },
9
- "eos_token": {
10
- "content": "</s>",
11
- "lstrip": false,
12
- "normalized": false,
13
- "rstrip": false,
14
- "single_word": false
15
- },
16
- "pad_token": "<unk>",
17
- "unk_token": {
18
- "content": "<unk>",
19
- "lstrip": false,
20
- "normalized": false,
21
- "rstrip": false,
22
- "single_word": false
23
- }
24
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tokenizer_config.json DELETED
@@ -1,44 +0,0 @@
1
- {
2
- "add_bos_token": true,
3
- "add_eos_token": false,
4
- "added_tokens_decoder": {
5
- "0": {
6
- "content": "<unk>",
7
- "lstrip": false,
8
- "normalized": false,
9
- "rstrip": false,
10
- "single_word": false,
11
- "special": true
12
- },
13
- "1": {
14
- "content": "<s>",
15
- "lstrip": false,
16
- "normalized": false,
17
- "rstrip": false,
18
- "single_word": false,
19
- "special": true
20
- },
21
- "2": {
22
- "content": "</s>",
23
- "lstrip": false,
24
- "normalized": false,
25
- "rstrip": false,
26
- "single_word": false,
27
- "special": true
28
- }
29
- },
30
- "additional_special_tokens": [],
31
- "bos_token": "<s>",
32
- "chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{{ messages[0]['content'].strip() }}{% else %}{% set loop_messages = messages %}{{ 'A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\\'s questions.' }}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{% if message['role'] == 'system' or message['role'] == 'user' %}{{ ' USER: ' + message['content'].strip() }}{% else %}{{ ' ASSISTANT: ' + message['content'].strip() + eos_token }}{% endif %}{% else %}{% if message['role'] == 'system' or message['role'] == 'user' %}{{ '\nUSER: ' + message['content'].strip() }}{% else %}{{ ' ASSISTANT: ' + message['content'].strip() + eos_token }}{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ ' ASSISTANT:' }}{% endif %}",
33
- "clean_up_tokenization_spaces": false,
34
- "eos_token": "</s>",
35
- "legacy": true,
36
- "model_max_length": 1000000000000000019884624838656,
37
- "pad_token": "<unk>",
38
- "padding_side": "right",
39
- "sp_model_kwargs": {},
40
- "spaces_between_special_tokens": false,
41
- "tokenizer_class": "LlamaTokenizer",
42
- "unk_token": "<unk>",
43
- "use_default_system_prompt": true
44
- }