paulilioaica commited on
Commit
958c7b2
1 Parent(s): 817811c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -9
README.md CHANGED
@@ -13,23 +13,38 @@ base_model:
13
  - rhysjones/phi-2-orange
14
  ---
15
 
16
- # PhiMiX-2x2B_embed
17
 
18
- PhiMiX-2x2B_embed is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
 
 
 
 
 
 
 
 
19
  * [cognitivecomputations/dolphin-2_6-phi-2](https://huggingface.co/cognitivecomputations/dolphin-2_6-phi-2)
20
  * [rhysjones/phi-2-orange](https://huggingface.co/rhysjones/phi-2-orange)
21
 
 
 
 
 
 
 
 
22
  ## 🧩 Configuration
23
 
24
  ```yaml
25
  base_model: rhysjones/phi-2-orange
26
- gate_mode: cheap_embed
27
  dtype: float16
28
  experts:
29
  - source_model: cognitivecomputations/dolphin-2_6-phi-2
30
- positive_prompts: ["research, logic, math, science"]
31
  - source_model: rhysjones/phi-2-orange
32
- positive_prompts: ["programming, reasoning"]
33
  ```
34
 
35
  ## 💻 Usage
@@ -47,11 +62,13 @@ tokenizer = AutoTokenizer.from_pretrained(model)
47
  pipeline = transformers.pipeline(
48
  "text-generation",
49
  model=model,
50
- model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
 
51
  )
52
 
53
- messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
54
- prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
55
- outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
56
  print(outputs[0]["generated_text"])
 
57
  ```
 
13
  - rhysjones/phi-2-orange
14
  ---
15
 
16
+ # PhiMiX-2x2B
17
 
18
+
19
+ ## Code is work in progress
20
+
21
+ <p align="center">
22
+ <img src="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11201acc-4089-416d-921b-cbd71fbf8ddb_1024x1024.jpeg" width="500" class="center"/>
23
+ </p>
24
+
25
+
26
+ PhiMiX-2x2B is a Mixure of Experts (MoE) made with the following models using mergekit:
27
  * [cognitivecomputations/dolphin-2_6-phi-2](https://huggingface.co/cognitivecomputations/dolphin-2_6-phi-2)
28
  * [rhysjones/phi-2-orange](https://huggingface.co/rhysjones/phi-2-orange)
29
 
30
+
31
+ ## ©️ Credits
32
+ * [mlabonne's phixtral](https://huggingface.co/mlabonne/phixtral-4x2_8) for the PhiConfig and inference code.
33
+ * [mergekit](https://github.com/cg123/mergekit) code which I tweaked (you can find the PhiConfig [here](https://github.com/cg123/mergekit/blob/508348ae34be17ea0a95d0a288a6e34491a2558a/mergekit/architecture.py#L289))
34
+ by mainly adding the config in the `moe_mixtral.py` script from `mixtral` branch.
35
+
36
+
37
  ## 🧩 Configuration
38
 
39
  ```yaml
40
  base_model: rhysjones/phi-2-orange
41
+ gate_mode: random
42
  dtype: float16
43
  experts:
44
  - source_model: cognitivecomputations/dolphin-2_6-phi-2
45
+ positive_prompts: [""]
46
  - source_model: rhysjones/phi-2-orange
47
+ positive_prompts: [""]
48
  ```
49
 
50
  ## 💻 Usage
 
62
  pipeline = transformers.pipeline(
63
  "text-generation",
64
  model=model,
65
+ trust_remote_code=True,
66
+ model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True,},
67
  )
68
 
69
+ prompt="How many continents are there?"
70
+ input = f"Instruct: <prompt>\nOutput:"
71
+ outputs = pipeline(input, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
72
  print(outputs[0]["generated_text"])
73
+
74
  ```