Migel Tissera commited on
Commit
14b68a1
1 Parent(s): e125241

edited readme

Browse files
Files changed (1) hide show
  1. README.md +125 -0
README.md CHANGED
@@ -1,3 +1,128 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ # HelixNet
6
+
7
+ HelixNet is a Deep Learning architecture consisting of 3 x Mistral-7B LLMs. It has an `actor`, a `critic`, and a `regenerator`. The `actor` LLM produces an initial response to a given system-context and a question. The `critic` then takes in as input, a tuple of (system-context, question, response) and provides a critique based on the provided answer to the given system-context and the question. Its job is not to criticize, but to provide an intelligent critique so that the answer can be modified/regenerated to address the question better. Finally, the `regenerator` takes in a tuple of (system-context, question, response, critique) and regenerates the answer.
8
+
9
+ HelixNet is insprired from an actor-critic architecture most prominent in Reinforcement Learning algorithms. The name derives from Helix, referring to the spiral structure of a DNA molecule. It symbolizes the intertwined nature of the three networks, working in tandem, much like the strands of a DNA molecule.
10
+
11
+ HelixNet regenerates very pleasing and accurate responses, due to the entropy preservation of the regenerator. The regenerator was only trained on a dataset of 1000 samples, similar to Meta's LIMA. The actor network here was trained on about 250K very high-quality samples, and the critic network was trained on further 10K samples.
12
+
13
+ # Training Methodology
14
+
15
+ ## Phase 1: Actor
16
+
17
+ The actor network was trained with Supervised Fine-Tuning, on 250K very high-quality samples. It has 75K of Open-Orca's Chain-of-Thought data, and a mixture of Dolphin (GPT-4), SynthIA's Tree-of-Thought data.
18
+
19
+ ## Phase 2: Critic
20
+
21
+ To train the critic, the following process was followed:
22
+ - Use Actor, and send 10K system-context and question pairs. Generate responses
23
+ - Use the (system-context, question, response) tuples to generate critiques. Used OpenAI's GPT-4.
24
+
25
+ Using the above training dataset, a Mistral-7B was fine-tuned.
26
+
27
+ ## Phase 3: Regenerator
28
+ - Use the (system-context, question, response, critique) tuples to regenerate the answers. Used OpenAI's GPT-4.
29
+
30
+ A thrid LLM was fine-tuned using the above data.
31
+
32
+
33
+ # Reusability of the `critic` and the `regenerator`
34
+
35
+ The `critic` and the `regenerator` was tested not only on the accopanying actor model, but 13B and 70B SynthIA models as well. They seem to be readily transferrable, as the function that it has learnt is to provide an intelligent critique and then a regeneration of the original response. Please feel free to try out other models as the `actor`. However, the architecture works best with all three as presented here in HelixNet.
36
+
37
+ # Prompt format:
38
+
39
+ ```
40
+ SYSTEM: Elaborate on the topic using a Tree of Thoughts and backtrack when necessary to construct a clear, cohesive Chain of Thought reasoning. Always answer without hesitation.
41
+ USER: What is the relationship between Earth's atmosphere, magnetic field and gravity?
42
+ ASSISTANT:
43
+ ```
44
+
45
+ # Example Usage
46
+
47
+ ## Code example:
48
+
49
+ The following is a code example on how to use HelixNet. No special system-context messages are needed for the `critic` and the `regenerator`.
50
+
51
+ ```python
52
+ import torch, json
53
+ from transformers import AutoModelForCausalLM, AutoTokenizer
54
+
55
+ model_path_actor = "/home/ubuntu/llm/HelixNet/actor"
56
+ model_path_critic = "/home/ubuntu/llm/HelixNet/critic"
57
+ model_path_regenerator = "/home/ubuntu/llm/HelixNet/regenerator"
58
+
59
+ def load_model(model_path):
60
+ model = AutoModelForCausalLM.from_pretrained(
61
+ model_path,
62
+ torch_dtype=torch.float16,
63
+ device_map="cuda",
64
+ load_in_4bit=False,
65
+ trust_remote_code=True,
66
+ )
67
+ return model
68
+
69
+ def load_tokenizer(model_path):
70
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
71
+ return tokenizer
72
+
73
+ model_actor = load_model(model_path_actor)
74
+ model_critic = load_model(model_path_critic)
75
+ model_regenerator = load_model(model_path_regenerator)
76
+
77
+ tokenizer_actor = load_tokenizer(model_path_actor)
78
+ tokenizer_critic = load_tokenizer(model_path_critic)
79
+ tokenizer_regenerator = load_tokenizer(model_path_regenerator)
80
+
81
+ def generate_text(instruction, model, tokenizer):
82
+ tokens = tokenizer.encode(instruction)
83
+ tokens = torch.LongTensor(tokens).unsqueeze(0)
84
+ tokens = tokens.to("cuda")
85
+
86
+ instance = {
87
+ "input_ids": tokens,
88
+ "top_p": 1.0,
89
+ "temperature": 0.75,
90
+ "generate_len": 1024,
91
+ "top_k": 50,
92
+ }
93
+
94
+ length = len(tokens[0])
95
+ with torch.no_grad():
96
+ rest = model.generate(
97
+ input_ids=tokens,
98
+ max_length=length + instance["generate_len"],
99
+ use_cache=True,
100
+ do_sample=True,
101
+ top_p=instance["top_p"],
102
+ temperature=instance["temperature"],
103
+ top_k=instance["top_k"],
104
+ num_return_sequences=1,
105
+ )
106
+ output = rest[0][length:]
107
+ string = tokenizer.decode(output, skip_special_tokens=True)
108
+ return f"{string}"
109
+
110
+ system_prompt = "You are HelixNet. Elaborate on the topic using a Tree of Thoughts and backtrack when necessary to construct a clear, cohesive Chain of Thought reasoning. Always answer without hesitation."
111
+
112
+
113
+ while True:
114
+ user_input = input("You: ")
115
+
116
+ prompt_actor = f"SYSTEM: {system_prompt} \nUSER: {user_input} \nASSISTANT: "
117
+ actor_response = generate_text(prompt_actor, model_actor, tokenizer_actor)
118
+ print(f"ACTOR: {actor_response}\n\n")
119
+
120
+ prompt_critic = f"SYSTEM: {system_prompt} \nUSER: {user_input} \nRESPONSE: {actor_response} \nCRITIQUE:"
121
+ critic_response = generate_text(prompt_critic, model_critic, tokenizer_critic)
122
+ print(f"CRITIQUE: {critic_response}\n\n")
123
+
124
+ prompt_regenerator = f"SYSTEM: {system_prompt} \nUSER: {user_input} \nRESPONSE: {actor_response} \nCRITIQUE: {critic_response} \nREGENERATOR:"
125
+ regenerator_response = generate_text(prompt_regenerator, model_regenerator, tokenizer_regenerator)
126
+ print(f"REGENERATION: {regenerator_response}")
127
+
128
+ ```