SpectreLynx
commited on
Commit
•
6c64927
1
Parent(s):
67fb966
Update README.md
Browse files
README.md
CHANGED
@@ -1,8 +1,6 @@
|
|
1 |
---
|
2 |
base_model:
|
3 |
- Undi95/Llama-3-Unholy-8B
|
4 |
-
- Locutusque/llama-3-neural-chat-v1-8b
|
5 |
-
- ruslanmv/Medical-Llama3-8B-16bit
|
6 |
library_name: transformers
|
7 |
tags:
|
8 |
- mergekit
|
@@ -18,7 +16,7 @@ datasets:
|
|
18 |
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
|
19 |
- ruslanmv/ai-medical-chatbot
|
20 |
model-index:
|
21 |
-
- name:
|
22 |
results:
|
23 |
- task:
|
24 |
type: text-generation
|
@@ -34,10 +32,7 @@ model-index:
|
|
34 |
- type: acc_norm
|
35 |
value: 59.13
|
36 |
name: normalized accuracy
|
37 |
-
|
38 |
-
url: >-
|
39 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
40 |
-
name: Open LLM Leaderboard
|
41 |
- task:
|
42 |
type: text-generation
|
43 |
name: Text Generation
|
@@ -51,10 +46,7 @@ model-index:
|
|
51 |
- type: acc_norm
|
52 |
value: 82.9
|
53 |
name: normalized accuracy
|
54 |
-
|
55 |
-
url: >-
|
56 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
57 |
-
name: Open LLM Leaderboard
|
58 |
- task:
|
59 |
type: text-generation
|
60 |
name: Text Generation
|
@@ -69,10 +61,7 @@ model-index:
|
|
69 |
- type: acc
|
70 |
value: 60.35
|
71 |
name: accuracy
|
72 |
-
|
73 |
-
url: >-
|
74 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
75 |
-
name: Open LLM Leaderboard
|
76 |
- task:
|
77 |
type: text-generation
|
78 |
name: Text Generation
|
@@ -86,10 +75,7 @@ model-index:
|
|
86 |
metrics:
|
87 |
- type: mc2
|
88 |
value: 49.65
|
89 |
-
|
90 |
-
url: >-
|
91 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
92 |
-
name: Open LLM Leaderboard
|
93 |
- task:
|
94 |
type: text-generation
|
95 |
name: Text Generation
|
@@ -104,10 +90,7 @@ model-index:
|
|
104 |
- type: acc
|
105 |
value: 78.93
|
106 |
name: accuracy
|
107 |
-
|
108 |
-
url: >-
|
109 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
110 |
-
name: Open LLM Leaderboard
|
111 |
- task:
|
112 |
type: text-generation
|
113 |
name: Text Generation
|
@@ -122,64 +105,36 @@ model-index:
|
|
122 |
- type: acc
|
123 |
value: 60.35
|
124 |
name: accuracy
|
125 |
-
|
126 |
-
url: >-
|
127 |
-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Medichat-Llama3-8B
|
128 |
-
name: Open LLM Leaderboard
|
129 |
language:
|
130 |
- en
|
131 |
---
|
132 |
|
133 |
-
###
|
134 |
|
135 |
Built upon the powerful LLaMa-3 architecture and fine-tuned on an extensive dataset of health information, this model leverages its vast medical knowledge to offer clear, comprehensive answers.
|
136 |
|
137 |
This model is generally better for accurate and informative responses, particularly for users seeking in-depth medical advice.
|
138 |
|
139 |
|
140 |
-
|
141 |
-
|
142 |
-
|
143 |
-
|
144 |
-
|
145 |
-
|
146 |
-
|
147 |
-
|
148 |
-
|
149 |
-
|
150 |
-
|
151 |
-
|
152 |
-
|
153 |
-
|
154 |
-
|
155 |
-
|
156 |
-
|
157 |
-
|
158 |
-
dtype: bfloat16
|
159 |
-
|
160 |
-
```
|
161 |
-
|
162 |
-
# Comparision Against Dr.Samantha 7B
|
163 |
-
|
164 |
-
| Subject | Medichat-Llama3-8B Accuracy (%) | Dr. Samantha Accuracy (%) |
|
165 |
-
|-------------------------|---------------------------------|---------------------------|
|
166 |
-
| Clinical Knowledge | 71.70 | 52.83 |
|
167 |
-
| Medical Genetics | 78.00 | 49.00 |
|
168 |
-
| Human Aging | 70.40 | 58.29 |
|
169 |
-
| Human Sexuality | 73.28 | 55.73 |
|
170 |
-
| College Medicine | 62.43 | 38.73 |
|
171 |
-
| Anatomy | 64.44 | 41.48 |
|
172 |
-
| College Biology | 72.22 | 52.08 |
|
173 |
-
| High School Biology | 77.10 | 53.23 |
|
174 |
-
| Professional Medicine | 63.97 | 38.73 |
|
175 |
-
| Nutrition | 73.86 | 50.33 |
|
176 |
-
| Professional Psychology | 68.95 | 46.57 |
|
177 |
-
| Virology | 54.22 | 41.57 |
|
178 |
-
| High School Psychology | 83.67 | 66.60 |
|
179 |
-
| **Average** | **70.33** | **48.85** |
|
180 |
-
|
181 |
-
|
182 |
-
The current model demonstrates a substantial improvement over the previous [Dr. Samantha](sethuiyer/Dr_Samantha-7b) model in terms of subject-specific knowledge and accuracy.
|
183 |
|
184 |
### Usage:
|
185 |
```python
|
@@ -187,7 +142,7 @@ import torch
|
|
187 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|
188 |
|
189 |
class MedicalAssistant:
|
190 |
-
def __init__(self, model_name="
|
191 |
self.device = device
|
192 |
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
|
193 |
self.model = AutoModelForCausalLM.from_pretrained(model_name).to(self.device)
|
@@ -224,12 +179,3 @@ if __name__ == "__main__":
|
|
224 |
print(response)
|
225 |
|
226 |
```
|
227 |
-
|
228 |
-
## Quants
|
229 |
-
Thanks to [Quant Factory](https://huggingface.co/QuantFactory), the quantized version of this model is available at [QuantFactory/Medichat-Llama3-8B-GGUF](https://huggingface.co/QuantFactory/Medichat-Llama3-8B-GGUF),
|
230 |
-
|
231 |
-
|
232 |
-
## Ollama
|
233 |
-
This model is now also available on Ollama. You can use it by running the command ```ollama run monotykamary/medichat-llama3``` in your
|
234 |
-
terminal. If you have limited computing resources, check out this [video](https://www.youtube.com/watch?v=Qa1h7ygwQq8) to learn how to run it on
|
235 |
-
a Google Colab backend.
|
|
|
1 |
---
|
2 |
base_model:
|
3 |
- Undi95/Llama-3-Unholy-8B
|
|
|
|
|
4 |
library_name: transformers
|
5 |
tags:
|
6 |
- mergekit
|
|
|
16 |
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
|
17 |
- ruslanmv/ai-medical-chatbot
|
18 |
model-index:
|
19 |
+
- name: Ryeta-0
|
20 |
results:
|
21 |
- task:
|
22 |
type: text-generation
|
|
|
32 |
- type: acc_norm
|
33 |
value: 59.13
|
34 |
name: normalized accuracy
|
35 |
+
|
|
|
|
|
|
|
36 |
- task:
|
37 |
type: text-generation
|
38 |
name: Text Generation
|
|
|
46 |
- type: acc_norm
|
47 |
value: 82.9
|
48 |
name: normalized accuracy
|
49 |
+
|
|
|
|
|
|
|
50 |
- task:
|
51 |
type: text-generation
|
52 |
name: Text Generation
|
|
|
61 |
- type: acc
|
62 |
value: 60.35
|
63 |
name: accuracy
|
64 |
+
|
|
|
|
|
|
|
65 |
- task:
|
66 |
type: text-generation
|
67 |
name: Text Generation
|
|
|
75 |
metrics:
|
76 |
- type: mc2
|
77 |
value: 49.65
|
78 |
+
|
|
|
|
|
|
|
79 |
- task:
|
80 |
type: text-generation
|
81 |
name: Text Generation
|
|
|
90 |
- type: acc
|
91 |
value: 78.93
|
92 |
name: accuracy
|
93 |
+
|
|
|
|
|
|
|
94 |
- task:
|
95 |
type: text-generation
|
96 |
name: Text Generation
|
|
|
105 |
- type: acc
|
106 |
value: 60.35
|
107 |
name: accuracy
|
108 |
+
|
|
|
|
|
|
|
109 |
language:
|
110 |
- en
|
111 |
---
|
112 |
|
113 |
+
### Ryeta-0
|
114 |
|
115 |
Built upon the powerful LLaMa-3 architecture and fine-tuned on an extensive dataset of health information, this model leverages its vast medical knowledge to offer clear, comprehensive answers.
|
116 |
|
117 |
This model is generally better for accurate and informative responses, particularly for users seeking in-depth medical advice.
|
118 |
|
119 |
|
120 |
+
# Benchmarks
|
121 |
+
|
122 |
+
| Subject | Medichat-Llama3-8B Accuracy (%) | |
|
123 |
+
|-------------------------|---------------------------------|
|
124 |
+
| Clinical Knowledge | 71.70 |
|
125 |
+
| Medical Genetics | 78.00 |
|
126 |
+
| Human Aging | 70.40 |
|
127 |
+
| Human Sexuality | 73.28 |
|
128 |
+
| College Medicine | 62.43 |
|
129 |
+
| Anatomy | 64.44 |
|
130 |
+
| College Biology | 72.22 |
|
131 |
+
| High School Biology | 77.10 |
|
132 |
+
| Professional Medicine | 63.97 |
|
133 |
+
| Nutrition | 73.86 |
|
134 |
+
| Professional Psychology | 68.95 |
|
135 |
+
| Virology | 54.22 |
|
136 |
+
| High School Psychology | 83.67 |
|
137 |
+
| **Average** | **70.33** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
138 |
|
139 |
### Usage:
|
140 |
```python
|
|
|
142 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|
143 |
|
144 |
class MedicalAssistant:
|
145 |
+
def __init__(self, model_name="SpectreLynx/Ryeta-0", device="cuda"):
|
146 |
self.device = device
|
147 |
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
|
148 |
self.model = AutoModelForCausalLM.from_pretrained(model_name).to(self.device)
|
|
|
179 |
print(response)
|
180 |
|
181 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|