|
--- |
|
base_model: |
|
- RozGrov/NemoDori-v0.2-12B-MN-BT |
|
- unsloth/Mistral-Nemo-Instruct-2407 |
|
- UsernameJustAnother/Nemo-12B-Marlin-v5 |
|
- crestf411/nemo-sunfall-v0.6.1 |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
pipeline_tag: text-generation |
|
--- |
|
# NemoDori-v0.2.1-12B-MN-BT |
|
|
|
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
|
|
|
The first child from [NemoDori-v0.2-12B-MN-BT](https://huggingface.co/RozGrov/NemoDori-v0.2-12B-MN-BT). |
|
|
|
**The purpose** is to find a way to increase v0.2 capability to stay **aware of the past conversations** and **follow instructions better**, especially the last one (depth-0), |
|
while keeping it's **creativity and capability to (E)RP**. |
|
This model is one of the few childs to try to fulfill that. |
|
|
|
In my very short testing so far, I haven't found anything that's different from the parent and worth mentioning. But I think this version is **slightly degraded** somehow, |
|
(I don't quite know it, I just felt like it did). Anyway, try it as you may, but I think it's parent (**v0.2**) is **better** than this one. |
|
|
|
The other child ([**v0.2.2**](https://huggingface.co/RozGrov/NemoDori-v0.2.2-12B-MN-ties)) is out. |
|
I tested it more than this model and it seems to be improved better than this model, but the response format is not very consistent. |
|
|
|
You may give me feedback on anything, or guide me how I can fulfill my-*ahem* it's purpose while keeping it as low as not-70B. |
|
<br> |
|
Fine-tune is... pretty expensive for me, and I'm not ready for that (yet, tho i'm interested). |
|
|
|
<p style="font-size: 11px; margin-top: 11px" id="heya-im-a-bit-of-a-programmer"> |
|
(listen, between you and me, i still don't get it. still learning this new hobby of mine, and it's kind of refreshing in a way. |
|
i'll be exploring more other architectures in the future. Yet, this is about how random i pick my straw, just to see how lucky i am.) |
|
<br> |
|
(although, i am interested to learn how to make a new merge method. |
|
similar to when i'm making a solution for solving specific problem just like good ol days. |
|
<span style="color: darkred">but hell, this llm stuff is really expensive.</span>) |
|
</p> |
|
|
|
|
|
## Merge Details |
|
|
|
### Merge Method |
|
|
|
This model was merged using the `breadcrumbs_ties` merge method using [RozGrov/NemoDori-v0.2-12B-MN-BT](https://huggingface.co/RozGrov/NemoDori-v0.2-12B-MN-BT) as a base. |
|
|
|
### Models Merged |
|
|
|
The following models were included in the merge: |
|
* [unsloth/Mistral-Nemo-Instruct-2407](https://huggingface.co/unsloth/Mistral-Nemo-Instruct-2407) |
|
* [UsernameJustAnother/Nemo-12B-Marlin-v5](https://huggingface.co/UsernameJustAnother/Nemo-12B-Marlin-v5) |
|
* [crestf411/nemo-sunfall-v0.6.1](https://huggingface.co/crestf411/nemo-sunfall-v0.6.1) |
|
|
|
### Configuration |
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
|
|
models: |
|
- model: crestf411/nemo-sunfall-v0.6.1 |
|
parameters: |
|
weight: 0.33 |
|
- model: UsernameJustAnother/Nemo-12B-Marlin-v5 |
|
parameters: |
|
weight: 0.2 |
|
- model: unsloth/Mistral-Nemo-Instruct-2407 |
|
parameters: |
|
weight: 0.37 |
|
- model: RozGrov/NemoDori-v0.2-12B-MN-BT |
|
parameters: |
|
weight: 1 |
|
merge_method: breadcrumbs_ties |
|
base_model: RozGrov/NemoDori-v0.2-12B-MN-BT |
|
parameters: |
|
density: 0.93 |
|
gamma: 0.015 |
|
dtype: bfloat16 |
|
|
|
``` |
