Q-bert

/

MetaMath-Cybertron-Starling

like 38

I love your work on merging models. Can I ask how can I reproduce it?

https://github.com/cg123/mergekit

You can use this.

Merge Config:

slices:
  - sources:
      - model: Q-bert/MetaMath-Cybertron
        layer_range: [0, 32]
      - model: berkeley-nest/Starling-LM-7B-alpha
        layer_range: [0, 32]
merge_method: slerp
base_model: Q-bert/MetaMath-Cybertron
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 
dtype: float16 

Its default template.

This is awsome. Thank you in advanced. Waiting for more good models from you