ayoubkirouane
commited on
Commit
•
17b3386
1
Parent(s):
ae052a0
Update README.md
Browse files
README.md
CHANGED
@@ -16,6 +16,7 @@ This model was merged using the [TIES](https://arxiv.org/abs/2306.01708) merge m
|
|
16 |
### TIES :
|
17 |
|
18 |
|
|
|
19 |
In Yadav et al.'s paper, the **TIES-Merging** technique is introduced as an efficient approach for consolidating multiple task-specific models into a unified multitask model. This method addresses two primary challenges associated with model merging. Firstly, it tackles redundancy in model parameters by identifying and eliminating redundant parameters within task-specific models. This is achieved by focusing on the changes made during fine-tuning, identifying the top-k% most significant changes, and discarding the remainder. Secondly, TIES-Merging addresses conflicts arising from disagreement between parameter signs, where different models suggest opposing adjustments to the same parameter. To resolve these conflicts, TIES-Merging creates a unified sign vector representing the most dominant direction of change across all models. The TIES-Merging process is structured into three key steps: Trim, which reduces redundancy by retaining a fraction of the most significant parameters; Elect Sign, which resolves sign conflicts by establishing a unified sign vector based on the dominant direction; and Disjoint Merge, which averages parameter values aligning with the unified sign vector while excluding zero values.
|
20 |
|
21 |
|
|
|
16 |
### TIES :
|
17 |
|
18 |
|
19 |
+
|
20 |
In Yadav et al.'s paper, the **TIES-Merging** technique is introduced as an efficient approach for consolidating multiple task-specific models into a unified multitask model. This method addresses two primary challenges associated with model merging. Firstly, it tackles redundancy in model parameters by identifying and eliminating redundant parameters within task-specific models. This is achieved by focusing on the changes made during fine-tuning, identifying the top-k% most significant changes, and discarding the remainder. Secondly, TIES-Merging addresses conflicts arising from disagreement between parameter signs, where different models suggest opposing adjustments to the same parameter. To resolve these conflicts, TIES-Merging creates a unified sign vector representing the most dominant direction of change across all models. The TIES-Merging process is structured into three key steps: Trim, which reduces redundancy by retaining a fraction of the most significant parameters; Elect Sign, which resolves sign conflicts by establishing a unified sign vector based on the dominant direction; and Disjoint Merge, which averages parameter values aligning with the unified sign vector while excluding zero values.
|
21 |
|
22 |
|