--- tags: - mteb model-index: - name: checkpoint-1431 results: - task: type: STS dataset: type: C-MTEB/AFQMC name: MTEB AFQMC config: default split: validation revision: None metrics: - type: cos_sim_pearson value: 56.306314279047875 - type: cos_sim_spearman value: 61.020227685004016 - type: euclidean_pearson value: 58.61821670933433 - type: euclidean_spearman value: 60.131457106640674 - type: manhattan_pearson value: 58.6189460369694 - type: manhattan_spearman value: 60.126350618526224 - task: type: STS dataset: type: C-MTEB/ATEC name: MTEB ATEC config: default split: test revision: None metrics: - type: cos_sim_pearson value: 55.8612958476143 - type: cos_sim_spearman value: 59.01977664864512 - type: euclidean_pearson value: 62.028094897243655 - type: euclidean_spearman value: 58.6046814257705 - type: manhattan_pearson value: 62.02580042431887 - type: manhattan_spearman value: 58.60626890004892 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) config: zh split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 49.496 - type: f1 value: 46.673963383873065 - task: type: STS dataset: type: C-MTEB/BQ name: MTEB BQ config: default split: test revision: None metrics: - type: cos_sim_pearson value: 70.73971622592535 - type: cos_sim_spearman value: 72.76102992060764 - type: euclidean_pearson value: 71.04525865868672 - type: euclidean_spearman value: 72.4032852155075 - type: manhattan_pearson value: 71.03693009336658 - type: manhattan_spearman value: 72.39635701224252 - task: type: Clustering dataset: type: C-MTEB/CLSClusteringP2P name: MTEB CLSClusteringP2P config: default split: test revision: None metrics: - type: v_measure value: 56.34751074520767 - task: type: Clustering dataset: type: C-MTEB/CLSClusteringS2S name: MTEB CLSClusteringS2S config: default split: test revision: None metrics: - type: v_measure value: 48.4856662121073 - task: type: Reranking dataset: type: C-MTEB/CMedQAv1-reranking name: MTEB CMedQAv1 config: default split: test revision: None metrics: - type: map value: 89.26384109024997 - type: mrr value: 91.27261904761905 - task: type: Reranking dataset: type: C-MTEB/CMedQAv2-reranking name: MTEB CMedQAv2 config: default split: test revision: None metrics: - type: map value: 90.0464058154547 - type: mrr value: 92.06480158730159 - task: type: Retrieval dataset: type: C-MTEB/CmedqaRetrieval name: MTEB CmedqaRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 27.236 - type: map_at_10 value: 40.778 - type: map_at_100 value: 42.692 - type: map_at_1000 value: 42.787 - type: map_at_3 value: 36.362 - type: map_at_5 value: 38.839 - type: mrr_at_1 value: 41.335 - type: mrr_at_10 value: 49.867 - type: mrr_at_100 value: 50.812999999999995 - type: mrr_at_1000 value: 50.848000000000006 - type: mrr_at_3 value: 47.354 - type: mrr_at_5 value: 48.718 - type: ndcg_at_1 value: 41.335 - type: ndcg_at_10 value: 47.642 - type: ndcg_at_100 value: 54.855 - type: ndcg_at_1000 value: 56.449000000000005 - type: ndcg_at_3 value: 42.203 - type: ndcg_at_5 value: 44.416 - type: precision_at_1 value: 41.335 - type: precision_at_10 value: 10.568 - type: precision_at_100 value: 1.6400000000000001 - type: precision_at_1000 value: 0.184 - type: precision_at_3 value: 23.998 - type: precision_at_5 value: 17.389 - type: recall_at_1 value: 27.236 - type: recall_at_10 value: 58.80800000000001 - type: recall_at_100 value: 88.411 - type: recall_at_1000 value: 99.032 - type: recall_at_3 value: 42.253 - type: recall_at_5 value: 49.118 - task: type: PairClassification dataset: type: C-MTEB/CMNLI name: MTEB Cmnli config: default split: validation revision: None metrics: - type: cos_sim_accuracy value: 86.03728202044498 - type: cos_sim_ap value: 92.49469583272597 - type: cos_sim_f1 value: 86.74095974528088 - type: cos_sim_precision value: 84.43657294664601 - type: cos_sim_recall value: 89.17465513210195 - type: dot_accuracy value: 72.21888153938664 - type: dot_ap value: 80.59377163340332 - type: dot_f1 value: 74.96686040583258 - type: dot_precision value: 66.4737793851718 - type: dot_recall value: 85.94809445873275 - type: euclidean_accuracy value: 85.47203848466627 - type: euclidean_ap value: 91.89152584749868 - type: euclidean_f1 value: 86.38105975197294 - type: euclidean_precision value: 83.40953625081646 - type: euclidean_recall value: 89.5721299976619 - type: manhattan_accuracy value: 85.3758268190018 - type: manhattan_ap value: 91.88989707722311 - type: manhattan_f1 value: 86.39767519839052 - type: manhattan_precision value: 82.76231263383298 - type: manhattan_recall value: 90.36707972878185 - type: max_accuracy value: 86.03728202044498 - type: max_ap value: 92.49469583272597 - type: max_f1 value: 86.74095974528088 - task: type: Retrieval dataset: type: C-MTEB/CovidRetrieval name: MTEB CovidRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 74.34100000000001 - type: map_at_10 value: 82.49499999999999 - type: map_at_100 value: 82.64200000000001 - type: map_at_1000 value: 82.643 - type: map_at_3 value: 81.142 - type: map_at_5 value: 81.95400000000001 - type: mrr_at_1 value: 74.71 - type: mrr_at_10 value: 82.553 - type: mrr_at_100 value: 82.699 - type: mrr_at_1000 value: 82.70100000000001 - type: mrr_at_3 value: 81.279 - type: mrr_at_5 value: 82.069 - type: ndcg_at_1 value: 74.605 - type: ndcg_at_10 value: 85.946 - type: ndcg_at_100 value: 86.607 - type: ndcg_at_1000 value: 86.669 - type: ndcg_at_3 value: 83.263 - type: ndcg_at_5 value: 84.71600000000001 - type: precision_at_1 value: 74.605 - type: precision_at_10 value: 9.758 - type: precision_at_100 value: 1.005 - type: precision_at_1000 value: 0.101 - type: precision_at_3 value: 29.996000000000002 - type: precision_at_5 value: 18.736 - type: recall_at_1 value: 74.34100000000001 - type: recall_at_10 value: 96.523 - type: recall_at_100 value: 99.473 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 89.278 - type: recall_at_5 value: 92.83500000000001 - task: type: Retrieval dataset: type: C-MTEB/DuRetrieval name: MTEB DuRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 26.950000000000003 - type: map_at_10 value: 82.408 - type: map_at_100 value: 85.057 - type: map_at_1000 value: 85.09100000000001 - type: map_at_3 value: 57.635999999999996 - type: map_at_5 value: 72.48 - type: mrr_at_1 value: 92.15 - type: mrr_at_10 value: 94.554 - type: mrr_at_100 value: 94.608 - type: mrr_at_1000 value: 94.61 - type: mrr_at_3 value: 94.292 - type: mrr_at_5 value: 94.459 - type: ndcg_at_1 value: 92.15 - type: ndcg_at_10 value: 89.108 - type: ndcg_at_100 value: 91.525 - type: ndcg_at_1000 value: 91.82900000000001 - type: ndcg_at_3 value: 88.44 - type: ndcg_at_5 value: 87.271 - type: precision_at_1 value: 92.15 - type: precision_at_10 value: 42.29 - type: precision_at_100 value: 4.812 - type: precision_at_1000 value: 0.48900000000000005 - type: precision_at_3 value: 79.14999999999999 - type: precision_at_5 value: 66.64 - type: recall_at_1 value: 26.950000000000003 - type: recall_at_10 value: 89.832 - type: recall_at_100 value: 97.921 - type: recall_at_1000 value: 99.471 - type: recall_at_3 value: 59.562000000000005 - type: recall_at_5 value: 76.533 - task: type: Retrieval dataset: type: C-MTEB/EcomRetrieval name: MTEB EcomRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 53.5 - type: map_at_10 value: 63.105999999999995 - type: map_at_100 value: 63.63100000000001 - type: map_at_1000 value: 63.641999999999996 - type: map_at_3 value: 60.617 - type: map_at_5 value: 62.132 - type: mrr_at_1 value: 53.5 - type: mrr_at_10 value: 63.105999999999995 - type: mrr_at_100 value: 63.63100000000001 - type: mrr_at_1000 value: 63.641999999999996 - type: mrr_at_3 value: 60.617 - type: mrr_at_5 value: 62.132 - type: ndcg_at_1 value: 53.5 - type: ndcg_at_10 value: 67.92200000000001 - type: ndcg_at_100 value: 70.486 - type: ndcg_at_1000 value: 70.777 - type: ndcg_at_3 value: 62.853 - type: ndcg_at_5 value: 65.59899999999999 - type: precision_at_1 value: 53.5 - type: precision_at_10 value: 8.309999999999999 - type: precision_at_100 value: 0.951 - type: precision_at_1000 value: 0.097 - type: precision_at_3 value: 23.1 - type: precision_at_5 value: 15.2 - type: recall_at_1 value: 53.5 - type: recall_at_10 value: 83.1 - type: recall_at_100 value: 95.1 - type: recall_at_1000 value: 97.39999999999999 - type: recall_at_3 value: 69.3 - type: recall_at_5 value: 76.0 - task: type: Classification dataset: type: C-MTEB/IFlyTek-classification name: MTEB IFlyTek config: default split: validation revision: None metrics: - type: accuracy value: 51.773759138130046 - type: f1 value: 40.38600802756481 - task: type: Classification dataset: type: C-MTEB/JDReview-classification name: MTEB JDReview config: default split: test revision: None metrics: - type: accuracy value: 88.48030018761726 - type: ap value: 59.2732541555627 - type: f1 value: 83.58836007358619 - task: type: STS dataset: type: C-MTEB/LCQMC name: MTEB LCQMC config: default split: test revision: None metrics: - type: cos_sim_pearson value: 73.67511194245922 - type: cos_sim_spearman value: 79.43347759067298 - type: euclidean_pearson value: 79.04491504318766 - type: euclidean_spearman value: 79.14478545356785 - type: manhattan_pearson value: 79.03365022867428 - type: manhattan_spearman value: 79.13172717619908 - task: type: Retrieval dataset: type: C-MTEB/MMarcoRetrieval name: MTEB MMarcoRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 67.184 - type: map_at_10 value: 76.24600000000001 - type: map_at_100 value: 76.563 - type: map_at_1000 value: 76.575 - type: map_at_3 value: 74.522 - type: map_at_5 value: 75.598 - type: mrr_at_1 value: 69.47 - type: mrr_at_10 value: 76.8 - type: mrr_at_100 value: 77.082 - type: mrr_at_1000 value: 77.093 - type: mrr_at_3 value: 75.29400000000001 - type: mrr_at_5 value: 76.24 - type: ndcg_at_1 value: 69.47 - type: ndcg_at_10 value: 79.81099999999999 - type: ndcg_at_100 value: 81.187 - type: ndcg_at_1000 value: 81.492 - type: ndcg_at_3 value: 76.536 - type: ndcg_at_5 value: 78.367 - type: precision_at_1 value: 69.47 - type: precision_at_10 value: 9.599 - type: precision_at_100 value: 1.026 - type: precision_at_1000 value: 0.105 - type: precision_at_3 value: 28.777 - type: precision_at_5 value: 18.232 - type: recall_at_1 value: 67.184 - type: recall_at_10 value: 90.211 - type: recall_at_100 value: 96.322 - type: recall_at_1000 value: 98.699 - type: recall_at_3 value: 81.556 - type: recall_at_5 value: 85.931 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) config: zh-CN split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 76.96032279757901 - type: f1 value: 73.48052314033545 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) config: zh-CN split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 84.64357767316744 - type: f1 value: 83.58250539497922 - task: type: Retrieval dataset: type: C-MTEB/MedicalRetrieval name: MTEB MedicalRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 56.00000000000001 - type: map_at_10 value: 62.066 - type: map_at_100 value: 62.553000000000004 - type: map_at_1000 value: 62.598 - type: map_at_3 value: 60.4 - type: map_at_5 value: 61.370000000000005 - type: mrr_at_1 value: 56.2 - type: mrr_at_10 value: 62.166 - type: mrr_at_100 value: 62.653000000000006 - type: mrr_at_1000 value: 62.699000000000005 - type: mrr_at_3 value: 60.5 - type: mrr_at_5 value: 61.47 - type: ndcg_at_1 value: 56.00000000000001 - type: ndcg_at_10 value: 65.199 - type: ndcg_at_100 value: 67.79899999999999 - type: ndcg_at_1000 value: 69.056 - type: ndcg_at_3 value: 61.814 - type: ndcg_at_5 value: 63.553000000000004 - type: precision_at_1 value: 56.00000000000001 - type: precision_at_10 value: 7.51 - type: precision_at_100 value: 0.878 - type: precision_at_1000 value: 0.098 - type: precision_at_3 value: 21.967 - type: precision_at_5 value: 14.02 - type: recall_at_1 value: 56.00000000000001 - type: recall_at_10 value: 75.1 - type: recall_at_100 value: 87.8 - type: recall_at_1000 value: 97.7 - type: recall_at_3 value: 65.9 - type: recall_at_5 value: 70.1 - task: type: Reranking dataset: type: C-MTEB/Mmarco-reranking name: MTEB MMarcoReranking config: default split: dev revision: None metrics: - type: map value: 32.74158258279793 - type: mrr value: 31.56071428571428 - task: type: Classification dataset: type: C-MTEB/MultilingualSentiment-classification name: MTEB MultilingualSentiment config: default split: validation revision: None metrics: - type: accuracy value: 78.96666666666667 - type: f1 value: 78.82528563818045 - task: type: PairClassification dataset: type: C-MTEB/OCNLI name: MTEB Ocnli config: default split: validation revision: None metrics: - type: cos_sim_accuracy value: 83.54087709799674 - type: cos_sim_ap value: 87.26170197077586 - type: cos_sim_f1 value: 84.7609561752988 - type: cos_sim_precision value: 80.20735155513667 - type: cos_sim_recall value: 89.86272439281943 - type: dot_accuracy value: 72.22523010286952 - type: dot_ap value: 79.51975358187732 - type: dot_f1 value: 76.32183908045977 - type: dot_precision value: 67.58957654723126 - type: dot_recall value: 87.64519535374869 - type: euclidean_accuracy value: 82.0249052517596 - type: euclidean_ap value: 85.32829948726406 - type: euclidean_f1 value: 83.24924318869829 - type: euclidean_precision value: 79.71014492753623 - type: euclidean_recall value: 87.11721224920802 - type: manhattan_accuracy value: 82.13318895506227 - type: manhattan_ap value: 85.28856869288006 - type: manhattan_f1 value: 83.34946757018393 - type: manhattan_precision value: 76.94369973190348 - type: manhattan_recall value: 90.91869060190075 - type: max_accuracy value: 83.54087709799674 - type: max_ap value: 87.26170197077586 - type: max_f1 value: 84.7609561752988 - task: type: Classification dataset: type: C-MTEB/OnlineShopping-classification name: MTEB OnlineShopping config: default split: test revision: None metrics: - type: accuracy value: 94.56 - type: ap value: 92.80848436710805 - type: f1 value: 94.54951966576111 - task: type: STS dataset: type: C-MTEB/PAWSX name: MTEB PAWSX config: default split: test revision: None metrics: - type: cos_sim_pearson value: 39.0866558287863 - type: cos_sim_spearman value: 45.9211126233312 - type: euclidean_pearson value: 44.86568743222145 - type: euclidean_spearman value: 45.63882757207507 - type: manhattan_pearson value: 44.89480036909126 - type: manhattan_spearman value: 45.65929449046206 - task: type: STS dataset: type: C-MTEB/QBQTC name: MTEB QBQTC config: default split: test revision: None metrics: - type: cos_sim_pearson value: 43.04701793979569 - type: cos_sim_spearman value: 44.87491033760315 - type: euclidean_pearson value: 36.2004061032567 - type: euclidean_spearman value: 41.44823909683865 - type: manhattan_pearson value: 36.136113427955095 - type: manhattan_spearman value: 41.39225495993949 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh) config: zh split: test revision: None metrics: - type: cos_sim_pearson value: 61.65611315777857 - type: cos_sim_spearman value: 64.4067673105648 - type: euclidean_pearson value: 61.814977248797184 - type: euclidean_spearman value: 63.99473350700169 - type: manhattan_pearson value: 61.684304629588624 - type: manhattan_spearman value: 63.97831213239316 - task: type: STS dataset: type: C-MTEB/STSB name: MTEB STSB config: default split: test revision: None metrics: - type: cos_sim_pearson value: 76.57324933064379 - type: cos_sim_spearman value: 79.23602286949782 - type: euclidean_pearson value: 80.28226284310948 - type: euclidean_spearman value: 80.32210477608423 - type: manhattan_pearson value: 80.27262188617811 - type: manhattan_spearman value: 80.31619185039723 - task: type: Reranking dataset: type: C-MTEB/T2Reranking name: MTEB T2Reranking config: default split: dev revision: None metrics: - type: map value: 67.05266891356277 - type: mrr value: 77.1906333623497 - task: type: Retrieval dataset: type: C-MTEB/T2Retrieval name: MTEB T2Retrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 28.212 - type: map_at_10 value: 78.932 - type: map_at_100 value: 82.51899999999999 - type: map_at_1000 value: 82.575 - type: map_at_3 value: 55.614 - type: map_at_5 value: 68.304 - type: mrr_at_1 value: 91.211 - type: mrr_at_10 value: 93.589 - type: mrr_at_100 value: 93.659 - type: mrr_at_1000 value: 93.662 - type: mrr_at_3 value: 93.218 - type: mrr_at_5 value: 93.453 - type: ndcg_at_1 value: 91.211 - type: ndcg_at_10 value: 86.24000000000001 - type: ndcg_at_100 value: 89.614 - type: ndcg_at_1000 value: 90.14 - type: ndcg_at_3 value: 87.589 - type: ndcg_at_5 value: 86.265 - type: precision_at_1 value: 91.211 - type: precision_at_10 value: 42.626 - type: precision_at_100 value: 5.043 - type: precision_at_1000 value: 0.517 - type: precision_at_3 value: 76.42 - type: precision_at_5 value: 64.045 - type: recall_at_1 value: 28.212 - type: recall_at_10 value: 85.223 - type: recall_at_100 value: 96.229 - type: recall_at_1000 value: 98.849 - type: recall_at_3 value: 57.30800000000001 - type: recall_at_5 value: 71.661 - task: type: Classification dataset: type: C-MTEB/TNews-classification name: MTEB TNews config: default split: validation revision: None metrics: - type: accuracy value: 54.385000000000005 - type: f1 value: 52.38762400903556 - task: type: Clustering dataset: type: C-MTEB/ThuNewsClusteringP2P name: MTEB ThuNewsClusteringP2P config: default split: test revision: None metrics: - type: v_measure value: 74.55283855942916 - task: type: Clustering dataset: type: C-MTEB/ThuNewsClusteringS2S name: MTEB ThuNewsClusteringS2S config: default split: test revision: None metrics: - type: v_measure value: 68.55115316700493 - task: type: Retrieval dataset: type: C-MTEB/VideoRetrieval name: MTEB VideoRetrieval config: default split: dev revision: None metrics: - type: map_at_1 value: 58.8 - type: map_at_10 value: 69.035 - type: map_at_100 value: 69.52000000000001 - type: map_at_1000 value: 69.529 - type: map_at_3 value: 67.417 - type: map_at_5 value: 68.407 - type: mrr_at_1 value: 58.8 - type: mrr_at_10 value: 69.035 - type: mrr_at_100 value: 69.52000000000001 - type: mrr_at_1000 value: 69.529 - type: mrr_at_3 value: 67.417 - type: mrr_at_5 value: 68.407 - type: ndcg_at_1 value: 58.8 - type: ndcg_at_10 value: 73.395 - type: ndcg_at_100 value: 75.62 - type: ndcg_at_1000 value: 75.90299999999999 - type: ndcg_at_3 value: 70.11800000000001 - type: ndcg_at_5 value: 71.87400000000001 - type: precision_at_1 value: 58.8 - type: precision_at_10 value: 8.68 - type: precision_at_100 value: 0.9690000000000001 - type: precision_at_1000 value: 0.099 - type: precision_at_3 value: 25.967000000000002 - type: precision_at_5 value: 16.42 - type: recall_at_1 value: 58.8 - type: recall_at_10 value: 86.8 - type: recall_at_100 value: 96.89999999999999 - type: recall_at_1000 value: 99.2 - type: recall_at_3 value: 77.9 - type: recall_at_5 value: 82.1 - task: type: Classification dataset: type: C-MTEB/waimai-classification name: MTEB Waimai config: default split: test revision: None metrics: - type: accuracy value: 89.42 - type: ap value: 75.35978503182068 - type: f1 value: 88.01006394348263 --- ## Yinka Yinka embedding 模型是在开源模型[stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d)上续训的，采用了[piccolo2](https://huggingface.co/sensenova/piccolo-large-zh-v2)提到的多任务混合损失(multi-task hybrid loss training)。同样本模型也支持了可变的向量维度。 ## 使用方法 该模型的使用方法同[stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d)一样, 无需任何前缀。 ```python from sentence_transformers import SentenceTransformer from sklearn.preprocessing import normalize model = SentenceTransformer("Classical/Yinka") # 注意先不要normalize! 选取前n维后再normalize vectors = model.encode(["text1", "text2"], normalize_embeddings=False) print(vectors.shape) # shape is [2,1792] n_dims = 768 cut_vecs = normalize(vectors[:, :n_dims]) ``` ## 结果 | Model Name | Model Size (GB) | Dimension | Sequence Length | Classification (9) | Clustering (4) | Pair Classification (2) | Reranking (4) | Retrieval (8) | STS (8) | Average (35) | |:----:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | [Yinka](https://huggingface.co/Classical/Yinka) | 1.21 | 1792 | 512 | 74.30 | 61.99 | 89.87 | 69.77 | 74.40 | 63.30 | 70.79 | | [stella-v3.5-mrl](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d) |1.21 | 1792 | 512 | 71.56 | 54.39 | 88.09 | 68.45 | 73.51 | 62.48 | 68.56 | | [piccolo-large-zh-v2](https://huggingface.co/sensenova/piccolo-large-zh-v2) | 1.21 | 1792 | 512 | 74.59 | 62.17 | 90.24 | 70 | 74.36 | 63.5 | 70.95 | ## 训练细节 TODO ## Licence 本模型采用MIT licence.