--- base_model: BAAI/bge-base-en-v1.5 datasets: [] language: - en library_name: sentence-transformers license: apache-2.0 metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:500 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: - source_sentence: 'Non-context LLM: Prompt LLM directly with True or False? without additional context. Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context. Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction. Retrieval→LLM + NP: Ensemble of two methods. Some interesting observations on model hallucination behavior: Error rates are higher for rarer entities in the task of biography generation. Error rates are higher for facts mentioned later in the generation. Using retrieval to ground the model generation significantly helps reduce hallucination.' sentences: - What is the impact of infrequent entities on the efficacy of language models in the context of biography generation? - In what ways does FActScore enhance the assessment of factual accuracy in long-form content generation when compared to conventional evaluation techniques? - What approaches does SelfCheckGPT implement when faced with questions it cannot answer, and how does this influence its overall reliability in delivering accurate information? - source_sentence: 'Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$. (1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$. (2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$. (3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$. Fig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022) When evaluating the revised text $y$, both attribution and preservation metrics matter.' sentences: - What impact does adjusting the sampling temperature have on the calibration of large language models, and how does this influence the uncertainty of their outputs? - How do unanswerable questions differ from answerable ones in the context of a language model's understanding of its own capabilities? - In what ways does the agreement model evaluate discrepancies between the provided evidence and the updated text, and how does this evaluation impact the reliability of AI-generated content modifications? - source_sentence: 'Non-context LLM: Prompt LLM directly with True or False? without additional context. Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context. Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction. Retrieval→LLM + NP: Ensemble of two methods. Some interesting observations on model hallucination behavior: Error rates are higher for rarer entities in the task of biography generation. Error rates are higher for facts mentioned later in the generation. Using retrieval to ground the model generation significantly helps reduce hallucination.' sentences: - In what ways can the acknowledgment of uncertainty by large language models (LLMs) contribute to the mitigation of hallucinations and enhance the overall factual accuracy of generated content? - In what ways does the process of retrieving related passages contribute to minimizing hallucinations in the outputs generated by language models, and how does this approach differ from the application of nonparametric probability methods? - How does the triplet structure $(c, y, y^*)$ play a crucial role in the categorization of errors, and in what ways does it enhance the training process of the editor model? - source_sentence: 'Fine-tuning New Knowledge# Fine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common technique for improving certain capabilities of the model like instruction following. Introducing new knowledge at the fine-tuning stage is hard to avoid. Fine-tuning usually consumes much less compute, making it debatable whether the model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge encourages hallucinations. They found that (1) LLMs learn fine-tuning examples with new knowledge slower than other examples with knowledge consistent with the pre-existing knowledge of the model; (2) Once the examples with new knowledge are eventually learned, they increase the model’s tendency to hallucinate.' sentences: - How do the intentionally designed questions in TruthfulQA highlight prevalent misunderstandings regarding AI responses in the healthcare domain? - What effect does the slower acquisition of new knowledge compared to established knowledge have on the effectiveness of large language models in practical scenarios? - How do the RARR methodology and the FAVA model compare in their approaches to mitigating hallucination errors in generated outputs, and what key distinctions can be identified between the two? - source_sentence: 'Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$. (1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$. (2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$. (3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$. Fig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022) When evaluating the revised text $y$, both attribution and preservation metrics matter.' sentences: - What mechanisms does the editing algorithm employ to maintain fidelity to the source material while simultaneously ensuring alignment with the supporting evidence? - What is the impact of constraining the dataset to a maximum of $M=5$ instances on the accuracy and reliability of the attribution report $A$ when analyzing AI-generated content? - In what ways does the implementation of a query generation model enhance the research phase when it comes to validating the accuracy of information? model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.8802083333333334 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.96875 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9895833333333334 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8802083333333334 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3229166666666667 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19791666666666666 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8802083333333334 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.96875 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9895833333333334 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9477255159324969 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9301711309523809 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.930171130952381 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.875 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.96875 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9947916666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.875 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3229166666666667 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19895833333333335 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.875 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.96875 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9947916666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9459628876705072 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9277405753968253 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9277405753968253 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.8802083333333334 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.96875 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9947916666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8802083333333334 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3229166666666667 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19895833333333335 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8802083333333334 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.96875 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9947916666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9458393511377685 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9277405753968254 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9277405753968253 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.8697916666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9895833333333334 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9947916666666666 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8697916666666666 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19791666666666666 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09947916666666667 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8697916666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9895833333333334 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9947916666666666 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9440191417149189 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9265252976190478 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.92687251984127 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.8541666666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9947916666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9947916666666666 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8541666666666666 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19895833333333335 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09947916666666667 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8541666666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9947916666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9947916666666666 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9380774892768095 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9184027777777778 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9186111111111112 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-500") # Run inference sentences = [ 'Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$.\n\n(1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \\to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$.\n(2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \\to \\text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$.\n(3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$.\n\n\n\n\nFig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022)\nWhen evaluating the revised text $y$, both attribution and preservation metrics matter.', 'What mechanisms does the editing algorithm employ to maintain fidelity to the source material while simultaneously ensuring alignment with the supporting evidence?', 'What is the impact of constraining the dataset to a maximum of $M=5$ instances on the accuracy and reliability of the attribution report $A$ when analyzing AI-generated content?', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8802 | | cosine_accuracy@3 | 0.9688 | | cosine_accuracy@5 | 0.9896 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.8802 | | cosine_precision@3 | 0.3229 | | cosine_precision@5 | 0.1979 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.8802 | | cosine_recall@3 | 0.9688 | | cosine_recall@5 | 0.9896 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9477 | | cosine_mrr@10 | 0.9302 | | **cosine_map@100** | **0.9302** | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.875 | | cosine_accuracy@3 | 0.9688 | | cosine_accuracy@5 | 0.9948 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.875 | | cosine_precision@3 | 0.3229 | | cosine_precision@5 | 0.199 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.875 | | cosine_recall@3 | 0.9688 | | cosine_recall@5 | 0.9948 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.946 | | cosine_mrr@10 | 0.9277 | | **cosine_map@100** | **0.9277** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8802 | | cosine_accuracy@3 | 0.9688 | | cosine_accuracy@5 | 0.9948 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.8802 | | cosine_precision@3 | 0.3229 | | cosine_precision@5 | 0.199 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.8802 | | cosine_recall@3 | 0.9688 | | cosine_recall@5 | 0.9948 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9458 | | cosine_mrr@10 | 0.9277 | | **cosine_map@100** | **0.9277** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8698 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 0.9896 | | cosine_accuracy@10 | 0.9948 | | cosine_precision@1 | 0.8698 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.1979 | | cosine_precision@10 | 0.0995 | | cosine_recall@1 | 0.8698 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 0.9896 | | cosine_recall@10 | 0.9948 | | cosine_ndcg@10 | 0.944 | | cosine_mrr@10 | 0.9265 | | **cosine_map@100** | **0.9269** | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8542 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 0.9948 | | cosine_accuracy@10 | 0.9948 | | cosine_precision@1 | 0.8542 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.199 | | cosine_precision@10 | 0.0995 | | cosine_recall@1 | 0.8542 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 0.9948 | | cosine_recall@10 | 0.9948 | | cosine_ndcg@10 | 0.9381 | | cosine_mrr@10 | 0.9184 | | **cosine_map@100** | **0.9186** | ## Training Details ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_eval_batch_size`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 5 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `load_best_model_at_end`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 8 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 5 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 | |:-------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:| | 0.0794 | 5 | 5.4149 | - | - | - | - | - | | 0.1587 | 10 | 4.8587 | - | - | - | - | - | | 0.2381 | 15 | 3.9711 | - | - | - | - | - | | 0.3175 | 20 | 3.4853 | - | - | - | - | - | | 0.3968 | 25 | 3.6227 | - | - | - | - | - | | 0.4762 | 30 | 3.3359 | - | - | - | - | - | | 0.5556 | 35 | 2.0868 | - | - | - | - | - | | 0.6349 | 40 | 2.256 | - | - | - | - | - | | 0.7143 | 45 | 2.2958 | - | - | - | - | - | | 0.7937 | 50 | 1.7128 | - | - | - | - | - | | 0.8730 | 55 | 2.029 | - | - | - | - | - | | 0.9524 | 60 | 1.9104 | - | - | - | - | - | | 1.0 | 63 | - | 0.8950 | 0.9042 | 0.9039 | 0.8640 | 0.8989 | | 1.0317 | 65 | 2.5929 | - | - | - | - | - | | 1.1111 | 70 | 1.4257 | - | - | - | - | - | | 1.1905 | 75 | 1.9956 | - | - | - | - | - | | 1.2698 | 80 | 1.5845 | - | - | - | - | - | | 1.3492 | 85 | 1.7383 | - | - | - | - | - | | 1.4286 | 90 | 1.4657 | - | - | - | - | - | | 1.5079 | 95 | 1.8461 | - | - | - | - | - | | 1.5873 | 100 | 1.8531 | - | - | - | - | - | | 1.6667 | 105 | 1.6504 | - | - | - | - | - | | 1.7460 | 110 | 2.7636 | - | - | - | - | - | | 1.8254 | 115 | 0.7195 | - | - | - | - | - | | 1.9048 | 120 | 1.2494 | - | - | - | - | - | | 1.9841 | 125 | 1.7331 | - | - | - | - | - | | 2.0 | 126 | - | 0.9170 | 0.9340 | 0.9167 | 0.9013 | 0.9179 | | 2.0635 | 130 | 1.1102 | - | - | - | - | - | | 2.1429 | 135 | 1.8586 | - | - | - | - | - | | 2.2222 | 140 | 1.4211 | - | - | - | - | - | | 2.3016 | 145 | 1.9531 | - | - | - | - | - | | 2.3810 | 150 | 1.9516 | - | - | - | - | - | | 2.4603 | 155 | 2.1174 | - | - | - | - | - | | 2.5397 | 160 | 1.7883 | - | - | - | - | - | | 2.6190 | 165 | 1.4537 | - | - | - | - | - | | 2.6984 | 170 | 1.3927 | - | - | - | - | - | | 2.7778 | 175 | 1.2559 | - | - | - | - | - | | 2.8571 | 180 | 1.8748 | - | - | - | - | - | | 2.9365 | 185 | 0.7509 | - | - | - | - | - | | 3.0 | 189 | - | 0.9312 | 0.9244 | 0.9241 | 0.9199 | 0.9349 | | 3.0159 | 190 | 0.947 | - | - | - | - | - | | 3.0952 | 195 | 1.9463 | - | - | - | - | - | | 3.1746 | 200 | 1.2077 | - | - | - | - | - | | 3.2540 | 205 | 0.7721 | - | - | - | - | - | | 3.3333 | 210 | 1.5633 | - | - | - | - | - | | 3.4127 | 215 | 1.5042 | - | - | - | - | - | | 3.4921 | 220 | 1.1531 | - | - | - | - | - | | 3.5714 | 225 | 1.2408 | - | - | - | - | - | | 3.6508 | 230 | 0.8085 | - | - | - | - | - | | 3.7302 | 235 | 1.1195 | - | - | - | - | - | | 3.8095 | 240 | 1.1843 | - | - | - | - | - | | 3.8889 | 245 | 0.7176 | - | - | - | - | - | | 3.9683 | 250 | 1.1715 | - | - | - | - | - | | 4.0 | 252 | - | 0.9244 | 0.9287 | 0.9251 | 0.9199 | 0.9300 | | 4.0476 | 255 | 1.3187 | - | - | - | - | - | | 4.1270 | 260 | 0.2891 | - | - | - | - | - | | 4.2063 | 265 | 1.5887 | - | - | - | - | - | | 4.2857 | 270 | 1.1227 | - | - | - | - | - | | 4.3651 | 275 | 1.5385 | - | - | - | - | - | | 4.4444 | 280 | 0.4732 | - | - | - | - | - | | 4.5238 | 285 | 1.2039 | - | - | - | - | - | | 4.6032 | 290 | 1.0755 | - | - | - | - | - | | 4.6825 | 295 | 1.5345 | - | - | - | - | - | | 4.7619 | 300 | 1.4255 | - | - | - | - | - | | 4.8413 | 305 | 1.7436 | - | - | - | - | - | | 4.9206 | 310 | 0.9408 | - | - | - | - | - | | **5.0** | **315** | **0.7724** | **0.9269** | **0.9277** | **0.9277** | **0.9186** | **0.9302** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.0.1 - Transformers: 4.42.4 - PyTorch: 2.3.1+cu121 - Accelerate: 0.32.1 - Datasets: 2.21.0 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```