metadata
base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:6300
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: >-
Total net additions to property and equipment for AWS in 2023 amounted to
$24,843 million.
sentences:
- >-
What technological feature helps protect digital transactions in the
Visa Token Service?
- >-
What was the total net addition to property and equipment for AWS in the
year 2023?
- >-
By what proportion did net cash used in financing activities increase
from 2022 to 2023?
- source_sentence: >-
Leases generally contain one or more of the following options, which the
Company can exercise at the end of the initial term: (a) renew the lease
for a defined number of years at the then-fair market rental rate or rate
stipulated in the lease agreement; (b) purchase the property at the
then-fair market value or purchase price stated in the agreement; or (c) a
right of first refusal in the event of a third-party offer.
sentences:
- >-
What are the requirements for health insurers and group health plans in
providing cost estimates to consumers?
- >-
What options does the company have at the end of the lease term for
their leased properties?
- >-
How much did the company incur in intangible amortization costs related
to the eOne acquisition in 2022?
- source_sentence: >-
We recorded an acquisition termination cost of $1.35 billion in fiscal
year 2023 reflecting the write-off of the prepayment provided at signing.
sentences:
- >-
How much did NVIDIA record as an acquisition termination cost in fiscal
year 2023 related to the Arm Share Purchase Agreement?
- >-
What is included in the consolidated financial statements and
accompanying notes mentioned in Part IV, Item 15(a)(1) of the Annual
Report on Form 10-K?
- >-
What risks are associated with projecting the effectiveness of internal
controls into future periods as mentioned?
- source_sentence: Item 8 is labeled as Financial Statements and Supplementary Data.
sentences:
- >-
What was the percentage of trading days in 2023 where trading-related
revenue was recorded as positive?
- >-
How is the discount rate for the Family Dollar goodwill impairment
evaluation determined?
- What is the title of Item 8 in the financial document?
- source_sentence: >-
Details about legal proceedings are included in Part II, Item 8,
"Financial Statements and Supplementary Data" of the Annual Report on Form
10-K, under the caption "Legal Proceedings".
sentences:
- >-
Where can details about legal proceedings be located in an Annual Report
on Form 10-K?
- >-
How many stores did AutoZone operate in the United States as of August
26, 2023?
- >-
In the context of Hewlett Packard Enterprise's recent financial
discussions, what factors are expected to impact their operational costs
and revenue growth moving forward?
model-index:
- name: BGE base Financial Matryoshka
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 768
type: dim_768
metrics:
- type: cosine_accuracy@1
value: 0.7071428571428572
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.8414285714285714
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.88
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.9314285714285714
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.7071428571428572
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.28047619047619043
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.176
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09314285714285712
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.7071428571428572
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.8414285714285714
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.88
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.9314285714285714
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.8207437059171859
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.7853486394557823
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.7881907906804949
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 512
type: dim_512
metrics:
- type: cosine_accuracy@1
value: 0.6957142857142857
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.8385714285714285
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.8757142857142857
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.93
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.6957142857142857
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.2795238095238095
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.17514285714285713
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09299999999999999
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.6957142857142857
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.8385714285714285
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.8757142857142857
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.93
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.8149439460863356
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.7780714285714285
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.781021025356189
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy@1
value: 0.6885714285714286
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.83
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.8742857142857143
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.9142857142857143
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.6885714285714286
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.27666666666666667
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.17485714285714282
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09142857142857141
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.6885714285714286
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.83
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.8742857142857143
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.9142857142857143
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.8060991379418679
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.7710873015873015
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.7751792513774886
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy@1
value: 0.6771428571428572
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.8214285714285714
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.8614285714285714
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.9142857142857143
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.6771428571428572
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.2738095238095238
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.17228571428571426
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.09142857142857141
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.6771428571428572
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.8214285714285714
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.8614285714285714
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.9142857142857143
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.7979494993398927
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.7605890022675734
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.7639633810343436
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy@1
value: 0.6557142857142857
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.7871428571428571
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.8271428571428572
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.8714285714285714
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.6557142857142857
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.2623809523809524
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.1654285714285714
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.08714285714285713
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.6557142857142857
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.7871428571428571
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.8271428571428572
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.8714285714285714
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.7664083634078753
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.7326604308390022
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.7375736792740525
name: Cosine Map@100
BGE base Financial Matryoshka
This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: BAAI/bge-base-en-v1.5
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
- Language: en
- License: apache-2.0
Model Sources
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("dustyatx/bge-base-financial-matryoshka")
sentences = [
'Details about legal proceedings are included in Part II, Item 8, "Financial Statements and Supplementary Data" of the Annual Report on Form 10-K, under the caption "Legal Proceedings".',
'Where can details about legal proceedings be located in an Annual Report on Form 10-K?',
'How many stores did AutoZone operate in the United States as of August 26, 2023?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
Evaluation
Metrics
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.7071 |
cosine_accuracy@3 |
0.8414 |
cosine_accuracy@5 |
0.88 |
cosine_accuracy@10 |
0.9314 |
cosine_precision@1 |
0.7071 |
cosine_precision@3 |
0.2805 |
cosine_precision@5 |
0.176 |
cosine_precision@10 |
0.0931 |
cosine_recall@1 |
0.7071 |
cosine_recall@3 |
0.8414 |
cosine_recall@5 |
0.88 |
cosine_recall@10 |
0.9314 |
cosine_ndcg@10 |
0.8207 |
cosine_mrr@10 |
0.7853 |
cosine_map@100 |
0.7882 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.6957 |
cosine_accuracy@3 |
0.8386 |
cosine_accuracy@5 |
0.8757 |
cosine_accuracy@10 |
0.93 |
cosine_precision@1 |
0.6957 |
cosine_precision@3 |
0.2795 |
cosine_precision@5 |
0.1751 |
cosine_precision@10 |
0.093 |
cosine_recall@1 |
0.6957 |
cosine_recall@3 |
0.8386 |
cosine_recall@5 |
0.8757 |
cosine_recall@10 |
0.93 |
cosine_ndcg@10 |
0.8149 |
cosine_mrr@10 |
0.7781 |
cosine_map@100 |
0.781 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.6886 |
cosine_accuracy@3 |
0.83 |
cosine_accuracy@5 |
0.8743 |
cosine_accuracy@10 |
0.9143 |
cosine_precision@1 |
0.6886 |
cosine_precision@3 |
0.2767 |
cosine_precision@5 |
0.1749 |
cosine_precision@10 |
0.0914 |
cosine_recall@1 |
0.6886 |
cosine_recall@3 |
0.83 |
cosine_recall@5 |
0.8743 |
cosine_recall@10 |
0.9143 |
cosine_ndcg@10 |
0.8061 |
cosine_mrr@10 |
0.7711 |
cosine_map@100 |
0.7752 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.6771 |
cosine_accuracy@3 |
0.8214 |
cosine_accuracy@5 |
0.8614 |
cosine_accuracy@10 |
0.9143 |
cosine_precision@1 |
0.6771 |
cosine_precision@3 |
0.2738 |
cosine_precision@5 |
0.1723 |
cosine_precision@10 |
0.0914 |
cosine_recall@1 |
0.6771 |
cosine_recall@3 |
0.8214 |
cosine_recall@5 |
0.8614 |
cosine_recall@10 |
0.9143 |
cosine_ndcg@10 |
0.7979 |
cosine_mrr@10 |
0.7606 |
cosine_map@100 |
0.764 |
Information Retrieval
Metric |
Value |
cosine_accuracy@1 |
0.6557 |
cosine_accuracy@3 |
0.7871 |
cosine_accuracy@5 |
0.8271 |
cosine_accuracy@10 |
0.8714 |
cosine_precision@1 |
0.6557 |
cosine_precision@3 |
0.2624 |
cosine_precision@5 |
0.1654 |
cosine_precision@10 |
0.0871 |
cosine_recall@1 |
0.6557 |
cosine_recall@3 |
0.7871 |
cosine_recall@5 |
0.8271 |
cosine_recall@10 |
0.8714 |
cosine_ndcg@10 |
0.7664 |
cosine_mrr@10 |
0.7327 |
cosine_map@100 |
0.7376 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 6,300 training samples
- Columns:
positive
and anchor
- Approximate statistics based on the first 1000 samples:
|
positive |
anchor |
type |
string |
string |
details |
- min: 8 tokens
- mean: 45.94 tokens
- max: 512 tokens
|
- min: 7 tokens
- mean: 20.7 tokens
- max: 42 tokens
|
- Samples:
positive |
anchor |
The company must continuously strengthen its capabilities in marketing and innovation to compete in a digital environment and maintain brand loyalty and marketallability. In addition, it is increasing its investments in e-commerce to support retail and meal delivery services, offering more package sizes that are fit-for-purpose for online sales and shifting more consumer and trade promotions to digital. |
What strategies is the company employing to enhance its competitiveness in a digital environment? |
Fedflowing expanded or relocated its hub and linehaul network, FedEx Ground also introduced new safety technologies, set new driver standards, and made operational enhancements for safer handling of heavy items. |
What specific changes has FedEx Ground made for vehicle and driver safety? |
The debt financing, which is being provided by a syndicate of Chinese financial institutions, contains certain covenants and a maximum borrowing limit of ¥29.7 billion RMB (approximately $4.2 billion). |
What is the maximum borrowing limit of the debt financing provided by the syndicate of Chinese financial institutions for Universal Beijing Resort? |
- Loss:
MatryoshkaLoss
with these parameters:{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: epoch
per_device_train_batch_size
: 32
per_device_eval_batch_size
: 16
gradient_accumulation_steps
: 16
learning_rate
: 2e-05
num_train_epochs
: 4
lr_scheduler_type
: cosine
warmup_ratio
: 0.1
bf16
: True
tf32
: True
load_best_model_at_end
: True
optim
: adamw_torch_fused
batch_sampler
: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir
: False
do_predict
: False
eval_strategy
: epoch
prediction_loss_only
: True
per_device_train_batch_size
: 32
per_device_eval_batch_size
: 16
per_gpu_train_batch_size
: None
per_gpu_eval_batch_size
: None
gradient_accumulation_steps
: 16
eval_accumulation_steps
: None
torch_empty_cache_steps
: None
learning_rate
: 2e-05
weight_decay
: 0.0
adam_beta1
: 0.9
adam_beta2
: 0.999
adam_epsilon
: 1e-08
max_grad_norm
: 1.0
num_train_epochs
: 4
max_steps
: -1
lr_scheduler_type
: cosine
lr_scheduler_kwargs
: {}
warmup_ratio
: 0.1
warmup_steps
: 0
log_level
: passive
log_level_replica
: warning
log_on_each_node
: True
logging_nan_inf_filter
: True
save_safetensors
: True
save_on_each_node
: False
save_only_model
: False
restore_callback_states_from_checkpoint
: False
no_cuda
: False
use_cpu
: False
use_mps_device
: False
seed
: 42
data_seed
: None
jit_mode_eval
: False
use_ipex
: False
bf16
: True
fp16
: False
fp16_opt_level
: O1
half_precision_backend
: auto
bf16_full_eval
: False
fp16_full_eval
: False
tf32
: True
local_rank
: 0
ddp_backend
: None
tpu_num_cores
: None
tpu_metrics_debug
: False
debug
: []
dataloader_drop_last
: False
dataloader_num_workers
: 0
dataloader_prefetch_factor
: None
past_index
: -1
disable_tqdm
: False
remove_unused_columns
: True
label_names
: None
load_best_model_at_end
: True
ignore_data_skip
: False
fsdp
: []
fsdp_min_num_params
: 0
fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap
: None
accelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed
: None
label_smoothing_factor
: 0.0
optim
: adamw_torch_fused
optim_args
: None
adafactor
: False
group_by_length
: False
length_column_name
: length
ddp_find_unused_parameters
: None
ddp_bucket_cap_mb
: None
ddp_broadcast_buffers
: False
dataloader_pin_memory
: True
dataloader_persistent_workers
: False
skip_memory_metrics
: True
use_legacy_prediction_loop
: False
push_to_hub
: False
resume_from_checkpoint
: None
hub_model_id
: None
hub_strategy
: every_save
hub_private_repo
: False
hub_always_push
: False
gradient_checkpointing
: False
gradient_checkpointing_kwargs
: None
include_inputs_for_metrics
: False
eval_do_concat_batches
: True
fp16_backend
: auto
push_to_hub_model_id
: None
push_to_hub_organization
: None
mp_parameters
:
auto_find_batch_size
: False
full_determinism
: False
torchdynamo
: None
ray_scope
: last
ddp_timeout
: 1800
torch_compile
: False
torch_compile_backend
: None
torch_compile_mode
: None
dispatch_batches
: None
split_batches
: None
include_tokens_per_second
: False
include_num_input_tokens_seen
: False
neftune_noise_alpha
: None
optim_target_modules
: None
batch_eval_metrics
: False
eval_on_start
: False
eval_use_gather_object
: False
batch_sampler
: no_duplicates
multi_dataset_batch_sampler
: proportional
Training Logs
Epoch |
Step |
Training Loss |
dim_128_cosine_map@100 |
dim_256_cosine_map@100 |
dim_512_cosine_map@100 |
dim_64_cosine_map@100 |
dim_768_cosine_map@100 |
0.8122 |
10 |
1.5212 |
- |
- |
- |
- |
- |
0.9746 |
12 |
- |
0.7439 |
0.7556 |
0.7670 |
0.7142 |
0.7717 |
1.6244 |
20 |
0.6418 |
- |
- |
- |
- |
- |
1.9492 |
24 |
- |
0.7592 |
0.7743 |
0.7787 |
0.7331 |
0.7839 |
2.4365 |
30 |
0.4411 |
- |
- |
- |
- |
- |
2.9239 |
36 |
- |
0.7623 |
0.7757 |
0.7816 |
0.7365 |
0.7902 |
3.2487 |
40 |
0.3917 |
- |
- |
- |
- |
- |
3.8985 |
48 |
- |
0.764 |
0.7752 |
0.781 |
0.7376 |
0.7882 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.11.9
- Sentence Transformers: 3.0.1
- Transformers: 4.44.2
- PyTorch: 2.4.0+cu121
- Accelerate: 0.33.0
- Datasets: 2.21.0
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}