cristuf/bge-base-financial-matryoshka

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cristuf/bge-base-financial-matryoshka")
# Run inference
sentences = [
    'During fiscal 2022, GameStop Corp increased its valuation allowances by approximately $70.2 million in various jurisdictions.',
    "How much did GameStop Corp's valuation allowances increase during fiscal 2022?",
    'How does Gilead ensure an inclusive and diverse workforce?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7186
cosine_accuracy@3	0.83
cosine_accuracy@5	0.8714
cosine_accuracy@10	0.91
cosine_precision@1	0.7186
cosine_precision@3	0.2767
cosine_precision@5	0.1743
cosine_precision@10	0.091
cosine_recall@1	0.7186
cosine_recall@3	0.83
cosine_recall@5	0.8714
cosine_recall@10	0.91
cosine_ndcg@10	0.8138
cosine_mrr@10	0.783
cosine_map@100	0.7867

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7114
cosine_accuracy@3	0.8314
cosine_accuracy@5	0.8729
cosine_accuracy@10	0.9143
cosine_precision@1	0.7114
cosine_precision@3	0.2771
cosine_precision@5	0.1746
cosine_precision@10	0.0914
cosine_recall@1	0.7114
cosine_recall@3	0.8314
cosine_recall@5	0.8729
cosine_recall@10	0.9143
cosine_ndcg@10	0.8124
cosine_mrr@10	0.7799
cosine_map@100	0.7832

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7
cosine_accuracy@3	0.8286
cosine_accuracy@5	0.8614
cosine_accuracy@10	0.9043
cosine_precision@1	0.7
cosine_precision@3	0.2762
cosine_precision@5	0.1723
cosine_precision@10	0.0904
cosine_recall@1	0.7
cosine_recall@3	0.8286
cosine_recall@5	0.8614
cosine_recall@10	0.9043
cosine_ndcg@10	0.8043
cosine_mrr@10	0.7722
cosine_map@100	0.7759

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.6857
cosine_accuracy@3	0.8071
cosine_accuracy@5	0.8571
cosine_accuracy@10	0.8971
cosine_precision@1	0.6857
cosine_precision@3	0.269
cosine_precision@5	0.1714
cosine_precision@10	0.0897
cosine_recall@1	0.6857
cosine_recall@3	0.8071
cosine_recall@5	0.8571
cosine_recall@10	0.8971
cosine_ndcg@10	0.7909
cosine_mrr@10	0.7569
cosine_map@100	0.7609

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.66
cosine_accuracy@3	0.7757
cosine_accuracy@5	0.8129
cosine_accuracy@10	0.8671
cosine_precision@1	0.66
cosine_precision@3	0.2586
cosine_precision@5	0.1626
cosine_precision@10	0.0867
cosine_recall@1	0.66
cosine_recall@3	0.7757
cosine_recall@5	0.8129
cosine_recall@10	0.8671
cosine_ndcg@10	0.7616
cosine_mrr@10	0.7281
cosine_map@100	0.7331

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,300 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 8 tokens mean: 46.36 tokens max: 439 tokens	min: 9 tokens mean: 20.41 tokens max: 51 tokens

• Samples:

  positive	anchor
  Japan's revenue for the year 2023 reached 2,367.0 million.	What was the revenue attributed to Japan in the year 2023?
  Our four reportable segments are: •the Data Center segment, which primarily includes server CPUs, GPUs, APUs, DPUs, FPGAs, SmartNICs, AI accelerators and Adaptive SoC products for data centers; •the Client segment, which primarily includes CPUs, APUs, and chipsets for desktop, notebook and handheld personal computers; •the Gaming segment, which primarily includes discrete GPUs, semi-custom SoC products and development services; and •the Embedded segment, which primarily includes embedded CPUs, GPUs, APUs, FPGAs, SOMs, and Adaptive SoC products.	What are the different segments that AMD reports financially?
  For detailed information about the company's legal proceedings, see Note 4 to the consolidated financial statements, included under the caption 'Contingencies' in the Annual Report on Form 10-K.	Where can detailed information about the company's legal proceedings be found in its financial statements?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.8122	10	1.5267	-	-	-	-	-
0.9746	12	-	0.7446	0.7639	0.7765	0.7039	0.7725
1.6244	20	0.6742	-	-	-	-	-
1.9492	24	-	0.7606	0.7795	0.7828	0.7297	0.7839
2.4365	30	0.4469	-	-	-	-	-
2.9239	36	-	0.7643	0.7758	0.7834	0.7332	0.7845
3.2487	40	0.3712	-	-	-	-	-
3.8985	48	-	0.7609	0.7759	0.7832	0.7331	0.7867

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.8
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.3.1+cu121
• Accelerate: 0.30.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}