joshuapb/fine-tuned-matryoshka-200

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-200")
# Run inference
sentences = [
    'Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)\nSome interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.\n\nUnknown examples are fitted substantially slower than Known.\nThe best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.\nAmong Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.',
    'In what ways does the fitting speed of examples that are not previously encountered differ from that of familiar examples, and how does this variation influence the overall accuracy of the model on the development set?',
    'How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8802
cosine_accuracy@3	0.9688
cosine_accuracy@5	0.9844
cosine_accuracy@10	0.9948
cosine_precision@1	0.8802
cosine_precision@3	0.3229
cosine_precision@5	0.1969
cosine_precision@10	0.0995
cosine_recall@1	0.8802
cosine_recall@3	0.9688
cosine_recall@5	0.9844
cosine_recall@10	0.9948
cosine_ndcg@10	0.9433
cosine_mrr@10	0.9261
cosine_map@100	0.9264

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8698
cosine_accuracy@3	0.974
cosine_accuracy@5	0.974
cosine_accuracy@10	0.9948
cosine_precision@1	0.8698
cosine_precision@3	0.3247
cosine_precision@5	0.1948
cosine_precision@10	0.0995
cosine_recall@1	0.8698
cosine_recall@3	0.974
cosine_recall@5	0.974
cosine_recall@10	0.9948
cosine_ndcg@10	0.94
cosine_mrr@10	0.9216
cosine_map@100	0.9221

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8698
cosine_accuracy@3	0.974
cosine_accuracy@5	0.9844
cosine_accuracy@10	1.0
cosine_precision@1	0.8698
cosine_precision@3	0.3247
cosine_precision@5	0.1969
cosine_precision@10	0.1
cosine_recall@1	0.8698
cosine_recall@3	0.974
cosine_recall@5	0.9844
cosine_recall@10	1.0
cosine_ndcg@10	0.942
cosine_mrr@10	0.9227
cosine_map@100	0.9227

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8542
cosine_accuracy@3	0.9583
cosine_accuracy@5	0.9688
cosine_accuracy@10	0.9948
cosine_precision@1	0.8542
cosine_precision@3	0.3194
cosine_precision@5	0.1937
cosine_precision@10	0.0995
cosine_recall@1	0.8542
cosine_recall@3	0.9583
cosine_recall@5	0.9688
cosine_recall@10	0.9948
cosine_ndcg@10	0.9306
cosine_mrr@10	0.9094
cosine_map@100	0.9099

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7917
cosine_accuracy@3	0.9531
cosine_accuracy@5	0.974
cosine_accuracy@10	0.9896
cosine_precision@1	0.7917
cosine_precision@3	0.3177
cosine_precision@5	0.1948
cosine_precision@10	0.099
cosine_recall@1	0.7917
cosine_recall@3	0.9531
cosine_recall@5	0.974
cosine_recall@10	0.9896
cosine_ndcg@10	0.9004
cosine_mrr@10	0.8706
cosine_map@100	0.8713

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.2	5	5.2225	-	-	-	-	-
0.4	10	4.956	-	-	-	-	-
0.6	15	3.6388	-	-	-	-	-
0.8	20	3.1957	-	-	-	-	-
1.0	25	2.6928	0.8661	0.8770	0.8754	0.8312	0.8871
1.2	30	2.5565	-	-	-	-	-
1.4	35	1.5885	-	-	-	-	-
1.6	40	2.1406	-	-	-	-	-
1.8	45	2.193	-	-	-	-	-
2.0	50	1.326	0.8944	0.9110	0.9028	0.8615	0.9037
2.2	55	2.6832	-	-	-	-	-
2.4	60	1.0584	-	-	-	-	-
2.6	65	0.8853	-	-	-	-	-
2.8	70	1.7129	-	-	-	-	-
3.0	75	2.1856	0.9106	0.9293	0.9075	0.8778	0.9266
3.2	80	1.7658	-	-	-	-	-
3.4	85	1.9783	-	-	-	-	-
3.6	90	1.9583	-	-	-	-	-
3.8	95	1.2396	-	-	-	-	-
4.0	100	1.1901	0.9073	0.9253	0.9151	0.8750	0.9312
4.2	105	2.6547	-	-	-	-	-
4.4	110	1.3485	-	-	-	-	-
4.6	115	1.0767	-	-	-	-	-
4.8	120	0.6663	-	-	-	-	-
5.0	125	1.3869	0.9099	0.9227	0.9221	0.8713	0.9264

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.4
• PyTorch: 2.3.1+cu121
• Accelerate: 0.32.1
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}