cferreiragonz/bge-base-fastdds-questions-30-epochs

Fine tuning poc1-30e

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cferreiragonz/bge-base-fastdds-questions-30-epochs")
# Run inference
sentences = [
    'After calling the "DataReader::read()" or "DataReader::take()"\noperations, accessing the data on the returned sequences is quite\neasy. The sequences API provides a **length()** operation returning\nthe number of elements in the collections. The application code just\nneeds to check this value and use the **[]** operator to access the\ncorresponding elements. Elements on the DDS data sequence should only\nbe accessed when the corresponding element on the SampleInfo sequence\nindicate that valid data is present. When using Data Sharing, it is\nalso important to check that the sample is valid (i.e, not replaced,\nrefer to DataReader and DataWriter history coupling for further\ninformation in this regard).',
    'What is the primary method described in the text for accessing data on returned sequences after calling "DataReader::read()" or "DataReader::take()" operations?',
    'What are the steps to install Fast DDS library, Python bindings, and Gen generation tool from sources in a Linux environment?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3427
cosine_accuracy@3	0.5291
cosine_accuracy@5	0.5758
cosine_accuracy@10	0.6643
cosine_precision@1	0.3427
cosine_precision@3	0.1764
cosine_precision@5	0.1152
cosine_precision@10	0.0664
cosine_recall@1	0.3427
cosine_recall@3	0.5291
cosine_recall@5	0.5758
cosine_recall@10	0.6643
cosine_ndcg@10	0.4999
cosine_mrr@10	0.4478
cosine_map@100	0.4573

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3333
cosine_accuracy@3	0.5315
cosine_accuracy@5	0.5804
cosine_accuracy@10	0.655
cosine_precision@1	0.3333
cosine_precision@3	0.1772
cosine_precision@5	0.1161
cosine_precision@10	0.0655
cosine_recall@1	0.3333
cosine_recall@3	0.5315
cosine_recall@5	0.5804
cosine_recall@10	0.655
cosine_ndcg@10	0.4931
cosine_mrr@10	0.4415
cosine_map@100	0.4521

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.331
cosine_accuracy@3	0.5385
cosine_accuracy@5	0.5734
cosine_accuracy@10	0.662
cosine_precision@1	0.331
cosine_precision@3	0.1795
cosine_precision@5	0.1147
cosine_precision@10	0.0662
cosine_recall@1	0.331
cosine_recall@3	0.5385
cosine_recall@5	0.5734
cosine_recall@10	0.662
cosine_ndcg@10	0.4946
cosine_mrr@10	0.4415
cosine_map@100	0.4518

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3287
cosine_accuracy@3	0.5291
cosine_accuracy@5	0.5781
cosine_accuracy@10	0.6643
cosine_precision@1	0.3287
cosine_precision@3	0.1764
cosine_precision@5	0.1156
cosine_precision@10	0.0664
cosine_recall@1	0.3287
cosine_recall@3	0.5291
cosine_recall@5	0.5781
cosine_recall@10	0.6643
cosine_ndcg@10	0.4917
cosine_mrr@10	0.4371
cosine_map@100	0.4465

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.317
cosine_accuracy@3	0.5058
cosine_accuracy@5	0.5734
cosine_accuracy@10	0.655
cosine_precision@1	0.317
cosine_precision@3	0.1686
cosine_precision@5	0.1147
cosine_precision@10	0.0655
cosine_recall@1	0.317
cosine_recall@3	0.5058
cosine_recall@5	0.5734
cosine_recall@10	0.655
cosine_ndcg@10	0.4805
cosine_mrr@10	0.4252
cosine_map@100	0.4342

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 30
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 30
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.6639	10	5.6138	-	-	-	-	-
0.9959	15	-	0.3594	0.3735	0.3723	0.3161	0.3807
1.3278	20	4.9173	-	-	-	-	-
1.9917	30	3.7581	0.3874	0.4014	0.4026	0.3729	0.4032
2.6556	40	3.0018	-	-	-	-	-
2.9876	45	-	0.4031	0.4200	0.4212	0.3858	0.4223
3.3195	50	2.5035	-	-	-	-	-
3.9834	60	1.9031	0.4187	0.4303	0.4178	0.3958	0.4291
4.6473	70	1.474	-	-	-	-	-
4.9793	75	-	0.4293	0.4332	0.4318	0.4172	0.4401
5.3112	80	1.2801	-	-	-	-	-
5.9751	90	0.9577	0.4397	0.4382	0.4444	0.4275	0.4518
6.6390	100	0.7539	-	-	-	-	-
6.9710	105	-	0.4434	0.4414	0.4496	0.4262	0.4466
7.3029	110	0.694	-	-	-	-	-
7.9668	120	0.5147	0.4423	0.4488	0.4507	0.4358	0.4495
8.6307	130	0.4589	-	-	-	-	-
8.9627	135	-	0.4488	0.4575	0.4544	0.4407	0.4493
9.2946	140	0.3843	-	-	-	-	-
9.9585	150	0.3506	0.4521	0.4465	0.4559	0.4420	0.4485
10.6224	160	0.2723	-	-	-	-	-
10.9544	165	-	0.4497	0.4435	0.4499	0.4304	0.4453
11.2863	170	0.2555	-	-	-	-	-
11.9502	180	0.2077	0.4448	0.4472	0.4468	0.4287	0.4453
12.6141	190	0.1894	-	-	-	-	-
12.9461	195	-	0.4516	0.4463	0.4566	0.4336	0.452
13.2780	200	0.1725	-	-	-	-	-
13.9419	210	0.1395	0.4528	0.4520	0.4561	0.4333	0.4534
14.6058	220	0.155	-	-	-	-	-
14.9378	225	-	0.4461	0.4491	0.4527	0.4369	0.4517
15.2697	230	0.132	-	-	-	-	-
15.9336	240	0.1148	-	-	-	-	-
16.0	241	-	0.4482	0.4537	0.4540	0.4303	0.4538
16.5975	250	0.1061	-	-	-	-	-
16.9959	256	-	0.4464	0.4538	0.4551	0.4294	0.4577
17.2614	260	0.0961	-	-	-	-	-
17.9253	270	0.087	-	-	-	-	-
17.9917	271	-	0.4485	0.4483	0.4495	0.4326	0.4568
18.5892	280	0.1009	-	-	-	-	-
18.9876	286	-	0.4483	0.4517	0.4545	0.4396	0.4565
19.2531	290	0.0854	-	-	-	-	-
19.9170	300	0.073	-	-	-	-	-
19.9834	301	-	0.4473	0.4502	0.4521	0.4349	0.4548
20.5809	310	0.0726	-	-	-	-	-
20.9793	316	-	0.4466	0.4525	0.4538	0.4341	0.4583
21.2448	320	0.0747	-	-	-	-	-
21.9087	330	0.0621	-	-	-	-	-
21.9751	331	-	0.4441	0.4537	0.4534	0.4388	0.4564
22.5726	340	0.0682	-	-	-	-	-
22.9710	346	-	0.4454	0.4529	0.4544	0.4385	0.4589
23.2365	350	0.0612	-	-	-	-	-
23.9004	360	0.0546	-	-	-	-	-
23.9668	361	-	0.4464	0.4494	0.4551	0.4381	0.4567
24.5643	370	0.0599	-	-	-	-	-
24.9627	376	-	0.4465	0.4506	0.4553	0.4363	0.4567
25.2282	380	0.0591	-	-	-	-	-
25.8921	390	0.0562	-	-	-	-	-
25.9585	391	-	0.4454	0.4515	0.4532	0.4343	0.4575
26.5560	400	0.0623	-	-	-	-	-
26.9544	406	-	0.4452	0.4531	0.4544	0.4342	0.4573
27.2199	410	0.061	-	-	-	-	-
27.8838	420	0.053	-	-	-	-	-
27.9502	421	-	0.4454	0.4514	0.4533	0.4330	0.4573
28.5477	430	0.0564	-	-	-	-	-
28.9461	436	-	0.4465	0.4516	0.4533	0.4338	0.4562
29.2116	440	0.056	-	-	-	-	-
29.8755	450	0.0586	0.4465	0.4518	0.4521	0.4342	0.4573

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2
• Accelerate: 0.30.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}