--- base_model: srikarvar/fine_tuned_model_5 library_name: sentence-transformers metrics: - cosine_accuracy - cosine_accuracy_threshold - cosine_f1 - cosine_f1_threshold - cosine_precision - cosine_recall - cosine_ap - dot_accuracy - dot_accuracy_threshold - dot_f1 - dot_f1_threshold - dot_precision - dot_recall - dot_ap - manhattan_accuracy - manhattan_accuracy_threshold - manhattan_f1 - manhattan_f1_threshold - manhattan_precision - manhattan_recall - manhattan_ap - euclidean_accuracy - euclidean_accuracy_threshold - euclidean_f1 - euclidean_f1_threshold - euclidean_precision - euclidean_recall - euclidean_ap - max_accuracy - max_accuracy_threshold - max_f1 - max_f1_threshold - max_precision - max_recall - max_ap pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:560 - loss:OnlineContrastiveLoss widget: - source_sentence: The `Garage` class has a `to_services` method which is used to transform tasks into a list of `ServiceRecord` objects that are scheduled. sentences: - The `to_services` method in the Garage class is used to convert Garage tasks to a list of scheduled `ServiceRecord` objects. - It returns a `Recipe` for the specified serving size. - The AI community is a group of individuals who collaborate on models, datasets, and tools to advance artificial intelligence research. - source_sentence: The main version of the guide contains the INSTALLATION page. Click the link to be directed there. sentences: - You can bake bread by following the Bake bread tutorial. - The base class for documents generated from a data stream is StreamBasedBuilder. - You can find the INSTALLATION page in the main version of the guide. Click on the provided link to redirect to the main version. - source_sentence: A major distinction between a ProductList and an InventoryList is that a ProductList allows for random access to the items, while an InventoryList updates gradually as it is navigated. sentences: - The how-to guides for the platform include Setup, Processing, Streaming, TensorFlow integration, PyTorch integration, Cache management, Cloud storage, Search index, Analytics, and Data Pipelines. - 'Yes, there is a tutorial for analyzing stock market data. You can find it at the link provided: /docs/stocks/v2.10.0/data_analysis.' - The main difference between a ProductList and an InventoryList is that a ProductList provides random access to the items, while an InventoryList updates progressively as you browse the list. - source_sentence: ImageFolder is a dataset builder that eliminates the need for coding to quickly load a dataset with thousands of image files. It will automatically incorporate any extra data such as resolution, format, or tags, provided that it is included in a metadata file (metadata.csv/metadata.jsonl). sentences: - The function `calc_and_sum` returns the calculated value and sum. - Some examples of supported network drives are Network File System (NFS), Server Message Block (SMB), and WebDAV. - ImageFolder is a dataset builder designed to quickly load an image dataset with several thousand image files without requiring you to write any code. It automatically loads any additional information about your dataset, such as image resolution, format, or image tags, as long as you include this information in a metadata file (metadata.csv/metadata.jsonl). - source_sentence: The `num_services` method gives the quantity of services in the garage. sentences: - A signature in the sales database is a unique identifier for a transaction that is updated every time a change is made. It is computed by combining the previous signature and a hash of the latest update applied. - The `num_services` method returns the number of services in the garage. - It returns the number of entries in the dataset. model-index: - name: SentenceTransformer based on srikarvar/fine_tuned_model_5 results: - task: type: binary-classification name: Binary Classification dataset: name: pair class dev type: pair-class-dev metrics: - type: cosine_accuracy value: 0.9821428571428571 name: Cosine Accuracy - type: cosine_accuracy_threshold value: 0.9922685623168945 name: Cosine Accuracy Threshold - type: cosine_f1 value: 0.9909909909909909 name: Cosine F1 - type: cosine_f1_threshold value: 0.9922685623168945 name: Cosine F1 Threshold - type: cosine_precision value: 1.0 name: Cosine Precision - type: cosine_recall value: 0.9821428571428571 name: Cosine Recall - type: cosine_ap value: 1.0 name: Cosine Ap - type: dot_accuracy value: 0.9821428571428571 name: Dot Accuracy - type: dot_accuracy_threshold value: 0.9922685623168945 name: Dot Accuracy Threshold - type: dot_f1 value: 0.9909909909909909 name: Dot F1 - type: dot_f1_threshold value: 0.9922685623168945 name: Dot F1 Threshold - type: dot_precision value: 1.0 name: Dot Precision - type: dot_recall value: 0.9821428571428571 name: Dot Recall - type: dot_ap value: 1.0 name: Dot Ap - type: manhattan_accuracy value: 0.9821428571428571 name: Manhattan Accuracy - type: manhattan_accuracy_threshold value: 1.8805665969848633 name: Manhattan Accuracy Threshold - type: manhattan_f1 value: 0.9909909909909909 name: Manhattan F1 - type: manhattan_f1_threshold value: 1.8805665969848633 name: Manhattan F1 Threshold - type: manhattan_precision value: 1.0 name: Manhattan Precision - type: manhattan_recall value: 0.9821428571428571 name: Manhattan Recall - type: manhattan_ap value: 1.0 name: Manhattan Ap - type: euclidean_accuracy value: 0.9821428571428571 name: Euclidean Accuracy - type: euclidean_accuracy_threshold value: 0.12164457887411118 name: Euclidean Accuracy Threshold - type: euclidean_f1 value: 0.9909909909909909 name: Euclidean F1 - type: euclidean_f1_threshold value: 0.12164457887411118 name: Euclidean F1 Threshold - type: euclidean_precision value: 1.0 name: Euclidean Precision - type: euclidean_recall value: 0.9821428571428571 name: Euclidean Recall - type: euclidean_ap value: 1.0 name: Euclidean Ap - type: max_accuracy value: 0.9821428571428571 name: Max Accuracy - type: max_accuracy_threshold value: 1.8805665969848633 name: Max Accuracy Threshold - type: max_f1 value: 0.9909909909909909 name: Max F1 - type: max_f1_threshold value: 1.8805665969848633 name: Max F1 Threshold - type: max_precision value: 1.0 name: Max Precision - type: max_recall value: 0.9821428571428571 name: Max Recall - type: max_ap value: 1.0 name: Max Ap - task: type: binary-classification name: Binary Classification dataset: name: pair class test type: pair-class-test metrics: - type: cosine_accuracy value: 0.9821428571428571 name: Cosine Accuracy - type: cosine_accuracy_threshold value: 0.9922685623168945 name: Cosine Accuracy Threshold - type: cosine_f1 value: 0.9909909909909909 name: Cosine F1 - type: cosine_f1_threshold value: 0.9922685623168945 name: Cosine F1 Threshold - type: cosine_precision value: 1.0 name: Cosine Precision - type: cosine_recall value: 0.9821428571428571 name: Cosine Recall - type: cosine_ap value: 1.0 name: Cosine Ap - type: dot_accuracy value: 0.9821428571428571 name: Dot Accuracy - type: dot_accuracy_threshold value: 0.9922685623168945 name: Dot Accuracy Threshold - type: dot_f1 value: 0.9909909909909909 name: Dot F1 - type: dot_f1_threshold value: 0.9922685623168945 name: Dot F1 Threshold - type: dot_precision value: 1.0 name: Dot Precision - type: dot_recall value: 0.9821428571428571 name: Dot Recall - type: dot_ap value: 1.0 name: Dot Ap - type: manhattan_accuracy value: 0.9821428571428571 name: Manhattan Accuracy - type: manhattan_accuracy_threshold value: 1.8805665969848633 name: Manhattan Accuracy Threshold - type: manhattan_f1 value: 0.9909909909909909 name: Manhattan F1 - type: manhattan_f1_threshold value: 1.8805665969848633 name: Manhattan F1 Threshold - type: manhattan_precision value: 1.0 name: Manhattan Precision - type: manhattan_recall value: 0.9821428571428571 name: Manhattan Recall - type: manhattan_ap value: 1.0 name: Manhattan Ap - type: euclidean_accuracy value: 0.9821428571428571 name: Euclidean Accuracy - type: euclidean_accuracy_threshold value: 0.12164457887411118 name: Euclidean Accuracy Threshold - type: euclidean_f1 value: 0.9909909909909909 name: Euclidean F1 - type: euclidean_f1_threshold value: 0.12164457887411118 name: Euclidean F1 Threshold - type: euclidean_precision value: 1.0 name: Euclidean Precision - type: euclidean_recall value: 0.9821428571428571 name: Euclidean Recall - type: euclidean_ap value: 1.0 name: Euclidean Ap - type: max_accuracy value: 0.9821428571428571 name: Max Accuracy - type: max_accuracy_threshold value: 1.8805665969848633 name: Max Accuracy Threshold - type: max_f1 value: 0.9909909909909909 name: Max F1 - type: max_f1_threshold value: 1.8805665969848633 name: Max F1 Threshold - type: max_precision value: 1.0 name: Max Precision - type: max_recall value: 0.9821428571428571 name: Max Recall - type: max_ap value: 1.0 name: Max Ap --- # SentenceTransformer based on srikarvar/fine_tuned_model_5 This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [srikarvar/fine_tuned_model_5](https://huggingface.co/srikarvar/fine_tuned_model_5) on the json dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [srikarvar/fine_tuned_model_5](https://huggingface.co/srikarvar/fine_tuned_model_5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 384 tokens - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("srikarvar/fine_tuned_model_12") # Run inference sentences = [ 'The `num_services` method gives the quantity of services in the garage.', 'The `num_services` method returns the number of services in the garage.', 'It returns the number of entries in the dataset.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 384] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Binary Classification * Dataset: `pair-class-dev` * Evaluated with [BinaryClassificationEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator) | Metric | Value | |:-----------------------------|:--------| | cosine_accuracy | 0.9821 | | cosine_accuracy_threshold | 0.9923 | | cosine_f1 | 0.991 | | cosine_f1_threshold | 0.9923 | | cosine_precision | 1.0 | | cosine_recall | 0.9821 | | cosine_ap | 1.0 | | dot_accuracy | 0.9821 | | dot_accuracy_threshold | 0.9923 | | dot_f1 | 0.991 | | dot_f1_threshold | 0.9923 | | dot_precision | 1.0 | | dot_recall | 0.9821 | | dot_ap | 1.0 | | manhattan_accuracy | 0.9821 | | manhattan_accuracy_threshold | 1.8806 | | manhattan_f1 | 0.991 | | manhattan_f1_threshold | 1.8806 | | manhattan_precision | 1.0 | | manhattan_recall | 0.9821 | | manhattan_ap | 1.0 | | euclidean_accuracy | 0.9821 | | euclidean_accuracy_threshold | 0.1216 | | euclidean_f1 | 0.991 | | euclidean_f1_threshold | 0.1216 | | euclidean_precision | 1.0 | | euclidean_recall | 0.9821 | | euclidean_ap | 1.0 | | max_accuracy | 0.9821 | | max_accuracy_threshold | 1.8806 | | max_f1 | 0.991 | | max_f1_threshold | 1.8806 | | max_precision | 1.0 | | max_recall | 0.9821 | | **max_ap** | **1.0** | #### Binary Classification * Dataset: `pair-class-test` * Evaluated with [BinaryClassificationEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator) | Metric | Value | |:-----------------------------|:--------| | cosine_accuracy | 0.9821 | | cosine_accuracy_threshold | 0.9923 | | cosine_f1 | 0.991 | | cosine_f1_threshold | 0.9923 | | cosine_precision | 1.0 | | cosine_recall | 0.9821 | | cosine_ap | 1.0 | | dot_accuracy | 0.9821 | | dot_accuracy_threshold | 0.9923 | | dot_f1 | 0.991 | | dot_f1_threshold | 0.9923 | | dot_precision | 1.0 | | dot_recall | 0.9821 | | dot_ap | 1.0 | | manhattan_accuracy | 0.9821 | | manhattan_accuracy_threshold | 1.8806 | | manhattan_f1 | 0.991 | | manhattan_f1_threshold | 1.8806 | | manhattan_precision | 1.0 | | manhattan_recall | 0.9821 | | manhattan_ap | 1.0 | | euclidean_accuracy | 0.9821 | | euclidean_accuracy_threshold | 0.1216 | | euclidean_f1 | 0.991 | | euclidean_f1_threshold | 0.1216 | | euclidean_precision | 1.0 | | euclidean_recall | 0.9821 | | euclidean_ap | 1.0 | | max_accuracy | 0.9821 | | max_accuracy_threshold | 1.8806 | | max_f1 | 0.991 | | max_f1_threshold | 1.8806 | | max_precision | 1.0 | | max_recall | 0.9821 | | **max_ap** | **1.0** | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 560 training samples * Columns: label, sentence2, and sentence1 * Approximate statistics based on the first 560 samples: | | label | sentence2 | sentence1 | |:--------|:-----------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | type | int | string | string | | details | | | | * Samples: | label | sentence2 | sentence1 | |:---------------|:---------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------| | 1 | It is not available in v2.10.0. | No, it doesn't exist in v2.10.0. | | 1 | You can become a member of the research forum and pose questions to the AI community. | You can join and ask questions in the AI research forum. | | 1 | No information regarding initializing a project for PyTorch is included in the guide. | The guide does not provide information on how to initialize a project for PyTorch. | * Loss: [OnlineContrastiveLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss) ### Evaluation Dataset #### json * Dataset: json * Size: 560 evaluation samples * Columns: label, sentence2, and sentence1 * Approximate statistics based on the first 560 samples: | | label | sentence2 | sentence1 | |:--------|:-----------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | int | string | string | | details | | | | * Samples: | label | sentence2 | sentence1 | |:---------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 | The how-to guides for the platform include instructions for Setup, Processing, Streaming, TensorFlow integration, PyTorch integration, Caching, Cloud storage, Indexing, Analytics, and Data Pipelines. | The how-to guides for the platform include Setup, Processing, Streaming, TensorFlow integration, PyTorch integration, Cache management, Cloud storage, Search index, Analytics, and Data Pipelines. | | 1 | In the absence of a model script, all files in the supported formats will be loaded. However, if a model script is present, it will be downloaded and executed in order to download and prepare the model. | If there’s no model script, all the files in the supported formats are loaded. If there’s a model script, it is downloaded and executed to download and prepare the model. | | 1 | React, Angular, and Vue are compatible with the Plugin library. | The Plugin library can be used with React, Angular, and Vue. | * Loss: [OnlineContrastiveLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss) ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 32 - `gradient_accumulation_steps`: 2 - `num_train_epochs`: 4 - `warmup_ratio`: 0.1 - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 32 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 2 - `eval_accumulation_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | loss | pair-class-dev_max_ap | pair-class-test_max_ap | |:-------:|:------:|:-------------:|:----------:|:---------------------:|:----------------------:| | 0 | 0 | - | - | 1.0 | - | | 1.0 | 8 | - | 0.0028 | 1.0 | - | | 1.25 | 10 | 0.1425 | - | - | - | | 2.0 | 16 | - | 0.0003 | 1.0 | - | | 2.5 | 20 | 0.002 | - | - | - | | 3.0 | 24 | - | 0.0001 | 1.0 | - | | 3.75 | 30 | 0.0008 | - | - | - | | **4.0** | **32** | **-** | **0.0001** | **1.0** | **1.0** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.1.0 - Transformers: 4.41.2 - PyTorch: 2.1.2+cu121 - Accelerate: 0.34.2 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```