SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct

This is a sentence-transformers model finetuned from Qwen/Qwen2.5-0.5B-Instruct. It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Qwen/Qwen2.5-0.5B-Instruct
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 896 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AlexWortega/qwen_emb_600_best_21.11")
# Run inference
sentences = [
    'Is a birth certificate a form of ID?',
    'Identity documents in the United States\nThe birth certificate is the initial identification document issued to parents shortly after the birth of their child. The birth certificate is typically issued by local governments, usually the city or county where a child is born. It is an important record, often called a "feeder document," because it establishes U.S. citizenship through birthright citizenship, which is then used to obtain, or is the basis for, all other identity documents.[2] By itself, the birth certificate is usually only considered proof of citizenship but not proof of identity, since it is issued without a photograph at birth, containing no identifying features. A birth certificate is normally produced along with proof of identity, such as a driver\'s license or the testimony of a third party (such as a parent), to establish identity or entitlement to a service.',
    "Identity document\nIn Canada, different forms of identification documentation are used, but there is no de jure national identity card. The Canadian passport is issued by the federal (national) government, and the provinces and territories issue various documents which can be used for identification purposes. The most commonly used forms of identification within Canada are the health card and driver's licence issued by provincial and territorial governments. The widespread usage of these two documents for identification purposes has made them de facto identity cards.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with InformationRetrievalEvaluator

Metric	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
cosine_accuracy@1	0.18	0.6	0.28	0.18	0.46	0.22	0.32	0.28	0.84	0.26	0.06	0.26	0.449
cosine_accuracy@3	0.28	0.72	0.56	0.28	0.56	0.34	0.42	0.4	0.92	0.38	0.42	0.38	0.7143
cosine_accuracy@5	0.32	0.82	0.66	0.4	0.6	0.34	0.46	0.5	0.92	0.48	0.52	0.5	0.7959
cosine_accuracy@10	0.4	0.88	0.82	0.5	0.68	0.44	0.52	0.58	0.98	0.62	0.68	0.6	0.9796
cosine_precision@1	0.18	0.6	0.28	0.18	0.46	0.22	0.32	0.28	0.84	0.26	0.06	0.26	0.449
cosine_precision@3	0.1	0.4333	0.1867	0.1133	0.2333	0.1133	0.28	0.1333	0.3667	0.1533	0.14	0.1333	0.3946
cosine_precision@5	0.072	0.416	0.136	0.096	0.148	0.068	0.24	0.1	0.236	0.148	0.104	0.104	0.3673
cosine_precision@10	0.052	0.34	0.086	0.062	0.09	0.044	0.18	0.058	0.128	0.118	0.068	0.066	0.3347
cosine_recall@1	0.07	0.0588	0.26	0.0969	0.23	0.22	0.0125	0.27	0.7407	0.053	0.06	0.235	0.0347
cosine_recall@3	0.1283	0.1203	0.52	0.1971	0.35	0.34	0.0367	0.37	0.8687	0.093	0.42	0.355	0.0924
cosine_recall@5	0.145	0.1722	0.63	0.275	0.37	0.34	0.0562	0.47	0.886	0.15	0.52	0.465	0.1378
cosine_recall@10	0.1933	0.2396	0.79	0.3352	0.45	0.44	0.0669	0.53	0.956	0.2407	0.68	0.58	0.2392
cosine_ndcg@10	0.1694	0.4329	0.5142	0.2365	0.4053	0.3192	0.2211	0.3978	0.889	0.2161	0.3708	0.3967	0.3738
cosine_mrr@10	0.2476	0.6865	0.4404	0.2642	0.5121	0.2824	0.3847	0.3705	0.8829	0.3527	0.2719	0.3504	0.6104
cosine_map@100	0.1356	0.2693	0.4253	0.1923	0.3491	0.2998	0.0815	0.3635	0.8629	0.1591	0.2829	0.3458	0.3108

Nano BEIR

Dataset: NanoBEIR_mean
Evaluated with NanoBEIREvaluator

Metric	Value
cosine_accuracy@1	0.3376
cosine_accuracy@3	0.4903
cosine_accuracy@5	0.5628
cosine_accuracy@10	0.6677
cosine_precision@1	0.3376
cosine_precision@3	0.2139
cosine_precision@5	0.1719
cosine_precision@10	0.1251
cosine_recall@1	0.1801
cosine_recall@3	0.2993
cosine_recall@5	0.3552
cosine_recall@10	0.4416
cosine_ndcg@10	0.3802
cosine_mrr@10	0.4351
cosine_map@100	0.3137

Training Details

Training Dataset

Unnamed Dataset

Size: 1,077,240 training samples
Columns: query, response, and negative

Approximate statistics based on the first 1000 samples:

	query	response	negative
type	string	string	string
details	min: 4 tokens mean: 8.76 tokens max: 26 tokens	min: 23 tokens mean: 141.88 tokens max: 532 tokens	min: 4 tokens mean: 134.02 tokens max: 472 tokens

Samples:

query	response	negative
`Was there a year 0?`	`Year zero Year zero does not exist in the anno Domini system usually used to number years in the Gregorian calendar and in its predecessor, the Julian calendar. In this system, the year 1 BC is followed by AD 1. However, there is a year zero in astronomical year numbering (where it coincides with the Julian year 1 BC) and in ISO 8601:2004 (where it coincides with the Gregorian year 1 BC) as well as in all Buddhist and Hindu calendars.`	`504 Year 504 (DIV) was a leap year starting on Thursday (link will display the full calendar) of the Julian calendar. At the time, it was known as the Year of the Consulship of Nicomachus without colleague (or, less frequently, year 1257 "Ab urbe condita"). The denomination 504 for this year has been used since the early medieval period, when the Anno Domini calendar era became the prevalent method in Europe for naming years.`
`When is the dialectical method used?`	Dialectic Dialectic or dialectics (Greek: διαλεκτική, dialektikḗ; related to dialogue), also known as the dialectical method, is at base a discourse between two or more people holding different points of view about a subject but wishing to establish the truth through reasoned arguments. Dialectic resembles debate, but the concept excludes subjective elements such as emotional appeal and the modern pejorative sense of rhetoric.[1][2] Dialectic may be contrasted with the didactic method, wherein one side of the conversation teaches the other. Dialectic is alternatively known as minor logic, as opposed to major logic or critique.	Derek Bentley case Another factor in the posthumous defence was that a "confession" recorded by Bentley, which was claimed by the prosecution to be a "verbatim record of dictated monologue", was shown by forensic linguistics methods to have been largely edited by policemen. Linguist Malcolm Coulthard showed that certain patterns, such as the frequency of the word "then" and the grammatical use of "then" after the grammatical subject ("I then" rather than "then I"), were not consistent with Bentley's use of language (his idiolect), as evidenced in court testimony. These patterns fit better the recorded testimony of the policemen involved. This is one of the earliest uses of forensic linguistics on record.
`What do Grasshoppers eat?`	Grasshopper Grasshoppers are plant-eaters, with a few species at times becoming serious pests of cereals, vegetables and pasture, especially when they swarm in their millions as locusts and destroy crops over wide areas. They protect themselves from predators by camouflage; when detected, many species attempt to startle the predator with a brilliantly-coloured wing-flash while jumping and (if adult) launching themselves into the air, usually flying for only a short distance. Other species such as the rainbow grasshopper have warning coloration which deters predators. Grasshoppers are affected by parasites and various diseases, and many predatory creatures feed on both nymphs and adults. The eggs are the subject of attack by parasitoids and predators.	Groundhog Very often the dens of groundhogs provide homes for other animals including skunks, red foxes, and cottontail rabbits. The fox and skunk feed upon field mice, grasshoppers, beetles and other creatures that destroy farm crops. In aiding these animals, the groundhog indirectly helps the farmer. In addition to providing homes for itself and other animals, the groundhog aids in soil improvement by bringing subsoil to the surface. The groundhog is also a valuable game animal and is considered a difficult sport when hunted in a fair manner. In some parts of Appalachia, they are eaten.

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
gradient_accumulation_steps: 8
learning_rate: 0.0001
max_grad_norm: 0.01
num_train_epochs: 2
warmup_ratio: 0.4
bf16: True
dataloader_num_workers: 8
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 8
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 0.0001
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 0.01
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.4
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: True
dataloader_num_workers: 8
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs