SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct

This is a sentence-transformers model finetuned from Qwen/Qwen2.5-0.5B-Instruct. It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Qwen/Qwen2.5-0.5B-Instruct
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 896 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AlexWortega/qwen1k")
# Run inference
sentences = [
    'When did the July Monarchy end?',
    'July Monarchy\nThe July Monarchy (French: Monarchie de Juillet) was a liberal constitutional monarchy in France under Louis Philippe I, starting with the July Revolution of 1830 and ending with the Revolution of 1848. It marks the end of the Bourbon Restoration (1814–1830). It began with the overthrow of the conservative government of Charles X, the last king of the House of Bourbon.',
    'July Monarchy\nDespite the return of the House of Bourbon to power, France was much changed from the era of the ancien régime. The egalitarianism and liberalism of the revolutionaries remained an important force and the autocracy and hierarchy of the earlier era could not be fully restored. Economic changes, which had been underway long before the revolution, had progressed further during the years of turmoil and were firmly entrenched by 1815. These changes had seen power shift from the noble landowners to the urban merchants. The administrative reforms of Napoleon, such as the Napoleonic Code and efficient bureaucracy, also remained in place. These changes produced a unified central government that was fiscally sound and had much control over all areas of French life, a sharp difference from the complicated mix of feudal and absolutist traditions and institutions of pre-Revolutionary Bourbons.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

Datasets: sts-dev-896 and sts-dev-768
Evaluated with EmbeddingSimilarityEvaluator

Metric	sts-dev-896	sts-dev-768
pearson_cosine	0.4573	0.4455
spearman_cosine	0.4965	0.4897

Training Details

Training Dataset

Unnamed Dataset

Size: 2,859,594 training samples
Columns: query, response, and negative

Approximate statistics based on the first 1000 samples:

	query	response	negative
type	string	string	string
details	min: 4 tokens mean: 8.76 tokens max: 26 tokens	min: 23 tokens mean: 141.88 tokens max: 532 tokens	min: 4 tokens mean: 134.02 tokens max: 472 tokens

Samples:

query	response	negative
`Was there a year 0?`	`Year zero Year zero does not exist in the anno Domini system usually used to number years in the Gregorian calendar and in its predecessor, the Julian calendar. In this system, the year 1 BC is followed by AD 1. However, there is a year zero in astronomical year numbering (where it coincides with the Julian year 1 BC) and in ISO 8601:2004 (where it coincides with the Gregorian year 1 BC) as well as in all Buddhist and Hindu calendars.`	`504 Year 504 (DIV) was a leap year starting on Thursday (link will display the full calendar) of the Julian calendar. At the time, it was known as the Year of the Consulship of Nicomachus without colleague (or, less frequently, year 1257 "Ab urbe condita"). The denomination 504 for this year has been used since the early medieval period, when the Anno Domini calendar era became the prevalent method in Europe for naming years.`
`When is the dialectical method used?`	Dialectic Dialectic or dialectics (Greek: διαλεκτική, dialektikḗ; related to dialogue), also known as the dialectical method, is at base a discourse between two or more people holding different points of view about a subject but wishing to establish the truth through reasoned arguments. Dialectic resembles debate, but the concept excludes subjective elements such as emotional appeal and the modern pejorative sense of rhetoric.[1][2] Dialectic may be contrasted with the didactic method, wherein one side of the conversation teaches the other. Dialectic is alternatively known as minor logic, as opposed to major logic or critique.	Derek Bentley case Another factor in the posthumous defence was that a "confession" recorded by Bentley, which was claimed by the prosecution to be a "verbatim record of dictated monologue", was shown by forensic linguistics methods to have been largely edited by policemen. Linguist Malcolm Coulthard showed that certain patterns, such as the frequency of the word "then" and the grammatical use of "then" after the grammatical subject ("I then" rather than "then I"), were not consistent with Bentley's use of language (his idiolect), as evidenced in court testimony. These patterns fit better the recorded testimony of the policemen involved. This is one of the earliest uses of forensic linguistics on record.
`What do Grasshoppers eat?`	Grasshopper Grasshoppers are plant-eaters, with a few species at times becoming serious pests of cereals, vegetables and pasture, especially when they swarm in their millions as locusts and destroy crops over wide areas. They protect themselves from predators by camouflage; when detected, many species attempt to startle the predator with a brilliantly-coloured wing-flash while jumping and (if adult) launching themselves into the air, usually flying for only a short distance. Other species such as the rainbow grasshopper have warning coloration which deters predators. Grasshoppers are affected by parasites and various diseases, and many predatory creatures feed on both nymphs and adults. The eggs are the subject of attack by parasitoids and predators.	Groundhog Very often the dens of groundhogs provide homes for other animals including skunks, red foxes, and cottontail rabbits. The fox and skunk feed upon field mice, grasshoppers, beetles and other creatures that destroy farm crops. In aiding these animals, the groundhog indirectly helps the farmer. In addition to providing homes for itself and other animals, the groundhog aids in soil improvement by bringing subsoil to the surface. The groundhog is also a valuable game animal and is considered a difficult sport when hunted in a fair manner. In some parts of Appalachia, they are eaten.

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        896,
        768
    ],
    "matryoshka_weights": [
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 12
per_device_eval_batch_size: 12
gradient_accumulation_steps: 4
num_train_epochs: 1
warmup_ratio: 0.3
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 12
per_device_eval_batch_size: 12
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 4
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.3
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	sts-dev-896_spearman_cosine	sts-dev-768_spearman_cosine
0.0002	10	4.4351	-	-
0.0003	20	4.6508	-	-
0.0005	30	4.7455	-	-
0.0007	40	4.5427	-	-
0.0008	50	4.3982	-	-
0.0010	60	4.3755	-	-
0.0012	70	4.4105	-	-
0.0013	80	5.2227	-	-
0.0015	90	5.8062	-	-
0.0017	100	5.7645	-	-
0.0018	110	5.9261	-	-
0.0020	120	5.8301	-	-
0.0022	130	5.7602	-	-
0.0023	140	5.9392	-	-
0.0025	150	5.7523	-	-
0.0027	160	5.8585	-	-
0.0029	170	5.7916	-	-
0.0030	180	5.8157	-	-
0.0032	190	5.7102	-	-
0.0034	200	5.5844	-	-
0.0035	210	5.5463	-	-
0.0037	220	5.5823	-	-
0.0039	230	5.5514	-	-
0.0040	240	5.5646	-	-
0.0042	250	5.5783	-	-
0.0044	260	5.5344	-	-
0.0045	270	5.523	-	-
0.0047	280	5.4969	-	-
0.0049	290	5.5407	-	-
0.0050	300	5.6171	-	-
0.0052	310	5.5581	-	-
0.0054	320	5.8903	-	-
0.0055	330	5.8675	-	-
0.0057	340	5.745	-	-
0.0059	350	5.6041	-	-
0.0060	360	5.5476	-	-
0.0062	370	5.3964	-	-
0.0064	380	5.3564	-	-
0.0065	390	5.3054	-	-
0.0067	400	5.2779	-	-
0.0069	410	5.206	-	-
0.0070	420	5.2168	-	-
0.0072	430	5.1645	-	-
0.0074	440	5.1797	-	-
0.0076	450	5.2526	-	-
0.0077	460	5.1768	-	-
0.0079	470	5.3519	-	-
0.0081	480	5.2982	-	-
0.0082	490	5.3229	-	-
0.0084	500	5.3758	-	-
0.0086	510	5.2478	-	-
0.0087	520	5.1799	-	-
0.0089	530	5.1088	-	-
0.0091	540	4.977	-	-
0.0092	550	4.9108	-	-
0.0094	560	4.811	-	-
0.0096	570	4.7203	-	-
0.0097	580	4.6499	-	-
0.0099	590	4.4548	-	-
0.0101	600	4.2891	-	-
0.0102	610	4.1881	-	-
0.0104	620	4.6	-	-
0.0106	630	4.5365	-	-
0.0107	640	4.3086	-	-
0.0109	650	4.0452	-	-
0.0111	660	3.9041	-	-
0.0112	670	4.3938	-	-
0.0114	680	4.3198	-	-
0.0116	690	4.1294	-	-
0.0117	700	4.077	-	-
0.0119	710	3.9174	-	-
0.0121	720	4.1629	-	-
0.0123	730	3.9611	-	-
0.0124	740	3.7768	-	-
0.0126	750	3.5842	-	-
0.0128	760	3.1196	-	-
0.0129	770	3.6288	-	-
0.0131	780	3.273	-	-
0.0133	790	2.7889	-	-
0.0134	800	2.5096	-	-
0.0136	810	1.8878	-	-
0.0138	820	2.3423	-	-
0.0139	830	1.7687	-	-
0.0141	840	2.0781	-	-
0.0143	850	2.4598	-	-
0.0144	860	1.7667	-	-
0.0146	870	2.6247	-	-
0.0148	880	1.916	-	-
0.0149	890	2.0817	-	-
0.0151	900	2.3679	-	-
0.0153	910	1.418	-	-
0.0154	920	2.7353	-	-
0.0156	930	1.992	-	-
0.0158	940	1.4564	-	-
0.0159	950	1.4154	-	-
0.0161	960	0.9499	-	-
0.0163	970	1.6304	-	-
0.0164	980	0.9264	-	-
0.0166	990	1.3278	-	-
0.0168	1000	1.686	0.4965	0.4897

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.3.0
Transformers: 4.46.2
PyTorch: 2.1.0+cu118
Accelerate: 1.1.1
Datasets: 3.1.0
Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

AlexWortega
/

qwen1k