Add new LinkTransformer model.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+pytorch_model.bin filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

LT_training_config.json

@@ -0,0 +1,26 @@
+{
+  "model_save_dir": "models",
+  "model_save_name": "linkage_ja_aliases",
+  "opt_model_description": "This model was trained on a dataset consisting of company aliases from wiki data using the LinkTransformer framework. \n                               It was trained for 100 epochs using other defaults that can be found in the repo's LinkTransformer config file - LT_training_config.json \n  ",
+  "opt_model_lang": "ja",
+  "train_batch_size": 64,
+  "num_epochs": 100,
+  "warm_up_perc": 1,
+  "learning_rate": 2e-06,
+  "val_perc": 0.2,
+  "wandb_names": {
+    "project": "linkage",
+    "id": "econabhishek",
+    "run": "linkage_ja_aliases",
+    "entity": "econabhishek"
+  },
+  "add_pooling_layer": false,
+  "large_val": true,
+  "eval_steps_perc": 0.1,
+  "test_at_end": true,
+  "save_val_test_pickles": true,
+  "eval_type": "retrieval",
+  "training_dataset": "dataframe",
+  "base_model_path": "oshizo/sbert-jsnli-luke-japanese-base-lite",
+  "best_model_path": "models/linkage_ja_aliases"
+}

README.md

@@ -0,0 +1,138 @@
+---
+pipeline_tag: sentence-similarity
+language: 
+- ja
+tags:
+- linktransformer
+- sentence-transformers
+- sentence-similarity
+- tabular-classification
+
+---
+
+# dell-research-harvard/lt-wikidata-comp-ja
+
+This is a [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) model. At its core this model this is a sentence transformer model [sentence-transformers](https://www.SBERT.net) model- it just wraps around the class. 
+It is designed for quick and easy record linkage (entity-matching) through the LinkTransformer package. The tasks include clustering, deduplication, linking, aggregation and more.
+Notwithstanding that, it can be used for any sentence similarity task within the sentence-transformers framework as well. 
+It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+Take a look at the documentation of [sentence-transformers](https://www.sbert.net/index.html) if you want to use this model for more than what we support in our applications. 
+
+
+This model has been fine-tuned on the model : oshizo/sbert-jsnli-luke-japanese-base-lite. It is pretrained for the language : - ja.
+
+
+This model was trained on a dataset consisting of company aliases from wiki data using the LinkTransformer framework. 
+                               It was trained for 100 epochs using other defaults that can be found in the repo's LinkTransformer config file - LT_training_config.json 
+  
+
+## Usage (LinkTransformer)
+
+Using this model becomes easy when you have [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) installed:
+
+```
+pip install -U linktransformer
+```
+
+Then you can use the model like this:
+
+```python
+import linktransformer as lt
+import pandas as pd
+
+##Load the two dataframes that you want to link. For example, 2 dataframes with company names that are written differently
+df1=pd.read_csv("data/df1.csv") ###This is the left dataframe with key CompanyName for instance
+df2=pd.read_csv("data/df2.csv") ###This is the right dataframe with key CompanyName for instance
+
+###Merge the two dataframes on the key column!
+df_merged = lt.merge(df1, df2, on="CompanyName", how="inner")
+
+##Done! The merged dataframe has a column called "score" that contains the similarity score between the two company names
+
+```
+
+
+## Training your own LinkTransformer model
+Any Sentence Transformers can be used as a backbone by simply adding a pooling layer.  Any other transformer on HuggingFace can also be used by specifying the option add_pooling_layer==True
+The model was trained using SupCon loss. 
+Usage can be found in the package docs. 
+The training config can be found in the repo with the name LT_training_config.json
+To replicate the training, you can download the file and specify the path in the config_path argument of the training function. You can also override the config by specifying the training_args argument.
+Here is an example. 
+
+
+```python
+
+##Consider the example in the paper that has a dataset of Mexican products and their tariff codes from 1947 and 1948 and we want train a model to link the two tariff codes.
+saved_model_path = train_model(
+        model_path="hiiamsid/sentence_similarity_spanish_es",
+        dataset_path=dataset_path,
+        left_col_names=["description47"],
+        right_col_names=['description48'],
+        left_id_name=['tariffcode47'],
+        right_id_name=['tariffcode48'],
+        log_wandb=False,
+        config_path=LINKAGE_CONFIG_PATH,
+        training_args={"num_epochs": 1}
+    )
+
+```
+
+
+You can also use this package for deduplication (clusters a df on the supplied key column). Merging a fine class (like product) to a coarse class (like HS code) is also possible.
+Read our paper and the documentation for more!
+
+
+
+## Evaluation Results
+
+<!--- Describe how your model was evaluated -->
+
+You can evaluate the model using the [LinkTransformer](https://github.com/dell-research-harvard/linktransformer) package's inference functions.
+We have provided a few datasets in the package for you to try out. We plan to host more datasets on Huggingface and our website (Coming soon) that you can take a look at.
+
+
+## Training
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 444 with parameters:
+```
+{'batch_size': 64, 'sampler': 'torch.utils.data.dataloader._InfiniteConstantSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+```
+
+**Loss**:
+
+`linktransformer.modified_sbert.losses.SupConLoss_wandb` 
+
+Parameters of the fit()-Method:
+```
+{
+    "epochs": 100,
+    "evaluation_steps": 4440,
+    "evaluator": "sentence_transformers.evaluation.SequentialEvaluator.SequentialEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
+    "optimizer_params": {
+        "lr": 2e-06
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 44400,
+    "weight_decay": 0.01
+}
+```
+
+
+
+
+LinkTransformer(
+  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: LukeModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

added_tokens.json

@@ -0,0 +1,4 @@
+{
+  "<ent2>": 32771,
+  "<ent>": 32770
+}

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "models/linkage_ja_aliases/",
+  "architectures": [
+    "LukeModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bert_model_name": "models/luke-japanese/hf_xlm_roberta",
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "cls_entity_prediction": false,
+  "entity_emb_size": 256,
+  "entity_vocab_size": 4,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "luke",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.31.0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "use_entity_aware_attention": true,
+  "vocab_size": 32772
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.25.1",
+    "pytorch": "1.13.0+cu116"
+  }
+}

entity_vocab.json

@@ -0,0 +1,6 @@
+{
+  "[MASK2]": 3,
+  "[MASK]": 0,
+  "[PAD]": 2,
+  "[UNK]": 1
+}

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:451d07b21cc069b7c2c7e8f6624f52da26f0e246f90abaedc8749782b54b0adf
+size 532357729

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 128,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b73a5e054936c920cf5b7d1ec21ce9c281977078269963beb821c6c86fbff7
+size 841889

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,110 @@
+{
+  "additional_special_tokens": [
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "__type": "AddedToken",
+      "content": "<ent2>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "entity_mask2_token": "[MASK2]",
+  "entity_mask_token": "[MASK]",
+  "entity_pad_token": "[PAD]",
+  "entity_token_1": {
+    "__type": "AddedToken",
+    "content": "<ent>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "entity_token_2": {
+    "__type": "AddedToken",
+    "content": "<ent2>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "entity_unk_token": "[UNK]",
+  "eos_token": "</s>",
+  "mask_token": {
+    "__type": "AddedToken",
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "max_entity_length": 32,
+  "max_mention_length": 30,
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "task": null,
+  "tokenizer_class": "MLukeTokenizer",
+  "tokenizer_file": "models/luke-japanese/hf_luke_japanese_lite_epoch20/tokenizer.json",
+  "unk_token": "<unk>"
+}