---
license: apache-2.0
base_model: flan-t5-small
tags:
- generated_from_trainer
model-index:
- name: shadow
  results: []
language:
- en
library_name: transformers
pipeline_tag: question-answering
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# SHADOW  ![alt text](https://fbi.cults3d.com/uploaders/17563507/illustration-file/b42b9737-93f0-4672-be1a-c8646e7ece39/Screenshot_1.png)

SHADOW (**S**ymbolic **H**igher-order **A**ssociative **D**eductive reasoning **O**n **W**ikidata) is a model that is a fine-tuned version of [flan-t5-small](https://huggingface.co/google/flan-t5-small) on the LM-KBC 2024 dataset. It achieves the following results on the validation set:
- Loss: 0.0003

## Model description

The model is trained to learn specific Wikidata patterns (e.g. type of relation, type of subject entity, etc.) to identify the correct template id that matches the SPARQL query that finds the appropriate object entities for the triple (subject entity, relation, object entity). The SPARQL queries are dynamically pre-defined for each template id and the model is not exposed to the queries themselves. The training of the model follows the associative reasoning pattern based on higher-order reasoning that deductively tries to associate a symbol (here, a template id) to data (here, the (subject entity, relation) tuple).

## Intended uses & limitations

This model is intended to be used with a (subject entity, relation) Wikidata pair to determine the best template id for the SPARQL query that fetches the matching object entities.

## Training and evaluation data

The LM-KBC dataset is used for training and evaluation.

## Templates

The templates are frames that define dynamic SPARQL queries to find the object entities to complete the triple (subject entity, relation, object entities). The model is currently trained on the following relations:

- **countryLandBordersCountry:** Null values possible (e.g., Iceland)
- **personHasCityOfDeath:** Null values possible
- **seriesHasNumberOfEpisodes:** Object is numeric
- **awardWonBy:** Many objects per subject (e.g., 224 Physics Nobel prize winners)
- **companyTradesAtStockExchange:** Null values possible

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-04
- train_batch_size: 4
- eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.6679        | 1.0   | 1000 | 0.0005          |
| 0.3425        | 2.0   | 2000 | 0.0002          |
| 0.2303        | 3.0   | 3000 | 0.0001          |
| 0.1735        | 4.0   | 4000 | 0.000           |
| 0.1394        | 5.0   | 5000 | 0.0001          |
| 0.1167        | 6.0   | 6000 | 0.00009         |
| 0.1006        | 7.0   | 7000 | 0.00008         |
| 0.0882        | 8.0   | 8000 | 0.00007         |
| 0.0785        | 9.0   | 9000 | 0.00006         |
| 0.0707        | 10.0  | 10000| 0.00006         |
| 0.0643        | 11.0  | 11000| 0.00005         |
| 0.0590        | 12.0  | 12000| 0.00005         |
| 0.0545        | 13.0  | 13000| 0.00004         |
| 0.0506        | 14.0  | 14000| 0.00004         |
| 0.0473        | 15.0  | 15000| 0.00004         |
| 0.0443        | 16.0  | 16000| 0.00003         |
| 0.0417        | 17.0  | 17000| 0.00003         |
| 0.0394        | 18.0  | 18000| 0.00003         |
| 0.0374        | 19.0  | 19000| 0.00003         |
| 0.0355        | 20.0  | 20000| 0.00003         |


```
@misc{akl2024projectshadowsymbolichigherorder,
      title={Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing}, 
      author={Hanna Abi Akl},
      year={2024},
      eprint={2408.14849},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.14849}, 
}
```