metadata

base_model: mrm8488/longformer-base-4096-finetuned-squadv2
tags:
  - generated_from_trainer
  - qsi
  - quote_speaker_identification
license: apache-2.0
datasets:
  - Kkordik/NovelQSI
language:
  - en
widget:
  - text: >-
      Which character said 'You know, I read somewhere that the brightest stars
      are those that have undergone the most turmoil. Maybe it's the same with
      us – our struggles make us shine brighter'?
    context: >-
      Characters:

      Alex: Aliases: {'Alex'}. Gender: Male, The character is: major Bella:
      Aliases: {'Bella'}. Gender: Female, The character is: major Charlie:
      Aliases: {'Charlie'}. Gender: Non-binary, The character is: major

      Summary: 

      In the novel's previous section, Alex, Bella, and Charlie, three friends
      in Luminara, engage in a deep conversation at The Starry Night café. They
      debate destiny, with Alex believing in fate, Bella advocating for
      self-made destiny, and Charlie suggesting a combination of both. Personal
      reflections emerge, such as Bella's musings on heartbreak and Alex's
      thoughts on longing. Charlie compares people's struggles to stars,
      implying that challenges enhance personal growth. The night progresses
      with their varied, meaningful discussions.

      Novel Text:

      In the bustling city of Luminara, three friends, Alex, Bella, and Charlie,
      often met at their favorite café, The Starry Night, to discuss life, love,
      and the mysteries of the universe. The café, with its warm ambiance and
      the soft hum of jazz in the background, provided the perfect setting for
      their deep conversations.

      One evening, as the city lights twinkled outside, the trio found
      themselves engrossed in a discussion about destiny.

      Alex, a firm believer in fate, argued passionately, "I truly believe that
      our paths are predestined. The universe has a plan for each of us, and all
      our choices lead us to our ultimate destiny."

      Bella, a skeptic, laughed softly and countered, "That's a romantic notion,
      Alex, but I think we make our own destiny. It's our decisions, not some
      cosmic plan, that shape our lives."

      Charlie, always the mediator, added thoughtfully, "Maybe it's a bit of
      both. Perhaps there's a grand design, but within it, we have the freedom
      to make choices that influence our journey."

      Their conversation drifted to other topics as the evening wore on. At one
      point, Bella, reflecting on a recent heartbreak, said, "Sometimes, I
      wonder if the heart ever truly heals from loss, or if it just learns to
      live with the pain."

      Alex, looking out the window at the starry sky, mused, "It's strange how
      the heart yearns for what it can't have. The unattainable always seems so
      much more alluring."

      Charlie, who had been quiet for a while, suddenly spoke up with a gleam in
      his eye, "You know, I read somewhere that the brightest stars are those
      that have undergone the most turmoil. Maybe it's the same with us – our
      struggles make us shine brighter."

      As the night deepened, their conversation meandered through various
      topics.
model-index:
  - name: Kkordik/test_longformer_4096_qsi
    results:
      - task:
          type: question-answering
        dataset:
          type: Kkordik/NovelQSI
          name: NovelQSI
          split: test
        metrics:
          - type: exact_match
            value: 20.346
            verified: false
          - type: f1
            value: 26.58
            verified: false

longformer_4096_qsi

This model is a fine-tuned version of mrm8488/longformer-base-4096-finetuned-squadv2 on a tiny NovelQSI dataset. It achieves the following results on the evaluation set:

Loss: 2.9598

Model description

This model is a test model for my research project. The idea of the model is to understand which novel character said the requested quote. It achieves a bit better results on the ´test´ split of the NovelQSI dataset than base longformer-base-4096-finetuned-squadv2 model on the same dataset split.

Base model results:

{
  "exact_match": {
    "confidence_interval": [8.754452551305853, 14.718614718614718],
    "score": 12.121212121212121,
    "standard_error": 1.8579217243778676
  },
  "f1": {
    "confidence_interval": [18.469101076147584, 28.28409063313956],
    "score": 22.799422799422796,
    "standard_error": 2.896728175757627
  },
  "latency_in_seconds": 0.7730605573419919,
  "samples_per_second": 1.2935597224598967,
  "total_time_in_seconds": 178.5769887460001
}

Achieved results:

{
  "exact_match": {
    "confidence_interval": [16.017316017316016, 24.242424242424242],
    "score": 20.346320346320347,
    "standard_error": 2.9434375492784994
  },
  "f1": {
    "confidence_interval": [23.123469058324783, 31.823648733317036],
    "score": 26.580086580086572,
    "standard_error": 2.593030474995015
  },
  "latency_in_seconds": 0.8093855569913422,
  "samples_per_second": 1.235505120349827,
  "total_time_in_seconds": 186.96806366500005
}

The results have shown, that the technique has its future.

Training and evaluation data

You can find training code in the github repo of my research:

https://github.com/Kkordik/NovelQSI

It was trained and evaluated in notebooks, so it is easy to reproduce.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 8
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	93	3.0886
No log	1.99	186	3.3755
No log	2.99	279	2.9598

Framework versions

Transformers 4.35.2
Pytorch 2.1.0+cu118
Datasets 2.15.0
Tokenizers 0.15.0