{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "en_st_process.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "732abb94-a6d1-4c04-88dc-a26146ee0fb6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"st\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "abe1d75c-11e1-45d7-b4b5-4cf563b60501",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-st-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "754640ca-0a48-41b0-d03f-b04ac11c1921",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: opustools-pkg in /usr/local/lib/python3.6/dist-packages (0.0.52)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "outputId": "6d6ac4fa-be78-40f4-ad69-aa547d80ddc9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-st.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            "   7 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-st.xml.gz\n",
            " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
            " 108 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/st.zip\n",
            "\n",
            " 378 MB Total size\n",
            "./JW300_latest_xml_en-st.xml.gz ... 100% of 7 MB\n",
            "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
            "./JW300_latest_xml_st.zip ... 100% of 108 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zA3elQXi82jv",
        "colab_type": "code",
        "outputId": "d658a840-16e4-420d-f69d-0b516133565c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        }
      },
      "source": [
        "! head -n 5 jw300.$src"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unity in Worship ​ — What Should It Mean for You ?\n",
            "As we draw closer and closer to the end of this system , pressures and tensions mount .\n",
            "So it is more difficult , and yet more urgent than ever , to stay on the road leading to life , a road that requires unity in worship .\n",
            "The new book United in Worship of the Only True God lovingly helps us to help ourselves and others through this critical period .\n",
            "Please send me , postpaid , the 192 - page , hardbound , illustrated book United in Worship of the Only True God .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "outputId": "0beac0ac-b31c-4de6-b127-4e6c86fecb3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 617
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-01-23 07:59:12--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2020-01-23 07:59:12 (6.61 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2020-01-23 07:59:15--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-st.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 203793 (199K) [text/plain]\n",
            "Saving to: ‘test.en-st.en’\n",
            "\n",
            "test.en-st.en       100%[===================>] 199.02K  --.-KB/s    in 0.03s   \n",
            "\n",
            "2020-01-23 07:59:15 (6.42 MB/s) - ‘test.en-st.en’ saved [203793/203793]\n",
            "\n",
            "--2020-01-23 07:59:20--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-st.st\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 223469 (218K) [text/plain]\n",
            "Saving to: ‘test.en-st.st’\n",
            "\n",
            "test.en-st.st       100%[===================>] 218.23K  --.-KB/s    in 0.04s   \n",
            "\n",
            "2020-01-23 07:59:20 (5.20 MB/s) - ‘test.en-st.st’ saved [223469/223469]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "outputId": "54d396f8-44b9-4e6f-aec1-ee0d054d26d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "26624e30-7b63-4fc1-a4e6-05cadcf0cfa7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 7135/927465 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Unity in Worship ​ — What Should It Mean for Y...</td>\n",
              "      <td>Ho Kopana Borapeling — Ho Lokela ho Bolela Eng...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>As we draw closer and closer to the end of thi...</td>\n",
              "      <td>Ha re ntse re atamela bofelo ba tsamaiso ena ,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>So it is more difficult , and yet more urgent ...</td>\n",
              "      <td>Lintho li ba boima le ho feta , ’ me leha ho l...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0  Unity in Worship ​ — What Should It Mean for Y...  Ho Kopana Borapeling — Ho Lokela ho Bolela Eng...\n",
              "1  As we draw closer and closer to the end of thi...  Ha re ntse re atamela bofelo ba tsamaiso ena ,...\n",
              "2  So it is more difficult , and yet more urgent ...  Lintho li ba boima le ho feta , ’ me leha ho l..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "6c7e833e-ba00-413a-ccc8-268598fa8a99",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 199
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "outputId": "3c3116e1-c24d-4152-c9b9-2770c2ca7ffd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: fuzzywuzzy in /usr/local/lib/python3.6/dist-packages (0.17.0)\n",
            "Requirement already satisfied: python-Levenshtein in /usr/local/lib/python3.6/dist-packages (0.12.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (42.0.2)\n",
            "00:00:00.30 0.00 percent complete\n",
            "00:00:21.69 0.12 percent complete\n",
            "00:00:43.18 0.24 percent complete\n",
            "00:01:05.38 0.36 percent complete\n",
            "00:01:27.09 0.48 percent complete\n",
            "00:01:49.34 0.60 percent complete\n",
            "00:02:10.64 0.72 percent complete\n",
            "00:02:34.87 0.84 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:02:56.86 0.96 percent complete\n",
            "00:03:18.60 1.07 percent complete\n",
            "00:03:39.93 1.19 percent complete\n",
            "00:04:01.79 1.31 percent complete\n",
            "00:04:23.34 1.43 percent complete\n",
            "00:04:43.58 1.55 percent complete\n",
            "00:05:03.72 1.67 percent complete\n",
            "00:05:25.94 1.79 percent complete\n",
            "00:05:45.13 1.91 percent complete\n",
            "00:06:04.05 2.03 percent complete\n",
            "00:06:22.94 2.15 percent complete\n",
            "00:06:41.45 2.27 percent complete\n",
            "00:07:00.40 2.39 percent complete\n",
            "00:07:19.11 2.51 percent complete\n",
            "00:07:38.01 2.63 percent complete\n",
            "00:07:58.49 2.75 percent complete\n",
            "00:08:18.56 2.87 percent complete\n",
            "00:08:39.29 2.98 percent complete\n",
            "00:09:00.46 3.10 percent complete\n",
            "00:09:21.68 3.22 percent complete\n",
            "00:09:44.15 3.34 percent complete\n",
            "00:10:06.43 3.46 percent complete\n",
            "00:10:29.21 3.58 percent complete\n",
            "00:10:52.27 3.70 percent complete\n",
            "00:11:13.86 3.82 percent complete\n",
            "00:11:35.30 3.94 percent complete\n",
            "00:11:57.55 4.06 percent complete\n",
            "00:12:19.65 4.18 percent complete\n",
            "00:12:41.32 4.30 percent complete\n",
            "00:13:02.93 4.42 percent complete\n",
            "00:13:24.47 4.54 percent complete\n",
            "00:13:47.26 4.66 percent complete\n",
            "00:14:09.17 4.78 percent complete\n",
            "00:14:31.86 4.89 percent complete\n",
            "00:14:53.30 5.01 percent complete\n",
            "00:15:15.53 5.13 percent complete\n",
            "00:15:37.83 5.25 percent complete\n",
            "00:15:59.48 5.37 percent complete\n",
            "00:16:22.10 5.49 percent complete\n",
            "00:16:45.21 5.61 percent complete\n",
            "00:17:07.02 5.73 percent complete\n",
            "00:17:29.25 5.85 percent complete\n",
            "00:17:51.10 5.97 percent complete\n",
            "00:18:13.32 6.09 percent complete\n",
            "00:18:35.51 6.21 percent complete\n",
            "00:18:57.47 6.33 percent complete\n",
            "00:19:19.86 6.45 percent complete\n",
            "00:19:42.89 6.57 percent complete\n",
            "00:20:04.91 6.69 percent complete\n",
            "00:20:27.42 6.80 percent complete\n",
            "00:20:49.29 6.92 percent complete\n",
            "00:21:11.40 7.04 percent complete\n",
            "00:21:33.54 7.16 percent complete\n",
            "00:21:55.50 7.28 percent complete\n",
            "00:22:17.39 7.40 percent complete\n",
            "00:22:40.00 7.52 percent complete\n",
            "00:23:03.48 7.64 percent complete\n",
            "00:23:25.69 7.76 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:23:47.92 7.88 percent complete\n",
            "00:24:09.44 8.00 percent complete\n",
            "00:24:31.61 8.12 percent complete\n",
            "00:24:53.52 8.24 percent complete\n",
            "00:25:14.98 8.36 percent complete\n",
            "00:25:37.45 8.48 percent complete\n",
            "00:26:01.03 8.60 percent complete\n",
            "00:26:23.18 8.71 percent complete\n",
            "00:26:44.70 8.83 percent complete\n",
            "00:27:06.63 8.95 percent complete\n",
            "00:27:28.53 9.07 percent complete\n",
            "00:27:50.25 9.19 percent complete\n",
            "00:28:11.74 9.31 percent complete\n",
            "00:28:33.00 9.43 percent complete\n",
            "00:28:53.66 9.55 percent complete\n",
            "00:29:12.40 9.67 percent complete\n",
            "00:29:31.87 9.79 percent complete\n",
            "00:29:50.41 9.91 percent complete\n",
            "00:30:08.87 10.03 percent complete\n",
            "00:30:28.43 10.15 percent complete\n",
            "00:30:47.93 10.27 percent complete\n",
            "00:31:07.05 10.39 percent complete\n",
            "00:31:27.66 10.51 percent complete\n",
            "00:31:46.60 10.62 percent complete\n",
            "00:32:06.09 10.74 percent complete\n",
            "00:32:25.66 10.86 percent complete\n",
            "00:32:46.61 10.98 percent complete\n",
            "00:33:08.45 11.10 percent complete\n",
            "00:33:29.97 11.22 percent complete\n",
            "00:33:51.07 11.34 percent complete\n",
            "00:34:13.71 11.46 percent complete\n",
            "00:34:35.69 11.58 percent complete\n",
            "00:34:57.50 11.70 percent complete\n",
            "00:35:19.10 11.82 percent complete\n",
            "00:35:41.25 11.94 percent complete\n",
            "00:36:03.18 12.06 percent complete\n",
            "00:36:25.85 12.18 percent complete\n",
            "00:36:48.51 12.30 percent complete\n",
            "00:37:12.35 12.42 percent complete\n",
            "00:37:34.28 12.54 percent complete\n",
            "00:37:56.20 12.65 percent complete\n",
            "00:38:18.36 12.77 percent complete\n",
            "00:38:40.50 12.89 percent complete\n",
            "00:39:02.48 13.01 percent complete\n",
            "00:39:24.53 13.13 percent complete\n",
            "00:39:46.10 13.25 percent complete\n",
            "00:40:09.69 13.37 percent complete\n",
            "00:40:32.00 13.49 percent complete\n",
            "00:40:53.40 13.61 percent complete\n",
            "00:41:15.16 13.73 percent complete\n",
            "00:41:37.39 13.85 percent complete\n",
            "00:41:58.70 13.97 percent complete\n",
            "00:42:20.97 14.09 percent complete\n",
            "00:42:43.03 14.21 percent complete\n",
            "00:43:06.70 14.33 percent complete\n",
            "00:43:28.97 14.45 percent complete\n",
            "00:43:50.71 14.56 percent complete\n",
            "00:44:12.26 14.68 percent complete\n",
            "00:44:34.56 14.80 percent complete\n",
            "00:44:56.13 14.92 percent complete\n",
            "00:45:17.75 15.04 percent complete\n",
            "00:45:39.26 15.16 percent complete\n",
            "00:46:02.60 15.28 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:46:23.91 15.40 percent complete\n",
            "00:46:46.15 15.52 percent complete\n",
            "00:47:08.22 15.64 percent complete\n",
            "00:47:30.36 15.76 percent complete\n",
            "00:47:52.72 15.88 percent complete\n",
            "00:48:15.04 16.00 percent complete\n",
            "00:48:37.87 16.12 percent complete\n",
            "00:48:59.92 16.24 percent complete\n",
            "00:49:22.13 16.36 percent complete\n",
            "00:49:44.19 16.47 percent complete\n",
            "00:50:05.34 16.59 percent complete\n",
            "00:50:26.90 16.71 percent complete\n",
            "00:50:49.00 16.83 percent complete\n",
            "00:51:10.77 16.95 percent complete\n",
            "00:51:32.95 17.07 percent complete\n",
            "00:51:54.98 17.19 percent complete\n",
            "00:52:18.59 17.31 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:52:40.71 17.43 percent complete\n",
            "00:53:02.29 17.55 percent complete\n",
            "00:53:23.95 17.67 percent complete\n",
            "00:53:46.47 17.79 percent complete\n",
            "00:54:08.48 17.91 percent complete\n",
            "00:54:30.33 18.03 percent complete\n",
            "00:54:52.72 18.15 percent complete\n",
            "00:55:17.05 18.27 percent complete\n",
            "00:55:39.08 18.38 percent complete\n",
            "00:56:00.76 18.50 percent complete\n",
            "00:56:22.82 18.62 percent complete\n",
            "00:56:44.95 18.74 percent complete\n",
            "00:57:06.81 18.86 percent complete\n",
            "00:57:28.90 18.98 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:57:51.12 19.10 percent complete\n",
            "00:58:14.80 19.22 percent complete\n",
            "00:58:37.40 19.34 percent complete\n",
            "00:58:57.70 19.46 percent complete\n",
            "00:59:17.86 19.58 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '$ $ $']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:59:37.26 19.70 percent complete\n",
            "00:59:56.89 19.82 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→ → →']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:00:17.07 19.94 percent complete\n",
            "01:00:37.84 20.06 percent complete\n",
            "01:01:01.70 20.18 percent complete\n",
            "01:01:24.28 20.29 percent complete\n",
            "01:01:45.82 20.41 percent complete\n",
            "01:02:06.80 20.53 percent complete\n",
            "01:02:28.87 20.65 percent complete\n",
            "01:02:50.91 20.77 percent complete\n",
            "01:03:13.00 20.89 percent complete\n",
            "01:03:35.45 21.01 percent complete\n",
            "01:03:58.42 21.13 percent complete\n",
            "01:04:20.36 21.25 percent complete\n",
            "01:04:42.13 21.37 percent complete\n",
            "01:05:03.77 21.49 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '\\']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:05:25.88 21.61 percent complete\n",
            "01:05:45.69 21.73 percent complete\n",
            "01:06:04.49 21.85 percent complete\n",
            "01:06:23.08 21.97 percent complete\n",
            "01:06:42.35 22.09 percent complete\n",
            "01:07:00.64 22.20 percent complete\n",
            "01:07:20.08 22.32 percent complete\n",
            "01:07:39.29 22.44 percent complete\n",
            "01:07:58.13 22.56 percent complete\n",
            "01:08:18.11 22.68 percent complete\n",
            "01:08:37.36 22.80 percent complete\n",
            "01:08:57.10 22.92 percent complete\n",
            "01:09:18.48 23.04 percent complete\n",
            "01:09:38.42 23.16 percent complete\n",
            "01:09:57.95 23.28 percent complete\n",
            "01:10:18.01 23.40 percent complete\n",
            "01:10:37.67 23.52 percent complete\n",
            "01:10:56.80 23.64 percent complete\n",
            "01:11:16.21 23.76 percent complete\n",
            "01:11:35.71 23.88 percent complete\n",
            "01:11:56.76 24.00 percent complete\n",
            "01:12:16.72 24.11 percent complete\n",
            "01:12:36.74 24.23 percent complete\n",
            "01:12:56.71 24.35 percent complete\n",
            "01:13:16.84 24.47 percent complete\n",
            "01:13:36.28 24.59 percent complete\n",
            "01:13:55.93 24.71 percent complete\n",
            "01:14:15.18 24.83 percent complete\n",
            "01:14:36.60 24.95 percent complete\n",
            "01:14:56.15 25.07 percent complete\n",
            "01:15:15.91 25.19 percent complete\n",
            "01:15:35.93 25.31 percent complete\n",
            "01:15:55.39 25.43 percent complete\n",
            "01:16:15.04 25.55 percent complete\n",
            "01:16:34.79 25.67 percent complete\n",
            "01:16:53.94 25.79 percent complete\n",
            "01:17:14.98 25.91 percent complete\n",
            "01:17:34.65 26.03 percent complete\n",
            "01:17:54.16 26.14 percent complete\n",
            "01:18:13.34 26.26 percent complete\n",
            "01:18:33.33 26.38 percent complete\n",
            "01:18:52.86 26.50 percent complete\n",
            "01:19:12.45 26.62 percent complete\n",
            "01:19:32.14 26.74 percent complete\n",
            "01:19:52.67 26.86 percent complete\n",
            "01:20:13.16 26.98 percent complete\n",
            "01:20:32.81 27.10 percent complete\n",
            "01:20:53.27 27.22 percent complete\n",
            "01:21:13.45 27.34 percent complete\n",
            "01:21:33.59 27.46 percent complete\n",
            "01:21:53.12 27.58 percent complete\n",
            "01:22:13.16 27.70 percent complete\n",
            "01:22:33.19 27.82 percent complete\n",
            "01:22:54.18 27.94 percent complete\n",
            "01:23:13.49 28.05 percent complete\n",
            "01:23:33.38 28.17 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '— ― ― ― ― ― ― ―']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:23:52.76 28.29 percent complete\n",
            "01:24:12.79 28.41 percent complete\n",
            "01:24:32.59 28.53 percent complete\n",
            "01:24:51.90 28.65 percent complete\n",
            "01:25:11.41 28.77 percent complete\n",
            "01:25:32.52 28.89 percent complete\n",
            "01:25:51.79 29.01 percent complete\n",
            "01:26:11.09 29.13 percent complete\n",
            "01:26:30.60 29.25 percent complete\n",
            "01:26:49.85 29.37 percent complete\n",
            "01:27:09.42 29.49 percent complete\n",
            "01:27:29.33 29.61 percent complete\n",
            "01:27:48.22 29.73 percent complete\n",
            "01:28:08.74 29.85 percent complete\n",
            "01:28:29.14 29.96 percent complete\n",
            "01:28:48.60 30.08 percent complete\n",
            "01:29:07.67 30.20 percent complete\n",
            "01:29:27.67 30.32 percent complete\n",
            "01:29:47.33 30.44 percent complete\n",
            "01:30:07.54 30.56 percent complete\n",
            "01:30:27.35 30.68 percent complete\n",
            "01:30:47.56 30.80 percent complete\n",
            "01:31:06.84 30.92 percent complete\n",
            "01:31:26.45 31.04 percent complete\n",
            "01:31:45.78 31.16 percent complete\n",
            "01:32:05.37 31.28 percent complete\n",
            "01:32:25.58 31.40 percent complete\n",
            "01:32:44.88 31.52 percent complete\n",
            "01:33:04.34 31.64 percent complete\n",
            "01:33:25.30 31.76 percent complete\n",
            "01:33:44.92 31.87 percent complete\n",
            "01:34:05.09 31.99 percent complete\n",
            "01:34:26.28 32.11 percent complete\n",
            "01:34:46.30 32.23 percent complete\n",
            "01:35:06.94 32.35 percent complete\n",
            "01:35:26.49 32.47 percent complete\n",
            "01:35:45.97 32.59 percent complete\n",
            "01:36:06.46 32.71 percent complete\n",
            "01:36:26.17 32.83 percent complete\n",
            "01:36:45.25 32.95 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇩']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:37:04.85 33.07 percent complete\n",
            "01:37:24.81 33.19 percent complete\n",
            "01:37:43.92 33.31 percent complete\n",
            "01:38:03.39 33.43 percent complete\n",
            "01:38:23.23 33.55 percent complete\n",
            "01:38:44.82 33.67 percent complete\n",
            "01:39:04.24 33.78 percent complete\n",
            "01:39:23.80 33.90 percent complete\n",
            "01:39:43.89 34.02 percent complete\n",
            "01:40:03.68 34.14 percent complete\n",
            "01:40:23.62 34.26 percent complete\n",
            "01:40:44.15 34.38 percent complete\n",
            "01:41:03.05 34.50 percent complete\n",
            "01:41:24.47 34.62 percent complete\n",
            "01:41:43.76 34.74 percent complete\n",
            "01:42:03.72 34.86 percent complete\n",
            "01:42:23.21 34.98 percent complete\n",
            "01:42:42.58 35.10 percent complete\n",
            "01:43:02.38 35.22 percent complete\n",
            "01:43:22.43 35.34 percent complete\n",
            "01:43:41.25 35.46 percent complete\n",
            "01:44:02.15 35.58 percent complete\n",
            "01:44:21.68 35.69 percent complete\n",
            "01:44:41.80 35.81 percent complete\n",
            "01:45:01.34 35.93 percent complete\n",
            "01:45:20.56 36.05 percent complete\n",
            "01:45:40.14 36.17 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ◯']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:45:59.59 36.29 percent complete\n",
            "01:46:19.62 36.41 percent complete\n",
            "01:46:39.65 36.53 percent complete\n",
            "01:47:00.14 36.65 percent complete\n",
            "01:47:20.14 36.77 percent complete\n",
            "01:47:40.15 36.89 percent complete\n",
            "01:48:00.83 37.01 percent complete\n",
            "01:48:21.66 37.13 percent complete\n",
            "01:48:42.05 37.25 percent complete\n",
            "01:49:02.11 37.37 percent complete\n",
            "01:49:22.70 37.49 percent complete\n",
            "01:49:42.97 37.61 percent complete\n",
            "01:50:02.71 37.72 percent complete\n",
            "01:50:22.71 37.84 percent complete\n",
            "01:50:42.52 37.96 percent complete\n",
            "01:51:02.16 38.08 percent complete\n",
            "01:51:21.80 38.20 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓ ↓ ↓']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "01:51:41.34 38.32 percent complete\n",
            "01:52:00.55 38.44 percent complete\n",
            "01:52:21.05 38.56 percent complete\n",
            "01:52:40.62 38.68 percent complete\n",
            "01:52:59.78 38.80 percent complete\n",
            "01:53:18.53 38.92 percent complete\n",
            "01:53:38.43 39.04 percent complete\n",
            "01:53:57.27 39.16 percent complete\n",
            "01:54:17.16 39.28 percent complete\n",
            "01:54:36.39 39.40 percent complete\n",
            "01:54:57.79 39.52 percent complete\n",
            "01:55:17.28 39.63 percent complete\n",
            "01:55:36.95 39.75 percent complete\n",
            "01:55:56.70 39.87 percent complete\n",
            "01:56:16.11 39.99 percent complete\n",
            "01:56:36.16 40.11 percent complete\n",
            "01:56:55.51 40.23 percent complete\n",
            "01:57:14.57 40.35 percent complete\n",
            "01:57:35.78 40.47 percent complete\n",
            "01:57:54.87 40.59 percent complete\n",
            "01:58:14.04 40.71 percent complete\n",
            "01:58:33.49 40.83 percent complete\n",
            "01:58:52.80 40.95 percent complete\n",
            "01:59:12.59 41.07 percent complete\n",
            "01:59:32.18 41.19 percent complete\n",
            "01:59:51.05 41.31 percent complete\n",
            "02:00:11.51 41.43 percent complete\n",
            "02:00:30.84 41.54 percent complete\n",
            "02:00:50.02 41.66 percent complete\n",
            "02:01:10.36 41.78 percent complete\n",
            "02:01:31.82 41.90 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ← ← ← ← ← ← ← ← ←']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:01:51.75 42.02 percent complete\n",
            "02:02:12.57 42.14 percent complete\n",
            "02:02:32.20 42.26 percent complete\n",
            "02:02:53.68 42.38 percent complete\n",
            "02:03:13.27 42.50 percent complete\n",
            "02:03:33.61 42.62 percent complete\n",
            "02:03:52.69 42.74 percent complete\n",
            "02:04:12.32 42.86 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:04:31.89 42.98 percent complete\n",
            "02:04:51.24 43.10 percent complete\n",
            "02:05:11.13 43.22 percent complete\n",
            "02:05:32.56 43.34 percent complete\n",
            "02:05:52.21 43.45 percent complete\n",
            "02:06:11.97 43.57 percent complete\n",
            "02:06:31.44 43.69 percent complete\n",
            "02:06:50.79 43.81 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '’']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:07:10.91 43.93 percent complete\n",
            "02:07:30.59 44.05 percent complete\n",
            "02:07:49.77 44.17 percent complete\n",
            "02:08:10.27 44.29 percent complete\n",
            "02:08:29.85 44.41 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '” *']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:08:49.65 44.53 percent complete\n",
            "02:09:09.56 44.65 percent complete\n",
            "02:09:28.94 44.77 percent complete\n",
            "02:09:48.98 44.89 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '. .']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:10:09.08 45.01 percent complete\n",
            "02:10:28.39 45.13 percent complete\n",
            "02:10:48.62 45.25 percent complete\n",
            "02:11:08.20 45.36 percent complete\n",
            "02:11:28.40 45.48 percent complete\n",
            "02:11:48.02 45.60 percent complete\n",
            "02:12:06.81 45.72 percent complete\n",
            "02:12:26.35 45.84 percent complete\n",
            "02:12:45.59 45.96 percent complete\n",
            "02:13:04.69 46.08 percent complete\n",
            "02:13:24.47 46.20 percent complete\n",
            "02:13:43.88 46.32 percent complete\n",
            "02:14:03.36 46.44 percent complete\n",
            "02:14:23.02 46.56 percent complete\n",
            "02:14:43.01 46.68 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '$ ․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:15:04.00 46.80 percent complete\n",
            "02:15:24.58 46.92 percent complete\n",
            "02:15:45.83 47.04 percent complete\n",
            "02:16:06.24 47.16 percent complete\n",
            "02:16:27.68 47.27 percent complete\n",
            "02:16:47.58 47.39 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '⇧']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:17:07.15 47.51 percent complete\n",
            "02:17:26.60 47.63 percent complete\n",
            "02:17:45.70 47.75 percent complete\n",
            "02:18:05.30 47.87 percent complete\n",
            "02:18:25.11 47.99 percent complete\n",
            "02:18:44.30 48.11 percent complete\n",
            "02:19:06.07 48.23 percent complete\n",
            "02:19:25.46 48.35 percent complete\n",
            "02:19:44.72 48.47 percent complete\n",
            "02:20:04.25 48.59 percent complete\n",
            "02:20:24.07 48.71 percent complete\n",
            "02:20:43.50 48.83 percent complete\n",
            "02:21:02.79 48.95 percent complete\n",
            "02:21:22.84 49.07 percent complete\n",
            "02:21:43.57 49.18 percent complete\n",
            "02:22:03.57 49.30 percent complete\n",
            "02:22:23.10 49.42 percent complete\n",
            "02:22:42.37 49.54 percent complete\n",
            "02:23:02.11 49.66 percent complete\n",
            "02:23:22.29 49.78 percent complete\n",
            "02:23:42.04 49.90 percent complete\n",
            "02:24:00.99 50.02 percent complete\n",
            "02:24:22.91 50.14 percent complete\n",
            "02:24:42.25 50.26 percent complete\n",
            "02:25:01.85 50.38 percent complete\n",
            "02:25:21.74 50.50 percent complete\n",
            "02:25:41.09 50.62 percent complete\n",
            "02:26:00.43 50.74 percent complete\n",
            "02:26:19.43 50.86 percent complete\n",
            "02:26:39.22 50.98 percent complete\n",
            "02:26:59.22 51.10 percent complete\n",
            "02:27:18.94 51.21 percent complete\n",
            "02:27:38.96 51.33 percent complete\n",
            "02:27:58.35 51.45 percent complete\n",
            "02:28:17.95 51.57 percent complete\n",
            "02:28:38.39 51.69 percent complete\n",
            "02:28:58.81 51.81 percent complete\n",
            "02:29:19.53 51.93 percent complete\n",
            "02:29:40.17 52.05 percent complete\n",
            "02:29:59.34 52.17 percent complete\n",
            "02:30:18.67 52.29 percent complete\n",
            "02:30:37.94 52.41 percent complete\n",
            "02:30:57.52 52.53 percent complete\n",
            "02:31:17.75 52.65 percent complete\n",
            "02:31:37.52 52.77 percent complete\n",
            "02:31:57.12 52.89 percent complete\n",
            "02:32:18.07 53.01 percent complete\n",
            "02:32:37.20 53.12 percent complete\n",
            "02:32:56.75 53.24 percent complete\n",
            "02:33:16.43 53.36 percent complete\n",
            "02:33:35.38 53.48 percent complete\n",
            "02:33:55.15 53.60 percent complete\n",
            "02:34:14.40 53.72 percent complete\n",
            "02:34:34.39 53.84 percent complete\n",
            "02:34:55.23 53.96 percent complete\n",
            "02:35:14.91 54.08 percent complete\n",
            "02:35:34.82 54.20 percent complete\n",
            "02:35:53.57 54.32 percent complete\n",
            "02:36:12.81 54.44 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓ ↓']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:36:32.76 54.56 percent complete\n",
            "02:36:51.99 54.68 percent complete\n",
            "02:37:11.72 54.80 percent complete\n",
            "02:37:32.51 54.92 percent complete\n",
            "02:37:51.84 55.03 percent complete\n",
            "02:38:11.80 55.15 percent complete\n",
            "02:38:31.03 55.27 percent complete\n",
            "02:38:49.93 55.39 percent complete\n",
            "02:39:09.13 55.51 percent complete\n",
            "02:39:28.56 55.63 percent complete\n",
            "02:39:47.58 55.75 percent complete\n",
            "02:40:08.35 55.87 percent complete\n",
            "02:40:27.68 55.99 percent complete\n",
            "02:40:47.00 56.11 percent complete\n",
            "02:41:06.03 56.23 percent complete\n",
            "02:41:25.39 56.35 percent complete\n",
            "02:41:45.26 56.47 percent complete\n",
            "02:42:05.54 56.59 percent complete\n",
            "02:42:26.43 56.71 percent complete\n",
            "02:42:46.28 56.83 percent complete\n",
            "02:43:05.52 56.94 percent complete\n",
            "02:43:24.77 57.06 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '* * *']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:43:43.85 57.18 percent complete\n",
            "02:44:02.81 57.30 percent complete\n",
            "02:44:22.25 57.42 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '▴']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:44:41.95 57.54 percent complete\n",
            "02:45:01.56 57.66 percent complete\n",
            "02:45:21.16 57.78 percent complete\n",
            "02:45:42.78 57.90 percent complete\n",
            "02:46:02.27 58.02 percent complete\n",
            "02:46:22.30 58.14 percent complete\n",
            "02:46:41.58 58.26 percent complete\n",
            "02:47:01.83 58.38 percent complete\n",
            "02:47:21.69 58.50 percent complete\n",
            "02:47:42.69 58.62 percent complete\n",
            "02:48:02.34 58.74 percent complete\n",
            "02:48:22.96 58.85 percent complete\n",
            "02:48:42.70 58.97 percent complete\n",
            "02:49:02.65 59.09 percent complete\n",
            "02:49:21.98 59.21 percent complete\n",
            "02:49:41.37 59.33 percent complete\n",
            "02:50:00.45 59.45 percent complete\n",
            "02:50:19.61 59.57 percent complete\n",
            "02:50:39.45 59.69 percent complete\n",
            "02:51:00.51 59.81 percent complete\n",
            "02:51:20.69 59.93 percent complete\n",
            "02:51:40.59 60.05 percent complete\n",
            "02:51:59.55 60.17 percent complete\n",
            "02:52:19.26 60.29 percent complete\n",
            "02:52:38.82 60.41 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "02:52:59.13 60.53 percent complete\n",
            "02:53:19.16 60.65 percent complete\n",
            "02:53:40.03 60.76 percent complete\n",
            "02:53:59.53 60.88 percent complete\n",
            "02:54:19.01 61.00 percent complete\n",
            "02:54:38.92 61.12 percent complete\n",
            "02:54:59.89 61.24 percent complete\n",
            "02:55:20.85 61.36 percent complete\n",
            "02:55:40.88 61.48 percent complete\n",
            "02:56:00.30 61.60 percent complete\n",
            "02:56:22.02 61.72 percent complete\n",
            "02:56:41.08 61.84 percent complete\n",
            "02:57:00.38 61.96 percent complete\n",
            "02:57:20.16 62.08 percent complete\n",
            "02:57:39.78 62.20 percent complete\n",
            "02:57:58.69 62.32 percent complete\n",
            "02:58:18.59 62.44 percent complete\n",
            "02:58:38.50 62.56 percent complete\n",
            "02:58:57.94 62.68 percent complete\n",
            "02:59:17.37 62.79 percent complete\n",
            "02:59:36.41 62.91 percent complete\n",
            "02:59:55.57 63.03 percent complete\n",
            "03:00:15.03 63.15 percent complete\n",
            "03:00:34.29 63.27 percent complete\n",
            "03:00:54.02 63.39 percent complete\n",
            "03:01:13.30 63.51 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '● ● ● ● ●']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:01:34.01 63.63 percent complete\n",
            "03:01:52.99 63.75 percent complete\n",
            "03:02:12.64 63.87 percent complete\n",
            "03:02:31.72 63.99 percent complete\n",
            "03:02:50.95 64.11 percent complete\n",
            "03:03:10.72 64.23 percent complete\n",
            "03:03:30.30 64.35 percent complete\n",
            "03:03:49.94 64.47 percent complete\n",
            "03:04:10.79 64.59 percent complete\n",
            "03:04:30.27 64.70 percent complete\n",
            "03:04:50.38 64.82 percent complete\n",
            "03:05:10.42 64.94 percent complete\n",
            "03:05:31.44 65.06 percent complete\n",
            "03:05:52.09 65.18 percent complete\n",
            "03:06:12.43 65.30 percent complete\n",
            "03:06:32.29 65.42 percent complete\n",
            "03:06:52.08 65.54 percent complete\n",
            "03:07:11.27 65.66 percent complete\n",
            "03:07:30.90 65.78 percent complete\n",
            "03:07:51.10 65.90 percent complete\n",
            "03:08:12.00 66.02 percent complete\n",
            "03:08:32.12 66.14 percent complete\n",
            "03:08:51.87 66.26 percent complete\n",
            "03:09:11.89 66.38 percent complete\n",
            "03:09:31.56 66.50 percent complete\n",
            "03:09:51.29 66.61 percent complete\n",
            "03:10:10.69 66.73 percent complete\n",
            "03:10:30.81 66.85 percent complete\n",
            "03:10:50.62 66.97 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '“ . . .']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:11:09.80 67.09 percent complete\n",
            "03:11:29.85 67.21 percent complete\n",
            "03:11:49.75 67.33 percent complete\n",
            "03:12:08.75 67.45 percent complete\n",
            "03:12:29.54 67.57 percent complete\n",
            "03:12:49.10 67.69 percent complete\n",
            "03:13:08.60 67.81 percent complete\n",
            "03:13:29.16 67.93 percent complete\n",
            "03:13:49.08 68.05 percent complete\n",
            "03:14:08.66 68.17 percent complete\n",
            "03:14:28.67 68.29 percent complete\n",
            "03:14:48.30 68.41 percent complete\n",
            "03:15:09.37 68.52 percent complete\n",
            "03:15:29.09 68.64 percent complete\n",
            "03:15:47.92 68.76 percent complete\n",
            "03:16:06.62 68.88 percent complete\n",
            "03:16:26.25 69.00 percent complete\n",
            "03:16:45.71 69.12 percent complete\n",
            "03:17:06.09 69.24 percent complete\n",
            "03:17:25.68 69.36 percent complete\n",
            "03:17:46.47 69.48 percent complete\n",
            "03:18:05.85 69.60 percent complete\n",
            "03:18:25.83 69.72 percent complete\n",
            "03:18:45.63 69.84 percent complete\n",
            "03:19:04.56 69.96 percent complete\n",
            "03:19:24.83 70.08 percent complete\n",
            "03:19:44.80 70.20 percent complete\n",
            "03:20:04.76 70.32 percent complete\n",
            "03:20:26.97 70.43 percent complete\n",
            "03:20:47.28 70.55 percent complete\n",
            "03:21:07.43 70.67 percent complete\n",
            "03:21:27.12 70.79 percent complete\n",
            "03:21:46.76 70.91 percent complete\n",
            "03:22:06.15 71.03 percent complete\n",
            "03:22:26.78 71.15 percent complete\n",
            "03:22:48.22 71.27 percent complete\n",
            "03:23:09.77 71.39 percent complete\n",
            "03:23:30.36 71.51 percent complete\n",
            "03:23:50.92 71.63 percent complete\n",
            "03:24:11.19 71.75 percent complete\n",
            "03:24:31.56 71.87 percent complete\n",
            "03:24:51.44 71.99 percent complete\n",
            "03:25:10.47 72.11 percent complete\n",
            "03:25:30.42 72.23 percent complete\n",
            "03:25:51.06 72.34 percent complete\n",
            "03:26:10.10 72.46 percent complete\n",
            "03:26:29.95 72.58 percent complete\n",
            "03:26:49.15 72.70 percent complete\n",
            "03:27:08.50 72.82 percent complete\n",
            "03:27:28.81 72.94 percent complete\n",
            "03:27:48.55 73.06 percent complete\n",
            "03:28:07.77 73.18 percent complete\n",
            "03:28:29.05 73.30 percent complete\n",
            "03:28:48.53 73.42 percent complete\n",
            "03:29:07.95 73.54 percent complete\n",
            "03:29:27.16 73.66 percent complete\n",
            "03:29:46.83 73.78 percent complete\n",
            "03:30:06.55 73.90 percent complete\n",
            "03:30:26.98 74.02 percent complete\n",
            "03:30:46.90 74.14 percent complete\n",
            "03:31:08.30 74.26 percent complete\n",
            "03:31:28.36 74.37 percent complete\n",
            "03:31:48.14 74.49 percent complete\n",
            "03:32:07.72 74.61 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:32:26.70 74.73 percent complete\n",
            "03:32:46.84 74.85 percent complete\n",
            "03:33:06.98 74.97 percent complete\n",
            "03:33:27.78 75.09 percent complete\n",
            "03:33:48.38 75.21 percent complete\n",
            "03:34:07.87 75.33 percent complete\n",
            "03:34:28.25 75.45 percent complete\n",
            "03:34:47.47 75.57 percent complete\n",
            "03:35:06.70 75.69 percent complete\n",
            "03:35:26.33 75.81 percent complete\n",
            "03:35:46.20 75.93 percent complete\n",
            "03:36:05.57 76.05 percent complete\n",
            "03:36:25.72 76.17 percent complete\n",
            "03:36:45.44 76.28 percent complete\n",
            "03:37:04.88 76.40 percent complete\n",
            "03:37:24.56 76.52 percent complete\n",
            "03:37:44.78 76.64 percent complete\n",
            "03:38:03.51 76.76 percent complete\n",
            "03:38:22.75 76.88 percent complete\n",
            "03:38:43.44 77.00 percent complete\n",
            "03:39:02.67 77.12 percent complete\n",
            "03:39:24.05 77.24 percent complete\n",
            "03:39:43.29 77.36 percent complete\n",
            "03:40:03.97 77.48 percent complete\n",
            "03:40:23.78 77.60 percent complete\n",
            "03:40:43.90 77.72 percent complete\n",
            "03:41:04.43 77.84 percent complete\n",
            "03:41:25.13 77.96 percent complete\n",
            "03:41:44.97 78.08 percent complete\n",
            "03:42:05.86 78.19 percent complete\n",
            "03:42:25.61 78.31 percent complete\n",
            "03:42:45.04 78.43 percent complete\n",
            "03:43:04.04 78.55 percent complete\n",
            "03:43:23.98 78.67 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '→ →']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:43:44.32 78.79 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↑']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:44:03.77 78.91 percent complete\n",
            "03:44:23.43 79.03 percent complete\n",
            "03:44:44.34 79.15 percent complete\n",
            "03:45:03.77 79.27 percent complete\n",
            "03:45:23.03 79.39 percent complete\n",
            "03:45:42.27 79.51 percent complete\n",
            "03:46:01.75 79.63 percent complete\n",
            "03:46:21.55 79.75 percent complete\n",
            "03:46:40.79 79.87 percent complete\n",
            "03:46:59.87 79.99 percent complete\n",
            "03:47:20.65 80.10 percent complete\n",
            "03:47:40.33 80.22 percent complete\n",
            "03:48:00.60 80.34 percent complete\n",
            "03:48:20.19 80.46 percent complete\n",
            "03:48:40.36 80.58 percent complete\n",
            "03:49:00.12 80.70 percent complete\n",
            "03:49:19.35 80.82 percent complete\n",
            "03:49:39.46 80.94 percent complete\n",
            "03:49:59.04 81.06 percent complete\n",
            "03:50:18.22 81.18 percent complete\n",
            "03:50:37.77 81.30 percent complete\n",
            "03:50:58.01 81.42 percent complete\n",
            "03:51:17.33 81.54 percent complete\n",
            "03:51:36.70 81.66 percent complete\n",
            "03:51:55.78 81.78 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '”']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:52:15.30 81.90 percent complete\n",
            "03:52:35.20 82.01 percent complete\n",
            "03:52:54.55 82.13 percent complete\n",
            "03:53:13.41 82.25 percent complete\n",
            "03:53:32.67 82.37 percent complete\n",
            "03:53:51.30 82.49 percent complete\n",
            "03:54:11.39 82.61 percent complete\n",
            "03:54:30.93 82.73 percent complete\n",
            "03:54:51.75 82.85 percent complete\n",
            "03:55:13.41 82.97 percent complete\n",
            "03:55:33.58 83.09 percent complete\n",
            "03:55:53.17 83.21 percent complete\n",
            "03:56:12.62 83.33 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '” ?']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:56:32.26 83.45 percent complete\n",
            "03:56:51.79 83.57 percent complete\n",
            "03:57:11.90 83.69 percent complete\n",
            "03:57:31.65 83.81 percent complete\n",
            "03:57:52.53 83.92 percent complete\n",
            "03:58:11.88 84.04 percent complete\n",
            "03:58:31.24 84.16 percent complete\n",
            "03:58:51.96 84.28 percent complete\n",
            "03:59:11.33 84.40 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '●']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "03:59:31.02 84.52 percent complete\n",
            "03:59:50.02 84.64 percent complete\n",
            "04:00:09.71 84.76 percent complete\n",
            "04:00:29.74 84.88 percent complete\n",
            "04:00:48.89 85.00 percent complete\n",
            "04:01:08.38 85.12 percent complete\n",
            "04:01:27.53 85.24 percent complete\n",
            "04:01:47.94 85.36 percent complete\n",
            "04:02:07.81 85.48 percent complete\n",
            "04:02:27.32 85.60 percent complete\n",
            "04:02:46.97 85.72 percent complete\n",
            "04:03:06.65 85.83 percent complete\n",
            "04:03:27.70 85.95 percent complete\n",
            "04:03:47.99 86.07 percent complete\n",
            "04:04:07.59 86.19 percent complete\n",
            "04:04:27.67 86.31 percent complete\n",
            "04:04:47.42 86.43 percent complete\n",
            "04:05:06.81 86.55 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '□ ․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "04:05:26.22 86.67 percent complete\n",
            "04:05:45.78 86.79 percent complete\n",
            "04:06:06.11 86.91 percent complete\n",
            "04:06:25.18 87.03 percent complete\n",
            "04:06:44.43 87.15 percent complete\n",
            "04:07:03.68 87.27 percent complete\n",
            "04:07:23.25 87.39 percent complete\n",
            "04:07:42.77 87.51 percent complete\n",
            "04:08:01.57 87.63 percent complete\n",
            "04:08:20.77 87.75 percent complete\n",
            "04:08:41.57 87.86 percent complete\n",
            "04:09:00.61 87.98 percent complete\n",
            "04:09:20.18 88.10 percent complete\n",
            "04:09:40.53 88.22 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '․ ․ ․ ․ ․ ․ ․ ․']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "04:10:00.93 88.34 percent complete\n",
            "04:10:21.56 88.46 percent complete\n",
            "04:10:42.34 88.58 percent complete\n",
            "04:11:02.69 88.70 percent complete\n",
            "04:11:25.70 88.82 percent complete\n",
            "04:11:47.07 88.94 percent complete\n",
            "04:12:06.90 89.06 percent complete\n",
            "04:12:26.67 89.18 percent complete\n",
            "04:12:46.21 89.30 percent complete\n",
            "04:13:05.99 89.42 percent complete\n",
            "04:13:25.88 89.54 percent complete\n",
            "04:13:45.80 89.66 percent complete\n",
            "04:14:06.67 89.77 percent complete\n",
            "04:14:26.45 89.89 percent complete\n",
            "04:14:46.81 90.01 percent complete\n",
            "04:15:06.71 90.13 percent complete\n",
            "04:15:26.34 90.25 percent complete\n",
            "04:15:46.29 90.37 percent complete\n",
            "04:16:05.73 90.49 percent complete\n",
            "04:16:25.47 90.61 percent complete\n",
            "04:16:46.51 90.73 percent complete\n",
            "04:17:05.38 90.85 percent complete\n",
            "04:17:25.29 90.97 percent complete\n",
            "04:17:44.40 91.09 percent complete\n",
            "04:18:03.82 91.21 percent complete\n",
            "04:18:23.56 91.33 percent complete\n",
            "04:18:42.84 91.45 percent complete\n",
            "04:19:02.21 91.57 percent complete\n",
            "04:19:22.98 91.68 percent complete\n",
            "04:19:43.12 91.80 percent complete\n",
            "04:20:02.68 91.92 percent complete\n",
            "04:20:22.91 92.04 percent complete\n",
            "04:20:42.58 92.16 percent complete\n",
            "04:21:02.19 92.28 percent complete\n",
            "04:21:21.89 92.40 percent complete\n",
            "04:21:41.44 92.52 percent complete\n",
            "04:22:01.64 92.64 percent complete\n",
            "04:22:21.20 92.76 percent complete\n",
            "04:22:41.45 92.88 percent complete\n",
            "04:23:01.23 93.00 percent complete\n",
            "04:23:20.89 93.12 percent complete\n",
            "04:23:40.09 93.24 percent complete\n",
            "04:23:59.66 93.36 percent complete\n",
            "04:24:20.74 93.48 percent complete\n",
            "04:24:42.10 93.59 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '←']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "04:25:02.19 93.71 percent complete\n",
            "04:25:22.20 93.83 percent complete\n",
            "04:25:41.65 93.95 percent complete\n",
            "04:26:01.42 94.07 percent complete\n",
            "04:26:21.07 94.19 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '← ←']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "04:26:40.36 94.31 percent complete\n",
            "04:26:59.27 94.43 percent complete\n",
            "04:27:19.39 94.55 percent complete\n",
            "04:27:39.37 94.67 percent complete\n",
            "04:27:58.34 94.79 percent complete\n",
            "04:28:18.19 94.91 percent complete\n",
            "04:28:37.38 95.03 percent complete\n",
            "04:28:56.39 95.15 percent complete\n",
            "04:29:16.15 95.27 percent complete\n",
            "04:29:35.67 95.39 percent complete\n",
            "04:29:55.31 95.50 percent complete\n",
            "04:30:16.26 95.62 percent complete\n",
            "04:30:35.68 95.74 percent complete\n",
            "04:30:55.61 95.86 percent complete\n",
            "04:31:15.15 95.98 percent complete\n",
            "04:31:35.05 96.10 percent complete\n",
            "04:31:54.41 96.22 percent complete\n",
            "04:32:13.97 96.34 percent complete\n",
            "04:32:33.96 96.46 percent complete\n",
            "04:32:54.95 96.58 percent complete\n",
            "04:33:14.70 96.70 percent complete\n",
            "04:33:34.60 96.82 percent complete\n",
            "04:33:54.05 96.94 percent complete\n",
            "04:34:13.07 97.06 percent complete\n",
            "04:34:32.93 97.18 percent complete\n",
            "04:34:52.62 97.30 percent complete\n",
            "04:35:12.12 97.41 percent complete\n",
            "04:35:33.47 97.53 percent complete\n",
            "04:35:53.19 97.65 percent complete\n",
            "04:36:12.86 97.77 percent complete\n",
            "04:36:32.64 97.89 percent complete\n",
            "04:36:53.32 98.01 percent complete\n",
            "04:37:13.71 98.13 percent complete\n",
            "04:37:33.55 98.25 percent complete\n",
            "04:37:52.75 98.37 percent complete\n",
            "04:38:12.98 98.49 percent complete\n",
            "04:38:32.59 98.61 percent complete\n",
            "04:38:51.98 98.73 percent complete\n",
            "04:39:11.33 98.85 percent complete\n",
            "04:39:31.56 98.97 percent complete\n",
            "04:39:51.31 99.09 percent complete\n",
            "04:40:11.05 99.21 percent complete\n",
            "04:40:31.11 99.33 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '↓ ↓']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "04:40:51.99 99.44 percent complete\n",
            "04:41:11.19 99.56 percent complete\n",
            "04:41:31.81 99.68 percent complete\n",
            "04:41:51.23 99.80 percent complete\n",
            "04:42:10.38 99.92 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "outputId": "618155af-0a29-47c7-e3df-d7f83fbc2076",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 873
        }
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "Little did I realize that in future years I would spend the major portion of my life continuing the work of these pilgrims by serving as a traveling overseer of Jehovah’s Witnesses .\n",
            "These were things I never imagined I would be able to get rid of . ”\n",
            "Ruins of the theater at Ephesus\n",
            "We were arrested and taken to the police station .\n",
            "Today , a more far - reaching destruction is looming , one that will bring an end to this entire system of things .\n",
            "The Bible provides the answer when it says that humans were created “ in God’s image , ” that is , they possess the same basic godly qualities . ​ — Genesis 1 : 27 .\n",
            "And whom did the king assign to sing out Mordecai’s praises ?\n",
            "Later , we began pioneering together in West Berlin .\n",
            "Just as he did in the past , today God is certainly able to deliver his servants from dangerous situations , and those who feel that they have benefited from his protection are not to be criticized .\n",
            "As long as we are being guided by Bible principles , however , we have every reason to be happy and at peace .\n",
            "\n",
            "==> train.st <==\n",
            "Ke ne ke sa hlokomele hore lilemong tse latelang ke ne ke tla sebelisa karolo e khōlō ea bophelo ba ka ke ntšetsa pele mosebetsi oa bana ba nkang maeto a bolumeli ka hore ke sebeletse ke le molebeli oa potoloho oa Lipaki tsa Jehova .\n",
            "Ke ne ke sa nahane hore ke tla ke ke khone ho tlosa menahano e joalo . ”\n",
            "Lithako tsa lebala la liketsahalo la Efese\n",
            "Re ile ra tšoaroa ’ me ra isoa motebong ea sepolesa .\n",
            "Kajeno , timetso e khōlō haholo e haufi , ’ me e tla felisa tsamaiso ena eohle ea lintho .\n",
            "Bibele e re re bopiloe ka “ setšoantšo sa Molimo ” kahoo re khona ho bontša makhabane a tšoanang le a hae . — Genese 1 : 27 .\n",
            "Joale morena o ile a khetha mang hore a roke Mordekai ?\n",
            "Hamorao , re ile ra qala bopula - maliboho Berlin Bophirimela .\n",
            "Joalokaha Molimo a ile a etsa nakong e fetileng , ka sebele kajeno o khona ho lopolla bahlanka ba hae maemong a kotsi , ’ me ba nahanang hore ba ruile molemo tšireletsong ea hae ha baa lokela ho tšoauoa phoso .\n",
            "Leha ho le joalo , hafeela re tataisoa ke melao - metheo ea Bibele , re na le motheo o tiileng oa ho thaba le ho ba le khotso .\n",
            "==> dev.en <==\n",
            "Has the situation improved since then ?\n",
            "Furthermore , as Jehovah reminded Job , animals testify to His strength .\n",
            "The wise man had one request : The gossiper was told to go and take a feather pillow and cut it open , scattering the feathers to the wind .\n",
            "Since Tischendorf’s time , more manuscripts have come to light .\n",
            "However , the Bible encourages us to use our “ thinking ability . ”\n",
            "We enjoy light refreshments , review the good points in the box “ Are You Equipped to Flee ? ”\n",
            "“ Like arrows in the hand of a mighty man , so are the sons of youth . ”\n",
            "But they do not always destroy all the harmful bacteria ; certain strains of bacteria withstand the attack .\n",
            "Daily he waters the flock .\n",
            "Thus , after about 120 days , your red blood cells begin to deteriorate and lose their elasticity .\n",
            "\n",
            "==> dev.st <==\n",
            "Na ho tloha ka nako eo boemo bo ntlafetse ?\n",
            "Ho feta moo , joalokaha Jehova a ile a hopotsa Jobo , liphoofolo li fana ka bopaki ba hore O matla .\n",
            "Monna ea bohlale o ile a kōpa mosebi eo ntho e le ’ ngoe : Hore a tsamaee ’ me a tle le mosamo oa masiba , a o bule ebe o hasa masiba ao moeeng .\n",
            "Ho tloha mehleng ea Tischendorf , ho fumanoe libuka tse ngotsoeng ka letsoho tse eketsehileng .\n",
            "Empa Bibele e re khothalletsa hore re sebelise ‘ matla a rōna a ho nahana . ’\n",
            "Re ile ra thabela lijo tse bobebe , ra hlahloba lintlha tsa bohlokoa tse fanoeng lebokoseng le reng , “ Na le se le Itokiselitse ho Baleha ? ”\n",
            "“ Joaloka metsu letsohong la monna ea matla , bara ba bocha ba joalo . ”\n",
            "Empa hase kamehla li ripitlang libaktheria tsohle tse lematsang ; mefuta e meng ea baktheria e ema ka tieo khahlanong le tlhaselo .\n",
            "O noesa mohlape ka matsatsi ’ ohle .\n",
            "Ka hona , ka mor’a matsatsi a 120 , lisele tse khubelu tsa mali li qala ho fokola ’ me li sitoa ho saroloha .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rMNKyorAG9QO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create directory, move everyone we care about to the google drive for training\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}