task_suggestions.py

"""
This is a file holding task-specific suggestions.

How to add a new suggestion:
1. Add a new constant at the bottom of the file with your suggestion. Please try to follow the same format as the
existing suggestions.
2. Add a new entry to the `TASK_SUGGESTIONS`, with format `((task tags,), (problem tag,), suggestion constant)`.
    a. See `app.py` for the existing task tags.
    b. See `app.py` for the existing problem tags.
    c. Make sure the task tags and the problem tags are a tuple.
"""

USE_SAMPLING = """
<details><summary>{match_emoji} {count}. Use sampling decoding strategies.</summary>
&nbsp;

🤔 Why? &nbsp;

The selected task benefits from creativity. Sampling-based decoding strategies typically yield better results in
creativity-based tasks &nbsp;


🤗 How? &nbsp;

Our text generation interfaces accept a `do_sample` argument. Set it to `True` to ensure sampling-based strategies
are used. &nbsp;
_________________
</details>
"""


BLOCK_LOW_PROBA_TOKENS = """
<details><summary>{match_emoji} {count}. Sampling - Block low probability tokens.</summary>
&nbsp;

🤔 Why? &nbsp;

When decoding with sampling-based strategies, ANY token in the model vocabulary can be selected. This means there is
always a chance that the generated text drifts off-topic. &nbsp;


🤗 How? &nbsp;

There are a few different strategies you can try. They all discard (i.e. set probability to 0) tokens at each
generation step. Unless stated otherwise, they can be used with each other.
1. Top K: discards all but the K most likely tokens (suggeted value: `top_k=50`);
2. Top P: sorts tokens by probability in descending order, computes the cumulative probability, then discards all
tokens with a cumulative probability above the threshold P (suggested value: `top_p=0.95`);
3. ETA Cutoff: A balance between Top K and Top P. See the [corresponding paper](https://arxiv.org/abs/2210.15191) for
more details (should not be used with others; suggested value: `eta_cutoff=1e-3`). &nbsp;
_________________
</details>
"""


USE_DETERMINISTIC = """
<details><summary>{match_emoji} {count}. Use greedy decoding strategies.</summary>
&nbsp;

🤔 Why? &nbsp;

The selected task is factual, it does not benefit from creativity. Greedy decoding strategies (like Greedy Search and
Beam Search) are preferred in those situations. &nbsp;


🤗 How? &nbsp;

Our text generation interfaces accept a `do_sample` argument. Set it to `False` to ensure greedy strategies
are used. &nbsp;
_________________
</details>
"""

# task tags that should use sampling and benefit from sampling-related advice
sampling = ("txtgen", "chat", "img2txt")
# task tags that should NOT use sampling and benefit from greedy/beam search advice
greedy = ("summ", "trans", "txtqa", "otherqa", "asr")

TASK_SUGGESTIONS = (
    (sampling, ("quality",), USE_SAMPLING),
    (sampling, ("quality", "hallucinations"), BLOCK_LOW_PROBA_TOKENS),
    (greedy, ("quality",), USE_DETERMINISTIC),
)
assert all(isinstance(problem_tags, tuple) for _, problem_tags, _ in TASK_SUGGESTIONS)
assert all(isinstance(task_tags, tuple) for task_tags, _, _ in TASK_SUGGESTIONS)