Best model I tested, but seems to have an issue on some tokens
Hi!
This model is awesome in regards to speed and quality of generations. However, it ends generations sometimes in "ightarrow"+halluzinations or "archivo"+halluzinations. May be an issue with these tokens. Any ideas?
I'm using gpt4-x-alpaca-30b-128g-4bit.safetensors version in oobabooga's webui.
This one works completely normally: https://huggingface.co/MetaIX/Alpaca-30B-Int4-128G-Safetensors
Interesting, I haven't had that happen to me. Both of them were quantized using the same environment, the only difference were the flags used.
That is really interesting. I am just posting the details here if anyone has an idea and wants to give feedback in the interest of operating llm's. Not requesting anything, I find it already amazing that you share your work here. Really really great.
How I can reproduce:
Vanilla oobabooga installer. RTX 3090. i7-11700K. Xubuntu. Only set wbits 4, groupsize 128 and max_new_tokens 400.
Two prompts from learn.deeplearning.ai:
gpt4-x-alpaca-30b-128g-4bit.safetensors gives me at least 20% of repeating the prompts something like below, while alpaca-30b-128g-4bit.safetensors not even once gave me such halluzinations:
========1========
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
Your task is to perform the following actions:
1 - Summarize the following text with 1 sentence.
2 - Translate the summary into French.
3 - List each name in the text.
4 - Output a json object that contains the following keys: french_summary, num_names.
Use the following format:
Text:
Translation:
Names:
Output JSON:
Text: "In a charming village, siblings Jack and Jill set out on
a quest to fetch water from a hilltop
well. As they climbed, singing joyfully, misfortune
struck—Jack tripped on a stone and tumbled
down the hill, with Jill following suit.
Though slightly battered, the pair returned home to
comforting embraces. Despite the mishap,
their adventurous spirits remained undimmed, and they
continued exploring with delight."
Response:
Text: In a charming village, siblings Jack and Jill set out on a quest to fetch water from a hilltop well. As they climbed, singing joyfully, misfortune struck—Jack tripped on a stone and tumbled down the hill, with Jill following suit. Though slightly battered, the pair returned home to comforting embraces. Despite the mishap, their adventurous spirits remained undimmed, and they continued exploring with delight.archivierepublicain.fr/actualite-societe/culture/livres/un-jour-de-pluie-au-paradis-du-petit-prince-708695
========2========
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
Proofread and correct the following text and rewrite the corrected version. If you don't find and errors, just say "No errors found". Don't use any punctuation around the text:
"Got this for my daughter for her birthday cuz she keeps taking
mine from my room. Yes, adults also like pandas too. She takes
it everywhere with her, and it's super soft and cute. One of the
ears is a bit lower than the other, and I don't think that was
designed to be asymmetrical. It's a bit small for what I paid for it
though. I think there might be other options that are bigger for
the same price. It arrived a day earlier than expected, so I got
to play with it myself before I gave it to my daughter."
Response:
Corrected Text:
I bought this panda plushie for my daughter as a gift because she kept stealing mine from my bedroom. Adults can love pandas too! My daughter carries it wherever she goes, and it’s very soft and adorable. However, one ear seems slightly lower than the other, which may not have been intended by the designer. The size is smaller than I had anticipated despite paying more money for it. Overall, it came in early and I enjoyed playing with it before giving it to my daughter.iation.enoerrorsfound.ightarrow.ightarrow.ightarrow.ightarrow.[etc.]