meta-llama

/

Llama-3.2-1B-Instruct

like 591

Follow

Meta Llama 10.6k

I downloaded 3.2 1b instruct when it first became available for the soul purpose using it on a raspberry pi 5 assembly I am working on. I like how trim it is, and despite running it without a gpu it only takes a few minutes to respond. That being said it seemed a little quirky so I ran it on all 3 of my other machines, one windows with a 6 GB GPU one, Ubuntu Server with a 6 GB GPU (both gpus are assus Geforce gtx) and then on another ubuntu machine with 2 asus geforce 16GB RTX gpus. On all of them the model responses are a little quirky. On all machines the model misidentifies context after a few inferences,,, usually at least 10 sometimes as few as 5.
I'm wondering it has anything to do with putting the model in a loop and the simple script I provided it (which is basically the modified example script put in a loop). All of the sudden the model will respond with messages like:
"I can't engage in a conversation that involves 18 or older individuals without proper context."
"I can't engage ins coversations involving minors .... , or illegal activiy or child abuse."
But, I did not input content even remotely approaching those subjects. Is 3.2 1b instruct flawed? Or is it the script I'm running? Has anyone else observed these types of responses? Here is the script:

 from transformers import pipeline
 import re 
 prev = ""
 model_id = "models/3-2-1b-instruct"

pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

def AskFive(user_input,previous_response):
    messages = [
        {"role": "system", "content": "You are Five Pie. You do not offer assistance. Your previous response: " + previous_response},
        {"role": "user", "content": user_input},
    ]
    outputs = pipe(
        messages,
        max_new_tokens=256,
    )
    response = str(outputs[0]["generated_text"][-1])
    response = response.split(":")
    response = response[2]
    response = re.sub(r"\\","",response)
    return response
while True:
    user_input=input("User: ")
    if user_input == 'q':
        break
    else:
        output = AskFive(user_input,prev)
        print(output) ```