metadata
license: mit
language:
- en
widget:
- text: >
In a busy city street, an autonomous car navigates around pedestrians and
cyclists, making a series of complex decisions to ensure safety.
example_title: Complex VQA Scenario
- text: >
On a clear highway, an autonomous vehicle adjusts its speed based on the
flow of traffic and the presence of a slower-moving truck ahead.
example_title: Simple VQA Scenario
tags:
- vision-language
- autonomous-driving
What is this?
Lingo-Judge, a novel evaluation metric that aligns closely with human judgment on the LingoQA evaluation suits.
How to use
# Import necessary libraries
from transformers import pipeline
# Define the model name to be used in the pipeline
model_name = 'wayveai/Lingo-Judge'
# Define the question and its corresponding answer and prediction
question = "Are there any pedestrians crossing the road? If yes, how many?"
answer = "1"
prediction = "Yes, there is one"
# Initialize the pipeline with the specified model, device, and other parameters
pipe = pipeline("text-classification", model=model_name)
# Format the input string with the question, answer, and prediction
input = f"[CLS]\nQuestion: {question}\nAnswer: {answer}\nStudent: {prediction}"
# Pass the input through the pipeline to get the result
result = pipe(input)
# Print the result and score
score = result[0]['score']
print(score > 0.5, score)