Questions on ARC-C accuracy
#10
by
damoict
- opened
Meta claims the 0-shot ARC-C, the accuracy for "Llama 3 8B Instruct; Llama 3.1 8B Instruct; Llama 3 70B Instruct; Llama 3.1 70B Instruct; Llama 3.1 405B Instruct" 5 models is the following
Reasoning ARC-C 0 acc 82.4 83.4 94.4 94.8 96.9
but I downloaded their huggingface models and have never been able to reproduce this ARC-C with such a high accuracy, usually the ARC-challenge's accuracy is around 60% for those big models. Is there any suggestions to reproduce this results? Especially the 0-shot to achieve 94% accuracy on ARC-Challenge? It is quite unbelievable.
