Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.9k

Running on CPU Upgrade

it looks like both of them have a 'failed' status:
65b: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/upstage/llama-65b-instruct_eval_request_False_False_False.json
70b: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/upstage/Llama-2-70b-instruct-1024_eval_request_False_float16_Original.json

@augtoma Thanks for the information and the links.

@clefourrier

We've verified that our config, model, and tokenizer are all loading properly, and we've passed the tests on the local lm-evaluation-harness. We used the pretrained argument as follows, for example, pretrained="upstage/llama-65b-instruct".

So, it's challenging for us to determine the cause of the failure. Could you provide any information about the errors that may have resulted in our submissions failing?
(Currently our transformers version is 4.31.0)

@Limerobot we're investigating, thank you for reporting this

Hello @clefourrier . Same case for MayaPH/GodziLLa-30B-instruct. I've also seen that you added it back manually hours ago at https://huggingface.co/datasets/open-llm-leaderboard/requests/tree/main/MayaPH, but the recent job also resulted in failure. I modified the config.json just now. If you could please re-run it again. Thank you.

@clefourrier , Any update on this? I really appreciate your help.

@jaspercatapang @Limerobot are all your models now on the leaderboard?

@clefourrier Thanks for asking. Still, https://huggingface.co/upstage/Llama-2-70b-instruct-v2 is in the queue.

Also upstage/Llama-2-70b-instruct-1024 and upstage/Llama-2-70b-instruct are duplicates. Feel free to remove upstage/Llama-2-70b-instruct-1024 in the leaderboard.

If it's in the queue it will be evaluated soon. As for duplicates models I will remove upstage/Llama-2-70b-instruct-1024.
Thanks for your patience !

@clefourrier @SaylorTwift Thank you for your hard work. :)