Spaces:
Running
on
CPU Upgrade
duplicated models in leaderboard
hi
There are some "duplicated" models in leaderboard, and system does not seem to capture them.
- duplicated means, model with same name
- tensor value , etc is not considered
is this real duplication? or just error of the system?
thanks for your kindness
same model with different precisions? fp16 bf16 int8 int4 gptq...
@JosephusCheung
Thank you for your sharing. really helped.
however, I think one model should be shown once, not shown for every possible precisions.
I wonder how admins think
Hi @leejunhyeok ,
As indicated in our FAQ (in the About tab) of the leaderboard, some models appear several times because we evaluated them in different precision.
I think that rule should be modified @clefourrier , as some submitters could mess up leaderboard!
- if model is trained with different precisions and parameters are different, that is fine and good to go
- in this case, model version will be different and their effort should be respected
- if model is trained with one precision and multiple submission of different precision is done, that is not fine.
- in this case, only one(probably the best) submission should be shown
thanks
You are talking nonsense, a model can be eval with different precisions, with not pre-quantized weight and pre-quantized GPTQ, no matter how it was trained. This should not be a problem.
Hi @leejunhyeok ,
Thanks a lot for your interest!
We convert the weight precision on the fly during the evaluation (as would people wanting to use a given model at a lower precision would do). We think it's important for the community to know if a model is still performing well at a lower precision than the one it was trained on, so we'll keep the feature.
Closing this issue.