Hugging Face's logo
Iwan Kawrakow's picture
13

Iwan Kawrakow

ikawrakow
BlueAquilae's profile picture eramax's profile picture zbruceli's profile picture
·

AI & ML interests

None yet

Recent Activity

replied to bartowski's post 2 months ago
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few) The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16 Had claude make me a script, using the new Reflection-70B, and these are the results: Total weights: 70553706496 Fully representable: 70530215524 Squashed: 23490972 Percentage squashed: 0.03% 0.03%!!!! A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8) This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers. Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
replied to bartowski's post 2 months ago
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few) The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16 Had claude make me a script, using the new Reflection-70B, and these are the results: Total weights: 70553706496 Fully representable: 70530215524 Squashed: 23490972 Percentage squashed: 0.03% 0.03%!!!! A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8) This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers. Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
replied to bartowski's post 2 months ago
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few) The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16 Had claude make me a script, using the new Reflection-70B, and these are the results: Total weights: 70553706496 Fully representable: 70530215524 Squashed: 23490972 Percentage squashed: 0.03% 0.03%!!!! A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8) This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers. Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
View all activity

Organizations

None yet

ikawrakow's activity

New activity in ikawrakow/validation-datasets-for-llama.cpp 3 months ago

Error with test data

7
#2 opened 3 months ago by fedric95
New activity in ikawrakow/various-2bit-sota-gguf 9 months ago

What 2bit quantization approach are you using?

1
#13 opened 9 months ago by ingridstevens
New activity in ikawrakow/mixtral-instruct-8x7b-quantized-gguf 10 months ago

Dataset for importance matrix?

4
#3 opened 10 months ago by RamoreRemora
New activity in ikawrakow/mixtral-8x7b-quantized-gguf 10 months ago

Broken M quants

1
#2 opened 10 months ago by Artefact2
New activity in ikawrakow/various-2bit-sota-gguf 10 months ago

What -ctx and -chunks parameters did you use to make the iMatrix of the Lllama 2 70b?

1
#9 opened 10 months ago by Nexesenex

Would love a deepseekcode 2bit quant. I bet others would love it too :)

2
#6 opened 10 months ago by subiculumforge

[Model request] Saily 100b, Saily 220b

1
#5 opened 11 months ago by Perpetuity7
New activity in ikawrakow/various-2bit-sota-gguf 11 months ago

Could We combine AWQ and Importance Matrix calculation together to further improve perplexity.

3
#4 opened 11 months ago by shing3232
New activity in ikawrakow/mixtral-instruct-8x7b-quantized-gguf 11 months ago

mixtral-instruct-8x7b-q3k-medium.gguf

4
#2 opened 11 months ago by cmh
New activity in ikawrakow/open-hermes-2.5-mistral-7b-quantized-gguf 11 months ago

SOTA 2-Quant of Nous-Heremes-Solar-10.7B and Nous-Hermes-2-Yi-34B, Please 🙏

4
#1 opened 11 months ago by Joseph717171
New activity in ikawrakow/various-2bit-sota-gguf 11 months ago

Please 3b model rocket 2bit?

17
#1 opened 11 months ago by Shqmil
New activity in ikawrakow/mixtral-instruct-8x7b-quantized-gguf 11 months ago

Q5_1 & Q5_K_M quant

3
#1 opened 11 months ago by muzika38
New activity in ikawrakow/mixtral-8x7b-quantized-gguf 11 months ago

Mixtral Instruct quant

3
#1 opened 11 months ago by cmh

Mixtral Instruct quant

3
#1 opened 11 months ago by cmh
New activity in ikawrakow/various-2bit-sota-gguf 11 months ago

Please 3b model rocket 2bit?

17
#1 opened 11 months ago by Shqmil

Please 3b model rocket 2bit?

17
#1 opened 11 months ago by Shqmil