Compilade's picture

7 6

Compilade

compilade

·

compilade

AI & ML interests

None yet

Recent Activity

liked a model about 2 months ago

ai21labs/Jamba-tiny-dev

replied to bartowski's post 2 months ago

Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few) The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16 Had claude make me a script, using the new Reflection-70B, and these are the results: Total weights: 70553706496 Fully representable: 70530215524 Squashed: 23490972 Percentage squashed: 0.03% 0.03%!!!! A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8) This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers. Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.

View all activity

Organizations

None yet

compilade's activity

New activity in HF1BitLLM/Llama3-8B-1.58-100B-tokens 2 months ago

GGUF conversion

#3 opened 2 months ago by

New activity in mistralai/Mamba-Codestral-7B-v0.1 4 months ago

Update hardcoded filenames

#1 opened 4 months ago by

New activity in ai21labs/Jamba-v0.1 8 months ago

Smaller version to ease implementation experiments?

#12 opened 8 months ago by

New activity in jondurbin/bagel-dpo-2.8b-v0.2 9 months ago

GGUF Please

#1 opened 11 months ago by

New activity in clibrain/mamba-2.8b-instruct-openhermes 9 months ago

gguf

#1 opened 12 months ago by

New activity in TheBloke/rocket-3B-GGUF 12 months ago

Wrong EOS token has been fixed in upstream tokenizer_config.json, consider reconverting

#2 opened about 1 year ago by

New activity in pansophic/rocket-3B about 1 year ago

Fix tokenizer_config.json EOS token

#3 opened about 1 year ago by

Fix tokenizer_config.json EOS token

#3 opened about 1 year ago by