Quants 4_NL vs 4K_S
#1
by
EloyOn
- opened
Did you use imatrix too to quant 4K_S? They weigh practically the same, do you recommend one over the other? I run local AI on my smartphone with Layla, so this Q4 zone is the sweet spot.
IQ quants tend to be a bit more CPU demanding. For a constrained hardware like this I'd assume the Q4_K_S would be a better bet as balanced quality/speed. All of them had the same Imatrix data used for calibration regardless. You can compare the speeds and use what feels better/faster in your hardware, honestly. Shouldn't be too big of a difference.
I'm adding both just because there could be a different in they react to the calibration data and as an extra choice for those that want to test.
Lewdiculous
changed discussion status to
closed
Nice. Thank you for the clarification.