--- base_model: google/flan-ul2 license: apache-2.0 tags: - flan - ul2 - candle - quant pipeline_tag: text2text-generation --- # flan-ul2: candle quants Quants of `google/flan-ul2` with [candle](https://github.com/huggingface/candle/tree/main/candle-examples/examples/quantized-t5) ```sh cargo run --example quantized-t5 --release -- \ --model-id pszemraj/candle-flanUL2-quantized \ --weight-file flan-ul2-q3k.gguf \ --prompt "Answer the following question by reasoning step by step. The cafeteria had 23 apples. If they used 20 for lunch, and bought 6 more, how many apples do they have?" \ --temperature 0 ``` On my laptop (CPU, running in WSL) I get: `45 tokens generated (0.48 token/s)` ## weights Below are the weights/file names in this repo: | Weight File Name | Quant Format | Size (GB) | |-------------------------|--------------|-----------| | flan-ul2-q2k.gguf | q2k | 6.39 | | flan-ul2-q3k.gguf | q3k | 8.36 | | flan-ul2-q4k.gguf | q4k | 10.9 | | flan-ul2-q6k.gguf | q6k | 16 | From initial testing: - it appears that q2k is too low precision and produces poor/incoherent output. The `q3k` and higher are coherent. - Interestingly, there is no noticeable increase in computation time (_again, on CPU_) when using higher precision quants. I get the same tok/sec for q3k and q6k +/- 0.02 ## setup > [!IMPORTANT] > this assumes you already have [rust installed](https://www.rust-lang.org/tools/install) ```sh git clone https://github.com/huggingface/candle.git cd candle cargo build ```