Could you please make a batch inference script?
#1
by
thesby
- opened
Batch inference will be much faster. Could you please make a batch inference script?
Can't we just use the official script?
https://github.com/fpgaminer/joycaption/tree/main/scripts
I made a modified version for my own use, but I have not released it until now because NF4 does not work well in my environment.
https://huggingface.co/John6666/llama-joycaption-alpha-two-hf-llava-nf4/blob/main/scripts/batch-caption.py