Text Generation
Transformers
PyTorch
TensorBoard
Safetensors
bloom
Eval Results
text-generation-inference
Inference Endpoints

Can I use Bloom on an a5000 GPU?

#257
by smcg1579 - opened

For an experiment I'm working on, we've been looking into the usage of Bloom as a large language model, but I'm not sure if it is feasible for it to work on the computer we have available to us. I'm still fairly new to LLMs, so I'm not sure how I can measure the capabilities against our computer. For the purposes of the experiment we don't need anything high-speed, and figure if we can get the model working to return query results after one or two days that works for us. We are working with a Nvidia RTX A5000 GPU and 128GB RAM, so I was just wondering if bloom would be at all feasible or if we should consider other options. Thanks!

smcg1579 changed discussion title from Bloom GPU Requirements to Can I use Bloom on an a5000 GPU?

Sign up or log in to comment