memory used

#6
by cdani - opened

Hello. I'm trying to execute this model and it's killed as it needs more memory than I have in the computer. How much memory it needs? I have to dedice how much to add.
Thanks.

BigScience Workshop org

Hi @cdani , are you talking about CPU memory or GPU memory?
Just to load the model, you will need ~90GB of CPU memory.

Ah! Ok, thanks, was just that memory. I understand that a cuda GPU its not necessary, isn't it? Will it be very slow without it?

BigScience Workshop org

Great!
Yes, technically, running inference on CPU is possible although it will be very slow (an order of magnitude slower than using GPU). There are a few pointers here: https://github.com/bigscience-workshop/t-zero/tree/master/inference

Could it be possible to upload the model weights in multiples bin files (as bloom does) to be able to load with less amount of memory?
Thanks!

BigScience Workshop org

Hi @juanluisdb ,
yes we can! and actually it has been done on the branch "sharded" -> https://huggingface.co/bigscience/T0pp/tree/sharded

You can use the sharded ones by adding a tag in the from_pretrained call: AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", revision="sharded").

christopher changed discussion status to closed

Sign up or log in to comment