Tidzo
/

falcon-7b-esg-instruct

Text Generation

Inference Endpoints

Model card Files Files and versions Community

Tidzo commited on Jul 4, 2023

Commit

8cb3d7b

•

1 Parent(s): 4b80708

Update handler.py

Files changed (1) hide show

handler.py +1 -0

handler.py CHANGED Viewed

@@ -16,6 +16,7 @@ class EndpointHandler():
         model = AutoModelForCausalLM.from_pretrained(
             config.base_model_name_or_path,
             trust_remote_code=True,
             device_map="auto"
         )

         model = AutoModelForCausalLM.from_pretrained(
             config.base_model_name_or_path,
+            load_in_8bit=True,
             trust_remote_code=True,
             device_map="auto"
         )