Error in promptformatter.py
I am trying this on our database in postgresql
File "/NSQL/examples/postgres.py", line 1, in
from db_connectors import PostgresConnector
File "NSQL/examples/db_connectors.py", line 8, in
from prompt_formatters import TableColumn, Table
File "/NSQL/examples/prompt_formatters.py", line 4, in
class TableColumn(BaseModel):
File "/NSQL/examples/prompt_formatters.py", line 8, in TableColumn
dtype: str | None
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
Not sure how to handle this. If I remove none than it pass from this statement but give another error later
pydantic.error_wrappers.ValidationError: 2 validation errors for Table
pks
field required (type=value_error.missing)
fks
field required (type=value_error.missing)
Can you please help me in this.
Which Python version are you using? The NSQL repo supports 3.10 or later.
Thanks. That worked. I am using Python 3.11 now.
I was using 3.9 before.
Can you also suggest hardware requirements to use 6B Model.
I am getting timeout errors on 6B model. 350M works fine.
File "/llm/env/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='', port=5000): Read timed out. (read timeout=60).
We use 1 A5000 GPU (24G) to host the model. Can you make sure your model is on GPU?
Is it necessary for 6B model to be on GPU?
I am using on centos with 128GB RAM and 1TB disk storage but it keeps throwing timeout error.
We highly recommend using GPU to host foundation models (e.g., 6B model). You can still host it on CPU but it will be very slow (e.g., a couple mins per query).
So is that the reason of getting below error?
File "/llm/env/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='', port=5000): Read timed out. (read timeout=60).
Timeout after 60secs
It means 60s is not enough for the model to generate the results.
Yes, I get it.
Is there a way to increase this timeout?
There is timeout defined in manifest.py -- "client_timeout": ("client_timeout", 60)
Can you give more information about how you call the model? In request package you can set the timeout in the post: https://requests.readthedocs.io/en/latest/user/quickstart/#timeouts
I am using manifest to run the model as mention in READme of github repo.
python3 -m manifest.api.app
--model_type huggingface
--model_generation_type text-generation
--model_name_or_path NumbersStation/nsql-6B
--device 0
and then running a python script similar to the example in github repo
Manifest has an argument called client_timeout and you can set the call timeout there. Here is an example:
manifest.run(
prompt,
return_response=True,
client_timeout=180,
**manifest_params,
)
That works. Thank you.
Yes it is slow but running.
Is manifest the best way to run this or is there any other way better than this?
Yes, we recommend using manifest to host the model on GPU machine and you can ping the model anywhere else.