The score of this model on spider and WikiSQL
I would like to know the score of this model on spider and WikiSQL. I'm not sure if you have submitted any results.
Also, do you have any scores on the test set！
Looking forward to a reply！
Here are some benchmark scores we have: https://www.numbersstation.ai/post/nsql-llama-2-7b and all numbers are on dev set.
Do you mind to share the evaluation script or the prompt template for us to duplicate the numbers? Many thanks
In the page https://www.numbersstation.ai/post/nsql-llama-2-7b, I found "Open Source Ours - Pretrain + Instruct". Could anyone explain what exactly is this "Instruct"? Only after this "Instruct", the accuracy can be close to chatgpt.
In the page https://www.numbersstation.ai/post/nsql-llama-2-7b, I found "Open Source Ours - Pretrain + Instruct". Could anyone explain what exactly is this "Instruct"? Only after this "Instruct", the accuracy can be close to chatgpt.
What they mean by pre-training is training it for Causal Language Modeling with SQL queries only. Instruct refers to the fine-tuning of the pretrained LLAMA model with text-to-SQL datasets, also known as instruction tuning.
Do you mind to share the evaluation script or the prompt template for us to duplicate the numbers? Many thanks
We've followed https://github.com/taoyds/spider for spider and geoquery evaluation and use Rajkumar prompt format. You can find examples here: https://github.com/NumbersStationAI/NSQL/tree/main/examples