[FEATURE] Tools

#470
by victor HF staff - opened
Hugging Chat org
โ€ข
edited Jun 17

Tools on HuggingChat

Learn more about available tools in this youtube video: https://www.youtube.com/watch?v=jRcheebdU5U

Today, we are excited to announce the beta release of Tools on HuggingChat! Tools open up a wide range of new possibilities, allowing the model to determine when a tool is needed, which tool to use, and what arguments to pass (via function calling).

  • For now, tools are only available on the default HuggingChat model: Cohere Command R+ because it's optimized for using tools and has performed well in our tests.
  • Tools use ZeroGPU spaces as endpoints, making it super convenient to add and test new tools!

Available tools

Tool name Description Host
Web Search Query the web and do some RAG on retrieved content against the user query HuggingChat internal tool
URL Fetcher Fetch text content from a given URL HuggingChat internal tool
Document Parser Parse content from PDF, text, csv, json and more ZeroGPU Space
Image Generation Generate images based on a given text prompt ZeroGPU Space
Image Editing Edit images based on a given text prompt ZeroGPU Space
Calculator A simple calculator for evaluating mathematical expressions HuggingChat internal tool

How we choose tools

  • A tool must be a ZeroGPU Space that comes by default with exposed API endpoints.
  • Tools need to be fast (~25 seconds max) to ensure a good user experience.
  • In general, we prefer simple and fun tools (like a new model) over complex workflows that are harder to test and more likely to fail.

Do you have an idea for a tool to add or to update one directly on HuggingChat? Share your thoughts in this ๐Ÿ‘ฅ community discussion.

Next Steps

  • Use previously generated files with tools (probably)
  • Add tools to Community Assistants: Making it possible for users to add their own ZeroGPU Spaces as tools in their Assistants.
  • Add more official tools on a regular basis.
  • Improve existing tools.
  • Support more models (maybe starting with Llama-3)
  • Add multi-step Tool Use (aka Agents)
  • Add ability to reference previous files from the conversation.
  • Add extra tools at runtime via OpenAPI specification.
victor changed discussion title from [FEATURE Tools] to [FEATURE] Tools

chat ui pauses

julien-c pinned discussion

chat ui pauses

https://huggingface.co/chat/ access it from here

Tried to do a Web search many times but I'm stuck with the loading icon and other tools seem to have different problems
Screenshot_2024-05-28-22-02-53-401_com.android.chrome.jpg

Screenshot_2024-05-28-22-07-29-152_com.android.chrome.jpg

Hugging Chat org

@Stefan171 Thanks for the report! Both issues should be fixed now, thanks to your screenshots!

@nsarrazin Pleasure. It's working now. Thanks for developing these tools.

deleted

How do we use the PDF parser?

deleted

Figured out how to use it, but PDF upload fails with error 413

Hugging Chat org

I think there's an issue with PDFs which are too big, will try to fix it but try to keep it under 1MB for now otherwise it might fail.

Error 413 even with PDFs under 500kb!

I think there's an issue with PDFs which are too big, will try to fix it but try to keep it under 1MB for now otherwise it might fail.

My smallest PDF on my phone is 4.5MB ๐Ÿ˜…

Editing an image conversation: https://hf.co/chat/r/zGGHUYc

Is the Calculator tool able to do Randomness? Like say, help to generate random numbers on a range, likes 1-6?

Screenshot-com.android.chrome-20240529115826.jpg

The image visualization tool is not displayed. Unable to create an image.

great tools and a lot of potential ,where or how we can get this as an API ?

Suggestion: Add "memories" to tools. "Memories" is just a function call which will be called by the model if it decides to save some memory which can be preserved across chats similar to chatgpt's memory feature.

Thank you for these tools, I personally appreciate them a lot. I wanted to say that I struggled with the document parser till I unchecked the other tools from the list of six. I don't know if it's a coincidence or if it's actually a fix though.

Screenshot_20240529-105219.png

Hugging Chat org
โ€ข
edited May 29

Thank you for these tools, I personally appreciate them a lot. I wanted to say that I struggled with the document parser till I unchecked the other tools from the list of six. I don't know if it's a coincidence or if it's actually a fix though.

Weird we'll look at it with @Saghen

Kudos to @victor and the whole team! Great way to start adding Tools and Function Calling to HuggingChat.

One question: I love Download prompt and parameters feature in HuggingChat. It gives transparency as what was the actual prompt in text going to the LLM. However, I see the available tools (all the 6 functions, schema, description, etc.) are missing from the prompt. It seems this is not a complete prompt which doesn't have System Preamble or Available Tools parts.

If we can see the entire results from tokenizer.apply_tool_use_template would be amazing!

image.png

Hugging Chat org

I see the available tools (all the 6 functions, schema, description, etc.) are missing from the prompt.

Yes we are going to add a better debug cc @Saghen

Also add private prompt settings to make it safer

Hi HuggingChat. Are there any plans to create detailed documentation covering the full range of functions (documentation including descriptions of various functions, parameters, limitations and use cases)?

Congrats! This is a great step, specially find the document parser feature really useful .
As you suggested I would love to see llama3 support for these tools since the licensing of command-r+ can be a bit restrictive.

One suggestion I would like to make, is to give users the flexibility to plug in their own document parser, similar to how you can configure your own LLM endpoints.

This comment has been hidden

I'm having an issue where I can't upload a document from my iPhone for the parser. Images will still upload, but no .txt .pdf or .docx

It still works fine from my laptop, but I use my phone far more frequently.

Also, none of the tools appear when using the Huggingchat app from the Apple App Store.

Update: .rtf also not selectable.

Hugging Chat org

We recently upgraded our image generation tool to use Stable Diffusion 3! Feel free to try it out and let us know how it works for you.

I check in here from time to time, and it's good to see that the service is developing. However, its usefulness is still almost negligible. Searching often doesn't work, errors appear, and the chat stops frequently, etc. At the moment, when it comes to free services it's better to use the free Copilot than HuggingChat (or Command R+ directly from the Cohere website). However, I'm keeping my fingers crossed.

I have found that the web search and image generation tools work really well and the document parser (when functional) is also really good. I would like to see an image viewing tool, which could emulate multi modal models by returning an image description from a different space like the current Florence 2.

Hugging Chat org

Yes @Smorty100 I agree. the problem I see is that it won't be super useful without multi-step tool calling? for this reason maybe it's better to wait for a true multimodal open model.

I'm noticing the SD 3 tool struggles with prompt alignment. Not sure if it's an SD 3 limitation, or the tool parsing the prompt differently. Is there a workaround improve prompt alignment? Below is an example:
inconsPrompAlign.PNG
Kirby is not visible in the image, and is not white from Level 37. There are no "negative" aspects in the prompt.

Oh also, command r+ states that it cannot generate images, despite clearly able to do so.

AI art look didn't work for the LLM
image.png

It often pauses and the only solution is to start a new chat. It would also be good to set the translation of promt for image generation to english by default. Otherwise, when you ask for an image in another language you get something what you didn't want. You can bypass this by setting an appropriate system prompt but automatic translation would be better I guess.

Error calling tool calculator

Please elaborate

It seems that it is impossible to compare the size of numbers
Screenshot_2024-07-24-09-05-05-124_mark.via.gp.jpg

I believe it can only do basic arithmetic (+,-,*,/, sqrt, that stuff) and not comparisons, which is likely just a limitation of the calculator tool. It should be replaced with something better like Sagemath or similar.

Best is to use python interpreter in this case.

Document Parser is not working for me, always getting an error

Hugging Chat org

Try using tools with the 70B model for now, or the 405B without tools.

We're working through issues with our API which is a bit overloaded on the 405B and when you use tools you need to call the API twice so you have twice as many chances of getting an error. The 70B should work fine for now though!

If you still have issues with document parsing with the 70B let me know and I can take a look.

I have a question. Why the Document Parser is returning the whole document instead of the relevant chunks?
llama-3.1 70B forgets the original instructions and returns a summary instead.

image.png

@nsarrazin may I ask if the image generation tool has been updated to Flux.1? If so, Schnell or Dev? Also where can I stay updated about the changes on HuggingChat and its tools? Thanks

Hugging Chat org

@sneedingface It's still stable diffusion but we were thinking of upgrading to Schnell! We'll keep you posted in this thread

Hugging Chat org

We just updated to Flux.1 Schnell! Let us know if it works well

@nsarrazin why not use flux dev?

Hugging Chat org

@flexagontnt it's a bigger model and with the load on HuggingChat, response time weren't great. We thought a smaller but faster model would be a better fit for a chat workflow :)

I already want daily updates. I would like to be able to add sharing of tools. Everyone can build and use each other's tools.

And I want to be able to use AI to create tools too.

@flexagontnt it's a bigger model and with the load on HuggingChat, response time weren't great. We thought a smaller but faster model would be a better fit for a chat workflow :)

Ye itโ€™s as expected. Just curious!

I just got this when I am pasting URLs, anything I can do differently?

Screenshot 2024-08-21 155055.png

How can I use the system prompt to tell the model to call (or not call) the tools specifically? And how can I make the model to specify the input to the functions? Eg. How can I use the system prompt to tell the AI to never search for the user's question but search "haha" instead?

This comment has been hidden
Hugging Chat org

@brianm94 those features are coming soon with community tools & tools for assistants!

@nsarrazin can we please have Flux Dev? Schnell can do so far but that's it, it doesn't give a solid result so it's kinda pointless imho

@sneedingface it was a bit too slow in our testing and was a bit frustrating to use in a chat format so we chose schnell as a default but you'll be able to create your own tools with some upcoming features :)

@nsarrazin can you share how exactly does the websearch work? does the llm generate the search term and "decide" to call the search tool to search it? or does the web search tool use a separate model (or a separate instance of the model) to automatically search the web and feed the result to the llm to be used?

Document parser (or the model) doesn't work as well as it should. e.g. If I upload an image or pdf of a table, it is not able to accurate convert it into text. While gpt40-mini or gemini flash 1.5 easily convert the image into table format. Can that be improved?

@toximod120 The current tools available in HuggingChat do not make the model able to interpret images. This would require either multimodal models, or parsing the image to a multimodal model first, just to then parse an image description to the main model. That second idea I has already proposed to victor, and he said that they'd rather gave actual multimodal functionality, than fake it with this combination approach.

Uploading images currently only allows for image editing.

Can you update Command R + to the lastest version? (https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024)

I love community tools. I created a very simple tool.

Will assistants support tools? It would be good to be able to call tools while using custom model parameters

nsarrazin unpinned discussion

So this models use up the quota of Huggingface GPU from the Logged in account? Only the premium members can use this new community tools after a few tries

Sign up or log in to comment