[FEATURE] Tools

#470
by victor HF staff - opened
Hugging Chat org
โ€ข
edited Jun 17

Tools on HuggingChat

Learn more about available tools in this youtube video: https://www.youtube.com/watch?v=jRcheebdU5U

Today, we are excited to announce the beta release of Tools on HuggingChat! Tools open up a wide range of new possibilities, allowing the model to determine when a tool is needed, which tool to use, and what arguments to pass (via function calling).

  • For now, tools are only available on the default HuggingChat model: Cohere Command R+ because it's optimized for using tools and has performed well in our tests.
  • Tools use ZeroGPU spaces as endpoints, making it super convenient to add and test new tools!

Available tools

Tool name Description Host
Web Search Query the web and do some RAG on retrieved content against the user query HuggingChat internal tool
URL Fetcher Fetch text content from a given URL HuggingChat internal tool
Document Parser Parse content from PDF, text, csv, json and more ZeroGPU Space
Image Generation Generate images based on a given text prompt ZeroGPU Space
Image Editing Edit images based on a given text prompt ZeroGPU Space
Calculator A simple calculator for evaluating mathematical expressions HuggingChat internal tool

How we choose tools

  • A tool must be a ZeroGPU Space that comes by default with exposed API endpoints.
  • Tools need to be fast (~25 seconds max) to ensure a good user experience.
  • In general, we prefer simple and fun tools (like a new model) over complex workflows that are harder to test and more likely to fail.

Do you have an idea for a tool to add or to update one directly on HuggingChat? Share your thoughts in this ๐Ÿ‘ฅ community discussion.

Next Steps

  • Use previously generated files with tools (probably)
  • Add tools to Community Assistants: Making it possible for users to add their own ZeroGPU Spaces as tools in their Assistants.
  • Add more official tools on a regular basis.
  • Improve existing tools.
  • Support more models (maybe starting with Llama-3)
  • Add multi-step Tool Use (aka Agents)
  • Add ability to reference previous files from the conversation.
  • Add extra tools at runtime via OpenAPI specification.
victor changed discussion title from [FEATURE Tools] to [FEATURE] Tools

chat ui pauses

julien-c pinned discussion

chat ui pauses

https://huggingface.co/chat/ access it from here

Tried to do a Web search many times but I'm stuck with the loading icon and other tools seem to have different problems
Screenshot_2024-05-28-22-02-53-401_com.android.chrome.jpg

Screenshot_2024-05-28-22-07-29-152_com.android.chrome.jpg

Hugging Chat org

@Stefan171 Thanks for the report! Both issues should be fixed now, thanks to your screenshots!

@nsarrazin Pleasure. It's working now. Thanks for developing these tools.

deleted

How do we use the PDF parser?

deleted

Figured out how to use it, but PDF upload fails with error 413

@sneedingface it was a bit too slow in our testing and was a bit frustrating to use in a chat format so we chose schnell as a default but you'll be able to create your own tools with some upcoming features :)

@nsarrazin can you share how exactly does the websearch work? does the llm generate the search term and "decide" to call the search tool to search it? or does the web search tool use a separate model (or a separate instance of the model) to automatically search the web and feed the result to the llm to be used?

Document parser (or the model) doesn't work as well as it should. e.g. If I upload an image or pdf of a table, it is not able to accurate convert it into text. While gpt40-mini or gemini flash 1.5 easily convert the image into table format. Can that be improved?

@toximod120 The current tools available in HuggingChat do not make the model able to interpret images. This would require either multimodal models, or parsing the image to a multimodal model first, just to then parse an image description to the main model. That second idea I has already proposed to victor, and he said that they'd rather gave actual multimodal functionality, than fake it with this combination approach.

Uploading images currently only allows for image editing.

Can you update Command R + to the lastest version? (https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024)

I love community tools. I created a very simple tool.

Will assistants support tools? It would be good to be able to call tools while using custom model parameters

nsarrazin unpinned discussion

So this models use up the quota of Huggingface GPU from the Logged in account? Only the premium members can use this new community tools after a few tries

Sign up or log in to comment