Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge
•
24
Aymeric Roucher
m-ric
AI & ML interests
MLE at Hugging Face 🤗
LLMs, Agents, RAG, Multimodal.
Recent Activity
updated
a model
about 11 hours ago
m-ric/Aria_hf_2
liked
a Space
about 16 hours ago
m-ric/llm-race-to-the-top
liked
a Space
1 day ago
data-is-better-together/image-preferences-leaderboard
Articles
Organizations
Posts
70
Post
981
🗞️ 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗔𝗜 𝟮𝟬𝟮𝟰: 𝗔𝗻𝘁𝗵𝗿𝗼𝗽𝗶𝗰 𝗲𝗮𝘁𝗶𝗻𝗴 𝘂𝗽 𝗢𝗽𝗲𝗻𝗔𝗜, 𝗔𝗴𝗲𝗻𝘁𝘀 𝗿𝗮𝗺𝗽 𝘂𝗽 𝘁𝗼 𝟭𝟮% 𝗼𝗳 𝘂𝘀𝗲-𝗰𝗮𝘀𝗲𝘀, 𝗼𝗽𝗲𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 𝗺𝗮𝗸𝗲 𝟭𝟵% 𝗼𝗳 𝘂𝘀𝗮𝗴𝗲
Menlo Ventures surveyed 600 enterprise IT decision-makers for their 2024 report. They reveal that AI spending surged to $13.8 billion this year, more than 6x the $2.3 billion spent in 2023!
Companies are shifting from experimentation to serious implementation.
👷 Top enterprise use cases by adoption:
‣ Code copilots (51%)
- GitHub Copilot hit $300M revenue run rate
‣ Support chatbots (31%)
‣ RAG (28%)
‣ Data extraction/transformation (27%)
‣ Meeting summarization (25%)
📈 Market dynamics:
‣ OpenAI's enterprise share dropped from 50% to 34% 👎
‣ Anthropic doubled presence from 12% to 24% 🚀
‣ Open-source makes up 19% of usage 🤗
😬 Implementation challenges:
‣ 26% failed due to unexpected implementation costs
‣ 21% failed due to data privacy issues
‣ 18% failed due to disappointing ROI
‣ 15% failed due to hallucinations
Read the full report here 👉 https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/
Menlo Ventures surveyed 600 enterprise IT decision-makers for their 2024 report. They reveal that AI spending surged to $13.8 billion this year, more than 6x the $2.3 billion spent in 2023!
Companies are shifting from experimentation to serious implementation.
👷 Top enterprise use cases by adoption:
‣ Code copilots (51%)
- GitHub Copilot hit $300M revenue run rate
‣ Support chatbots (31%)
‣ RAG (28%)
‣ Data extraction/transformation (27%)
‣ Meeting summarization (25%)
📈 Market dynamics:
‣ OpenAI's enterprise share dropped from 50% to 34% 👎
‣ Anthropic doubled presence from 12% to 24% 🚀
‣ Open-source makes up 19% of usage 🤗
😬 Implementation challenges:
‣ 26% failed due to unexpected implementation costs
‣ 21% failed due to data privacy issues
‣ 18% failed due to disappointing ROI
‣ 15% failed due to hallucinations
Read the full report here 👉 https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/
Post
1141
Made a new app to visualize the LLM race ⇒ 𝗡𝗼 𝗘𝘂𝗿𝗼𝗽𝗲𝗮𝗻 𝗰𝗼𝗺𝗽𝗮𝗻𝘆 𝗶𝗻 𝘁𝗵𝗲 𝘁𝗼𝗽 𝟭𝟬 🇪🇺❌
See the app here 👉 m-ric/llm-race-to-the-top
I've adapted an app by @andrewrreed that tracks progress of LLMs ( andrewrreed/closed-vs-open-arena-elo), on the Chatbot Arena leaderboard, to compare companies from different countries.
The outcome is quite sad, as a Frenchman and European.
The top 10 is exclusively US 🇺🇸 and Chinese 🇨🇳 companies (after great Chinese LLM releases recently, like the Qwen2.5 series), with the notable exception of Mistral AI 🇫🇷.
American companies are making fast progress, Chinese ones even faster. Europe is at risk of being left behind. And the EU AI Act hasn't even come into force yet to slow down the EU market. We need to wake up 😬
⚠️ Caution: This Chatbot Arena ELO ranking is not the most accurate, especially at high scores like this, because LLM makers can game it to some extent.
See the app here 👉 m-ric/llm-race-to-the-top
I've adapted an app by @andrewrreed that tracks progress of LLMs ( andrewrreed/closed-vs-open-arena-elo), on the Chatbot Arena leaderboard, to compare companies from different countries.
The outcome is quite sad, as a Frenchman and European.
The top 10 is exclusively US 🇺🇸 and Chinese 🇨🇳 companies (after great Chinese LLM releases recently, like the Qwen2.5 series), with the notable exception of Mistral AI 🇫🇷.
American companies are making fast progress, Chinese ones even faster. Europe is at risk of being left behind. And the EU AI Act hasn't even come into force yet to slow down the EU market. We need to wake up 😬
⚠️ Caution: This Chatbot Arena ELO ranking is not the most accurate, especially at high scores like this, because LLM makers can game it to some extent.
Collections
14
-
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Paper • 2206.10789 • Published • 4 -
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
Paper • 2401.00448 • Published • 28 -
Training Compute-Optimal Large Language Models
Paper • 2203.15556 • Published • 10 -
Scaling Laws for Neural Language Models
Paper • 2001.08361 • Published • 6
spaces
23
pinned
Running
66
🗺️🏕️
AI Travel Planner
Plan your next vacation with the help of an AI!
pinned
Running
187
🍫👀
Chunk Visualizer
Pick a text splitter => visualize chunks. Great for RAG.
pinned
Running
on
Zero
118
✍👀
Beam Search Visualizer
View how beam search decoding works, in detail!
pinned
Running
101
🤔📊
Agent Data Analyst
Need to analyze data? Let a Llama-3.1 agent do it for you!
Running
on
CPU Upgrade
11
📈🚀
Llm Race To The Top
View Chatbot Arena ELO of top models increasing
Running
👦🏻👧🏼
Face Swap
Face-Swap-Roop
models
2
datasets
15
m-ric/newsletter
Updated
•
44
m-ric/songs_july_24
Viewer
•
Updated
•
85.6k
•
41
m-ric/huggingface_doc_qa_eval
Viewer
•
Updated
•
65
•
406
•
6
m-ric/agents_medium_benchmark
Viewer
•
Updated
•
172
•
50
•
3
m-ric/songs
Preview
•
Updated
•
19
m-ric/rag_scores_cookbook
Viewer
•
Updated
•
5
•
83
m-ric/images
Viewer
•
Updated
•
1
•
15
m-ric/agents_small_benchmark
Viewer
•
Updated
•
100
•
51
•
8
m-ric/huggingface_doc
Viewer
•
Updated
•
2.65k
•
2.06k
•
9
m-ric/transformers_documentation_en
Viewer
•
Updated
•
359
•
46