ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models
•
16
Ahmed Masry PRO
ahmed-masry
AI & ML interests
Multimodal Chart Understanding,
Multimodal Document AI,
Multimodal Vision - Language Models,
Recent Activity
New activity
22 days ago
ahmed-masry/ColFlor:Update README.md
updated
a model
about 1 month ago
ahmed-masry/ColFlor
Articles
Organizations
None yet
ahmed-masry's activity
Update README.md
1
#1 opened 23 days ago
by
omkar334
omkar334
updated
a
model
about 1 month ago
Reacted to
merve's
post with 🚀
about 1 month ago
Post
1966
It's raining depth estimation models ☔️
DepthPro is a zero-shot depth estimation model by Apple, it's fast, sharp and accurate 🔥
Demo: akhaliq/depth-pro
Model: apple/DepthPro
Paper page: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2410.02073)
The model consists of two encoders: an encoder for patches and an image encoder 🖼️ The outputs of both are merged to decode to depth maps and get the focal length.
The model outperforms the previous state-of-the-art models in average of various benchmarks 📑
DepthPro is a zero-shot depth estimation model by Apple, it's fast, sharp and accurate 🔥
Demo: akhaliq/depth-pro
Model: apple/DepthPro
Paper page: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2410.02073)
The model consists of two encoders: an encoder for patches and an image encoder 🖼️ The outputs of both are merged to decode to depth maps and get the focal length.
The model outperforms the previous state-of-the-art models in average of various benchmarks 📑
posted
an
update
about 1 month ago
Post
1156
🚀 Introducing ColFlor: An Efficient, OCR-Free Vision-Language Document Retrieval Model 🌟
Earlier this year, ColPali revolutionized document retrieval by eliminating the need for error-prone OCR pipelines. Instead, it directly processes the document images. However, with its 3 billion parameters, ColPali is computationally heavy for large-scale applications.
That’s where ColFlor comes in—a smaller, faster alternative! 🎉 At 17x smaller than ColPali, ColFlor offers a more efficient, OCR-free document retrieval solution, making it ideal for users with limited computing resources (GPU Poor). 💡
Key Highlights:
🧠 174M parameters (vs. 3B for ColPali)
⚡ 9.8x faster query encoding, 5.25x faster image encoding
📉 Only 1.8% performance drop on text-rich English documents
Check out the full blog post for more insights on modeling, training, and evaluations across various document retrieval tasks! 🚀
Also, feel free to try our demo on huggingface 🤗
🔗 Resources:
📄 Blog post: https://huggingface.co/blog/ahmed-masry/colflor
🧠 Model: ahmed-masry/ColFlor
💻 Demo: ahmed-masry/ColFlor-Demo
🏋️♂️ Training code: https://github.com/AhmedMasryKU/colflor
📊 Evaluation code: https://github.com/AhmedMasryKU/vidore-benchmark-colflor
Earlier this year, ColPali revolutionized document retrieval by eliminating the need for error-prone OCR pipelines. Instead, it directly processes the document images. However, with its 3 billion parameters, ColPali is computationally heavy for large-scale applications.
That’s where ColFlor comes in—a smaller, faster alternative! 🎉 At 17x smaller than ColPali, ColFlor offers a more efficient, OCR-free document retrieval solution, making it ideal for users with limited computing resources (GPU Poor). 💡
Key Highlights:
🧠 174M parameters (vs. 3B for ColPali)
⚡ 9.8x faster query encoding, 5.25x faster image encoding
📉 Only 1.8% performance drop on text-rich English documents
Check out the full blog post for more insights on modeling, training, and evaluations across various document retrieval tasks! 🚀
Also, feel free to try our demo on huggingface 🤗
🔗 Resources:
📄 Blog post: https://huggingface.co/blog/ahmed-masry/colflor
🧠 Model: ahmed-masry/ColFlor
💻 Demo: ahmed-masry/ColFlor-Demo
🏋️♂️ Training code: https://github.com/AhmedMasryKU/colflor
📊 Evaluation code: https://github.com/AhmedMasryKU/vidore-benchmark-colflor
updated
a
Space
about 1 month ago
liked
a
dataset
about 2 months ago
published
an
article
about 2 months ago
Article
ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models
By
•
•
16
how the handle hallucination?
1
#2 opened 4 months ago
by
RaushanA
RaushanA
OlenaVyn
updated
a
dataset
4 months ago
liked
a
Space
4 months ago
updated
a
dataset
4 months ago
Missing license
1
#1 opened 4 months ago
by
maskedviper
maskedviper
updated
a
model
4 months ago
commented
a
paper
5 months ago
updated
a
dataset
5 months ago
posted
an
update
5 months ago
Post
3370
📢 Exciting News! Our latest paper "ChartGemma" is out! 📊
🧵1/3: ChartGemma overcomes existing chart models key limitations that rely too much on data tables. Instead, it is trained on data generated directly from chart images, capturing crucial visual trends📸🔍
🧵2/3: ChartGemma builds upon PaliGemma from Google Research and is fine-tuned on a high-quality visual instruction tuning dataset generated from Gemini Flash 1.5. 🌟📊
🧵3/3: Achieves state-of-the-art results in chart summarization, question answering, and fact-checking tasks. 🏅📊 It can also generate more accurate and realistic chart summaries. 📝🔍
Our model and data are publicly available. We also have a cool web demo. Check it out! 🚀
Demo: ahmed-masry/ChartGemma
Code: https://github.com/vis-nlp/ChartGemma
Paper: ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild (2407.04172)
🧵1/3: ChartGemma overcomes existing chart models key limitations that rely too much on data tables. Instead, it is trained on data generated directly from chart images, capturing crucial visual trends📸🔍
🧵2/3: ChartGemma builds upon PaliGemma from Google Research and is fine-tuned on a high-quality visual instruction tuning dataset generated from Gemini Flash 1.5. 🌟📊
🧵3/3: Achieves state-of-the-art results in chart summarization, question answering, and fact-checking tasks. 🏅📊 It can also generate more accurate and realistic chart summaries. 📝🔍
Our model and data are publicly available. We also have a cool web demo. Check it out! 🚀
Demo: ahmed-masry/ChartGemma
Code: https://github.com/vis-nlp/ChartGemma
Paper: ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild (2407.04172)
Reacted to
fdaudens's
post with 🔥
5 months ago
Post
587
Just updated the Journalists on 🤗 community with two new AI tools! 🚀📊
Check them out:
- Free video transcription & smart summary tool artificialguybr/Video-Transcription-Smart-Summary
- ChartGemma - next-level chart analysis focusing on visual trends ahmed-masry/ChartGemma
Media folks: Join our community for more tools! https://huggingface.co/JournalistsonHF
Check them out:
- Free video transcription & smart summary tool artificialguybr/Video-Transcription-Smart-Summary
- ChartGemma - next-level chart analysis focusing on visual trends ahmed-masry/ChartGemma
Media folks: Join our community for more tools! https://huggingface.co/JournalistsonHF
authored
2
papers
5 months ago