SmolVLM - small yet mighty Vision Language Model
โข
93
Merve Noyan
merve
AI & ML interests
VLMs, vision & co
Recent Activity
New activity
2 days ago
HuggingFaceTB/SmolVLM:Actual text streaming
updated
a Space
3 days ago
HuggingFaceTB/SmolVLM
Articles
Organizations
merve's activity
posted
an
update
3 days ago
Post
1891
The authors of ColPali trained a retrieval model based on SmolVLM ๐ค
vidore/colsmolvlm-alpha
TLDR;
- ColSmolVLM performs better than ColPali and DSE-Qwen2 on all English tasks
- ColSmolVLM is more memory efficient than ColQwen2 ๐
TLDR;
- ColSmolVLM performs better than ColPali and DSE-Qwen2 on all English tasks
- ColSmolVLM is more memory efficient than ColQwen2 ๐
posted
an
update
4 days ago
Post
3579
Small yet mighty! ๐ซ
We are releasing SmolVLM: a new 2B small vision language made for on-device use, fine-tunable on consumer GPU, immensely memory efficient ๐ค
We release three checkpoints under Apache 2.0: SmolVLM-Instruct, SmolVLM-Synthetic and SmolVLM-Base HuggingFaceTB/smolvlm-6740bd584b2dcbf51ecb1f39
Learn more from our blog here: huggingface.co/blog/smolvlm
This release comes with a demo, fine-tuning code, MLX integration and TRL integration for DPO ๐
Try the demo: HuggingFaceTB/SmolVLM
Fine-tuning Recipe: https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
Also TRL integration for DPO ๐
We are releasing SmolVLM: a new 2B small vision language made for on-device use, fine-tunable on consumer GPU, immensely memory efficient ๐ค
We release three checkpoints under Apache 2.0: SmolVLM-Instruct, SmolVLM-Synthetic and SmolVLM-Base HuggingFaceTB/smolvlm-6740bd584b2dcbf51ecb1f39
Learn more from our blog here: huggingface.co/blog/smolvlm
This release comes with a demo, fine-tuning code, MLX integration and TRL integration for DPO ๐
Try the demo: HuggingFaceTB/SmolVLM
Fine-tuning Recipe: https://github.com/huggingface/smollm/blob/main/finetuning/Smol_VLM_FT.ipynb
Also TRL integration for DPO ๐
posted
an
update
8 days ago
Post
2496
What a week! A recap for everything you missed โ๏ธ
merve/nov-22-releases-673fbbcfc1c97c4f411def07
Multimodal โจ
> Mistral AI
released Pixtral 124B, a gigantic open vision language model
> Llava-CoT (formerly known as Llava-o1) was released, a multimodal reproduction of o1 model by PKU
> OpenGVLab released MMPR: a new multimodal reasoning dataset
> Jina has released Jina-CLIP-v2 0.98B multilingual multimodal embeddings
> Apple released new SotA vision encoders AIMv2
LLMs ๐ฆ
> AllenAI dropped a huge release of models, datasets and scripts for Tรผlu, a family of models based on Llama 3.1 aligned with SFT, DPO and a new technique they have developed called RLVR
> Jina has released embeddings-v3: new multilingual embeddings with longer context
> Hugging Face released SmolTalk: synthetic dataset used to align SmolLM2 using supervised fine-tuning
> Microsoft released orca-agentinstruct-1M-v1: a gigantic instruction dataset of 1M synthetic instruction pairs
Image Generation ๐ผ๏ธ
> Black Forest Labs released Flux 1. tools: four new models for different image modifications and two LoRAs to do image conditioning and better steer generations
Lastly Hugging Face released a new library Observers: a lightweight SDK for monitoring interactions with AI APIs and easily store and browse them ๐
$ pip install observers
merve/nov-22-releases-673fbbcfc1c97c4f411def07
Multimodal โจ
> Mistral AI
released Pixtral 124B, a gigantic open vision language model
> Llava-CoT (formerly known as Llava-o1) was released, a multimodal reproduction of o1 model by PKU
> OpenGVLab released MMPR: a new multimodal reasoning dataset
> Jina has released Jina-CLIP-v2 0.98B multilingual multimodal embeddings
> Apple released new SotA vision encoders AIMv2
LLMs ๐ฆ
> AllenAI dropped a huge release of models, datasets and scripts for Tรผlu, a family of models based on Llama 3.1 aligned with SFT, DPO and a new technique they have developed called RLVR
> Jina has released embeddings-v3: new multilingual embeddings with longer context
> Hugging Face released SmolTalk: synthetic dataset used to align SmolLM2 using supervised fine-tuning
> Microsoft released orca-agentinstruct-1M-v1: a gigantic instruction dataset of 1M synthetic instruction pairs
Image Generation ๐ผ๏ธ
> Black Forest Labs released Flux 1. tools: four new models for different image modifications and two LoRAs to do image conditioning and better steer generations
Lastly Hugging Face released a new library Observers: a lightweight SDK for monitoring interactions with AI APIs and easily store and browse them ๐
$ pip install observers
- 3 replies
posted
an
update
8 days ago
Post
1446
Apple released AIMv2 ๐ a family of state-of-the-art open-set vision encoders
apple/aimv2-6720fe1558d94c7805f7688c
> like CLIP, but add a decoder and train on autoregression ๐คฏ
> 19 open models come in 300M, 600M, 1.2B, 2.7B with resolutions of 224, 336, 448
> Load and use with ๐ค transformers
apple/aimv2-6720fe1558d94c7805f7688c
> like CLIP, but add a decoder and train on autoregression ๐คฏ
> 19 open models come in 300M, 600M, 1.2B, 2.7B with resolutions of 224, 336, 448
> Load and use with ๐ค transformers
posted
an
update
9 days ago
Post
2892
your hugging face profile now has your recent activities ๐ค
posted
an
update
12 days ago
Post
1773
๐ค transformers pipelines now support vision language models for easy local inference ๐ซฐ๐ป
h/t @yonigozlan for shipping this ๐ฉ๐
you can also use inference API to infer hosted vision LMs (via Python, JS and cURL) https://huggingface.co/docs/api-inference/en/tasks/image-text-to-text
h/t @yonigozlan for shipping this ๐ฉ๐
you can also use inference API to infer hosted vision LMs (via Python, JS and cURL) https://huggingface.co/docs/api-inference/en/tasks/image-text-to-text
Reacted to
sayakpaul's
post with โค๏ธ
12 days ago
Post
2409
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers ๐งจWe currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
Reacted to
BlinkDL's
post with ๐ฅ
12 days ago
Post
2994
RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now:
BlinkDL/rwkv-6-world
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
Reacted to
davidberenstein1957's
post with ๐ฅ๐
12 days ago
Post
1898
For anyone who struggles with NER or information extraction with LLM.
We showed an efficient workflow for token classification including zero-shot suggestions and model fine-tuning with Argilla, GliNER, the NuMind NuExtract LLM and SpanMarker. @argilla
Video: https://youtu.be/JvLpaYgNd84?feature=shared
Notebooks and slides included to try it yourself ๐
We showed an efficient workflow for token classification including zero-shot suggestions and model fine-tuning with Argilla, GliNER, the NuMind NuExtract LLM and SpanMarker. @argilla
Video: https://youtu.be/JvLpaYgNd84?feature=shared
Notebooks and slides included to try it yourself ๐
Reacted to
erikkaum's
post with ๐๐ฅ
12 days ago
Post
1675
A while ago I started experimenting with compiling the Python interpreter to WASM.
To build a secure, fast, and lightweight sandbox for code execution โ ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
To build a secure, fast, and lightweight sandbox for code execution โ ideal for running LLM-generated Python code.
- Send code simply as a POST request
- 1-2ms startup times
Hack away:
https://github.com/ErikKaum/runner
Reacted to
AdinaY's
post with ๐
12 days ago
Post
1449
LLaMA Mesh ๐ฅ Unifying 3D Mesh Generation with Language Models
Model: Zhengyi/LLaMA-Mesh
Demo: Zhengyi/LLaMA-Mesh
Paper: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models (2411.09595)
โจ Unified 3D generation & text understanding.
โจ 3D meshes as plain text for seamless LLM integration.
โจ High-quality 3D outputs rivaling specialized models.
Model: Zhengyi/LLaMA-Mesh
Demo: Zhengyi/LLaMA-Mesh
Paper: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models (2411.09595)
โจ Unified 3D generation & text understanding.
โจ 3D meshes as plain text for seamless LLM integration.
โจ High-quality 3D outputs rivaling specialized models.
Reacted to
prithivMLmods's
post with ๐ค
12 days ago
Post
3909
Minimalistic Adapters ๐
๐Demo Here:
prithivMLmods/FLUX-LoRA-DLC
๐Model:
{ Quote Tuner } : prithivMLmods/Flux.1-Dev-Quote-LoRA
{ Stamp Art } : prithivMLmods/Flux.1-Dev-Stamp-Art-LoRA
{ Hand Sticky } : prithivMLmods/Flux.1-Dev-Hand-Sticky-LoRA
{ Poster HQ } : prithivMLmods/Flux.1-Dev-Poster-HQ-LoRA
{ Ctoon Min } : prithivMLmods/Flux.1-Dev-Ctoon-LoRA
๐Collection:
{ Flux LoRA Collection} : prithivMLmods/flux-lora-collections-66dd5908be2206cfaa8519be
{ LoRA Space Collection } : prithivMLmods/lora-space-collections-6714b72e0d49e1c97fbd6a32
๐For More Visit
https://huggingface.co/strangerzonehf
.
.
.
๐ค@prithivMLmods
๐Demo Here:
prithivMLmods/FLUX-LoRA-DLC
๐Model:
{ Quote Tuner } : prithivMLmods/Flux.1-Dev-Quote-LoRA
{ Stamp Art } : prithivMLmods/Flux.1-Dev-Stamp-Art-LoRA
{ Hand Sticky } : prithivMLmods/Flux.1-Dev-Hand-Sticky-LoRA
{ Poster HQ } : prithivMLmods/Flux.1-Dev-Poster-HQ-LoRA
{ Ctoon Min } : prithivMLmods/Flux.1-Dev-Ctoon-LoRA
๐Collection:
{ Flux LoRA Collection} : prithivMLmods/flux-lora-collections-66dd5908be2206cfaa8519be
{ LoRA Space Collection } : prithivMLmods/lora-space-collections-6714b72e0d49e1c97fbd6a32
๐For More Visit
https://huggingface.co/strangerzonehf
.
.
.
๐ค@prithivMLmods
- 3 replies
Reacted to
hexgrad's
post with ๐ฅ
13 days ago
Post
3191
Kokoro: a small, fast 80M param TTS model hosted on ZeroGPU at
hexgrad/Kokoro-TTS
- 3 replies
posted
an
update
14 days ago
Post
4816
OmniVision-968M: a new local VLM for edge devices, fast & small but performant
๐จ a new vision language model with 9x less image tokens, super efficient
๐ aligned with DPO for reducing hallucinations
โก๏ธ Apache 2.0 license ๐ฅ
Demo hf.co/spaces/NexaAIDev/omnivlm-dpo-demo
Model NexaAIDev/omnivision-968M
๐จ a new vision language model with 9x less image tokens, super efficient
๐ aligned with DPO for reducing hallucinations
โก๏ธ Apache 2.0 license ๐ฅ
Demo hf.co/spaces/NexaAIDev/omnivlm-dpo-demo
Model NexaAIDev/omnivision-968M
- 4 replies
Reacted to
AdinaY's
post with ๐๐ฅ
15 days ago
Post
2519
Letโs dive into the exciting releases from the Chinese community last week ๐ฅ๐
More details ๐ https://huggingface.co/zh-ai-community
Code model:
โจQwen 2.5 coder by Alibaba Qwen
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
โจOpenCoder by InflyAI - Fully open code model๐
infly/opencoder-672cec44bbb86c39910fb55e
Image model:
โจHunyuan3D-1.0 by Tencent
tencent/Hunyuan3D-1
MLLM:
โจJanusFlow by DeepSeek
deepseek-ai/JanusFlow-1.3B
deepseek-ai/JanusFlow-1.3B
โจMono-InternVL-2B by OpenGVlab
OpenGVLab/Mono-InternVL-2B
Video model:
โจCogVideoX 1.5 by ChatGLM
THUDM/CogVideoX1.5-5B-SAT
Audio model:
โจFish Agent by FishAudio
fishaudio/fish-agent-v0.1-3b
Dataset:
โจOPI dataset by BAAIBeijing
BAAI/OPI
More details ๐ https://huggingface.co/zh-ai-community
Code model:
โจQwen 2.5 coder by Alibaba Qwen
Qwen/qwen25-coder-66eaa22e6f99801bf65b0c2f
โจOpenCoder by InflyAI - Fully open code model๐
infly/opencoder-672cec44bbb86c39910fb55e
Image model:
โจHunyuan3D-1.0 by Tencent
tencent/Hunyuan3D-1
MLLM:
โจJanusFlow by DeepSeek
deepseek-ai/JanusFlow-1.3B
deepseek-ai/JanusFlow-1.3B
โจMono-InternVL-2B by OpenGVlab
OpenGVLab/Mono-InternVL-2B
Video model:
โจCogVideoX 1.5 by ChatGLM
THUDM/CogVideoX1.5-5B-SAT
Audio model:
โจFish Agent by FishAudio
fishaudio/fish-agent-v0.1-3b
Dataset:
โจOPI dataset by BAAIBeijing
BAAI/OPI
posted
an
update
15 days ago
Post
1953
Amazing past days at open ML, it's raining coding models, let's have a recap ๐ง๏ธ Find all models and datasets here
merve/nov-15-releases-67372d0ebdc354756a52ecd0
Models
๐ป Coding: Qwen team released two Qwen2.5-Coder checkpoints of 32B and 7B. Infly released OpenCoder: 1.5B and 8B coding models with instruction SFT'd versions and their datasets! ๐
๐ผ๏ธ Image/Video Gen: Alibaba vision lab released In-context LoRA -- 10 LoRA models on different themes based on Flux. Also Mochi the sota video generation model with A2.0 license now comes natively supported in diffusers ๐
๐ผ๏ธ VLMs/Multimodal: NexaAIDev released Omnivision 968M a new vision language model aligned with DPO for reducing hallucinations, also comes with GGUF ckpts ๐ Microsoft released LLM2CLIP, a new CLIP-like model with longer context window allowing complex text inputs and better search
๐ฎ AGI?: Etched released Oasis 500M, a diffusion based open world model that takes keyboard input and outputs gameplay ๐คฏ
Datasets
Common Corpus: A text dataset with 2T tokens with permissive license for EN/FR on various sources: code, science, finance, culture ๐
Models
๐ป Coding: Qwen team released two Qwen2.5-Coder checkpoints of 32B and 7B. Infly released OpenCoder: 1.5B and 8B coding models with instruction SFT'd versions and their datasets! ๐
๐ผ๏ธ Image/Video Gen: Alibaba vision lab released In-context LoRA -- 10 LoRA models on different themes based on Flux. Also Mochi the sota video generation model with A2.0 license now comes natively supported in diffusers ๐
๐ผ๏ธ VLMs/Multimodal: NexaAIDev released Omnivision 968M a new vision language model aligned with DPO for reducing hallucinations, also comes with GGUF ckpts ๐ Microsoft released LLM2CLIP, a new CLIP-like model with longer context window allowing complex text inputs and better search
๐ฎ AGI?: Etched released Oasis 500M, a diffusion based open world model that takes keyboard input and outputs gameplay ๐คฏ
Datasets
Common Corpus: A text dataset with 2T tokens with permissive license for EN/FR on various sources: code, science, finance, culture ๐
Reacted to
maxiw's
post with ๐
16 days ago
Post
4581
I was curious to see what people post here on HF so I created a dataset with all HF Posts:
maxiw/hf-posts
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
Some interesting stats:
Top 5 Authors by Total Impressions:
-----------------------------------
@merve : 171,783 impressions (68 posts)
@fdaudens : 135,253 impressions (81 posts)
@singhsidhukuldeep : 122,591 impressions (81 posts)
@akhaliq : 119,526 impressions (78 posts)
@MonsterMMORPG : 112,500 impressions (45 posts)
Top 5 Users by Number of Reactions Given:
----------------------------------------
@osanseviero : 1278 reactions
@clem : 910 reactions
@John6666 : 899 reactions
@victor : 674 reactions
@samusenps : 655 reactions
Top 5 Most Used Reactions:
-------------------------
โค๏ธ: 7048 times
๐ฅ: 5921 times
๐: 4856 times
๐: 2549 times
๐ค: 2065 times
- 10 replies