base_model: HuggingFaceTB/SmolVLM-Instruct
library_name: peft
license: apache-2.0
datasets:
- HuggingFaceH4/rlaif-v_formatted
language:
- en
pipeline_tag: image-text-to-text
tags:
- trl
- dpo
SmolVLM-Instruct-DPO
SmolVLM is a compact open multimodal model that accepts arbitrary sequences of image and text inputs to produce text outputs. Designed for efficiency, SmolVLM can answer questions about images, describe visual content, create stories grounded on multiple images, or function as a pure language model without visual inputs. Its lightweight architecture makes it suitable for on-device applications while maintaining strong performance on multimodal tasks.
Model Summary
- Developed by: Hugging Face 🤗
- Model type: Multi-modal model (image+text)
- Language(s) (NLP): English
- License: Apache 2.0
- Architecture: Based on Idefics3 (see technical summary)
Resources
- Demo: SmolVLM Demo
- Blog: [More Information Needed]
- Technical Report: [More Information Needed]
- Repository: [More Information Needed]
Uses
SmolVLM can be used for inference on multimodal (image + text) tasks where the input comprises text queries along with one or more images. Text and images can be interleaved arbitrarily, enabling tasks like image captioning, visual question answering, and storytelling based on visual content. The model does not support image generation.
Direct Use
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
Training Details
Training Data
HuggingFaceH4/rlaif-v_formatted
Training Procedure
accelerate launch --config_file examples/accelerate_configs/multi_gpu.yaml examples/scripts/dpo_vlm.py --dataset_name HuggingFaceH4/rlaif-v_formatted --model_name_or_path HuggingFaceTB/SmolVLM-Instruct --per_device_train_batch_size 8 --gradient_accumulation_steps 32 --dataset_num_proc 32 --output_dir dpo_smolvlm_rlaif-v --bf16 --torch_dtype bfloat16 --use_peft --lora_target_modules=all-linear exit
Framework versions
- PEFT 0.13.2