|
--- |
|
language: |
|
- ru |
|
- en |
|
library_name: transformers |
|
pipeline_tag: feature-extraction |
|
--- |
|
|
|
# ruclip-vit-base-patch32-384 |
|
|
|
**RuCLIP** (**Ru**ssian **C**ontrastive **L**anguage–**I**mage **P**retraining) is a multimodal model |
|
for obtaining images and text similarities and rearranging captions and pictures. |
|
RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and |
|
multimodal learning. |
|
|
|
Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices](https://sberdevices.ru/) teams. |
|
|
|
- Task: `text ranking`; `image ranking`; `zero-shot image classification`; |
|
- Type: `encoder` |
|
- Num Parameters: `150M` |
|
- Training Data Volume: `240 million text-image pairs` |
|
- Language: `Russian` |
|
- Context Length: `77` |
|
- Transformer Layers: `12` |
|
- Transformer Width: `512` |
|
- Transformer Heads: `8` |
|
- Image Size: `384` |
|
- Vision Layers: `12` |
|
- Vision Width: `768` |
|
- Vision Patch Size: `32` |
|
|
|
## Usage [Github](https://github.com/sberbank-ai/ru-clip) |
|
|
|
``` |
|
pip install ruclip |
|
``` |
|
|
|
```python |
|
clip, processor = ruclip.load("ruclip-vit-base-patch32-384", device="cuda") |
|
``` |
|
|
|
## Performance |
|
|
|
We have evaluated the performance on the following datasets: |
|
|
|
| Dataset | Metric Name | Metric Result | |
|
| :------------ | :------------- | :------------ | |
|
| Food101 | acc | 0.642 | |
|
| CIFAR10 | acc | 0.862 | |
|
| CIFAR100 | acc | 0.529 | |
|
| Birdsnap | acc | 0.161 | |
|
| SUN397 | acc | 0.510 | |
|
| Stanford Cars | acc | 0.572 | |
|
| DTD | acc | 0.390 | |
|
| MNIST | acc | 0.404 | |
|
| STL10 | acc | 0.946 | |
|
| PCam | acc | 0.506 | |
|
| CLEVR | acc | 0.188 | |
|
| Rendered SST2 | acc | 0.508 | |
|
| ImageNet | acc | 0.451 | |
|
| FGVC Aircraft | mean-per-class | 0.053 | |
|
| Oxford Pets | mean-per-class | 0.587 | |
|
| Caltech101 | mean-per-class | 0.834 | |
|
| Flowers102 | mean-per-class | 0.449 | |
|
| HatefulMemes | roc-auc | 0.537 | |
|
|
|
# Authors |
|
|
|
- Alex Shonenkov: [Github](https://github.com/shonenkov), [Kaggle GM](https://www.kaggle.com/shonenkov) |
|
- Daniil Chesakov: [Github](https://github.com/Danyache) |
|
- Denis Dimitrov: [Github](https://github.com/denndimitrov) |
|
- Igor Pavlov: [Github](https://github.com/boomb0om) |
