Upload 351 files

.gitattributes

@@ -35,3 +35,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 Text2LIVE-main/Text2LIVE-main/data/data/pretrained_nla_models/blackswan/checkpoint filter=lfs diff=lfs merge=lfs -text
 Text2LIVE-main/Text2LIVE-main/data/data/pretrained_nla_models/car-turn/checkpoint filter=lfs diff=lfs merge=lfs -text
 Text2LIVE-main/Text2LIVE-main/data/data/pretrained_nla_models/libby/checkpoint filter=lfs diff=lfs merge=lfs -text
+Text2LIVE-main/data/data/pretrained_nla_models/blackswan/checkpoint filter=lfs diff=lfs merge=lfs -text
+Text2LIVE-main/data/data/pretrained_nla_models/car-turn/checkpoint filter=lfs diff=lfs merge=lfs -text
+Text2LIVE-main/data/data/pretrained_nla_models/libby/checkpoint filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,3 @@
+/wandb/
+__pycache__/
+/idea

CLIP/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2021 OpenAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

CLIP/MANIFEST.in

@@ -0,0 +1 @@
+include clip/bpe_simple_vocab_16e6.txt.gz

CLIP/README.md

@@ -0,0 +1,193 @@
+# CLIP
+
+[[Blog]](https://openai.com/blog/clip/) [[Paper]](https://arxiv.org/abs/2103.00020) [[Model Card]](model-card.md) [[Colab]](https://colab.research.google.com/github/openai/clip/blob/master/notebooks/Interacting_with_CLIP.ipynb)
+
+CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.
+
+
+
+## Approach
+
+![CLIP](CLIP.png)
+
+
+
+## Usage
+
+First, [install PyTorch 1.7.1](https://pytorch.org/get-started/locally/) and torchvision, as well as small additional dependencies, and then install this repo as a Python package. On a CUDA GPU machine, the following will do the trick:
+
+```bash
+$ conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0
+$ pip install ftfy regex tqdm
+$ pip install git+https://github.com/openai/CLIP.git
+```
+
+Replace `cudatoolkit=11.0` above with the appropriate CUDA version on your machine or `cpuonly` when installing on a machine without a GPU.
+
+```python
+import torch
+import clip
+from PIL import Image
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model, preprocess = clip.load("ViT-B/32", device=device)
+
+image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
+text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)
+
+with torch.no_grad():
+    image_features = model.encode_image(image)
+    text_features = model.encode_text(text)
+    
+    logits_per_image, logits_per_text = model(image, text)
+    probs = logits_per_image.softmax(dim=-1).cpu().numpy()
+
+print("Label probs:", probs)  # prints: [[0.9927937  0.00421068 0.00299572]]
+```
+
+
+## API
+
+The CLIP module `clip` provides the following methods:
+
+#### `clip.available_models()`
+
+Returns the names of the available CLIP models.
+
+#### `clip.load(name, device=..., jit=False)`
+
+Returns the model and the TorchVision transform needed by the model, specified by the model name returned by `clip.available_models()`. It will download the model as necessary. The `name` argument can also be a path to a local checkpoint.
+
+The device to run the model can be optionally specified, and the default is to use the first CUDA device if there is any, otherwise the CPU. When `jit` is `False`, a non-JIT version of the model will be loaded.
+
+#### `clip.tokenize(text: Union[str, List[str]], context_length=77)`
+
+Returns a LongTensor containing tokenized sequences of given text input(s). This can be used as the input to the model
+
+---
+
+The model returned by `clip.load()` supports the following methods:
+
+#### `model.encode_image(image: Tensor)`
+
+Given a batch of images, returns the image features encoded by the vision portion of the CLIP model.
+
+#### `model.encode_text(text: Tensor)`
+
+Given a batch of text tokens, returns the text features encoded by the language portion of the CLIP model.
+
+#### `model(image: Tensor, text: Tensor)`
+
+Given a batch of images and a batch of text tokens, returns two Tensors, containing the logit scores corresponding to each image and text input. The values are cosine similarities between the corresponding image and text features, times 100.
+
+
+
+## More Examples
+
+### Zero-Shot Prediction
+
+The code below performs zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html), and predicts the most likely labels among the 100 textual labels from the dataset.
+
+```python
+import os
+import clip
+import torch
+from torchvision.datasets import CIFAR100
+
+# Load the model
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model, preprocess = clip.load('ViT-B/32', device)
+
+# Download the dataset
+cifar100 = CIFAR100(root=os.path.expanduser("~/.cache"), download=True, train=False)
+
+# Prepare the inputs
+image, class_id = cifar100[3637]
+image_input = preprocess(image).unsqueeze(0).to(device)
+text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device)
+
+# Calculate features
+with torch.no_grad():
+    image_features = model.encode_image(image_input)
+    text_features = model.encode_text(text_inputs)
+
+# Pick the top 5 most similar labels for the image
+image_features /= image_features.norm(dim=-1, keepdim=True)
+text_features /= text_features.norm(dim=-1, keepdim=True)
+similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+values, indices = similarity[0].topk(5)
+
+# Print the result
+print("\nTop predictions:\n")
+for value, index in zip(values, indices):
+    print(f"{cifar100.classes[index]:>16s}: {100 * value.item():.2f}%")
+```
+
+The output will look like the following (the exact numbers may be slightly different depending on the compute device):
+
+```
+Top predictions:
+
+           snake: 65.31%
+          turtle: 12.29%
+    sweet_pepper: 3.83%
+          lizard: 1.88%
+       crocodile: 1.75%
+```
+
+Note that this example uses the `encode_image()` and `encode_text()` methods that return the encoded features of given inputs.
+
+
+### Linear-probe evaluation
+
+The example below uses [scikit-learn](https://scikit-learn.org/) to perform logistic regression on image features.
+
+```python
+import os
+import clip
+import torch
+
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from torch.utils.data import DataLoader
+from torchvision.datasets import CIFAR100
+from tqdm import tqdm
+
+# Load the model
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model, preprocess = clip.load('ViT-B/32', device)
+
+# Load the dataset
+root = os.path.expanduser("~/.cache")
+train = CIFAR100(root, download=True, train=True, transform=preprocess)
+test = CIFAR100(root, download=True, train=False, transform=preprocess)
+
+
+def get_features(dataset):
+    all_features = []
+    all_labels = []
+    
+    with torch.no_grad():
+        for images, labels in tqdm(DataLoader(dataset, batch_size=100)):
+            features = model.encode_image(images.to(device))
+
+            all_features.append(features)
+            all_labels.append(labels)
+
+    return torch.cat(all_features).cpu().numpy(), torch.cat(all_labels).cpu().numpy()
+
+# Calculate the image features
+train_features, train_labels = get_features(train)
+test_features, test_labels = get_features(test)
+
+# Perform logistic regression
+classifier = LogisticRegression(random_state=0, C=0.316, max_iter=1000, verbose=1)
+classifier.fit(train_features, train_labels)
+
+# Evaluate using the logistic regression classifier
+predictions = classifier.predict(test_features)
+accuracy = np.mean((test_labels == predictions).astype(np.float)) * 100.
+print(f"Accuracy = {accuracy:.3f}")
+```
+
+Note that the `C` value should be determined via a hyperparameter sweep using a validation split.

CLIP/clip/__init__.py

@@ -0,0 +1 @@
+from .clip import *

CLIP/clip/bpe_simple_vocab_16e6.txt.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a
+size 1356917

CLIP/clip/clip.py

@@ -0,0 +1,231 @@
+import hashlib
+import os
+import urllib
+import warnings
+from typing import Any, Union, List
+from pkg_resources import packaging
+
+import torch
+from PIL import Image
+from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize
+from tqdm import tqdm
+
+from .model import build_model
+from .simple_tokenizer import SimpleTokenizer as _Tokenizer
+
+try:
+    from torchvision.transforms import InterpolationMode
+    BICUBIC = InterpolationMode.BICUBIC
+except ImportError:
+    BICUBIC = Image.BICUBIC
+
+
+if packaging.version.parse(torch.__version__) < packaging.version.parse("1.7.1"):
+    warnings.warn("PyTorch version 1.7.1 or higher is recommended")
+
+
+__all__ = ["available_models", "load", "tokenize"]
+_tokenizer = _Tokenizer()
+
+_MODELS = {
+    "RN50": "https://openaipublic.azureedge.net/clip/models/afeb0e10f9e5a86da6080e35cf09123aca3b358a0c3e3b6c78a7b63bc04b6762/RN50.pt",
+    "RN101": "https://openaipublic.azureedge.net/clip/models/8fa8567bab74a42d41c5915025a8e4538c3bdbe8804a470a72f30b0d94fab599/RN101.pt",
+    "RN50x4": "https://openaipublic.azureedge.net/clip/models/7e526bd135e493cef0776de27d5f42653e6b4c8bf9e0f653bb11773263205fdd/RN50x4.pt",
+    "RN50x16": "https://openaipublic.azureedge.net/clip/models/52378b407f34354e150460fe41077663dd5b39c54cd0bfd2b27167a4a06ec9aa/RN50x16.pt",
+    "RN50x64": "https://openaipublic.azureedge.net/clip/models/be1cfb55d75a9666199fb2206c106743da0f6468c9d327f3e0d0a543a9919d9c/RN50x64.pt",
+    "ViT-B/32": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt",
+    "ViT-B/16": "https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt",
+    "ViT-L/14": "https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt",
+}
+
+
+def _download(url: str, root: str):
+    os.makedirs(root, exist_ok=True)
+    filename = os.path.basename(url)
+
+    expected_sha256 = url.split("/")[-2]
+    download_target = os.path.join(root, filename)
+
+    if os.path.exists(download_target) and not os.path.isfile(download_target):
+        raise RuntimeError(f"{download_target} exists and is not a regular file")
+
+    if os.path.isfile(download_target):
+        if hashlib.sha256(open(download_target, "rb").read()).hexdigest() == expected_sha256:
+            return download_target
+        else:
+            warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file")
+
+    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
+        with tqdm(total=int(source.info().get("Content-Length")), ncols=80, unit='iB', unit_scale=True, unit_divisor=1024) as loop:
+            while True:
+                buffer = source.read(8192)
+                if not buffer:
+                    break
+
+                output.write(buffer)
+                loop.update(len(buffer))
+
+    if hashlib.sha256(open(download_target, "rb").read()).hexdigest() != expected_sha256:
+        raise RuntimeError(f"Model has been downloaded but the SHA256 checksum does not not match")
+
+    return download_target
+
+
+def _convert_image_to_rgb(image):
+    return image.convert("RGB")
+
+
+def _transform(n_px):
+    return Compose([
+        Resize(n_px, interpolation=BICUBIC),
+        CenterCrop(n_px),
+        _convert_image_to_rgb,
+        ToTensor(),
+        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
+    ])
+
+
+def available_models() -> List[str]:
+    """Returns the names of available CLIP models"""
+    return list(_MODELS.keys())
+
+
+def load(name: str, device: Union[str, torch.device] = "cuda" if torch.cuda.is_available() else "cpu", jit: bool = False, download_root: str = None):
+    """Load a CLIP model
+
+    Parameters
+    ----------
+    name : str
+        A model name listed by `clip.available_models()`, or the path to a model checkpoint containing the state_dict
+
+    device : Union[str, torch.device]
+        The device to put the loaded model
+
+    jit : bool
+        Whether to load the optimized JIT model or more hackable non-JIT model (default).
+
+    download_root: str
+        path to download the model files; by default, it uses "~/.cache/clip"
+
+    Returns
+    -------
+    model : torch.nn.Module
+        The CLIP model
+
+    preprocess : Callable[[PIL.Image], torch.Tensor]
+        A torchvision transform that converts a PIL image into a tensor that the returned model can take as its input
+    """
+    if name in _MODELS:
+        model_path = _download(_MODELS[name], download_root or os.path.expanduser("~/.cache/clip"))
+    elif os.path.isfile(name):
+        model_path = name
+    else:
+        raise RuntimeError(f"Model {name} not found; available models = {available_models()}")
+
+    try:
+        # loading JIT archive
+        model = torch.jit.load(model_path, map_location=device if jit else "cpu").eval()
+        state_dict = None
+    except RuntimeError:
+        # loading saved state dict
+        if jit:
+            warnings.warn(f"File {model_path} is not a JIT archive. Loading as a state dict instead")
+            jit = False
+        state_dict = torch.load(model_path, map_location="cpu")
+
+    if not jit:
+        model = build_model(state_dict or model.state_dict()).to(device)
+        if str(device) == "cpu":
+            model.float()
+        return model, _transform(model.visual.input_resolution)
+
+    # patch the device names
+    device_holder = torch.jit.trace(lambda: torch.ones([]).to(torch.device(device)), example_inputs=[])
+    device_node = [n for n in device_holder.graph.findAllNodes("prim::Constant") if "Device" in repr(n)][-1]
+
+    def patch_device(module):
+        try:
+            graphs = [module.graph] if hasattr(module, "graph") else []
+        except RuntimeError:
+            graphs = []
+
+        if hasattr(module, "forward1"):
+            graphs.append(module.forward1.graph)
+
+        for graph in graphs:
+            for node in graph.findAllNodes("prim::Constant"):
+                if "value" in node.attributeNames() and str(node["value"]).startswith("cuda"):
+                    node.copyAttributes(device_node)
+
+    model.apply(patch_device)
+    patch_device(model.encode_image)
+    patch_device(model.encode_text)
+
+    # patch dtype to float32 on CPU
+    if str(device) == "cpu":
+        float_holder = torch.jit.trace(lambda: torch.ones([]).float(), example_inputs=[])
+        float_input = list(float_holder.graph.findNode("aten::to").inputs())[1]
+        float_node = float_input.node()
+
+        def patch_float(module):
+            try:
+                graphs = [module.graph] if hasattr(module, "graph") else []
+            except RuntimeError:
+                graphs = []
+
+            if hasattr(module, "forward1"):
+                graphs.append(module.forward1.graph)
+
+            for graph in graphs:
+                for node in graph.findAllNodes("aten::to"):
+                    inputs = list(node.inputs())
+                    for i in [1, 2]:  # dtype can be the second or third argument to aten::to()
+                        if inputs[i].node()["value"] == 5:
+                            inputs[i].node().copyAttributes(float_node)
+
+        model.apply(patch_float)
+        patch_float(model.encode_image)
+        patch_float(model.encode_text)
+
+        model.float()
+
+    return model, _transform(model.input_resolution.item())
+
+
+def tokenize(texts: Union[str, List[str]], context_length: int = 77, truncate: bool = False) -> torch.LongTensor:
+    """
+    Returns the tokenized representation of given input string(s)
+
+    Parameters
+    ----------
+    texts : Union[str, List[str]]
+        An input string or a list of input strings to tokenize
+
+    context_length : int
+        The context length to use; all CLIP models use 77 as the context length
+
+    truncate: bool
+        Whether to truncate the text in case its encoding is longer than the context length
+
+    Returns
+    -------
+    A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]
+    """
+    if isinstance(texts, str):
+        texts = [texts]
+
+    sot_token = _tokenizer.encoder["<|startoftext|>"]
+    eot_token = _tokenizer.encoder["<|endoftext|>"]
+    all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
+    result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)
+
+    for i, tokens in enumerate(all_tokens):
+        if len(tokens) > context_length:
+            if truncate:
+                tokens = tokens[:context_length]
+                tokens[-1] = eot_token
+            else:
+                raise RuntimeError(f"Input {texts[i]} is too long for context length {context_length}")
+        result[i, :len(tokens)] = torch.tensor(tokens)
+
+    return result

CLIP/clip/model.py

@@ -0,0 +1,484 @@
+from collections import OrderedDict
+from typing import Tuple, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from torch import nn
+import math
+
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1):
+        super().__init__()
+
+        # all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
+        self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+
+        self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+
+        self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()
+
+        self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = None
+        self.stride = stride
+
+        if stride > 1 or inplanes != planes * Bottleneck.expansion:
+            # downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
+            self.downsample = nn.Sequential(OrderedDict([
+                ("-1", nn.AvgPool2d(stride)),
+                ("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
+                ("1", nn.BatchNorm2d(planes * self.expansion))
+            ]))
+
+    def forward(self, x: torch.Tensor):
+        identity = x
+
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.relu(self.bn2(self.conv2(out)))
+        out = self.avgpool(out)
+        out = self.bn3(self.conv3(out))
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu(out)
+        return out
+
+
+class AttentionPool2d(nn.Module):
+    def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
+        super().__init__()
+        self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+        self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
+        self.num_heads = num_heads
+
+    def forward(self, x):
+        x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1)  # NCHW -> (HW)NC
+        x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
+        x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
+        x, _ = F.multi_head_attention_forward(
+            query=x, key=x, value=x,
+            embed_dim_to_check=x.shape[-1],
+            num_heads=self.num_heads,
+            q_proj_weight=self.q_proj.weight,
+            k_proj_weight=self.k_proj.weight,
+            v_proj_weight=self.v_proj.weight,
+            in_proj_weight=None,
+            in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
+            bias_k=None,
+            bias_v=None,
+            add_zero_attn=False,
+            dropout_p=0,
+            out_proj_weight=self.c_proj.weight,
+            out_proj_bias=self.c_proj.bias,
+            use_separate_proj_weight=True,
+            training=self.training,
+            need_weights=False
+        )
+
+        return x[0]
+
+
+class ModifiedResNet(nn.Module):
+    """
+    A ResNet class that is similar to torchvision's but contains the following changes:
+    - There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
+    - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
+    - The final pooling layer is a QKV attention instead of an average pool
+    """
+
+    def __init__(self, layers, output_dim, heads, input_resolution=224, width=64):
+        super().__init__()
+        self.output_dim = output_dim
+        self.input_resolution = input_resolution
+
+        # the 3-layer stem
+        self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(width // 2)
+        self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(width // 2)
+        self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(width)
+        self.avgpool = nn.AvgPool2d(2)
+        self.relu = nn.ReLU(inplace=True)
+
+        # residual layers
+        self._inplanes = width  # this is a *mutable* variable used during construction
+        self.layer1 = self._make_layer(width, layers[0])
+        self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
+        self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
+        self.layer4 = self._make_layer(width * 8, layers[3], stride=2)
+
+        embed_dim = width * 32  # the ResNet feature dimension
+        self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)
+
+    def _make_layer(self, planes, blocks, stride=1):
+        layers = [Bottleneck(self._inplanes, planes, stride)]
+
+        self._inplanes = planes * Bottleneck.expansion
+        for _ in range(1, blocks):
+            layers.append(Bottleneck(self._inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        def stem(x):
+            for conv, bn in [(self.conv1, self.bn1), (self.conv2, self.bn2), (self.conv3, self.bn3)]:
+                x = self.relu(bn(conv(x)))
+            x = self.avgpool(x)
+            return x
+
+        x = x.type(self.conv1.weight.dtype)
+        x = stem(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.attnpool(x)
+
+        return x
+
+
+class LayerNorm(nn.LayerNorm):
+    """Subclass torch's LayerNorm to handle fp16."""
+
+    def forward(self, x: torch.Tensor):
+        orig_type = x.dtype
+        ret = super().forward(x.type(torch.float32))
+        return ret.type(orig_type)
+
+
+class QuickGELU(nn.Module):
+    def forward(self, x: torch.Tensor):
+        return x * torch.sigmoid(1.702 * x)
+
+
+class ResidualAttentionBlock(nn.Module):
+    def __init__(self, d_model: int, n_head: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+
+        self.attn = nn.MultiheadAttention(d_model, n_head)
+        self.ln_1 = LayerNorm(d_model)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, d_model * 4)),
+            ("gelu", QuickGELU()),
+            ("c_proj", nn.Linear(d_model * 4, d_model))
+        ]))
+        self.ln_2 = LayerNorm(d_model)
+        self.attn_mask = attn_mask
+
+    def attention(self, x: torch.Tensor):
+        self.attn_mask = self.attn_mask.to(dtype=x.dtype, device=x.device) if self.attn_mask is not None else None
+        return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]
+
+    def forward(self, x: torch.Tensor):
+        x = x + self.attention(self.ln_1(x))
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Transformer(nn.Module):
+    def __init__(self, width: int, layers: int, heads: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+        self.resblocks = nn.Sequential(*[ResidualAttentionBlock(width, heads, attn_mask) for _ in range(layers)])
+
+    def forward(self, x: torch.Tensor):
+        return self.resblocks(x)
+
+
+class VisionTransformer(nn.Module):
+    def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int):
+        super().__init__()
+        self.input_resolution = input_resolution
+        self.output_dim = output_dim
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+
+        scale = width ** -0.5
+        self.class_embedding = nn.Parameter(scale * torch.randn(width))
+        self.positional_embedding = nn.Parameter(scale * torch.randn((input_resolution // patch_size) ** 2 + 1, width))
+        self.ln_pre = LayerNorm(width)
+
+        self.transformer = Transformer(width, layers, heads)
+
+        self.ln_post = LayerNorm(width)
+        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))
+
+    # https://github.com/facebookresearch/dino
+    def interpolate_pos_encoding(self, x, w, h):
+        positional_embedding = self.positional_embedding.unsqueeze(0)
+        patch_size = self.conv1.kernel_size[0]
+
+        npatch = x.shape[1] - 1
+        N = positional_embedding.shape[1] - 1
+        if npatch == N and w == h:
+            return positional_embedding
+        class_pos_embed = positional_embedding[:, 0]
+        patch_pos_embed = positional_embedding[:, 1:]
+        dim = x.shape[-1]
+
+        w0 = w // patch_size
+        h0 = h // patch_size
+
+        # we add a small number to avoid floating point error in the interpolation
+        # see discussion at https://github.com/facebookresearch/dino/issues/8
+        w0, h0 = w0 + 0.1, h0 + 0.1
+        patch_pos_embed = nn.functional.interpolate(
+            patch_pos_embed.reshape(1, int(math.sqrt(N)), int(math.sqrt(N)), dim).permute(0, 3, 1, 2),
+            scale_factor=(w0 / math.sqrt(N), h0 / math.sqrt(N)),
+            mode='bicubic',
+        )
+        assert int(w0) == patch_pos_embed.shape[-2] and int(h0) == patch_pos_embed.shape[-1]
+        patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)
+        return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1)
+
+    def forward(self, x: torch.Tensor):
+        x = self.transformer_first_blocks_forward(x)
+        x = self.transformer.resblocks[-1](x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_post(x[:, 0, :])
+
+        if self.proj is not None:
+            x = x @ self.proj
+
+        return x
+
+    def transformer_first_blocks_forward(self, x):
+        h, w = x.shape[-2:]
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+        x = torch.cat(
+            [self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device),
+             x], dim=1)  # shape = [*, grid ** 2 + 1, width]
+        positional_embedding = self.interpolate_pos_encoding(x, w, h)
+        x = x + positional_embedding.to(x.dtype)
+        # x = x + self.positional_embedding.to(x.dtype)
+        x = self.ln_pre(x)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer.resblocks[:-1](x)
+        return x
+
+    @staticmethod
+    def attn_cosine_sim(x, eps=1e-08):
+        norm = x.norm(dim=2, keepdim=True)
+        factor = torch.clamp(norm @ norm.permute(0, 2, 1), min=eps)  # shape [1, t, t]
+        sim_matrix = (x @ x.permute(0, 2, 1)) / factor  # shape [1, t, t]
+        return sim_matrix
+
+
+class CLIP(nn.Module):
+    def __init__(self,
+                 embed_dim: int,
+                 # vision
+                 image_resolution: int,
+                 vision_layers: Union[Tuple[int, int, int, int], int],
+                 vision_width: int,
+                 vision_patch_size: int,
+                 # text
+                 context_length: int,
+                 vocab_size: int,
+                 transformer_width: int,
+                 transformer_heads: int,
+                 transformer_layers: int
+                 ):
+        super().__init__()
+
+        self.context_length = context_length
+
+        if isinstance(vision_layers, (tuple, list)):
+            vision_heads = vision_width * 32 // 64
+            self.visual = ModifiedResNet(
+                layers=vision_layers,
+                output_dim=embed_dim,
+                heads=vision_heads,
+                input_resolution=image_resolution,
+                width=vision_width
+            )
+        else:
+            vision_heads = vision_width // 64
+            self.visual = VisionTransformer(
+                input_resolution=image_resolution,
+                patch_size=vision_patch_size,
+                width=vision_width,
+                layers=vision_layers,
+                heads=vision_heads,
+                output_dim=embed_dim
+            )
+
+        self.transformer = Transformer(
+            width=transformer_width,
+            layers=transformer_layers,
+            heads=transformer_heads,
+            attn_mask=self.build_attention_mask()
+        )
+
+        self.vocab_size = vocab_size
+        self.token_embedding = nn.Embedding(vocab_size, transformer_width)
+        self.positional_embedding = nn.Parameter(torch.empty(self.context_length, transformer_width))
+        self.ln_final = LayerNorm(transformer_width)
+
+        self.text_projection = nn.Parameter(torch.empty(transformer_width, embed_dim))
+        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
+
+        self.initialize_parameters()
+
+    def initialize_parameters(self):
+        nn.init.normal_(self.token_embedding.weight, std=0.02)
+        nn.init.normal_(self.positional_embedding, std=0.01)
+
+        if isinstance(self.visual, ModifiedResNet):
+            if self.visual.attnpool is not None:
+                std = self.visual.attnpool.c_proj.in_features ** -0.5
+                nn.init.normal_(self.visual.attnpool.q_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.k_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.v_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.c_proj.weight, std=std)
+
+            for resnet_block in [self.visual.layer1, self.visual.layer2, self.visual.layer3, self.visual.layer4]:
+                for name, param in resnet_block.named_parameters():
+                    if name.endswith("bn3.weight"):
+                        nn.init.zeros_(param)
+
+        proj_std = (self.transformer.width ** -0.5) * ((2 * self.transformer.layers) ** -0.5)
+        attn_std = self.transformer.width ** -0.5
+        fc_std = (2 * self.transformer.width) ** -0.5
+        for block in self.transformer.resblocks:
+            nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
+            nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
+            nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
+            nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)
+
+        if self.text_projection is not None:
+            nn.init.normal_(self.text_projection, std=self.transformer.width ** -0.5)
+
+    def build_attention_mask(self):
+        # lazily create causal attention mask, with full attention between the vision tokens
+        # pytorch uses additive attention mask; fill with -inf
+        mask = torch.empty(self.context_length, self.context_length)
+        mask.fill_(float("-inf"))
+        mask.triu_(1)  # zero out the lower diagonal
+        return mask
+
+    @property
+    def dtype(self):
+        return self.visual.conv1.weight.dtype
+
+    def calculate_self_sim(self, x: torch.Tensor):
+        tokens = self.visual.transformer_first_blocks_forward(
+            x.type(self.dtype))  # shape = [batch, tokens, emb_dim] tokens include class token
+        tokens = tokens.permute(1, 0, 2)
+        ssim = self.visual.attn_cosine_sim(tokens)  # shape = [batch, tokens, tokens]
+        return ssim
+
+    def encode_image(self, image):
+        return self.visual(image.type(self.dtype))
+
+    def encode_text(self, text):
+        x = self.token_embedding(text).type(self.dtype)  # [batch_size, n_ctx, d_model]
+        x = x + self.positional_embedding.type(self.dtype)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_final(x).type(self.dtype)
+
+        # x.shape = [batch_size, n_ctx, transformer.width]
+        # take features from the eot embedding (eot_token is the highest number in each sequence)
+        x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection
+
+        return x
+
+    def forward(self, image, text):
+        image_features = self.encode_image(image)
+        text_features = self.encode_text(text)
+
+        # normalized features
+        image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+        text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+
+        # cosine similarity as logits
+        logit_scale = self.logit_scale.exp()
+        logits_per_image = logit_scale * image_features @ text_features.t()
+        logits_per_text = logits_per_image.t()
+
+        # shape = [global_batch_size, global_batch_size]
+        return logits_per_image, logits_per_text
+
+
+def convert_weights(model: nn.Module):
+    """Convert applicable model parameters to fp16"""
+
+    def _convert_weights_to_fp16(l):
+        if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
+            l.weight.data = l.weight.data.half()
+            if l.bias is not None:
+                l.bias.data = l.bias.data.half()
+
+        if isinstance(l, nn.MultiheadAttention):
+            for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
+                tensor = getattr(l, attr)
+                if tensor is not None:
+                    tensor.data = tensor.data.half()
+
+        for name in ["text_projection", "proj"]:
+            if hasattr(l, name):
+                attr = getattr(l, name)
+                if attr is not None:
+                    attr.data = attr.data.half()
+
+    model.apply(_convert_weights_to_fp16)
+
+
+def build_model(state_dict: dict):
+    vit = "visual.proj" in state_dict
+
+    if vit:
+        vision_width = state_dict["visual.conv1.weight"].shape[0]
+        vision_layers = len(
+            [k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
+        vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
+        grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
+        image_resolution = vision_patch_size * grid_size
+    else:
+        counts: list = [len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in
+                        [1, 2, 3, 4]]
+        vision_layers = tuple(counts)
+        vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
+        output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
+        vision_patch_size = None
+        assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
+        image_resolution = output_width * 32
+
+    embed_dim = state_dict["text_projection"].shape[1]
+    context_length = state_dict["positional_embedding"].shape[0]
+    vocab_size = state_dict["token_embedding.weight"].shape[0]
+    transformer_width = state_dict["ln_final.weight"].shape[0]
+    transformer_heads = transformer_width // 64
+    transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith(f"transformer.resblocks")))
+
+    model = CLIP(
+        embed_dim,
+        image_resolution, vision_layers, vision_width, vision_patch_size,
+        context_length, vocab_size, transformer_width, transformer_heads, transformer_layers
+    )
+
+    for key in ["input_resolution", "context_length", "vocab_size"]:
+        if key in state_dict:
+            del state_dict[key]
+
+    convert_weights(model)
+    model.load_state_dict(state_dict)
+    return model.eval()

CLIP/clip/simple_tokenizer.py

@@ -0,0 +1,132 @@
+import gzip
+import html
+import os
+from functools import lru_cache
+
+import ftfy
+import regex as re
+
+
+@lru_cache()
+def default_bpe():
+    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "bpe_simple_vocab_16e6.txt.gz")
+
+
+@lru_cache()
+def bytes_to_unicode():
+    """
+    Returns list of utf-8 byte and a corresponding list of unicode strings.
+    The reversible bpe codes work on unicode strings.
+    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
+    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
+    This is a signficant percentage of your normal, say, 32K bpe vocab.
+    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
+    And avoids mapping to whitespace/control characters the bpe code barfs on.
+    """
+    bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8+n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+
+def basic_clean(text):
+    text = ftfy.fix_text(text)
+    text = html.unescape(html.unescape(text))
+    return text.strip()
+
+
+def whitespace_clean(text):
+    text = re.sub(r'\s+', ' ', text)
+    text = text.strip()
+    return text
+
+
+class SimpleTokenizer(object):
+    def __init__(self, bpe_path: str = default_bpe()):
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        merges = gzip.open(bpe_path).read().decode("utf-8").split('\n')
+        merges = merges[1:49152-256-2+1]
+        merges = [tuple(merge.split()) for merge in merges]
+        vocab = list(bytes_to_unicode().values())
+        vocab = vocab + [v+'</w>' for v in vocab]
+        for merge in merges:
+            vocab.append(''.join(merge))
+        vocab.extend(['<|startoftext|>', '<|endoftext|>'])
+        self.encoder = dict(zip(vocab, range(len(vocab))))
+        self.decoder = {v: k for k, v in self.encoder.items()}
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {'<|startoftext|>': '<|startoftext|>', '<|endoftext|>': '<|endoftext|>'}
+        self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token[:-1]) + ( token[-1] + '</w>',)
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token+'</w>'
+
+        while True:
+            bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                    new_word.extend(word[i:j])
+                    i = j
+                except:
+                    new_word.extend(word[i:])
+                    break
+
+                if word[i] == first and i < len(word)-1 and word[i+1] == second:
+                    new_word.append(first+second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = ' '.join(word)
+        self.cache[token] = word
+        return word
+
+    def encode(self, text):
+        bpe_tokens = []
+        text = whitespace_clean(basic_clean(text)).lower()
+        for token in re.findall(self.pat, text):
+            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
+            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
+        return bpe_tokens
+
+    def decode(self, tokens):
+        text = ''.join([self.decoder[token] for token in tokens])
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors="replace").replace('</w>', ' ')
+        return text

CLIP/clip_explainability/__init__.py

@@ -0,0 +1 @@
+from .clip import *

CLIP/clip_explainability/auxilary.py

@@ -0,0 +1,422 @@
+import torch
+import warnings
+from typing import Tuple, Optional
+
+import torch
+from torch import Tensor
+from torch.nn.init import xavier_uniform_
+from torch.nn.init import constant_
+from torch.nn.init import xavier_normal_
+from torch.nn.parameter import Parameter
+from torch.nn import functional as F
+
+# We define this function as _pad because it takes an argument
+# named pad, which clobbers the recursive reference to the pad
+# function needed for __torch_function__ support
+pad = F._pad
+
+# This class exists solely for Transformer; it has an annotation stating
+# that bias is never None, which appeases TorchScript
+class _LinearWithBias(torch.nn.Linear):
+    bias: Tensor
+
+    def __init__(self, in_features: int, out_features: int) -> None:
+        super().__init__(in_features, out_features, bias=True)
+
+def multi_head_attention_forward(query: Tensor,
+                                 key: Tensor,
+                                 value: Tensor,
+                                 embed_dim_to_check: int,
+                                 num_heads: int,
+                                 in_proj_weight: Tensor,
+                                 in_proj_bias: Tensor,
+                                 bias_k: Optional[Tensor],
+                                 bias_v: Optional[Tensor],
+                                 add_zero_attn: bool,
+                                 dropout_p: float,
+                                 out_proj_weight: Tensor,
+                                 out_proj_bias: Tensor,
+                                 training: bool = True,
+                                 key_padding_mask: Optional[Tensor] = None,
+                                 need_weights: bool = True,
+                                 attn_mask: Optional[Tensor] = None,
+                                 use_separate_proj_weight: bool = False,
+                                 q_proj_weight: Optional[Tensor] = None,
+                                 k_proj_weight: Optional[Tensor] = None,
+                                 v_proj_weight: Optional[Tensor] = None,
+                                 static_k: Optional[Tensor] = None,
+                                 static_v: Optional[Tensor] = None,
+                                 attention_probs_forward_hook = None,
+                                 attention_probs_backwards_hook = None,
+                                 ) -> Tuple[Tensor, Optional[Tensor]]:
+    if not torch.jit.is_scripting():
+        tens_ops = (query, key, value, in_proj_weight, in_proj_bias, bias_k, bias_v,
+                    out_proj_weight, out_proj_bias)
+        if any([type(t) is not Tensor for t in tens_ops]) and F.has_torch_function(tens_ops):
+            return F.handle_torch_function(
+                multi_head_attention_forward, tens_ops, query, key, value,
+                embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias,
+                bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight,
+                out_proj_bias, training=training, key_padding_mask=key_padding_mask,
+                need_weights=need_weights, attn_mask=attn_mask,
+                use_separate_proj_weight=use_separate_proj_weight,
+                q_proj_weight=q_proj_weight, k_proj_weight=k_proj_weight,
+                v_proj_weight=v_proj_weight, static_k=static_k, static_v=static_v)
+    tgt_len, bsz, embed_dim = query.size()
+    assert embed_dim == embed_dim_to_check
+    # allow MHA to have different sizes for the feature dimension
+    assert key.size(0) == value.size(0) and key.size(1) == value.size(1)
+
+    head_dim = embed_dim // num_heads
+    assert head_dim * num_heads == embed_dim, "embed_dim must be divisible by num_heads"
+    scaling = float(head_dim) ** -0.5
+
+    if not use_separate_proj_weight:
+        if torch.equal(query, key) and torch.equal(key, value):
+            # self-attention
+            q, k, v = F.linear(query, in_proj_weight, in_proj_bias).chunk(3, dim=-1)
+
+        elif torch.equal(key, value):
+            # encoder-decoder attention
+            # This is inline in_proj function with in_proj_weight and in_proj_bias
+            _b = in_proj_bias
+            _start = 0
+            _end = embed_dim
+            _w = in_proj_weight[_start:_end, :]
+            if _b is not None:
+                _b = _b[_start:_end]
+            q = F.linear(query, _w, _b)
+
+            if key is None:
+                assert value is None
+                k = None
+                v = None
+            else:
+
+                # This is inline in_proj function with in_proj_weight and in_proj_bias
+                _b = in_proj_bias
+                _start = embed_dim
+                _end = None
+                _w = in_proj_weight[_start:, :]
+                if _b is not None:
+                    _b = _b[_start:]
+                k, v = F.linear(key, _w, _b).chunk(2, dim=-1)
+
+        else:
+            # This is inline in_proj function with in_proj_weight and in_proj_bias
+            _b = in_proj_bias
+            _start = 0
+            _end = embed_dim
+            _w = in_proj_weight[_start:_end, :]
+            if _b is not None:
+                _b = _b[_start:_end]
+            q = F.linear(query, _w, _b)
+
+            # This is inline in_proj function with in_proj_weight and in_proj_bias
+            _b = in_proj_bias
+            _start = embed_dim
+            _end = embed_dim * 2
+            _w = in_proj_weight[_start:_end, :]
+            if _b is not None:
+                _b = _b[_start:_end]
+            k = F.linear(key, _w, _b)
+
+            # This is inline in_proj function with in_proj_weight and in_proj_bias
+            _b = in_proj_bias
+            _start = embed_dim * 2
+            _end = None
+            _w = in_proj_weight[_start:, :]
+            if _b is not None:
+                _b = _b[_start:]
+            v = F.linear(value, _w, _b)
+    else:
+        q_proj_weight_non_opt = torch.jit._unwrap_optional(q_proj_weight)
+        len1, len2 = q_proj_weight_non_opt.size()
+        assert len1 == embed_dim and len2 == query.size(-1)
+
+        k_proj_weight_non_opt = torch.jit._unwrap_optional(k_proj_weight)
+        len1, len2 = k_proj_weight_non_opt.size()
+        assert len1 == embed_dim and len2 == key.size(-1)
+
+        v_proj_weight_non_opt = torch.jit._unwrap_optional(v_proj_weight)
+        len1, len2 = v_proj_weight_non_opt.size()
+        assert len1 == embed_dim and len2 == value.size(-1)
+
+        if in_proj_bias is not None:
+            q = F.linear(query, q_proj_weight_non_opt, in_proj_bias[0:embed_dim])
+            k = F.linear(key, k_proj_weight_non_opt, in_proj_bias[embed_dim:(embed_dim * 2)])
+            v = F.linear(value, v_proj_weight_non_opt, in_proj_bias[(embed_dim * 2):])
+        else:
+            q = F.linear(query, q_proj_weight_non_opt, in_proj_bias)
+            k = F.linear(key, k_proj_weight_non_opt, in_proj_bias)
+            v = F.linear(value, v_proj_weight_non_opt, in_proj_bias)
+    q = q * scaling
+
+    if attn_mask is not None:
+        assert attn_mask.dtype == torch.float32 or attn_mask.dtype == torch.float64 or \
+            attn_mask.dtype == torch.float16 or attn_mask.dtype == torch.uint8 or attn_mask.dtype == torch.bool, \
+            'Only float, byte, and bool types are supported for attn_mask, not {}'.format(attn_mask.dtype)
+        if attn_mask.dtype == torch.uint8:
+            warnings.warn("Byte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
+            attn_mask = attn_mask.to(torch.bool)
+
+        if attn_mask.dim() == 2:
+            attn_mask = attn_mask.unsqueeze(0)
+            if list(attn_mask.size()) != [1, query.size(0), key.size(0)]:
+                raise RuntimeError('The size of the 2D attn_mask is not correct.')
+        elif attn_mask.dim() == 3:
+            if list(attn_mask.size()) != [bsz * num_heads, query.size(0), key.size(0)]:
+                raise RuntimeError('The size of the 3D attn_mask is not correct.')
+        else:
+            raise RuntimeError("attn_mask's dimension {} is not supported".format(attn_mask.dim()))
+        # attn_mask's dim is 3 now.
+
+    # convert ByteTensor key_padding_mask to bool
+    if key_padding_mask is not None and key_padding_mask.dtype == torch.uint8:
+        warnings.warn("Byte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
+        key_padding_mask = key_padding_mask.to(torch.bool)
+
+    if bias_k is not None and bias_v is not None:
+        if static_k is None and static_v is None:
+            k = torch.cat([k, bias_k.repeat(1, bsz, 1)])
+            v = torch.cat([v, bias_v.repeat(1, bsz, 1)])
+            if attn_mask is not None:
+                attn_mask = pad(attn_mask, (0, 1))
+            if key_padding_mask is not None:
+                key_padding_mask = pad(key_padding_mask, (0, 1))
+        else:
+            assert static_k is None, "bias cannot be added to static key."
+            assert static_v is None, "bias cannot be added to static value."
+    else:
+        assert bias_k is None
+        assert bias_v is None
+
+    q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1)
+    if k is not None:
+        k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
+    if v is not None:
+        v = v.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
+
+    if static_k is not None:
+        assert static_k.size(0) == bsz * num_heads
+        assert static_k.size(2) == head_dim
+        k = static_k
+
+    if static_v is not None:
+        assert static_v.size(0) == bsz * num_heads
+        assert static_v.size(2) == head_dim
+        v = static_v
+
+    src_len = k.size(1)
+
+    if key_padding_mask is not None:
+        assert key_padding_mask.size(0) == bsz
+        assert key_padding_mask.size(1) == src_len
+
+    if add_zero_attn:
+        src_len += 1
+        k = torch.cat([k, torch.zeros((k.size(0), 1) + k.size()[2:], dtype=k.dtype, device=k.device)], dim=1)
+        v = torch.cat([v, torch.zeros((v.size(0), 1) + v.size()[2:], dtype=v.dtype, device=v.device)], dim=1)
+        if attn_mask is not None:
+            attn_mask = pad(attn_mask, (0, 1))
+        if key_padding_mask is not None:
+            key_padding_mask = pad(key_padding_mask, (0, 1))
+
+    attn_output_weights = torch.bmm(q, k.transpose(1, 2))
+    assert list(attn_output_weights.size()) == [bsz * num_heads, tgt_len, src_len]
+
+    if attn_mask is not None:
+        if attn_mask.dtype == torch.bool:
+            attn_output_weights.masked_fill_(attn_mask, float('-inf'))
+        else:
+            attn_output_weights += attn_mask
+
+
+    if key_padding_mask is not None:
+        attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
+        attn_output_weights = attn_output_weights.masked_fill(
+            key_padding_mask.unsqueeze(1).unsqueeze(2),
+            float('-inf'),
+        )
+        attn_output_weights = attn_output_weights.view(bsz * num_heads, tgt_len, src_len)
+
+    attn_output_weights = F.softmax(
+        attn_output_weights, dim=-1)
+    attn_output_weights = F.dropout(attn_output_weights, p=dropout_p, training=training)
+
+    # use hooks for the attention weights if necessary
+    if attention_probs_forward_hook is not None and attention_probs_backwards_hook is not None:
+        attention_probs_forward_hook(attn_output_weights)
+        attn_output_weights.register_hook(attention_probs_backwards_hook)
+
+    attn_output = torch.bmm(attn_output_weights, v)
+    assert list(attn_output.size()) == [bsz * num_heads, tgt_len, head_dim]
+    attn_output = attn_output.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim)
+    attn_output = F.linear(attn_output, out_proj_weight, out_proj_bias)
+
+    if need_weights:
+        # average attention weights over heads
+        attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
+        return attn_output, attn_output_weights.sum(dim=1) / num_heads
+    else:
+        return attn_output, None
+
+
+class MultiheadAttention(torch.nn.Module):
+    r"""Allows the model to jointly attend to information
+    from different representation subspaces.
+    See reference: Attention Is All You Need
+
+    .. math::
+        \text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O
+        \text{where} head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)
+
+    Args:
+        embed_dim: total dimension of the model.
+        num_heads: parallel attention heads.
+        dropout: a Dropout layer on attn_output_weights. Default: 0.0.
+        bias: add bias as module parameter. Default: True.
+        add_bias_kv: add bias to the key and value sequences at dim=0.
+        add_zero_attn: add a new batch of zeros to the key and
+                       value sequences at dim=1.
+        kdim: total number of features in key. Default: None.
+        vdim: total number of features in value. Default: None.
+
+        Note: if kdim and vdim are None, they will be set to embed_dim such that
+        query, key, and value have the same number of features.
+
+    Examples::
+
+        >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
+        >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
+    """
+    bias_k: Optional[torch.Tensor]
+    bias_v: Optional[torch.Tensor]
+
+    def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None):
+        super(MultiheadAttention, self).__init__()
+        self.embed_dim = embed_dim
+        self.kdim = kdim if kdim is not None else embed_dim
+        self.vdim = vdim if vdim is not None else embed_dim
+        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim
+
+        self.num_heads = num_heads
+        self.dropout = dropout
+        self.head_dim = embed_dim // num_heads
+        assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"
+
+        if self._qkv_same_embed_dim is False:
+            self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))
+            self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim))
+            self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim))
+            self.register_parameter('in_proj_weight', None)
+        else:
+            self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim))
+            self.register_parameter('q_proj_weight', None)
+            self.register_parameter('k_proj_weight', None)
+            self.register_parameter('v_proj_weight', None)
+
+        if bias:
+            self.in_proj_bias = Parameter(torch.empty(3 * embed_dim))
+        else:
+            self.register_parameter('in_proj_bias', None)
+        self.out_proj = _LinearWithBias(embed_dim, embed_dim)
+
+        if add_bias_kv:
+            self.bias_k = Parameter(torch.empty(1, 1, embed_dim))
+            self.bias_v = Parameter(torch.empty(1, 1, embed_dim))
+        else:
+            self.bias_k = self.bias_v = None
+
+        self.add_zero_attn = add_zero_attn
+
+        self._reset_parameters()
+
+    def _reset_parameters(self):
+        if self._qkv_same_embed_dim:
+            xavier_uniform_(self.in_proj_weight)
+        else:
+            xavier_uniform_(self.q_proj_weight)
+            xavier_uniform_(self.k_proj_weight)
+            xavier_uniform_(self.v_proj_weight)
+
+        if self.in_proj_bias is not None:
+            constant_(self.in_proj_bias, 0.)
+            constant_(self.out_proj.bias, 0.)
+        if self.bias_k is not None:
+            xavier_normal_(self.bias_k)
+        if self.bias_v is not None:
+            xavier_normal_(self.bias_v)
+
+    def __setstate__(self, state):
+        # Support loading old MultiheadAttention checkpoints generated by v1.1.0
+        if '_qkv_same_embed_dim' not in state:
+            state['_qkv_same_embed_dim'] = True
+
+        super(MultiheadAttention, self).__setstate__(state)
+
+    def forward(self, query, key, value, key_padding_mask=None,
+                need_weights=True, attn_mask=None, attention_probs_forward_hook=None, attention_probs_backwards_hook=None):
+        r"""
+    Args:
+        query, key, value: map a query and a set of key-value pairs to an output.
+            See "Attention Is All You Need" for more details.
+        key_padding_mask: if provided, specified padding elements in the key will
+            be ignored by the attention. When given a binary mask and a value is True,
+            the corresponding value on the attention layer will be ignored. When given
+            a byte mask and a value is non-zero, the corresponding value on the attention
+            layer will be ignored
+        need_weights: output attn_output_weights.
+        attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
+            the batches while a 3D mask allows to specify a different mask for the entries of each batch.
+
+    Shape:
+        - Inputs:
+        - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
+          the embedding dimension.
+        - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
+          the embedding dimension.
+        - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
+          the embedding dimension.
+        - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
+          If a ByteTensor is provided, the non-zero positions will be ignored while the position
+          with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the
+          value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
+        - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
+          3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
+          S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked
+          positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
+          while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
+          is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
+          is provided, it will be added to the attention weight.
+
+        - Outputs:
+        - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
+          E is the embedding dimension.
+        - attn_output_weights: :math:`(N, L, S)` where N is the batch size,
+          L is the target sequence length, S is the source sequence length.
+        """
+        if not self._qkv_same_embed_dim:
+            return multi_head_attention_forward(
+                query, key, value, self.embed_dim, self.num_heads,
+                self.in_proj_weight, self.in_proj_bias,
+                self.bias_k, self.bias_v, self.add_zero_attn,
+                self.dropout, self.out_proj.weight, self.out_proj.bias,
+                training=self.training,
+                key_padding_mask=key_padding_mask, need_weights=need_weights,
+                attn_mask=attn_mask, use_separate_proj_weight=True,
+                q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
+                v_proj_weight=self.v_proj_weight,
+                attention_probs_forward_hook=attention_probs_forward_hook,
+                attention_probs_backwards_hook=attention_probs_backwards_hook)
+        else:
+            return multi_head_attention_forward(
+                query, key, value, self.embed_dim, self.num_heads,
+                self.in_proj_weight, self.in_proj_bias,
+                self.bias_k, self.bias_v, self.add_zero_attn,
+                self.dropout, self.out_proj.weight, self.out_proj.bias,
+                training=self.training,
+                key_padding_mask=key_padding_mask, need_weights=need_weights,
+                attn_mask=attn_mask,
+                attention_probs_forward_hook=attention_probs_forward_hook,
+                attention_probs_backwards_hook=attention_probs_backwards_hook)

CLIP/clip_explainability/bpe_simple_vocab_16e6.txt.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a
+size 1356917

CLIP/clip_explainability/clip.py

@@ -0,0 +1,196 @@
+import hashlib
+import os
+import urllib
+import warnings
+from typing import Union, List
+
+import torch
+from PIL import Image
+from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize
+from tqdm import tqdm
+
+from .model import build_model
+from .simple_tokenizer import SimpleTokenizer as _Tokenizer
+
+__all__ = ["available_models", "load", "tokenize"]
+_tokenizer = _Tokenizer()
+
+_MODELS = {
+    "RN50": "https://openaipublic.azureedge.net/clip/models/afeb0e10f9e5a86da6080e35cf09123aca3b358a0c3e3b6c78a7b63bc04b6762/RN50.pt",
+    "RN101": "https://openaipublic.azureedge.net/clip/models/8fa8567bab74a42d41c5915025a8e4538c3bdbe8804a470a72f30b0d94fab599/RN101.pt",
+    "RN50x4": "https://openaipublic.azureedge.net/clip/models/7e526bd135e493cef0776de27d5f42653e6b4c8bf9e0f653bb11773263205fdd/RN50x4.pt",
+    "RN50x16": "https://openaipublic.azureedge.net/clip/models/52378b407f34354e150460fe41077663dd5b39c54cd0bfd2b27167a4a06ec9aa/RN50x16.pt",
+    "RN50x64": "https://openaipublic.azureedge.net/clip/models/be1cfb55d75a9666199fb2206c106743da0f6468c9d327f3e0d0a543a9919d9c/RN50x64.pt",
+    "ViT-B/32": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt",
+    "ViT-B/16": "https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt",
+    "ViT-L/14": "https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt",
+}
+
+def _download(url: str, root: str = os.path.expanduser("~/.cache/clip")):
+    os.makedirs(root, exist_ok=True)
+    filename = os.path.basename(url)
+
+    expected_sha256 = url.split("/")[-2]
+    download_target = os.path.join(root, filename)
+
+    if os.path.exists(download_target) and not os.path.isfile(download_target):
+        raise RuntimeError(f"{download_target} exists and is not a regular file")
+
+    if os.path.isfile(download_target):
+        if hashlib.sha256(open(download_target, "rb").read()).hexdigest() == expected_sha256:
+            return download_target
+        else:
+            warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file")
+
+    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
+        with tqdm(total=int(source.info().get("Content-Length")), ncols=80, unit='iB', unit_scale=True) as loop:
+            while True:
+                buffer = source.read(8192)
+                if not buffer:
+                    break
+
+                output.write(buffer)
+                loop.update(len(buffer))
+
+    if hashlib.sha256(open(download_target, "rb").read()).hexdigest() != expected_sha256:
+        raise RuntimeError(f"Model has been downloaded but the SHA256 checksum does not not match")
+
+    return download_target
+
+
+def _transform(n_px):
+    return Compose([
+        Resize(n_px, interpolation=Image.BICUBIC),
+        CenterCrop(n_px),
+        lambda image: image.convert("RGB"),
+        ToTensor(),
+        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
+    ])
+
+
+def available_models() -> List[str]:
+    """Returns the names of available CLIP models"""
+    return list(_MODELS.keys())
+
+
+def load(name: str, device: Union[str, torch.device] = "cuda" if torch.cuda.is_available() else "cpu", jit=True):
+    """Load a CLIP model
+
+    Parameters
+    ----------
+    name : str
+        A model name listed by `clip.available_models()`, or the path to a model checkpoint containing the state_dict
+
+    device : Union[str, torch.device]
+        The device to put the loaded model
+
+    jit : bool
+        Whether to load the optimized JIT model (default) or more hackable non-JIT model.
+
+    Returns
+    -------
+    model : torch.nn.Module
+        The CLIP model
+
+    preprocess : Callable[[PIL.Image], torch.Tensor]
+        A torchvision transform that converts a PIL image into a tensor that the returned model can take as its input
+    """
+    if name in _MODELS:
+        model_path = _download(_MODELS[name])
+    elif os.path.isfile(name):
+        model_path = name
+    else:
+        raise RuntimeError(f"Model {name} not found; available models = {available_models()}")
+
+    try:
+        # loading JIT archive
+        model = torch.jit.load(model_path, map_location=device if jit else "cpu").eval()
+        state_dict = None
+    except RuntimeError:
+        # loading saved state dict
+        if jit:
+            warnings.warn(f"File {model_path} is not a JIT archive. Loading as a state dict instead")
+            jit = False
+        state_dict = torch.load(model_path, map_location="cpu")
+
+    if not jit:
+        model = build_model(state_dict or model.state_dict()).to(device)
+        if str(device) == "cpu":
+            model.float()
+        return model, _transform(model.visual.input_resolution)
+
+    # patch the device names
+    device_holder = torch.jit.trace(lambda: torch.ones([]).to(torch.device(device)), example_inputs=[])
+    device_node = [n for n in device_holder.graph.findAllNodes("prim::Constant") if "Device" in repr(n)][-1]
+
+    def patch_device(module):
+        graphs = [module.graph] if hasattr(module, "graph") else []
+        if hasattr(module, "forward1"):
+            graphs.append(module.forward1.graph)
+
+        for graph in graphs:
+            for node in graph.findAllNodes("prim::Constant"):
+                if "value" in node.attributeNames() and str(node["value"]).startswith("cuda"):
+                    node.copyAttributes(device_node)
+
+    model.apply(patch_device)
+    patch_device(model.encode_image)
+    patch_device(model.encode_text)
+
+    # patch dtype to float32 on CPU
+    if str(device) == "cpu":
+        float_holder = torch.jit.trace(lambda: torch.ones([]).float(), example_inputs=[])
+        float_input = list(float_holder.graph.findNode("aten::to").inputs())[1]
+        float_node = float_input.node()
+
+        def patch_float(module):
+            graphs = [module.graph] if hasattr(module, "graph") else []
+            if hasattr(module, "forward1"):
+                graphs.append(module.forward1.graph)
+
+            for graph in graphs:
+                for node in graph.findAllNodes("aten::to"):
+                    inputs = list(node.inputs())
+                    for i in [1, 2]:  # dtype can be the second or third argument to aten::to()
+                        if inputs[i].node()["value"] == 5:
+                            inputs[i].node().copyAttributes(float_node)
+
+        model.apply(patch_float)
+        patch_float(model.encode_image)
+        patch_float(model.encode_text)
+
+        model.float()
+
+    return model, _transform(model.input_resolution.item())
+
+
+def tokenize(texts: Union[str, List[str]], context_length: int = 77) -> torch.LongTensor:
+    """
+    Returns the tokenized representation of given input string(s)
+
+    Parameters
+    ----------
+    texts : Union[str, List[str]]
+        An input string or a list of input strings to tokenize
+
+    context_length : int
+        The context length to use; all CLIP models use 77 as the context length
+
+    Returns
+    -------
+    A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]
+    """
+    if isinstance(texts, str):
+        texts = [texts]
+
+    sot_token = _tokenizer.encoder["<|startoftext|>"]
+    eot_token = _tokenizer.encoder["<|endoftext|>"]
+    all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
+    result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)
+
+    for i, tokens in enumerate(all_tokens):
+        if len(tokens) > context_length:
+            raise RuntimeError(f"Input {texts[i]} is too long for context length {context_length}")
+        result[i, :len(tokens)] = torch.tensor(tokens)
+
+    return result

CLIP/clip_explainability/model.py

@@ -0,0 +1,442 @@
+from collections import OrderedDict
+from typing import Tuple, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from torch import nn
+from .auxilary import *
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1):
+        super().__init__()
+
+        # all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
+        self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+
+        self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+
+        self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()
+
+        self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = None
+        self.stride = stride
+
+        if stride > 1 or inplanes != planes * Bottleneck.expansion:
+            # downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
+            self.downsample = nn.Sequential(OrderedDict([
+                ("-1", nn.AvgPool2d(stride)),
+                ("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
+                ("1", nn.BatchNorm2d(planes * self.expansion))
+            ]))
+
+    def forward(self, x: torch.Tensor):
+        identity = x
+
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.relu(self.bn2(self.conv2(out)))
+        out = self.avgpool(out)
+        out = self.bn3(self.conv3(out))
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu(out)
+        return out
+
+
+class AttentionPool2d(nn.Module):
+    def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
+        super().__init__()
+        self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+        self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
+        self.num_heads = num_heads
+
+    def forward(self, x):
+        x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1)  # NCHW -> (HW)NC
+        x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
+        x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
+        x, _ = multi_head_attention_forward(
+            query=x, key=x, value=x,
+            embed_dim_to_check=x.shape[-1],
+            num_heads=self.num_heads,
+            q_proj_weight=self.q_proj.weight,
+            k_proj_weight=self.k_proj.weight,
+            v_proj_weight=self.v_proj.weight,
+            in_proj_weight=None,
+            in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
+            bias_k=None,
+            bias_v=None,
+            add_zero_attn=False,
+            dropout_p=0,
+            out_proj_weight=self.c_proj.weight,
+            out_proj_bias=self.c_proj.bias,
+            use_separate_proj_weight=True,
+            training=self.training,
+            need_weights=False
+        )
+
+        return x[0]
+
+
+class ModifiedResNet(nn.Module):
+    """
+    A ResNet class that is similar to torchvision's but contains the following changes:
+    - There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
+    - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
+    - The final pooling layer is a QKV attention instead of an average pool
+    """
+
+    def __init__(self, layers, output_dim, heads, input_resolution=224, width=64):
+        super().__init__()
+        self.output_dim = output_dim
+        self.input_resolution = input_resolution
+
+        # the 3-layer stem
+        self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(width // 2)
+        self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(width // 2)
+        self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(width)
+        self.avgpool = nn.AvgPool2d(2)
+        self.relu = nn.ReLU(inplace=True)
+
+        # residual layers
+        self._inplanes = width  # this is a *mutable* variable used during construction
+        self.layer1 = self._make_layer(width, layers[0])
+        self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
+        self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
+        self.layer4 = self._make_layer(width * 8, layers[3], stride=2)
+
+        embed_dim = width * 32  # the ResNet feature dimension
+        self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)
+
+    def _make_layer(self, planes, blocks, stride=1):
+        layers = [Bottleneck(self._inplanes, planes, stride)]
+
+        self._inplanes = planes * Bottleneck.expansion
+        for _ in range(1, blocks):
+            layers.append(Bottleneck(self._inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        def stem(x):
+            for conv, bn in [(self.conv1, self.bn1), (self.conv2, self.bn2), (self.conv3, self.bn3)]:
+                x = self.relu(bn(conv(x)))
+            x = self.avgpool(x)
+            return x
+
+        x = x.type(self.conv1.weight.dtype)
+        x = stem(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.attnpool(x)
+
+        return x
+
+
+class LayerNorm(nn.LayerNorm):
+    """Subclass torch's LayerNorm to handle fp16."""
+
+    def forward(self, x: torch.Tensor):
+        orig_type = x.dtype
+        ret = super().forward(x.type(torch.float32))
+        return ret.type(orig_type)
+
+
+class QuickGELU(nn.Module):
+    def forward(self, x: torch.Tensor):
+        return x * torch.sigmoid(1.702 * x)
+
+
+class ResidualAttentionBlock(nn.Module):
+    def __init__(self, d_model: int, n_head: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+
+        self.attn = MultiheadAttention(d_model, n_head)
+        self.ln_1 = LayerNorm(d_model)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, d_model * 4)),
+            ("gelu", QuickGELU()),
+            ("c_proj", nn.Linear(d_model * 4, d_model))
+        ]))
+        self.ln_2 = LayerNorm(d_model)
+        self.attn_mask = attn_mask
+
+        self.attn_probs = None
+        self.attn_grad = None
+
+    def set_attn_probs(self, attn_probs):
+        self.attn_probs = attn_probs
+
+    def set_attn_grad(self, attn_grad):
+        self.attn_grad = attn_grad
+
+    def attention(self, x: torch.Tensor):
+        self.attn_mask = self.attn_mask.to(dtype=x.dtype, device=x.device) if self.attn_mask is not None else None
+        return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask, attention_probs_forward_hook=self.set_attn_probs,
+                         attention_probs_backwards_hook=self.set_attn_grad)[0]
+
+    def forward(self, x: torch.Tensor):
+        x = x + self.attention(self.ln_1(x))
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Transformer(nn.Module):
+    def __init__(self, width: int, layers: int, heads: int, attn_mask: torch.Tensor = None):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+        self.resblocks = nn.Sequential(*[ResidualAttentionBlock(width, heads, attn_mask) for _ in range(layers)])
+
+    def forward(self, x: torch.Tensor):
+        return self.resblocks(x)
+
+
+class VisualTransformer(nn.Module):
+    def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int):
+        super().__init__()
+        self.input_resolution = input_resolution
+        self.output_dim = output_dim
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+
+        scale = width ** -0.5
+        self.class_embedding = nn.Parameter(scale * torch.randn(width))
+        self.positional_embedding = nn.Parameter(scale * torch.randn((input_resolution // patch_size) ** 2 + 1, width))
+        self.ln_pre = LayerNorm(width)
+
+        self.transformer = Transformer(width, layers, heads)
+
+        self.ln_post = LayerNorm(width)
+        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))
+
+    def forward(self, x: torch.Tensor):
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+        x = torch.cat([self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1)  # shape = [*, grid ** 2 + 1, width]
+        x = x + self.positional_embedding.to(x.dtype)
+        x = self.ln_pre(x)
+
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+
+        x = self.ln_post(x[:, 0, :])
+
+        if self.proj is not None:
+            x = x @ self.proj
+
+        return x
+
+
+class CLIP(nn.Module):
+    def __init__(self,
+                 embed_dim: int,
+                 # vision
+                 image_resolution: int,
+                 vision_layers: Union[Tuple[int, int, int, int], int],
+                 vision_width: int,
+                 vision_patch_size: int,
+                 # text
+                 context_length: int,
+                 vocab_size: int,
+                 transformer_width: int,
+                 transformer_heads: int,
+                 transformer_layers: int
+                 ):
+        super().__init__()
+
+        self.context_length = context_length
+
+        if isinstance(vision_layers, (tuple, list)):
+            vision_heads = vision_width * 32 // 64
+            self.visual = ModifiedResNet(
+                layers=vision_layers,
+                output_dim=embed_dim,
+                heads=vision_heads,
+                input_resolution=image_resolution,
+                width=vision_width
+            )
+        else:
+            vision_heads = vision_width // 64
+            self.visual = VisualTransformer(
+                input_resolution=image_resolution,
+                patch_size=vision_patch_size,
+                width=vision_width,
+                layers=vision_layers,
+                heads=vision_heads,
+                output_dim=embed_dim
+            )
+
+        self.transformer = Transformer(
+            width=transformer_width,
+            layers=transformer_layers,
+            heads=transformer_heads,
+            attn_mask=self.build_attention_mask()
+        )
+
+        self.vocab_size = vocab_size
+        self.token_embedding = nn.Embedding(vocab_size, transformer_width)
+        self.positional_embedding = nn.Parameter(torch.empty(self.context_length, transformer_width))
+        self.ln_final = LayerNorm(transformer_width)
+
+        self.text_projection = nn.Parameter(torch.empty(transformer_width, embed_dim))
+        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
+
+        self.initialize_parameters()
+
+    def initialize_parameters(self):
+        nn.init.normal_(self.token_embedding.weight, std=0.02)
+        nn.init.normal_(self.positional_embedding, std=0.01)
+
+        if isinstance(self.visual, ModifiedResNet):
+            if self.visual.attnpool is not None:
+                std = self.visual.attnpool.c_proj.in_features ** -0.5
+                nn.init.normal_(self.visual.attnpool.q_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.k_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.v_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.c_proj.weight, std=std)
+
+            for resnet_block in [self.visual.layer1, self.visual.layer2, self.visual.layer3, self.visual.layer4]:
+                for name, param in resnet_block.named_parameters():
+                    if name.endswith("bn3.weight"):
+                        nn.init.zeros_(param)
+
+        proj_std = (self.transformer.width ** -0.5) * ((2 * self.transformer.layers) ** -0.5)
+        attn_std = self.transformer.width ** -0.5
+        fc_std = (2 * self.transformer.width) ** -0.5
+        for block in self.transformer.resblocks:
+            nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
+            nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
+            nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
+            nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)
+
+        if self.text_projection is not None:
+            nn.init.normal_(self.text_projection, std=self.transformer.width ** -0.5)
+
+    def build_attention_mask(self):
+        # lazily create causal attention mask, with full attention between the vision tokens
+        # pytorch uses additive attention mask; fill with -inf
+        mask = torch.empty(self.context_length, self.context_length)
+        mask.fill_(float("-inf"))
+        mask.triu_(1)  # zero out the lower diagonal
+        return mask
+
+    @property
+    def dtype(self):
+        return self.visual.conv1.weight.dtype
+
+    def encode_image(self, image):
+        return self.visual(image.type(self.dtype))
+
+    def encode_text(self, text):
+        x = self.token_embedding(text).type(self.dtype)  # [batch_size, n_ctx, d_model]
+
+        x = x + self.positional_embedding.type(self.dtype)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_final(x).type(self.dtype)
+
+        # x.shape = [batch_size, n_ctx, transformer.width]
+        # take features from the eot embedding (eot_token is the highest number in each sequence)
+        x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection
+
+        return x
+
+    def forward(self, image, text):
+        image_features = self.encode_image(image)
+        text_features = self.encode_text(text)
+
+        # normalized features
+        image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+        text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+
+        # cosine similarity as logits
+        logit_scale = self.logit_scale.exp()
+        logits_per_image = logit_scale * image_features @ text_features.t()
+        logits_per_text = logit_scale * text_features @ image_features.t()
+
+        # shape = [global_batch_size, global_batch_size]
+        return logits_per_image, logits_per_text
+
+
+def convert_weights(model: nn.Module):
+    """Convert applicable model parameters to fp16"""
+
+    def _convert_weights_to_fp16(l):
+        if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
+            l.weight.data = l.weight.data.half()
+            if l.bias is not None:
+                l.bias.data = l.bias.data.half()
+
+        if isinstance(l, MultiheadAttention):
+            for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
+                tensor = getattr(l, attr)
+                if tensor is not None:
+                    tensor.data = tensor.data.half()
+
+        for name in ["text_projection", "proj"]:
+            if hasattr(l, name):
+                attr = getattr(l, name)
+                if attr is not None:
+                    attr.data = attr.data.half()
+
+    model.apply(_convert_weights_to_fp16)
+
+
+def build_model(state_dict: dict):
+    vit = "visual.proj" in state_dict
+
+    if vit:
+        vision_width = state_dict["visual.conv1.weight"].shape[0]
+        vision_layers = len([k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
+        vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
+        grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
+        image_resolution = vision_patch_size * grid_size
+    else:
+        counts: list = [len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in [1, 2, 3, 4]]
+        vision_layers = tuple(counts)
+        vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
+        output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
+        vision_patch_size = None
+        assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
+        image_resolution = output_width * 32
+
+    embed_dim = state_dict["text_projection"].shape[1]
+    context_length = state_dict["positional_embedding"].shape[0]
+    vocab_size = state_dict["token_embedding.weight"].shape[0]
+    transformer_width = state_dict["ln_final.weight"].shape[0]
+    transformer_heads = transformer_width // 64
+    transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith(f"transformer.resblocks")))
+
+    model = CLIP(
+        embed_dim,
+        image_resolution, vision_layers, vision_width, vision_patch_size,
+        context_length, vocab_size, transformer_width, transformer_heads, transformer_layers
+    )
+
+    for key in ["input_resolution", "context_length", "vocab_size"]:
+        if key in state_dict:
+            del state_dict[key]
+
+    convert_weights(model)
+    model.load_state_dict(state_dict)
+    return model.eval()

CLIP/clip_explainability/simple_tokenizer.py

@@ -0,0 +1,132 @@
+import gzip
+import html
+import os
+from functools import lru_cache
+
+import ftfy
+import regex as re
+
+
+@lru_cache()
+def default_bpe():
+    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "bpe_simple_vocab_16e6.txt.gz")
+
+
+@lru_cache()
+def bytes_to_unicode():
+    """
+    Returns list of utf-8 byte and a corresponding list of unicode strings.
+    The reversible bpe codes work on unicode strings.
+    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
+    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
+    This is a signficant percentage of your normal, say, 32K bpe vocab.
+    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
+    And avoids mapping to whitespace/control characters the bpe code barfs on.
+    """
+    bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8+n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+
+def basic_clean(text):
+    text = ftfy.fix_text(text)
+    text = html.unescape(html.unescape(text))
+    return text.strip()
+
+
+def whitespace_clean(text):
+    text = re.sub(r'\s+', ' ', text)
+    text = text.strip()
+    return text
+
+
+class SimpleTokenizer(object):
+    def __init__(self, bpe_path: str = default_bpe()):
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        merges = gzip.open(bpe_path).read().decode("utf-8").split('\n')
+        merges = merges[1:49152-256-2+1]
+        merges = [tuple(merge.split()) for merge in merges]
+        vocab = list(bytes_to_unicode().values())
+        vocab = vocab + [v+'</w>' for v in vocab]
+        for merge in merges:
+            vocab.append(''.join(merge))
+        vocab.extend(['<|startoftext|>', '<|endoftext|>'])
+        self.encoder = dict(zip(vocab, range(len(vocab))))
+        self.decoder = {v: k for k, v in self.encoder.items()}
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {'<|startoftext|>': '<|startoftext|>', '<|endoftext|>': '<|endoftext|>'}
+        self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token[:-1]) + ( token[-1] + '</w>',)
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token+'</w>'
+
+        while True:
+            bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                    new_word.extend(word[i:j])
+                    i = j
+                except:
+                    new_word.extend(word[i:])
+                    break
+
+                if word[i] == first and i < len(word)-1 and word[i+1] == second:
+                    new_word.append(first+second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = ' '.join(word)
+        self.cache[token] = word
+        return word
+
+    def encode(self, text):
+        bpe_tokens = []
+        text = whitespace_clean(basic_clean(text)).lower()
+        for token in re.findall(self.pat, text):
+            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
+            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
+        return bpe_tokens
+
+    def decode(self, tokens):
+        text = ''.join([self.decoder[token] for token in tokens])
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors="replace").replace('</w>', ' ')
+        return text

CLIP/data/country211.md

@@ -0,0 +1,12 @@
+# The Country211 Dataset
+
+In the paper, we used an image classification dataset called Country211, to evaluate the model's capability on geolocation. To do so, we filtered the YFCC100m dataset that have GPS coordinate corresponding to a [ISO-3166 country code](https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes) and created a balanced dataset by sampling 150 train images, 50 validation images, and 100 test images images for each country.
+
+The following command will download an 11GB archive countaining the images and extract into a subdirectory `country211`:
+
+```bash
+wget https://openaipublic.azureedge.net/clip/data/country211.tgz
+tar zxvf country211.tgz
+```
+
+These images are a subset of the YFCC100m dataset. Use of the underlying media files is subject to the Creative Commons licenses chosen by their creators/uploaders. For more information about the YFCC100M dataset, visit [the official website](https://multimediacommons.wordpress.com/yfcc100m-core-dataset/).

CLIP/data/prompts.md

@@ -0,0 +1,3401 @@
+# Prompts for Image Classification
+
+Below are the class names and templates that are used for collecting the zero-shot classification scores in the paper. Each dataset has two lists `classes` and `templates`, where the string `{}` in the template is to be replaced with the corresponding class names. For the Facial Emotion Recognition 2013 dataset specifically, we used multiple class names for certain classes.
+
+This file contains prompt data for 26 of the 27 datasets shown in Table 9 of the paper; the text prompts for ImageNet (as well as other [ImageNet Testbed](https://modestyachts.github.io/imagenet-testbed/) datasets in Figure 13) can be found in [this notebook](https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb), as well as how to ensemble predictions from multiple prompts using these templates.
+
+If you are viewing this document on GitHub, use the table of contents icon at the upper left to browse the datasets.
+
+
+## Birdsnap
+
+```bash
+classes = [
+    'Acadian Flycatcher',
+    'Acorn Woodpecker',
+    'Alder Flycatcher',
+    'Allens Hummingbird',
+    'Altamira Oriole',
+    'American Avocet',
+    'American Bittern',
+    'American Black Duck',
+    'American Coot',
+    'American Crow',
+    'American Dipper',
+    'American Golden Plover',
+    'American Goldfinch',
+    'American Kestrel',
+    'American Oystercatcher',
+    'American Pipit',
+    'American Redstart',
+    'American Robin',
+    'American Three toed Woodpecker',
+    'American Tree Sparrow',
+    'American White Pelican',
+    'American Wigeon',
+    'American Woodcock',
+    'Anhinga',
+    'Annas Hummingbird',
+    'Arctic Tern',
+    'Ash throated Flycatcher',
+    'Audubons Oriole',
+    'Bairds Sandpiper',
+    'Bald Eagle',
+    'Baltimore Oriole',
+    'Band tailed Pigeon',
+    'Barn Swallow',
+    'Barred Owl',
+    'Barrows Goldeneye',
+    'Bay breasted Warbler',
+    'Bells Vireo',
+    'Belted Kingfisher',
+    'Bewicks Wren',
+    'Black Guillemot',
+    'Black Oystercatcher',
+    'Black Phoebe',
+    'Black Rosy Finch',
+    'Black Scoter',
+    'Black Skimmer',
+    'Black Tern',
+    'Black Turnstone',
+    'Black Vulture',
+    'Black and white Warbler',
+    'Black backed Woodpecker',
+    'Black bellied Plover',
+    'Black billed Cuckoo',
+    'Black billed Magpie',
+    'Black capped Chickadee',
+    'Black chinned Hummingbird',
+    'Black chinned Sparrow',
+    'Black crested Titmouse',
+    'Black crowned Night Heron',
+    'Black headed Grosbeak',
+    'Black legged Kittiwake',
+    'Black necked Stilt',
+    'Black throated Blue Warbler',
+    'Black throated Gray Warbler',
+    'Black throated Green Warbler',
+    'Black throated Sparrow',
+    'Blackburnian Warbler',
+    'Blackpoll Warbler',
+    'Blue Grosbeak',
+    'Blue Jay',
+    'Blue gray Gnatcatcher',
+    'Blue headed Vireo',
+    'Blue winged Teal',
+    'Blue winged Warbler',
+    'Boat tailed Grackle',
+    'Bobolink',
+    'Bohemian Waxwing',
+    'Bonapartes Gull',
+    'Boreal Chickadee',
+    'Brandts Cormorant',
+    'Brant',
+    'Brewers Blackbird',
+    'Brewers Sparrow',
+    'Bridled Titmouse',
+    'Broad billed Hummingbird',
+    'Broad tailed Hummingbird',
+    'Broad winged Hawk',
+    'Bronzed Cowbird',
+    'Brown Creeper',
+    'Brown Pelican',
+    'Brown Thrasher',
+    'Brown capped Rosy Finch',
+    'Brown crested Flycatcher',
+    'Brown headed Cowbird',
+    'Brown headed Nuthatch',
+    'Bufflehead',
+    'Bullocks Oriole',
+    'Burrowing Owl',
+    'Bushtit',
+    'Cackling Goose',
+    'Cactus Wren',
+    'California Gull',
+    'California Quail',
+    'California Thrasher',
+    'California Towhee',
+    'Calliope Hummingbird',
+    'Canada Goose',
+    'Canada Warbler',
+    'Canvasback',
+    'Canyon Towhee',
+    'Canyon Wren',
+    'Cape May Warbler',
+    'Carolina Chickadee',
+    'Carolina Wren',
+    'Caspian Tern',
+    'Cassins Finch',
+    'Cassins Kingbird',
+    'Cassins Sparrow',
+    'Cassins Vireo',
+    'Cattle Egret',
+    'Cave Swallow',
+    'Cedar Waxwing',
+    'Cerulean Warbler',
+    'Chestnut backed Chickadee',
+    'Chestnut collared Longspur',
+    'Chestnut sided Warbler',
+    'Chihuahuan Raven',
+    'Chimney Swift',
+    'Chipping Sparrow',
+    'Cinnamon Teal',
+    'Clapper Rail',
+    'Clarks Grebe',
+    'Clarks Nutcracker',
+    'Clay colored Sparrow',
+    'Cliff Swallow',
+    'Common Black Hawk',
+    'Common Eider',
+    'Common Gallinule',
+    'Common Goldeneye',
+    'Common Grackle',
+    'Common Ground Dove',
+    'Common Loon',
+    'Common Merganser',
+    'Common Murre',
+    'Common Nighthawk',
+    'Common Raven',
+    'Common Redpoll',
+    'Common Tern',
+    'Common Yellowthroat',
+    'Connecticut Warbler',
+    'Coopers Hawk',
+    'Cordilleran Flycatcher',
+    'Costas Hummingbird',
+    'Couchs Kingbird',
+    'Crested Caracara',
+    'Curve billed Thrasher',
+    'Dark eyed Junco',
+    'Dickcissel',
+    'Double crested Cormorant',
+    'Downy Woodpecker',
+    'Dunlin',
+    'Dusky Flycatcher',
+    'Dusky Grouse',
+    'Eared Grebe',
+    'Eastern Bluebird',
+    'Eastern Kingbird',
+    'Eastern Meadowlark',
+    'Eastern Phoebe',
+    'Eastern Screech Owl',
+    'Eastern Towhee',
+    'Eastern Wood Pewee',
+    'Elegant Trogon',
+    'Elf Owl',
+    'Eurasian Collared Dove',
+    'Eurasian Wigeon',
+    'European Starling',
+    'Evening Grosbeak',
+    'Ferruginous Hawk',
+    'Ferruginous Pygmy Owl',
+    'Field Sparrow',
+    'Fish Crow',
+    'Florida Scrub Jay',
+    'Forsters Tern',
+    'Fox Sparrow',
+    'Franklins Gull',
+    'Fulvous Whistling Duck',
+    'Gadwall',
+    'Gambels Quail',
+    'Gila Woodpecker',
+    'Glaucous Gull',
+    'Glaucous winged Gull',
+    'Glossy Ibis',
+    'Golden Eagle',
+    'Golden crowned Kinglet',
+    'Golden crowned Sparrow',
+    'Golden fronted Woodpecker',
+    'Golden winged Warbler',
+    'Grasshopper Sparrow',
+    'Gray Catbird',
+    'Gray Flycatcher',
+    'Gray Jay',
+    'Gray Kingbird',
+    'Gray cheeked Thrush',
+    'Gray crowned Rosy Finch',
+    'Great Black backed Gull',
+    'Great Blue Heron',
+    'Great Cormorant',
+    'Great Crested Flycatcher',
+    'Great Egret',
+    'Great Gray Owl',
+    'Great Horned Owl',
+    'Great Kiskadee',
+    'Great tailed Grackle',
+    'Greater Prairie Chicken',
+    'Greater Roadrunner',
+    'Greater Sage Grouse',
+    'Greater Scaup',
+    'Greater White fronted Goose',
+    'Greater Yellowlegs',
+    'Green Jay',
+    'Green tailed Towhee',
+    'Green winged Teal',
+    'Groove billed Ani',
+    'Gull billed Tern',
+    'Hairy Woodpecker',
+    'Hammonds Flycatcher',
+    'Harlequin Duck',
+    'Harriss Hawk',
+    'Harriss Sparrow',
+    'Heermanns Gull',
+    'Henslows Sparrow',
+    'Hepatic Tanager',
+    'Hermit Thrush',
+    'Herring Gull',
+    'Hoary Redpoll',
+    'Hooded Merganser',
+    'Hooded Oriole',
+    'Hooded Warbler',
+    'Horned Grebe',
+    'Horned Lark',
+    'House Finch',
+    'House Sparrow',
+    'House Wren',
+    'Huttons Vireo',
+    'Iceland Gull',
+    'Inca Dove',
+    'Indigo Bunting',
+    'Killdeer',
+    'King Rail',
+    'Ladder backed Woodpecker',
+    'Lapland Longspur',
+    'Lark Bunting',
+    'Lark Sparrow',
+    'Laughing Gull',
+    'Lazuli Bunting',
+    'Le Contes Sparrow',
+    'Least Bittern',
+    'Least Flycatcher',
+    'Least Grebe',
+    'Least Sandpiper',
+    'Least Tern',
+    'Lesser Goldfinch',
+    'Lesser Nighthawk',
+    'Lesser Scaup',
+    'Lesser Yellowlegs',
+    'Lewiss Woodpecker',
+    'Limpkin',
+    'Lincolns Sparrow',
+    'Little Blue Heron',
+    'Loggerhead Shrike',
+    'Long billed Curlew',
+    'Long billed Dowitcher',
+    'Long billed Thrasher',
+    'Long eared Owl',
+    'Long tailed Duck',
+    'Louisiana Waterthrush',
+    'Magnificent Frigatebird',
+    'Magnolia Warbler',
+    'Mallard',
+    'Marbled Godwit',
+    'Marsh Wren',
+    'Merlin',
+    'Mew Gull',
+    'Mexican Jay',
+    'Mississippi Kite',
+    'Monk Parakeet',
+    'Mottled Duck',
+    'Mountain Bluebird',
+    'Mountain Chickadee',
+    'Mountain Plover',
+    'Mourning Dove',
+    'Mourning Warbler',
+    'Muscovy Duck',
+    'Mute Swan',
+    'Nashville Warbler',
+    'Nelsons Sparrow',
+    'Neotropic Cormorant',
+    'Northern Bobwhite',
+    'Northern Cardinal',
+    'Northern Flicker',
+    'Northern Gannet',
+    'Northern Goshawk',
+    'Northern Harrier',
+    'Northern Hawk Owl',
+    'Northern Mockingbird',
+    'Northern Parula',
+    'Northern Pintail',
+    'Northern Rough winged Swallow',
+    'Northern Saw whet Owl',
+    'Northern Shrike',
+    'Northern Waterthrush',
+    'Nuttalls Woodpecker',
+    'Oak Titmouse',
+    'Olive Sparrow',
+    'Olive sided Flycatcher',
+    'Orange crowned Warbler',
+    'Orchard Oriole',
+    'Osprey',
+    'Ovenbird',
+    'Pacific Golden Plover',
+    'Pacific Loon',
+    'Pacific Wren',
+    'Pacific slope Flycatcher',
+    'Painted Bunting',
+    'Painted Redstart',
+    'Palm Warbler',
+    'Pectoral Sandpiper',
+    'Peregrine Falcon',
+    'Phainopepla',
+    'Philadelphia Vireo',
+    'Pied billed Grebe',
+    'Pigeon Guillemot',
+    'Pileated Woodpecker',
+    'Pine Grosbeak',
+    'Pine Siskin',
+    'Pine Warbler',
+    'Piping Plover',
+    'Plumbeous Vireo',
+    'Prairie Falcon',
+    'Prairie Warbler',
+    'Prothonotary Warbler',
+    'Purple Finch',
+    'Purple Gallinule',
+    'Purple Martin',
+    'Purple Sandpiper',
+    'Pygmy Nuthatch',
+    'Pyrrhuloxia',
+    'Red Crossbill',
+    'Red Knot',
+    'Red Phalarope',
+    'Red bellied Woodpecker',
+    'Red breasted Merganser',
+    'Red breasted Nuthatch',
+    'Red breasted Sapsucker',
+    'Red cockaded Woodpecker',
+    'Red eyed Vireo',
+    'Red headed Woodpecker',
+    'Red naped Sapsucker',
+    'Red necked Grebe',
+    'Red necked Phalarope',
+    'Red shouldered Hawk',
+    'Red tailed Hawk',
+    'Red throated Loon',
+    'Red winged Blackbird',
+    'Reddish Egret',
+    'Redhead',
+    'Ring billed Gull',
+    'Ring necked Duck',
+    'Ring necked Pheasant',
+    'Rock Pigeon',
+    'Rock Ptarmigan',
+    'Rock Sandpiper',
+    'Rock Wren',
+    'Rose breasted Grosbeak',
+    'Roseate Tern',
+    'Rosss Goose',
+    'Rough legged Hawk',
+    'Royal Tern',
+    'Ruby crowned Kinglet',
+    'Ruby throated Hummingbird',
+    'Ruddy Duck',
+    'Ruddy Turnstone',
+    'Ruffed Grouse',
+    'Rufous Hummingbird',
+    'Rufous crowned Sparrow',
+    'Rusty Blackbird',
+    'Sage Thrasher',
+    'Saltmarsh Sparrow',
+    'Sanderling',
+    'Sandhill Crane',
+    'Sandwich Tern',
+    'Says Phoebe',
+    'Scaled Quail',
+    'Scarlet Tanager',
+    'Scissor tailed Flycatcher',
+    'Scotts Oriole',
+    'Seaside Sparrow',
+    'Sedge Wren',
+    'Semipalmated Plover',
+    'Semipalmated Sandpiper',
+    'Sharp shinned Hawk',
+    'Sharp tailed Grouse',
+    'Short billed Dowitcher',
+    'Short eared Owl',
+    'Snail Kite',
+    'Snow Bunting',
+    'Snow Goose',
+    'Snowy Egret',
+    'Snowy Owl',
+    'Snowy Plover',
+    'Solitary Sandpiper',
+    'Song Sparrow',
+    'Sooty Grouse',
+    'Sora',
+    'Spotted Owl',
+    'Spotted Sandpiper',
+    'Spotted Towhee',
+    'Spruce Grouse',
+    'Stellers Jay',
+    'Stilt Sandpiper',
+    'Summer Tanager',
+    'Surf Scoter',
+    'Surfbird',
+    'Swainsons Hawk',
+    'Swainsons Thrush',
+    'Swallow tailed Kite',
+    'Swamp Sparrow',
+    'Tennessee Warbler',
+    'Thayers Gull',
+    'Townsends Solitaire',
+    'Townsends Warbler',
+    'Tree Swallow',
+    'Tricolored Heron',
+    'Tropical Kingbird',
+    'Trumpeter Swan',
+    'Tufted Titmouse',
+    'Tundra Swan',
+    'Turkey Vulture',
+    'Upland Sandpiper',
+    'Varied Thrush',
+    'Veery',
+    'Verdin',
+    'Vermilion Flycatcher',
+    'Vesper Sparrow',
+    'Violet green Swallow',
+    'Virginia Rail',
+    'Wandering Tattler',
+    'Warbling Vireo',
+    'Western Bluebird',
+    'Western Grebe',
+    'Western Gull',
+    'Western Kingbird',
+    'Western Meadowlark',
+    'Western Sandpiper',
+    'Western Screech Owl',
+    'Western Scrub Jay',
+    'Western Tanager',
+    'Western Wood Pewee',
+    'Whimbrel',
+    'White Ibis',
+    'White breasted Nuthatch',
+    'White crowned Sparrow',
+    'White eyed Vireo',
+    'White faced Ibis',
+    'White headed Woodpecker',
+    'White rumped Sandpiper',
+    'White tailed Hawk',
+    'White tailed Kite',
+    'White tailed Ptarmigan',
+    'White throated Sparrow',
+    'White throated Swift',
+    'White winged Crossbill',
+    'White winged Dove',
+    'White winged Scoter',
+    'Wild Turkey',
+    'Willet',
+    'Williamsons Sapsucker',
+    'Willow Flycatcher',
+    'Willow Ptarmigan',
+    'Wilsons Phalarope',
+    'Wilsons Plover',
+    'Wilsons Snipe',
+    'Wilsons Warbler',
+    'Winter Wren',
+    'Wood Stork',
+    'Wood Thrush',
+    'Worm eating Warbler',
+    'Wrentit',
+    'Yellow Warbler',
+    'Yellow bellied Flycatcher',
+    'Yellow bellied Sapsucker',
+    'Yellow billed Cuckoo',
+    'Yellow billed Magpie',
+    'Yellow breasted Chat',
+    'Yellow crowned Night Heron',
+    'Yellow eyed Junco',
+    'Yellow headed Blackbird',
+    'Yellow rumped Warbler',
+    'Yellow throated Vireo',
+    'Yellow throated Warbler',
+    'Zone tailed Hawk',
+]
+
+templates = [
+    'a photo of a {}, a type of bird.',
+]
+```
+
+
+
+## CIFAR10
+
+```bash
+classes = [
+    'airplane',
+    'automobile',
+    'bird',
+    'cat',
+    'deer',
+    'dog',
+    'frog',
+    'horse',
+    'ship',
+    'truck',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a blurry photo of a {}.',
+    'a black and white photo of a {}.',
+    'a low contrast photo of a {}.',
+    'a high contrast photo of a {}.',
+    'a bad photo of a {}.',
+    'a good photo of a {}.',
+    'a photo of a small {}.',
+    'a photo of a big {}.',
+    'a photo of the {}.',
+    'a blurry photo of the {}.',
+    'a black and white photo of the {}.',
+    'a low contrast photo of the {}.',
+    'a high contrast photo of the {}.',
+    'a bad photo of the {}.',
+    'a good photo of the {}.',
+    'a photo of the small {}.',
+    'a photo of the big {}.',
+]
+```
+
+
+
+## CIFAR100
+
+```bash
+classes = [
+    'apple',
+    'aquarium fish',
+    'baby',
+    'bear',
+    'beaver',
+    'bed',
+    'bee',
+    'beetle',
+    'bicycle',
+    'bottle',
+    'bowl',
+    'boy',
+    'bridge',
+    'bus',
+    'butterfly',
+    'camel',
+    'can',
+    'castle',
+    'caterpillar',
+    'cattle',
+    'chair',
+    'chimpanzee',
+    'clock',
+    'cloud',
+    'cockroach',
+    'couch',
+    'crab',
+    'crocodile',
+    'cup',
+    'dinosaur',
+    'dolphin',
+    'elephant',
+    'flatfish',
+    'forest',
+    'fox',
+    'girl',
+    'hamster',
+    'house',
+    'kangaroo',
+    'keyboard',
+    'lamp',
+    'lawn mower',
+    'leopard',
+    'lion',
+    'lizard',
+    'lobster',
+    'man',
+    'maple tree',
+    'motorcycle',
+    'mountain',
+    'mouse',
+    'mushroom',
+    'oak tree',
+    'orange',
+    'orchid',
+    'otter',
+    'palm tree',
+    'pear',
+    'pickup truck',
+    'pine tree',
+    'plain',
+    'plate',
+    'poppy',
+    'porcupine',
+    'possum',
+    'rabbit',
+    'raccoon',
+    'ray',
+    'road',
+    'rocket',
+    'rose',
+    'sea',
+    'seal',
+    'shark',
+    'shrew',
+    'skunk',
+    'skyscraper',
+    'snail',
+    'snake',
+    'spider',
+    'squirrel',
+    'streetcar',
+    'sunflower',
+    'sweet pepper',
+    'table',
+    'tank',
+    'telephone',
+    'television',
+    'tiger',
+    'tractor',
+    'train',
+    'trout',
+    'tulip',
+    'turtle',
+    'wardrobe',
+    'whale',
+    'willow tree',
+    'wolf',
+    'woman',
+    'worm',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a blurry photo of a {}.',
+    'a black and white photo of a {}.',
+    'a low contrast photo of a {}.',
+    'a high contrast photo of a {}.',
+    'a bad photo of a {}.',
+    'a good photo of a {}.',
+    'a photo of a small {}.',
+    'a photo of a big {}.',
+    'a photo of the {}.',
+    'a blurry photo of the {}.',
+    'a black and white photo of the {}.',
+    'a low contrast photo of the {}.',
+    'a high contrast photo of the {}.',
+    'a bad photo of the {}.',
+    'a good photo of the {}.',
+    'a photo of the small {}.',
+    'a photo of the big {}.',
+]
+```
+
+
+
+## CLEVRCounts
+
+```bash
+classes = [
+    '10',
+    '3',
+    '4',
+    '5',
+    '6',
+    '7',
+    '8',
+    '9',
+]
+
+templates = [
+    'a photo of {} objects.',
+]
+```
+
+
+
+## Caltech101
+
+```bash
+classes = [
+    'background',
+    'off-center face',
+    'centered face',
+    'leopard',
+    'motorbike',
+    'accordion',
+    'airplane',
+    'anchor',
+    'ant',
+    'barrel',
+    'bass',
+    'beaver',
+    'binocular',
+    'bonsai',
+    'brain',
+    'brontosaurus',
+    'buddha',
+    'butterfly',
+    'camera',
+    'cannon',
+    'side of a car',
+    'ceiling fan',
+    'cellphone',
+    'chair',
+    'chandelier',
+    'body of a cougar cat',
+    'face of a cougar cat',
+    'crab',
+    'crayfish',
+    'crocodile',
+    'head of a  crocodile',
+    'cup',
+    'dalmatian',
+    'dollar bill',
+    'dolphin',
+    'dragonfly',
+    'electric guitar',
+    'elephant',
+    'emu',
+    'euphonium',
+    'ewer',
+    'ferry',
+    'flamingo',
+    'head of a flamingo',
+    'garfield',
+    'gerenuk',
+    'gramophone',
+    'grand piano',
+    'hawksbill',
+    'headphone',
+    'hedgehog',
+    'helicopter',
+    'ibis',
+    'inline skate',
+    'joshua tree',
+    'kangaroo',
+    'ketch',
+    'lamp',
+    'laptop',
+    'llama',
+    'lobster',
+    'lotus',
+    'mandolin',
+    'mayfly',
+    'menorah',
+    'metronome',
+    'minaret',
+    'nautilus',
+    'octopus',
+    'okapi',
+    'pagoda',
+    'panda',
+    'pigeon',
+    'pizza',
+    'platypus',
+    'pyramid',
+    'revolver',
+    'rhino',
+    'rooster',
+    'saxophone',
+    'schooner',
+    'scissors',
+    'scorpion',
+    'sea horse',
+    'snoopy (cartoon beagle)',
+    'soccer ball',
+    'stapler',
+    'starfish',
+    'stegosaurus',
+    'stop sign',
+    'strawberry',
+    'sunflower',
+    'tick',
+    'trilobite',
+    'umbrella',
+    'watch',
+    'water lilly',
+    'wheelchair',
+    'wild cat',
+    'windsor chair',
+    'wrench',
+    'yin and yang symbol',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a painting of a {}.',
+    'a plastic {}.',
+    'a sculpture of a {}.',
+    'a sketch of a {}.',
+    'a tattoo of a {}.',
+    'a toy {}.',
+    'a rendition of a {}.',
+    'a embroidered {}.',
+    'a cartoon {}.',
+    'a {} in a video game.',
+    'a plushie {}.',
+    'a origami {}.',
+    'art of a {}.',
+    'graffiti of a {}.',
+    'a drawing of a {}.',
+    'a doodle of a {}.',
+    'a photo of the {}.',
+    'a painting of the {}.',
+    'the plastic {}.',
+    'a sculpture of the {}.',
+    'a sketch of the {}.',
+    'a tattoo of the {}.',
+    'the toy {}.',
+    'a rendition of the {}.',
+    'the embroidered {}.',
+    'the cartoon {}.',
+    'the {} in a video game.',
+    'the plushie {}.',
+    'the origami {}.',
+    'art of the {}.',
+    'graffiti of the {}.',
+    'a drawing of the {}.',
+    'a doodle of the {}.',
+]
+```
+
+
+
+## Country211
+
+```bash
+classes = [
+    'Andorra',
+    'United Arab Emirates',
+    'Afghanistan',
+    'Antigua and Barbuda',
+    'Anguilla',
+    'Albania',
+    'Armenia',
+    'Angola',
+    'Antarctica',
+    'Argentina',
+    'Austria',
+    'Australia',
+    'Aruba',
+    'Aland Islands',
+    'Azerbaijan',
+    'Bosnia and Herzegovina',
+    'Barbados',
+    'Bangladesh',
+    'Belgium',
+    'Burkina Faso',
+    'Bulgaria',
+    'Bahrain',
+    'Benin',
+    'Bermuda',
+    'Brunei Darussalam',
+    'Bolivia',
+    'Bonaire, Saint Eustatius and Saba',
+    'Brazil',
+    'Bahamas',
+    'Bhutan',
+    'Botswana',
+    'Belarus',
+    'Belize',
+    'Canada',
+    'DR Congo',
+    'Central African Republic',
+    'Switzerland',
+    "Cote d'Ivoire",
+    'Cook Islands',
+    'Chile',
+    'Cameroon',
+    'China',
+    'Colombia',
+    'Costa Rica',
+    'Cuba',
+    'Cabo Verde',
+    'Curacao',
+    'Cyprus',
+    'Czech Republic',
+    'Germany',
+    'Denmark',
+    'Dominica',
+    'Dominican Republic',
+    'Algeria',
+    'Ecuador',
+    'Estonia',
+    'Egypt',
+    'Spain',
+    'Ethiopia',
+    'Finland',
+    'Fiji',
+    'Falkland Islands',
+    'Faeroe Islands',
+    'France',
+    'Gabon',
+    'United Kingdom',
+    'Grenada',
+    'Georgia',
+    'French Guiana',
+    'Guernsey',
+    'Ghana',
+    'Gibraltar',
+    'Greenland',
+    'Gambia',
+    'Guadeloupe',
+    'Greece',
+    'South Georgia and South Sandwich Is.',
+    'Guatemala',
+    'Guam',
+    'Guyana',
+    'Hong Kong',
+    'Honduras',
+    'Croatia',
+    'Haiti',
+    'Hungary',
+    'Indonesia',
+    'Ireland',
+    'Israel',
+    'Isle of Man',
+    'India',
+    'Iraq',
+    'Iran',
+    'Iceland',
+    'Italy',
+    'Jersey',
+    'Jamaica',
+    'Jordan',
+    'Japan',
+    'Kenya',
+    'Kyrgyz Republic',
+    'Cambodia',
+    'St. Kitts and Nevis',
+    'North Korea',
+    'South Korea',
+    'Kuwait',
+    'Cayman Islands',
+    'Kazakhstan',
+    'Laos',
+    'Lebanon',
+    'St. Lucia',
+    'Liechtenstein',
+    'Sri Lanka',
+    'Liberia',
+    'Lithuania',
+    'Luxembourg',
+    'Latvia',
+    'Libya',
+    'Morocco',
+    'Monaco',
+    'Moldova',
+    'Montenegro',
+    'Saint-Martin',
+    'Madagascar',
+    'Macedonia',
+    'Mali',
+    'Myanmar',
+    'Mongolia',
+    'Macau',
+    'Martinique',
+    'Mauritania',
+    'Malta',
+    'Mauritius',
+    'Maldives',
+    'Malawi',
+    'Mexico',
+    'Malaysia',
+    'Mozambique',
+    'Namibia',
+    'New Caledonia',
+    'Nigeria',
+    'Nicaragua',
+    'Netherlands',
+    'Norway',
+    'Nepal',
+    'New Zealand',
+    'Oman',
+    'Panama',
+    'Peru',
+    'French Polynesia',
+    'Papua New Guinea',
+    'Philippines',
+    'Pakistan',
+    'Poland',
+    'Puerto Rico',
+    'Palestine',
+    'Portugal',
+    'Palau',
+    'Paraguay',
+    'Qatar',
+    'Reunion',
+    'Romania',
+    'Serbia',
+    'Russia',
+    'Rwanda',
+    'Saudi Arabia',
+    'Solomon Islands',
+    'Seychelles',
+    'Sudan',
+    'Sweden',
+    'Singapore',
+    'St. Helena',
+    'Slovenia',
+    'Svalbard and Jan Mayen Islands',
+    'Slovakia',
+    'Sierra Leone',
+    'San Marino',
+    'Senegal',
+    'Somalia',
+    'South Sudan',
+    'El Salvador',
+    'Sint Maarten',
+    'Syria',
+    'Eswatini',
+    'Togo',
+    'Thailand',
+    'Tajikistan',
+    'Timor-Leste',
+    'Turkmenistan',
+    'Tunisia',
+    'Tonga',
+    'Turkey',
+    'Trinidad and Tobago',
+    'Taiwan',
+    'Tanzania',
+    'Ukraine',
+    'Uganda',
+    'United States',
+    'Uruguay',
+    'Uzbekistan',
+    'Vatican',
+    'Venezuela',
+    'British Virgin Islands',
+    'United States Virgin Islands',
+    'Vietnam',
+    'Vanuatu',
+    'Samoa',
+    'Kosovo',
+    'Yemen',
+    'South Africa',
+    'Zambia',
+    'Zimbabwe',
+]
+
+templates = [
+    'a photo i took in {}.',
+    'a photo i took while visiting {}.',
+    'a photo from my home country of {}.',
+    'a photo from my visit to {}.',
+    'a photo showing the country of {}.',
+]
+```
+
+
+
+## DescribableTextures
+
+```bash
+classes = [
+    'banded',
+    'blotchy',
+    'braided',
+    'bubbly',
+    'bumpy',
+    'chequered',
+    'cobwebbed',
+    'cracked',
+    'crosshatched',
+    'crystalline',
+    'dotted',
+    'fibrous',
+    'flecked',
+    'freckled',
+    'frilly',
+    'gauzy',
+    'grid',
+    'grooved',
+    'honeycombed',
+    'interlaced',
+    'knitted',
+    'lacelike',
+    'lined',
+    'marbled',
+    'matted',
+    'meshed',
+    'paisley',
+    'perforated',
+    'pitted',
+    'pleated',
+    'polka-dotted',
+    'porous',
+    'potholed',
+    'scaly',
+    'smeared',
+    'spiralled',
+    'sprinkled',
+    'stained',
+    'stratified',
+    'striped',
+    'studded',
+    'swirly',
+    'veined',
+    'waffled',
+    'woven',
+    'wrinkled',
+    'zigzagged',
+]
+
+templates = [
+    'a photo of a {} texture.',
+    'a photo of a {} pattern.',
+    'a photo of a {} thing.',
+    'a photo of a {} object.',
+    'a photo of the {} texture.',
+    'a photo of the {} pattern.',
+    'a photo of the {} thing.',
+    'a photo of the {} object.',
+]
+```
+
+
+
+## EuroSAT
+
+```bash
+classes = [
+    'forest',
+    'permanent crop land',
+    'residential buildings or homes or apartments',
+    'river',
+    'pasture land',
+    'lake or sea',
+    'brushland or shrubland',
+    'annual crop land',
+    'industrial buildings or commercial buildings',
+    'highway or road',
+]
+
+templates = [
+    'a centered satellite photo of {}.',
+    'a centered satellite photo of a {}.',
+    'a centered satellite photo of the {}.',
+]
+```
+
+
+
+## FGVCAircraft
+
+```bash
+classes = [
+    '707-320',
+    '727-200',
+    '737-200',
+    '737-300',
+    '737-400',
+    '737-500',
+    '737-600',
+    '737-700',
+    '737-800',
+    '737-900',
+    '747-100',
+    '747-200',
+    '747-300',
+    '747-400',
+    '757-200',
+    '757-300',
+    '767-200',
+    '767-300',
+    '767-400',
+    '777-200',
+    '777-300',
+    'A300B4',
+    'A310',
+    'A318',
+    'A319',
+    'A320',
+    'A321',
+    'A330-200',
+    'A330-300',
+    'A340-200',
+    'A340-300',
+    'A340-500',
+    'A340-600',
+    'A380',
+    'ATR-42',
+    'ATR-72',
+    'An-12',
+    'BAE 146-200',
+    'BAE 146-300',
+    'BAE-125',
+    'Beechcraft 1900',
+    'Boeing 717',
+    'C-130',
+    'C-47',
+    'CRJ-200',
+    'CRJ-700',
+    'CRJ-900',
+    'Cessna 172',
+    'Cessna 208',
+    'Cessna 525',
+    'Cessna 560',
+    'Challenger 600',
+    'DC-10',
+    'DC-3',
+    'DC-6',
+    'DC-8',
+    'DC-9-30',
+    'DH-82',
+    'DHC-1',
+    'DHC-6',
+    'DHC-8-100',
+    'DHC-8-300',
+    'DR-400',
+    'Dornier 328',
+    'E-170',
+    'E-190',
+    'E-195',
+    'EMB-120',
+    'ERJ 135',
+    'ERJ 145',
+    'Embraer Legacy 600',
+    'Eurofighter Typhoon',
+    'F-16A/B',
+    'F/A-18',
+    'Falcon 2000',
+    'Falcon 900',
+    'Fokker 100',
+    'Fokker 50',
+    'Fokker 70',
+    'Global Express',
+    'Gulfstream IV',
+    'Gulfstream V',
+    'Hawk T1',
+    'Il-76',
+    'L-1011',
+    'MD-11',
+    'MD-80',
+    'MD-87',
+    'MD-90',
+    'Metroliner',
+    'Model B200',
+    'PA-28',
+    'SR-20',
+    'Saab 2000',
+    'Saab 340',
+    'Spitfire',
+    'Tornado',
+    'Tu-134',
+    'Tu-154',
+    'Yak-42',
+]
+
+templates = [
+    'a photo of a {}, a type of aircraft.',
+    'a photo of the {}, a type of aircraft.',
+]
+```
+
+
+
+## FacialEmotionRecognition2013
+
+```bash
+classes = [
+    ['angry'],
+    ['disgusted'],
+    ['fearful'],
+    ['happy', 'smiling'],
+    ['sad', 'depressed'],
+    ['surprised', 'shocked', 'spooked'],
+    ['neutral', 'bored'],
+]
+
+templates = [
+    'a photo of a {} looking face.',
+    'a photo of a face showing the emotion: {}.',
+    'a photo of a face looking {}.',
+    'a face that looks {}.',
+    'they look {}.',
+    'look at how {} they are.',
+]
+```
+
+
+
+## Flowers102
+
+```bash
+classes = [
+    'pink primrose',
+    'hard-leaved pocket orchid',
+    'canterbury bells',
+    'sweet pea',
+    'english marigold',
+    'tiger lily',
+    'moon orchid',
+    'bird of paradise',
+    'monkshood',
+    'globe thistle',
+    'snapdragon',
+    "colt's foot",
+    'king protea',
+    'spear thistle',
+    'yellow iris',
+    'globe flower',
+    'purple coneflower',
+    'peruvian lily',
+    'balloon flower',
+    'giant white arum lily',
+    'fire lily',
+    'pincushion flower',
+    'fritillary',
+    'red ginger',
+    'grape hyacinth',
+    'corn poppy',
+    'prince of wales feathers',
+    'stemless gentian',
+    'artichoke',
+    'sweet william',
+    'carnation',
+    'garden phlox',
+    'love in the mist',
+    'mexican aster',
+    'alpine sea holly',
+    'ruby-lipped cattleya',
+    'cape flower',
+    'great masterwort',
+    'siam tulip',
+    'lenten rose',
+    'barbeton daisy',
+    'daffodil',
+    'sword lily',
+    'poinsettia',
+    'bolero deep blue',
+    'wallflower',
+    'marigold',
+    'buttercup',
+    'oxeye daisy',
+    'common dandelion',
+    'petunia',
+    'wild pansy',
+    'primula',
+    'sunflower',
+    'pelargonium',
+    'bishop of llandaff',
+    'gaura',
+    'geranium',
+    'orange dahlia',
+    'pink and yellow dahlia',
+    'cautleya spicata',
+    'japanese anemone',
+    'black-eyed susan',
+    'silverbush',
+    'californian poppy',
+    'osteospermum',
+    'spring crocus',
+    'bearded iris',
+    'windflower',
+    'tree poppy',
+    'gazania',
+    'azalea',
+    'water lily',
+    'rose',
+    'thorn apple',
+    'morning glory',
+    'passion flower',
+    'lotus',
+    'toad lily',
+    'anthurium',
+    'frangipani',
+    'clematis',
+    'hibiscus',
+    'columbine',
+    'desert-rose',
+    'tree mallow',
+    'magnolia',
+    'cyclamen',
+    'watercress',
+    'canna lily',
+    'hippeastrum',
+    'bee balm',
+    'air plant',
+    'foxglove',
+    'bougainvillea',
+    'camellia',
+    'mallow',
+    'mexican petunia',
+    'bromelia',
+    'blanket flower',
+    'trumpet creeper',
+    'blackberry lily',
+]
+
+templates = [
+    'a photo of a {}, a type of flower.',
+]
+```
+
+
+
+## Food101
+
+```bash
+classes = [
+    'apple pie',
+    'baby back ribs',
+    'baklava',
+    'beef carpaccio',
+    'beef tartare',
+    'beet salad',
+    'beignets',
+    'bibimbap',
+    'bread pudding',
+    'breakfast burrito',
+    'bruschetta',
+    'caesar salad',
+    'cannoli',
+    'caprese salad',
+    'carrot cake',
+    'ceviche',
+    'cheese plate',
+    'cheesecake',
+    'chicken curry',
+    'chicken quesadilla',
+    'chicken wings',
+    'chocolate cake',
+    'chocolate mousse',
+    'churros',
+    'clam chowder',
+    'club sandwich',
+    'crab cakes',
+    'creme brulee',
+    'croque madame',
+    'cup cakes',
+    'deviled eggs',
+    'donuts',
+    'dumplings',
+    'edamame',
+    'eggs benedict',
+    'escargots',
+    'falafel',
+    'filet mignon',
+    'fish and chips',
+    'foie gras',
+    'french fries',
+    'french onion soup',
+    'french toast',
+    'fried calamari',
+    'fried rice',
+    'frozen yogurt',
+    'garlic bread',
+    'gnocchi',
+    'greek salad',
+    'grilled cheese sandwich',
+    'grilled salmon',
+    'guacamole',
+    'gyoza',
+    'hamburger',
+    'hot and sour soup',
+    'hot dog',
+    'huevos rancheros',
+    'hummus',
+    'ice cream',
+    'lasagna',
+    'lobster bisque',
+    'lobster roll sandwich',
+    'macaroni and cheese',
+    'macarons',
+    'miso soup',
+    'mussels',
+    'nachos',
+    'omelette',
+    'onion rings',
+    'oysters',
+    'pad thai',
+    'paella',
+    'pancakes',
+    'panna cotta',
+    'peking duck',
+    'pho',
+    'pizza',
+    'pork chop',
+    'poutine',
+    'prime rib',
+    'pulled pork sandwich',
+    'ramen',
+    'ravioli',
+    'red velvet cake',
+    'risotto',
+    'samosa',
+    'sashimi',
+    'scallops',
+    'seaweed salad',
+    'shrimp and grits',
+    'spaghetti bolognese',
+    'spaghetti carbonara',
+    'spring rolls',
+    'steak',
+    'strawberry shortcake',
+    'sushi',
+    'tacos',
+    'takoyaki',
+    'tiramisu',
+    'tuna tartare',
+    'waffles',
+]
+
+templates = [
+    'a photo of {}, a type of food.',
+]
+```
+
+
+
+## GTSRB
+
+```bash
+classes = [
+    'red and white circle 20 kph speed limit',
+    'red and white circle 30 kph speed limit',
+    'red and white circle 50 kph speed limit',
+    'red and white circle 60 kph speed limit',
+    'red and white circle 70 kph speed limit',
+    'red and white circle 80 kph speed limit',
+    'end / de-restriction of 80 kph speed limit',
+    'red and white circle 100 kph speed limit',
+    'red and white circle 120 kph speed limit',
+    'red and white circle red car and black car no passing',
+    'red and white circle red truck and black car no passing',
+    'red and white triangle road intersection warning',
+    'white and yellow diamond priority road',
+    'red and white upside down triangle yield right-of-way',
+    'stop',
+    'empty red and white circle',
+    'red and white circle no truck entry',
+    'red circle with white horizonal stripe no entry',
+    'red and white triangle with exclamation mark warning',
+    'red and white triangle with black left curve approaching warning',
+    'red and white triangle with black right curve approaching warning',
+    'red and white triangle with black double curve approaching warning',
+    'red and white triangle rough / bumpy road warning',
+    'red and white triangle car skidding / slipping warning',
+    'red and white triangle with merging / narrow lanes warning',
+    'red and white triangle with person digging / construction / road work warning',
+    'red and white triangle with traffic light approaching warning',
+    'red and white triangle with person walking warning',
+    'red and white triangle with child and person walking warning',
+    'red and white triangle with bicyle warning',
+    'red and white triangle with snowflake / ice warning',
+    'red and white triangle with deer warning',
+    'white circle with gray strike bar no speed limit',
+    'blue circle with white right turn arrow mandatory',
+    'blue circle with white left turn arrow mandatory',
+    'blue circle with white forward arrow mandatory',
+    'blue circle with white forward or right turn arrow mandatory',
+    'blue circle with white forward or left turn arrow mandatory',
+    'blue circle with white keep right arrow mandatory',
+    'blue circle with white keep left arrow mandatory',
+    'blue circle with white arrows indicating a traffic circle',
+    'white circle with gray strike bar indicating no passing for cars has ended',
+    'white circle with gray strike bar indicating no passing for trucks has ended',
+]
+
+templates = [
+    'a zoomed in photo of a "{}" traffic sign.',
+    'a centered photo of a "{}" traffic sign.',
+    'a close up photo of a "{}" traffic sign.',
+]
+```
+
+
+
+## HatefulMemes
+
+```bash
+classes = [
+    'meme',
+    'hatespeech meme',
+]
+
+templates = [
+    'a {}.',
+]
+```
+
+
+
+## KITTI
+
+```bash
+classes = [
+    'a photo i took of a car on my left or right side.',
+    'a photo i took with a car nearby.',
+    'a photo i took with a car in the distance.',
+    'a photo i took with no car.',
+]
+
+templates = [
+    '{}',
+]
+```
+
+
+
+## Kinetics700
+
+```bash
+classes = [
+    'abseiling',
+    'acting in play',
+    'adjusting glasses',
+    'air drumming',
+    'alligator wrestling',
+    'answering questions',
+    'applauding',
+    'applying cream',
+    'archaeological excavation',
+    'archery',
+    'arguing',
+    'arm wrestling',
+    'arranging flowers',
+    'arresting',
+    'assembling bicycle',
+    'assembling computer',
+    'attending conference',
+    'auctioning',
+    'baby waking up',
+    'backflip (human)',
+    'baking cookies',
+    'bandaging',
+    'barbequing',
+    'bartending',
+    'base jumping',
+    'bathing dog',
+    'battle rope training',
+    'beatboxing',
+    'bee keeping',
+    'being excited',
+    'being in zero gravity',
+    'belly dancing',
+    'bench pressing',
+    'bending back',
+    'bending metal',
+    'biking through snow',
+    'blasting sand',
+    'blending fruit',
+    'blowdrying hair',
+    'blowing bubble gum',
+    'blowing glass',
+    'blowing leaves',
+    'blowing nose',
+    'blowing out candles',
+    'bobsledding',
+    'bodysurfing',
+    'bookbinding',
+    'bottling',
+    'bouncing ball (not juggling)',
+    'bouncing on bouncy castle',
+    'bouncing on trampoline',
+    'bowling',
+    'braiding hair',
+    'breading or breadcrumbing',
+    'breakdancing',
+    'breaking boards',
+    'breaking glass',
+    'breathing fire',
+    'brush painting',
+    'brushing floor',
+    'brushing hair',
+    'brushing teeth',
+    'building cabinet',
+    'building lego',
+    'building sandcastle',
+    'building shed',
+    'bulldozing',
+    'bungee jumping',
+    'burping',
+    'busking',
+    'calculating',
+    'calligraphy',
+    'canoeing or kayaking',
+    'capoeira',
+    'capsizing',
+    'card stacking',
+    'card throwing',
+    'carrying baby',
+    'carrying weight',
+    'cartwheeling',
+    'carving ice',
+    'carving marble',
+    'carving pumpkin',
+    'carving wood with a knife',
+    'casting fishing line',
+    'catching fish',
+    'catching or throwing baseball',
+    'catching or throwing frisbee',
+    'catching or throwing softball',
+    'celebrating',
+    'changing gear in car',
+    'changing oil',
+    'changing wheel (not on bike)',
+    'chasing',
+    'checking tires',
+    'checking watch',
+    'cheerleading',
+    'chewing gum',
+    'chiseling stone',
+    'chiseling wood',
+    'chopping meat',
+    'chopping wood',
+    'clam digging',
+    'clapping',
+    'clay pottery making',
+    'clean and jerk',
+    'cleaning gutters',
+    'cleaning pool',
+    'cleaning shoes',
+    'cleaning toilet',
+    'cleaning windows',
+    'climbing a rope',
+    'climbing ladder',
+    'climbing tree',
+    'closing door',
+    'coloring in',
+    'combing hair',
+    'contact juggling',
+    'contorting',
+    'cooking chicken',
+    'cooking egg',
+    'cooking on campfire',
+    'cooking sausages (not on barbeque)',
+    'cooking scallops',
+    'cosplaying',
+    'coughing',
+    'counting money',
+    'country line dancing',
+    'cracking back',
+    'cracking knuckles',
+    'cracking neck',
+    'crawling baby',
+    'crocheting',
+    'crossing eyes',
+    'crossing river',
+    'crying',
+    'cumbia',
+    'curling (sport)',
+    'curling eyelashes',
+    'curling hair',
+    'cutting apple',
+    'cutting cake',
+    'cutting nails',
+    'cutting orange',
+    'cutting pineapple',
+    'cutting watermelon',
+    'dancing ballet',
+    'dancing charleston',
+    'dancing gangnam style',
+    'dancing macarena',
+    'deadlifting',
+    'dealing cards',
+    'decorating the christmas tree',
+    'decoupage',
+    'delivering mail',
+    'digging',
+    'dining',
+    'directing traffic',
+    'disc golfing',
+    'diving cliff',
+    'docking boat',
+    'dodgeball',
+    'doing aerobics',
+    'doing jigsaw puzzle',
+    'doing laundry',
+    'doing nails',
+    'doing sudoku',
+    'drawing',
+    'dribbling basketball',
+    'drinking shots',
+    'driving car',
+    'driving tractor',
+    'drooling',
+    'drop kicking',
+    'drumming fingers',
+    'dumpster diving',
+    'dunking basketball',
+    'dyeing eyebrows',
+    'dyeing hair',
+    'eating burger',
+    'eating cake',
+    'eating carrots',
+    'eating chips',
+    'eating doughnuts',
+    'eating hotdog',
+    'eating ice cream',
+    'eating nachos',
+    'eating spaghetti',
+    'eating watermelon',
+    'egg hunting',
+    'embroidering',
+    'entering church',
+    'exercising arm',
+    'exercising with an exercise ball',
+    'extinguishing fire',
+    'faceplanting',
+    'falling off bike',
+    'falling off chair',
+    'feeding birds',
+    'feeding fish',
+    'feeding goats',
+    'fencing (sport)',
+    'fidgeting',
+    'filling cake',
+    'filling eyebrows',
+    'finger snapping',
+    'fixing bicycle',
+    'fixing hair',
+    'flint knapping',
+    'flipping bottle',
+    'flipping pancake',
+    'fly tying',
+    'flying kite',
+    'folding clothes',
+    'folding napkins',
+    'folding paper',
+    'front raises',
+    'frying vegetables',
+    'gargling',
+    'geocaching',
+    'getting a haircut',
+    'getting a piercing',
+    'getting a tattoo',
+    'giving or receiving award',
+    'gold panning',
+    'golf chipping',
+    'golf driving',
+    'golf putting',
+    'gospel singing in church',
+    'grinding meat',
+    'grooming cat',
+    'grooming dog',
+    'grooming horse',
+    'gymnastics tumbling',
+    'hammer throw',
+    'hand washing clothes',
+    'head stand',
+    'headbanging',
+    'headbutting',
+    'helmet diving',
+    'herding cattle',
+    'high fiving',
+    'high jump',
+    'high kick',
+    'historical reenactment',
+    'hitting baseball',
+    'hockey stop',
+    'holding snake',
+    'home roasting coffee',
+    'hopscotch',
+    'hoverboarding',
+    'huddling',
+    'hugging (not baby)',
+    'hugging baby',
+    'hula hooping',
+    'hurdling',
+    'hurling (sport)',
+    'ice climbing',
+    'ice fishing',
+    'ice skating',
+    'ice swimming',
+    'inflating balloons',
+    'installing carpet',
+    'ironing',
+    'ironing hair',
+    'javelin throw',
+    'jaywalking',
+    'jetskiing',
+    'jogging',
+    'juggling balls',
+    'juggling fire',
+    'juggling soccer ball',
+    'jumping bicycle',
+    'jumping into pool',
+    'jumping jacks',
+    'jumping sofa',
+    'jumpstyle dancing',
+    'karaoke',
+    'kicking field goal',
+    'kicking soccer ball',
+    'kissing',
+    'kitesurfing',
+    'knitting',
+    'krumping',
+    'land sailing',
+    'laughing',
+    'lawn mower racing',
+    'laying bricks',
+    'laying concrete',
+    'laying decking',
+    'laying stone',
+    'laying tiles',
+    'leatherworking',
+    'letting go of balloon',
+    'licking',
+    'lifting hat',
+    'lighting candle',
+    'lighting fire',
+    'listening with headphones',
+    'lock picking',
+    'long jump',
+    'longboarding',
+    'looking at phone',
+    'looking in mirror',
+    'luge',
+    'lunge',
+    'making a cake',
+    'making a sandwich',
+    'making balloon shapes',
+    'making bubbles',
+    'making cheese',
+    'making horseshoes',
+    'making jewelry',
+    'making latte art',
+    'making paper aeroplanes',
+    'making pizza',
+    'making slime',
+    'making snowman',
+    'making sushi',
+    'making tea',
+    'making the bed',
+    'marching',
+    'marriage proposal',
+    'massaging back',
+    'massaging feet',
+    'massaging legs',
+    'massaging neck',
+    "massaging person's head",
+    'metal detecting',
+    'milking cow',
+    'milking goat',
+    'mixing colours',
+    'moon walking',
+    'mopping floor',
+    'mosh pit dancing',
+    'motorcycling',
+    'mountain climber (exercise)',
+    'moving baby',
+    'moving child',
+    'moving furniture',
+    'mowing lawn',
+    'mushroom foraging',
+    'needle felting',
+    'news anchoring',
+    'opening bottle (not wine)',
+    'opening coconuts',
+    'opening door',
+    'opening present',
+    'opening refrigerator',
+    'opening wine bottle',
+    'packing',
+    'paragliding',
+    'parasailing',
+    'parkour',
+    'passing American football (in game)',
+    'passing American football (not in game)',
+    'passing soccer ball',
+    'peeling apples',
+    'peeling banana',
+    'peeling potatoes',
+    'person collecting garbage',
+    'petting animal (not cat)',
+    'petting cat',
+    'petting horse',
+    'photobombing',
+    'photocopying',
+    'picking apples',
+    'picking blueberries',
+    'pillow fight',
+    'pinching',
+    'pirouetting',
+    'planing wood',
+    'planting trees',
+    'plastering',
+    'playing accordion',
+    'playing american football',
+    'playing badminton',
+    'playing bagpipes',
+    'playing basketball',
+    'playing bass guitar',
+    'playing beer pong',
+    'playing billiards',
+    'playing blackjack',
+    'playing cards',
+    'playing cello',
+    'playing checkers',
+    'playing chess',
+    'playing clarinet',
+    'playing controller',
+    'playing cricket',
+    'playing cymbals',
+    'playing darts',
+    'playing didgeridoo',
+    'playing dominoes',
+    'playing drums',
+    'playing field hockey',
+    'playing flute',
+    'playing gong',
+    'playing guitar',
+    'playing hand clapping games',
+    'playing harmonica',
+    'playing harp',
+    'playing ice hockey',
+    'playing keyboard',
+    'playing kickball',
+    'playing laser tag',
+    'playing lute',
+    'playing mahjong',
+    'playing maracas',
+    'playing marbles',
+    'playing monopoly',
+    'playing netball',
+    'playing nose flute',
+    'playing oboe',
+    'playing ocarina',
+    'playing organ',
+    'playing paintball',
+    'playing pan pipes',
+    'playing piano',
+    'playing piccolo',
+    'playing pinball',
+    'playing ping pong',
+    'playing poker',
+    'playing polo',
+    'playing recorder',
+    'playing road hockey',
+    'playing rounders',
+    'playing rubiks cube',
+    'playing saxophone',
+    'playing scrabble',
+    'playing shuffleboard',
+    'playing slot machine',
+    'playing squash or racquetball',
+    'playing tennis',
+    'playing trombone',
+    'playing trumpet',
+    'playing ukulele',
+    'playing violin',
+    'playing volleyball',
+    'playing with trains',
+    'playing xylophone',
+    'poaching eggs',
+    'poking bellybutton',
+    'pole vault',
+    'polishing furniture',
+    'polishing metal',
+    'popping balloons',
+    'pouring beer',
+    'pouring milk',
+    'pouring wine',
+    'preparing salad',
+    'presenting weather forecast',
+    'pretending to be a statue',
+    'pull ups',
+    'pulling espresso shot',
+    'pulling rope (game)',
+    'pumping fist',
+    'pumping gas',
+    'punching bag',
+    'punching person (boxing)',
+    'push up',
+    'pushing car',
+    'pushing cart',
+    'pushing wheelbarrow',
+    'pushing wheelchair',
+    'putting in contact lenses',
+    'putting on eyeliner',
+    'putting on foundation',
+    'putting on lipstick',
+    'putting on mascara',
+    'putting on sari',
+    'putting on shoes',
+    'putting wallpaper on wall',
+    'raising eyebrows',
+    'reading book',
+    'reading newspaper',
+    'recording music',
+    'repairing puncture',
+    'riding a bike',
+    'riding camel',
+    'riding elephant',
+    'riding mechanical bull',
+    'riding mule',
+    'riding or walking with horse',
+    'riding scooter',
+    'riding snow blower',
+    'riding unicycle',
+    'ripping paper',
+    'roasting marshmallows',
+    'roasting pig',
+    'robot dancing',
+    'rock climbing',
+    'rock scissors paper',
+    'roller skating',
+    'rolling eyes',
+    'rolling pastry',
+    'rope pushdown',
+    'running on treadmill',
+    'sailing',
+    'salsa dancing',
+    'saluting',
+    'sanding floor',
+    'sanding wood',
+    'sausage making',
+    'sawing wood',
+    'scrambling eggs',
+    'scrapbooking',
+    'scrubbing face',
+    'scuba diving',
+    'seasoning food',
+    'separating eggs',
+    'setting table',
+    'sewing',
+    'shaking hands',
+    'shaking head',
+    'shaping bread dough',
+    'sharpening knives',
+    'sharpening pencil',
+    'shaving head',
+    'shaving legs',
+    'shearing sheep',
+    'shining flashlight',
+    'shining shoes',
+    'shoot dance',
+    'shooting basketball',
+    'shooting goal (soccer)',
+    'shooting off fireworks',
+    'shopping',
+    'shot put',
+    'shouting',
+    'shoveling snow',
+    'shredding paper',
+    'shucking oysters',
+    'shuffling cards',
+    'shuffling feet',
+    'side kick',
+    'sieving',
+    'sign language interpreting',
+    'silent disco',
+    'singing',
+    'sipping cup',
+    'situp',
+    'skateboarding',
+    'ski ballet',
+    'ski jumping',
+    'skiing crosscountry',
+    'skiing mono',
+    'skiing slalom',
+    'skipping rope',
+    'skipping stone',
+    'skydiving',
+    'slacklining',
+    'slapping',
+    'sled dog racing',
+    'sleeping',
+    'slicing onion',
+    'smashing',
+    'smelling feet',
+    'smoking',
+    'smoking hookah',
+    'smoking pipe',
+    'snatch weight lifting',
+    'sneezing',
+    'snorkeling',
+    'snowboarding',
+    'snowkiting',
+    'snowmobiling',
+    'somersaulting',
+    'spelunking',
+    'spinning plates',
+    'spinning poi',
+    'splashing water',
+    'spray painting',
+    'spraying',
+    'springboard diving',
+    'square dancing',
+    'squat',
+    'squeezing orange',
+    'stacking cups',
+    'stacking dice',
+    'standing on hands',
+    'staring',
+    'steer roping',
+    'steering car',
+    'sticking tongue out',
+    'stomping grapes',
+    'stretching arm',
+    'stretching leg',
+    'sucking lolly',
+    'surfing crowd',
+    'surfing water',
+    'surveying',
+    'sweeping floor',
+    'swimming backstroke',
+    'swimming breast stroke',
+    'swimming butterfly stroke',
+    'swimming front crawl',
+    'swimming with dolphins',
+    'swimming with sharks',
+    'swing dancing',
+    'swinging baseball bat',
+    'swinging on something',
+    'sword fighting',
+    'sword swallowing',
+    'tackling',
+    'tagging graffiti',
+    'tai chi',
+    'taking photo',
+    'talking on cell phone',
+    'tango dancing',
+    'tap dancing',
+    'tapping guitar',
+    'tapping pen',
+    'tasting beer',
+    'tasting food',
+    'tasting wine',
+    'testifying',
+    'texting',
+    'threading needle',
+    'throwing axe',
+    'throwing ball (not baseball or American football)',
+    'throwing discus',
+    'throwing knife',
+    'throwing snowballs',
+    'throwing tantrum',
+    'throwing water balloon',
+    'tickling',
+    'tie dying',
+    'tightrope walking',
+    'tiptoeing',
+    'tobogganing',
+    'tossing coin',
+    'tossing salad',
+    'training dog',
+    'trapezing',
+    'treating wood',
+    'trimming or shaving beard',
+    'trimming shrubs',
+    'trimming trees',
+    'triple jump',
+    'twiddling fingers',
+    'tying bow tie',
+    'tying knot (not on a tie)',
+    'tying necktie',
+    'tying shoe laces',
+    'unboxing',
+    'uncorking champagne',
+    'unloading truck',
+    'using a microscope',
+    'using a paint roller',
+    'using a power drill',
+    'using a sledge hammer',
+    'using a wrench',
+    'using atm',
+    'using bagging machine',
+    'using circular saw',
+    'using inhaler',
+    'using megaphone',
+    'using puppets',
+    'using remote controller (not gaming)',
+    'using segway',
+    'vacuuming car',
+    'vacuuming floor',
+    'visiting the zoo',
+    'wading through mud',
+    'wading through water',
+    'waiting in line',
+    'waking up',
+    'walking on stilts',
+    'walking the dog',
+    'walking through snow',
+    'walking with crutches',
+    'washing dishes',
+    'washing feet',
+    'washing hair',
+    'washing hands',
+    'watching tv',
+    'water skiing',
+    'water sliding',
+    'watering plants',
+    'waving hand',
+    'waxing armpits',
+    'waxing back',
+    'waxing chest',
+    'waxing eyebrows',
+    'waxing legs',
+    'weaving basket',
+    'weaving fabric',
+    'welding',
+    'whistling',
+    'windsurfing',
+    'winking',
+    'wood burning (art)',
+    'wrapping present',
+    'wrestling',
+    'writing',
+    'yarn spinning',
+    'yawning',
+    'yoga',
+    'zumba'
+]
+
+templates = [
+    'a photo of {}.',
+    'a photo of a person {}.',
+    'a photo of a person using {}.',
+    'a photo of a person doing {}.',
+    'a photo of a person during {}.',
+    'a photo of a person performing {}.',
+    'a photo of a person practicing {}.',
+    'a video of {}.',
+    'a video of a person {}.',
+    'a video of a person using {}.',
+    'a video of a person doing {}.',
+    'a video of a person during {}.',
+    'a video of a person performing {}.',
+    'a video of a person practicing {}.',
+    'a example of {}.',
+    'a example of a person {}.',
+    'a example of a person using {}.',
+    'a example of a person doing {}.',
+    'a example of a person during {}.',
+    'a example of a person performing {}.',
+    'a example of a person practicing {}.',
+    'a demonstration of {}.',
+    'a demonstration of a person {}.',
+    'a demonstration of a person using {}.',
+    'a demonstration of a person doing {}.',
+    'a demonstration of a person during {}.',
+    'a demonstration of a person performing {}.',
+    'a demonstration of a person practicing {}.',
+]
+```
+
+
+
+## MNIST
+
+```bash
+classes = [
+    '0',
+    '1',
+    '2',
+    '3',
+    '4',
+    '5',
+    '6',
+    '7',
+    '8',
+    '9',
+]
+
+templates = [
+    'a photo of the number: "{}".',
+]
+```
+
+
+
+## OxfordPets
+
+```bash
+classes = [
+    'Abyssinian',
+    'Bengal',
+    'Birman',
+    'Bombay',
+    'British Shorthair',
+    'Egyptian Mau',
+    'Maine Coon',
+    'Persian',
+    'Ragdoll',
+    'Russian Blue',
+    'Siamese',
+    'Sphynx',
+    'american bulldog',
+    'american pit bull terrier',
+    'basset hound',
+    'beagle',
+    'boxer',
+    'chihuahua',
+    'english cocker spaniel',
+    'english setter',
+    'german shorthaired',
+    'great pyrenees',
+    'havanese',
+    'japanese chin',
+    'keeshond',
+    'leonberger',
+    'miniature pinscher',
+    'newfoundland',
+    'pomeranian',
+    'pug',
+    'saint bernard',
+    'samoyed',
+    'scottish terrier',
+    'shiba inu',
+    'staffordshire bull terrier',
+    'wheaten terrier',
+    'yorkshire terrier',
+]
+
+templates = [
+    'a photo of a {}, a type of pet.',
+]
+```
+
+
+
+## PascalVOC2007
+
+```bash
+classes = [
+    'aeroplane',
+    'bicycle',
+    'bird',
+    'boat',
+    'bottle',
+    'bus',
+    'car',
+    'cat',
+    'chair',
+    'cow',
+    'dog',
+    'horse',
+    'motorbike',
+    'person',
+    'sheep',
+    'sofa',
+    'diningtable',
+    'pottedplant',
+    'train',
+    'tvmonitor',
+]
+
+templates = [
+    'a photo of a {}.',
+]
+```
+
+
+
+## PatchCamelyon
+
+```bash
+classes = [
+    'lymph node',
+    'lymph node containing metastatic tumor tissue',
+]
+
+templates = [
+    'this is a photo of {}',
+]
+```
+
+
+
+## RESISC45
+
+```bash
+classes = [
+    'airplane',
+    'airport',
+    'baseball diamond',
+    'basketball court',
+    'beach',
+    'bridge',
+    'chaparral',
+    'church',
+    'circular farmland',
+    'cloud',
+    'commercial area',
+    'dense residential',
+    'desert',
+    'forest',
+    'freeway',
+    'golf course',
+    'ground track field',
+    'harbor',
+    'industrial area',
+    'intersection',
+    'island',
+    'lake',
+    'meadow',
+    'medium residential',
+    'mobile home park',
+    'mountain',
+    'overpass',
+    'palace',
+    'parking lot',
+    'railway',
+    'railway station',
+    'rectangular farmland',
+    'river',
+    'roundabout',
+    'runway',
+    'sea ice',
+    'ship',
+    'snowberg',
+    'sparse residential',
+    'stadium',
+    'storage tank',
+    'tennis court',
+    'terrace',
+    'thermal power station',
+    'wetland',
+]
+
+templates = [
+    'satellite imagery of {}.',
+    'aerial imagery of {}.',
+    'satellite photo of {}.',
+    'aerial photo of {}.',
+    'satellite view of {}.',
+    'aerial view of {}.',
+    'satellite imagery of a {}.',
+    'aerial imagery of a {}.',
+    'satellite photo of a {}.',
+    'aerial photo of a {}.',
+    'satellite view of a {}.',
+    'aerial view of a {}.',
+    'satellite imagery of the {}.',
+    'aerial imagery of the {}.',
+    'satellite photo of the {}.',
+    'aerial photo of the {}.',
+    'satellite view of the {}.',
+    'aerial view of the {}.',
+]
+```
+
+
+
+## SST2
+
+```bash
+classes = [
+    'negative',
+    'positive',
+]
+
+templates = [
+    'a {} review of a movie.',
+]
+```
+
+
+
+## STL10
+
+```bash
+classes = [
+    'airplane',
+    'bird',
+    'car',
+    'cat',
+    'deer',
+    'dog',
+    'horse',
+    'monkey',
+    'ship',
+    'truck',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a photo of the {}.',
+]
+```
+
+
+
+## SUN397
+
+```bash
+classes = [
+    'abbey',
+    'airplane cabin',
+    'airport terminal',
+    'alley',
+    'amphitheater',
+    'amusement arcade',
+    'amusement park',
+    'anechoic chamber',
+    'apartment building outdoor',
+    'apse indoor',
+    'aquarium',
+    'aqueduct',
+    'arch',
+    'archive',
+    'arrival gate outdoor',
+    'art gallery',
+    'art school',
+    'art studio',
+    'assembly line',
+    'athletic field outdoor',
+    'atrium public',
+    'attic',
+    'auditorium',
+    'auto factory',
+    'badlands',
+    'badminton court indoor',
+    'baggage claim',
+    'bakery shop',
+    'balcony exterior',
+    'balcony interior',
+    'ball pit',
+    'ballroom',
+    'bamboo forest',
+    'banquet hall',
+    'bar',
+    'barn',
+    'barndoor',
+    'baseball field',
+    'basement',
+    'basilica',
+    'basketball court outdoor',
+    'bathroom',
+    'batters box',
+    'bayou',
+    'bazaar indoor',
+    'bazaar outdoor',
+    'beach',
+    'beauty salon',
+    'bedroom',
+    'berth',
+    'biology laboratory',
+    'bistro indoor',
+    'boardwalk',
+    'boat deck',
+    'boathouse',
+    'bookstore',
+    'booth indoor',
+    'botanical garden',
+    'bow window indoor',
+    'bow window outdoor',
+    'bowling alley',
+    'boxing ring',
+    'brewery indoor',
+    'bridge',
+    'building facade',
+    'bullring',
+    'burial chamber',
+    'bus interior',
+    'butchers shop',
+    'butte',
+    'cabin outdoor',
+    'cafeteria',
+    'campsite',
+    'campus',
+    'canal natural',
+    'canal urban',
+    'candy store',
+    'canyon',
+    'car interior backseat',
+    'car interior frontseat',
+    'carrousel',
+    'casino indoor',
+    'castle',
+    'catacomb',
+    'cathedral indoor',
+    'cathedral outdoor',
+    'cavern indoor',
+    'cemetery',
+    'chalet',
+    'cheese factory',
+    'chemistry lab',
+    'chicken coop indoor',
+    'chicken coop outdoor',
+    'childs room',
+    'church indoor',
+    'church outdoor',
+    'classroom',
+    'clean room',
+    'cliff',
+    'cloister indoor',
+    'closet',
+    'clothing store',
+    'coast',
+    'cockpit',
+    'coffee shop',
+    'computer room',
+    'conference center',
+    'conference room',
+    'construction site',
+    'control room',
+    'control tower outdoor',
+    'corn field',
+    'corral',
+    'corridor',
+    'cottage garden',
+    'courthouse',
+    'courtroom',
+    'courtyard',
+    'covered bridge exterior',
+    'creek',
+    'crevasse',
+    'crosswalk',
+    'cubicle office',
+    'dam',
+    'delicatessen',
+    'dentists office',
+    'desert sand',
+    'desert vegetation',
+    'diner indoor',
+    'diner outdoor',
+    'dinette home',
+    'dinette vehicle',
+    'dining car',
+    'dining room',
+    'discotheque',
+    'dock',
+    'doorway outdoor',
+    'dorm room',
+    'driveway',
+    'driving range outdoor',
+    'drugstore',
+    'electrical substation',
+    'elevator door',
+    'elevator interior',
+    'elevator shaft',
+    'engine room',
+    'escalator indoor',
+    'excavation',
+    'factory indoor',
+    'fairway',
+    'fastfood restaurant',
+    'field cultivated',
+    'field wild',
+    'fire escape',
+    'fire station',
+    'firing range indoor',
+    'fishpond',
+    'florist shop indoor',
+    'food court',
+    'forest broadleaf',
+    'forest needleleaf',
+    'forest path',
+    'forest road',
+    'formal garden',
+    'fountain',
+    'galley',
+    'game room',
+    'garage indoor',
+    'garbage dump',
+    'gas station',
+    'gazebo exterior',
+    'general store indoor',
+    'general store outdoor',
+    'gift shop',
+    'golf course',
+    'greenhouse indoor',
+    'greenhouse outdoor',
+    'gymnasium indoor',
+    'hangar indoor',
+    'hangar outdoor',
+    'harbor',
+    'hayfield',
+    'heliport',
+    'herb garden',
+    'highway',
+    'hill',
+    'home office',
+    'hospital',
+    'hospital room',
+    'hot spring',
+    'hot tub outdoor',
+    'hotel outdoor',
+    'hotel room',
+    'house',
+    'hunting lodge outdoor',
+    'ice cream parlor',
+    'ice floe',
+    'ice shelf',
+    'ice skating rink indoor',
+    'ice skating rink outdoor',
+    'iceberg',
+    'igloo',
+    'industrial area',
+    'inn outdoor',
+    'islet',
+    'jacuzzi indoor',
+    'jail cell',
+    'jail indoor',
+    'jewelry shop',
+    'kasbah',
+    'kennel indoor',
+    'kennel outdoor',
+    'kindergarden classroom',
+    'kitchen',
+    'kitchenette',
+    'labyrinth outdoor',
+    'lake natural',
+    'landfill',
+    'landing deck',
+    'laundromat',
+    'lecture room',
+    'library indoor',
+    'library outdoor',
+    'lido deck outdoor',
+    'lift bridge',
+    'lighthouse',
+    'limousine interior',
+    'living room',
+    'lobby',
+    'lock chamber',
+    'locker room',
+    'mansion',
+    'manufactured home',
+    'market indoor',
+    'market outdoor',
+    'marsh',
+    'martial arts gym',
+    'mausoleum',
+    'medina',
+    'moat water',
+    'monastery outdoor',
+    'mosque indoor',
+    'mosque outdoor',
+    'motel',
+    'mountain',
+    'mountain snowy',
+    'movie theater indoor',
+    'museum indoor',
+    'music store',
+    'music studio',
+    'nuclear power plant outdoor',
+    'nursery',
+    'oast house',
+    'observatory outdoor',
+    'ocean',
+    'office',
+    'office building',
+    'oil refinery outdoor',
+    'oilrig',
+    'operating room',
+    'orchard',
+    'outhouse outdoor',
+    'pagoda',
+    'palace',
+    'pantry',
+    'park',
+    'parking garage indoor',
+    'parking garage outdoor',
+    'parking lot',
+    'parlor',
+    'pasture',
+    'patio',
+    'pavilion',
+    'pharmacy',
+    'phone booth',
+    'physics laboratory',
+    'picnic area',
+    'pilothouse indoor',
+    'planetarium outdoor',
+    'playground',
+    'playroom',
+    'plaza',
+    'podium indoor',
+    'podium outdoor',
+    'pond',
+    'poolroom establishment',
+    'poolroom home',
+    'power plant outdoor',
+    'promenade deck',
+    'pub indoor',
+    'pulpit',
+    'putting green',
+    'racecourse',
+    'raceway',
+    'raft',
+    'railroad track',
+    'rainforest',
+    'reception',
+    'recreation room',
+    'residential neighborhood',
+    'restaurant',
+    'restaurant kitchen',
+    'restaurant patio',
+    'rice paddy',
+    'riding arena',
+    'river',
+    'rock arch',
+    'rope bridge',
+    'ruin',
+    'runway',
+    'sandbar',
+    'sandbox',
+    'sauna',
+    'schoolhouse',
+    'sea cliff',
+    'server room',
+    'shed',
+    'shoe shop',
+    'shopfront',
+    'shopping mall indoor',
+    'shower',
+    'skatepark',
+    'ski lodge',
+    'ski resort',
+    'ski slope',
+    'sky',
+    'skyscraper',
+    'slum',
+    'snowfield',
+    'squash court',
+    'stable',
+    'stadium baseball',
+    'stadium football',
+    'stage indoor',
+    'staircase',
+    'street',
+    'subway interior',
+    'subway station platform',
+    'supermarket',
+    'sushi bar',
+    'swamp',
+    'swimming pool indoor',
+    'swimming pool outdoor',
+    'synagogue indoor',
+    'synagogue outdoor',
+    'television studio',
+    'temple east asia',
+    'temple south asia',
+    'tennis court indoor',
+    'tennis court outdoor',
+    'tent outdoor',
+    'theater indoor procenium',
+    'theater indoor seats',
+    'thriftshop',
+    'throne room',
+    'ticket booth',
+    'toll plaza',
+    'topiary garden',
+    'tower',
+    'toyshop',
+    'track outdoor',
+    'train railway',
+    'train station platform',
+    'tree farm',
+    'tree house',
+    'trench',
+    'underwater coral reef',
+    'utility room',
+    'valley',
+    'van interior',
+    'vegetable garden',
+    'veranda',
+    'veterinarians office',
+    'viaduct',
+    'videostore',
+    'village',
+    'vineyard',
+    'volcano',
+    'volleyball court indoor',
+    'volleyball court outdoor',
+    'waiting room',
+    'warehouse indoor',
+    'water tower',
+    'waterfall block',
+    'waterfall fan',
+    'waterfall plunge',
+    'watering hole',
+    'wave',
+    'wet bar',
+    'wheat field',
+    'wind farm',
+    'windmill',
+    'wine cellar barrel storage',
+    'wine cellar bottle storage',
+    'wrestling ring indoor',
+    'yard',
+    'youth hostel',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a photo of the {}.',
+]
+```
+
+
+
+## StanfordCars
+
+```bash
+classes = [
+    'AM General Hummer SUV 2000',
+    'Acura RL Sedan 2012',
+    'Acura TL Sedan 2012',
+    'Acura TL Type-S 2008',
+    'Acura TSX Sedan 2012',
+    'Acura Integra Type R 2001',
+    'Acura ZDX Hatchback 2012',
+    'Aston Martin V8 Vantage Convertible 2012',
+    'Aston Martin V8 Vantage Coupe 2012',
+    'Aston Martin Virage Convertible 2012',
+    'Aston Martin Virage Coupe 2012',
+    'Audi RS 4 Convertible 2008',
+    'Audi A5 Coupe 2012',
+    'Audi TTS Coupe 2012',
+    'Audi R8 Coupe 2012',
+    'Audi V8 Sedan 1994',
+    'Audi 100 Sedan 1994',
+    'Audi 100 Wagon 1994',
+    'Audi TT Hatchback 2011',
+    'Audi S6 Sedan 2011',
+    'Audi S5 Convertible 2012',
+    'Audi S5 Coupe 2012',
+    'Audi S4 Sedan 2012',
+    'Audi S4 Sedan 2007',
+    'Audi TT RS Coupe 2012',
+    'BMW ActiveHybrid 5 Sedan 2012',
+    'BMW 1 Series Convertible 2012',
+    'BMW 1 Series Coupe 2012',
+    'BMW 3 Series Sedan 2012',
+    'BMW 3 Series Wagon 2012',
+    'BMW 6 Series Convertible 2007',
+    'BMW X5 SUV 2007',
+    'BMW X6 SUV 2012',
+    'BMW M3 Coupe 2012',
+    'BMW M5 Sedan 2010',
+    'BMW M6 Convertible 2010',
+    'BMW X3 SUV 2012',
+    'BMW Z4 Convertible 2012',
+    'Bentley Continental Supersports Conv. Convertible 2012',
+    'Bentley Arnage Sedan 2009',
+    'Bentley Mulsanne Sedan 2011',
+    'Bentley Continental GT Coupe 2012',
+    'Bentley Continental GT Coupe 2007',
+    'Bentley Continental Flying Spur Sedan 2007',
+    'Bugatti Veyron 16.4 Convertible 2009',
+    'Bugatti Veyron 16.4 Coupe 2009',
+    'Buick Regal GS 2012',
+    'Buick Rainier SUV 2007',
+    'Buick Verano Sedan 2012',
+    'Buick Enclave SUV 2012',
+    'Cadillac CTS-V Sedan 2012',
+    'Cadillac SRX SUV 2012',
+    'Cadillac Escalade EXT Crew Cab 2007',
+    'Chevrolet Silverado 1500 Hybrid Crew Cab 2012',
+    'Chevrolet Corvette Convertible 2012',
+    'Chevrolet Corvette ZR1 2012',
+    'Chevrolet Corvette Ron Fellows Edition Z06 2007',
+    'Chevrolet Traverse SUV 2012',
+    'Chevrolet Camaro Convertible 2012',
+    'Chevrolet HHR SS 2010',
+    'Chevrolet Impala Sedan 2007',
+    'Chevrolet Tahoe Hybrid SUV 2012',
+    'Chevrolet Sonic Sedan 2012',
+    'Chevrolet Express Cargo Van 2007',
+    'Chevrolet Avalanche Crew Cab 2012',
+    'Chevrolet Cobalt SS 2010',
+    'Chevrolet Malibu Hybrid Sedan 2010',
+    'Chevrolet TrailBlazer SS 2009',
+    'Chevrolet Silverado 2500HD Regular Cab 2012',
+    'Chevrolet Silverado 1500 Classic Extended Cab 2007',
+    'Chevrolet Express Van 2007',
+    'Chevrolet Monte Carlo Coupe 2007',
+    'Chevrolet Malibu Sedan 2007',
+    'Chevrolet Silverado 1500 Extended Cab 2012',
+    'Chevrolet Silverado 1500 Regular Cab 2012',
+    'Chrysler Aspen SUV 2009',
+    'Chrysler Sebring Convertible 2010',
+    'Chrysler Town and Country Minivan 2012',
+    'Chrysler 300 SRT-8 2010',
+    'Chrysler Crossfire Convertible 2008',
+    'Chrysler PT Cruiser Convertible 2008',
+    'Daewoo Nubira Wagon 2002',
+    'Dodge Caliber Wagon 2012',
+    'Dodge Caliber Wagon 2007',
+    'Dodge Caravan Minivan 1997',
+    'Dodge Ram Pickup 3500 Crew Cab 2010',
+    'Dodge Ram Pickup 3500 Quad Cab 2009',
+    'Dodge Sprinter Cargo Van 2009',
+    'Dodge Journey SUV 2012',
+    'Dodge Dakota Crew Cab 2010',
+    'Dodge Dakota Club Cab 2007',
+    'Dodge Magnum Wagon 2008',
+    'Dodge Challenger SRT8 2011',
+    'Dodge Durango SUV 2012',
+    'Dodge Durango SUV 2007',
+    'Dodge Charger Sedan 2012',
+    'Dodge Charger SRT-8 2009',
+    'Eagle Talon Hatchback 1998',
+    'FIAT 500 Abarth 2012',
+    'FIAT 500 Convertible 2012',
+    'Ferrari FF Coupe 2012',
+    'Ferrari California Convertible 2012',
+    'Ferrari 458 Italia Convertible 2012',
+    'Ferrari 458 Italia Coupe 2012',
+    'Fisker Karma Sedan 2012',
+    'Ford F-450 Super Duty Crew Cab 2012',
+    'Ford Mustang Convertible 2007',
+    'Ford Freestar Minivan 2007',
+    'Ford Expedition EL SUV 2009',
+    'Ford Edge SUV 2012',
+    'Ford Ranger SuperCab 2011',
+    'Ford GT Coupe 2006',
+    'Ford F-150 Regular Cab 2012',
+    'Ford F-150 Regular Cab 2007',
+    'Ford Focus Sedan 2007',
+    'Ford E-Series Wagon Van 2012',
+    'Ford Fiesta Sedan 2012',
+    'GMC Terrain SUV 2012',
+    'GMC Savana Van 2012',
+    'GMC Yukon Hybrid SUV 2012',
+    'GMC Acadia SUV 2012',
+    'GMC Canyon Extended Cab 2012',
+    'Geo Metro Convertible 1993',
+    'HUMMER H3T Crew Cab 2010',
+    'HUMMER H2 SUT Crew Cab 2009',
+    'Honda Odyssey Minivan 2012',
+    'Honda Odyssey Minivan 2007',
+    'Honda Accord Coupe 2012',
+    'Honda Accord Sedan 2012',
+    'Hyundai Veloster Hatchback 2012',
+    'Hyundai Santa Fe SUV 2012',
+    'Hyundai Tucson SUV 2012',
+    'Hyundai Veracruz SUV 2012',
+    'Hyundai Sonata Hybrid Sedan 2012',
+    'Hyundai Elantra Sedan 2007',
+    'Hyundai Accent Sedan 2012',
+    'Hyundai Genesis Sedan 2012',
+    'Hyundai Sonata Sedan 2012',
+    'Hyundai Elantra Touring Hatchback 2012',
+    'Hyundai Azera Sedan 2012',
+    'Infiniti G Coupe IPL 2012',
+    'Infiniti QX56 SUV 2011',
+    'Isuzu Ascender SUV 2008',
+    'Jaguar XK XKR 2012',
+    'Jeep Patriot SUV 2012',
+    'Jeep Wrangler SUV 2012',
+    'Jeep Liberty SUV 2012',
+    'Jeep Grand Cherokee SUV 2012',
+    'Jeep Compass SUV 2012',
+    'Lamborghini Reventon Coupe 2008',
+    'Lamborghini Aventador Coupe 2012',
+    'Lamborghini Gallardo LP 570-4 Superleggera 2012',
+    'Lamborghini Diablo Coupe 2001',
+    'Land Rover Range Rover SUV 2012',
+    'Land Rover LR2 SUV 2012',
+    'Lincoln Town Car Sedan 2011',
+    'MINI Cooper Roadster Convertible 2012',
+    'Maybach Landaulet Convertible 2012',
+    'Mazda Tribute SUV 2011',
+    'McLaren MP4-12C Coupe 2012',
+    'Mercedes-Benz 300-Class Convertible 1993',
+    'Mercedes-Benz C-Class Sedan 2012',
+    'Mercedes-Benz SL-Class Coupe 2009',
+    'Mercedes-Benz E-Class Sedan 2012',
+    'Mercedes-Benz S-Class Sedan 2012',
+    'Mercedes-Benz Sprinter Van 2012',
+    'Mitsubishi Lancer Sedan 2012',
+    'Nissan Leaf Hatchback 2012',
+    'Nissan NV Passenger Van 2012',
+    'Nissan Juke Hatchback 2012',
+    'Nissan 240SX Coupe 1998',
+    'Plymouth Neon Coupe 1999',
+    'Porsche Panamera Sedan 2012',
+    'Ram C/V Cargo Van Minivan 2012',
+    'Rolls-Royce Phantom Drophead Coupe Convertible 2012',
+    'Rolls-Royce Ghost Sedan 2012',
+    'Rolls-Royce Phantom Sedan 2012',
+    'Scion xD Hatchback 2012',
+    'Spyker C8 Convertible 2009',
+    'Spyker C8 Coupe 2009',
+    'Suzuki Aerio Sedan 2007',
+    'Suzuki Kizashi Sedan 2012',
+    'Suzuki SX4 Hatchback 2012',
+    'Suzuki SX4 Sedan 2012',
+    'Tesla Model S Sedan 2012',
+    'Toyota Sequoia SUV 2012',
+    'Toyota Camry Sedan 2012',
+    'Toyota Corolla Sedan 2012',
+    'Toyota 4Runner SUV 2012',
+    'Volkswagen Golf Hatchback 2012',
+    'Volkswagen Golf Hatchback 1991',
+    'Volkswagen Beetle Hatchback 2012',
+    'Volvo C30 Hatchback 2012',
+    'Volvo 240 Sedan 1993',
+    'Volvo XC90 SUV 2007',
+    'smart fortwo Convertible 2012',
+]
+
+templates = [
+    'a photo of a {}.',
+    'a photo of the {}.',
+    'a photo of my {}.',
+    'i love my {}!',
+    'a photo of my dirty {}.',
+    'a photo of my clean {}.',
+    'a photo of my new {}.',
+    'a photo of my old {}.',
+]
+```
+
+
+
+## UCF101
+
+```bash
+classes = [
+    'Apply Eye Makeup',
+    'Apply Lipstick',
+    'Archery',
+    'Baby Crawling',
+    'Balance Beam',
+    'Band Marching',
+    'Baseball Pitch',
+    'Basketball',
+    'Basketball Dunk',
+    'Bench Press',
+    'Biking',
+    'Billiards',
+    'Blow Dry Hair',
+    'Blowing Candles',
+    'Body Weight Squats',
+    'Bowling',
+    'Boxing Punching Bag',
+    'Boxing Speed Bag',
+    'Breast Stroke',
+    'Brushing Teeth',
+    'Clean And Jerk',
+    'Cliff Diving',
+    'Cricket Bowling',
+    'Cricket Shot',
+    'Cutting In Kitchen',
+    'Diving',
+    'Drumming',
+    'Fencing',
+    'Field Hockey Penalty',
+    'Floor Gymnastics',
+    'Frisbee Catch',
+    'Front Crawl',
+    'Golf Swing',
+    'Haircut',
+    'Hammer Throw',
+    'Hammering',
+    'Hand Stand Pushups',
+    'Handstand Walking',
+    'Head Massage',
+    'High Jump',
+    'Horse Race',
+    'Horse Riding',
+    'Hula Hoop',
+    'Ice Dancing',
+    'Javelin Throw',
+    'Juggling Balls',
+    'Jump Rope',
+    'Jumping Jack',
+    'Kayaking',
+    'Knitting',
+    'Long Jump',
+    'Lunges',
+    'Military Parade',
+    'Mixing',
+    'Mopping Floor',
+    'Nunchucks',
+    'Parallel Bars',
+    'Pizza Tossing',
+    'Playing Cello',
+    'Playing Daf',
+    'Playing Dhol',
+    'Playing Flute',
+    'Playing Guitar',
+    'Playing Piano',
+    'Playing Sitar',
+    'Playing Tabla',
+    'Playing Violin',
+    'Pole Vault',
+    'Pommel Horse',
+    'Pull Ups',
+    'Punch',
+    'Push Ups',
+    'Rafting',
+    'Rock Climbing Indoor',
+    'Rope Climbing',
+    'Rowing',
+    'Salsa Spin',
+    'Shaving Beard',
+    'Shotput',
+    'Skate Boarding',
+    'Skiing',
+    'Skijet',
+    'Sky Diving',
+    'Soccer Juggling',
+    'Soccer Penalty',
+    'Still Rings',
+    'Sumo Wrestling',
+    'Surfing',
+    'Swing',
+    'Table Tennis Shot',
+    'Tai Chi',
+    'Tennis Swing',
+    'Throw Discus',
+    'Trampoline Jumping',
+    'Typing',
+    'Uneven Bars',
+    'Volleyball Spiking',
+    'Walking With Dog',
+    'Wall Pushups',
+    'Writing On Board',
+    'Yo Yo',
+]
+
+templates = [
+    'a photo of a person {}.',
+    'a video of a person {}.',
+    'a example of a person {}.',
+    'a demonstration of a person {}.',
+    'a photo of the person {}.',
+    'a video of the person {}.',
+    'a example of the person {}.',
+    'a demonstration of the person {}.',
+    'a photo of a person using {}.',
+    'a video of a person using {}.',
+    'a example of a person using {}.',
+    'a demonstration of a person using {}.',
+    'a photo of the person using {}.',
+    'a video of the person using {}.',
+    'a example of the person using {}.',
+    'a demonstration of the person using {}.',
+    'a photo of a person doing {}.',
+    'a video of a person doing {}.',
+    'a example of a person doing {}.',
+    'a demonstration of a person doing {}.',
+    'a photo of the person doing {}.',
+    'a video of the person doing {}.',
+    'a example of the person doing {}.',
+    'a demonstration of the person doing {}.',
+    'a photo of a person during {}.',
+    'a video of a person during {}.',
+    'a example of a person during {}.',
+    'a demonstration of a person during {}.',
+    'a photo of the person during {}.',
+    'a video of the person during {}.',
+    'a example of the person during {}.',
+    'a demonstration of the person during {}.',
+    'a photo of a person performing {}.',
+    'a video of a person performing {}.',
+    'a example of a person performing {}.',
+    'a demonstration of a person performing {}.',
+    'a photo of the person performing {}.',
+    'a video of the person performing {}.',
+    'a example of the person performing {}.',
+    'a demonstration of the person performing {}.',
+    'a photo of a person practicing {}.',
+    'a video of a person practicing {}.',
+    'a example of a person practicing {}.',
+    'a demonstration of a person practicing {}.',
+    'a photo of the person practicing {}.',
+    'a video of the person practicing {}.',
+    'a example of the person practicing {}.',
+    'a demonstration of the person practicing {}.',
+]
+```
+
+

CLIP/data/rendered-sst2.md

@@ -0,0 +1,11 @@
+# The Rendered SST2 Dataset
+
+In the paper, we used an image classification dataset called Rendered SST2, to evaluate the model's capability on optical character recognition. To do so, we rendered the sentences in the [Standford Sentiment Treebank v2](https://nlp.stanford.edu/sentiment/treebank.html) dataset and used those as the input to the CLIP image encoder.
+
+The following command will download a 131MB archive countaining the images and extract into a subdirectory `rendered-sst2`:
+
+```bash
+wget https://openaipublic.azureedge.net/clip/data/rendered-sst2.tgz
+tar zxvf rendered-sst2.tgz
+```
+

CLIP/data/yfcc100m.md

@@ -0,0 +1,14 @@
+# The YFCC100M Subset
+
+In the paper, we performed a dataset ablation using a subset of the YFCC100M dataset and showed that the performance remained largely similar. 
+
+The subset contains 14,829,396 images, about 15% of the full dataset, which have been filtered to only keep those with natural languag titles and/or descriptions in English.
+
+We provide the list of (line number, photo identifier, photo hash) of each image contained in this subset. These correspond to the first three columns in the dataset's metadata TSV file.
+
+```bash
+wget https://openaipublic.azureedge.net/clip/data/yfcc100m_subset_data.tsv.bz2
+bunzip2 yfcc100m_subset_data.tsv.bz2
+```
+
+Use of the underlying media files is subject to the Creative Commons licenses chosen by their creators/uploaders. For more information about the YFCC100M dataset, visit [the official website](https://multimediacommons.wordpress.com/yfcc100m-core-dataset/).

CLIP/model-card.md

@@ -0,0 +1,120 @@
+# Model Card: CLIP
+
+Inspired by [Model Cards for Model Reporting (Mitchell et al.)](https://arxiv.org/abs/1810.03993) and [Lessons from Archives (Jo & Gebru)](https://arxiv.org/pdf/1912.10389.pdf), we’re providing some accompanying information about the multimodal model.
+
+## Model Details
+
+The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.
+
+### Model Date
+
+January 2021
+
+### Model Type
+
+The base model uses a ResNet50 with several modifications as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. There is also a variant of the model where the ResNet image encoder is replaced with a Vision Transformer.
+
+### Model Versions
+
+Initially, we’ve released one CLIP model based on the Vision Transformer architecture equivalent to ViT-B/32, along with the RN50 model, using the architecture equivalent to ResNet-50.
+
+As part of the staged release process, we have also released the RN101 model, as well as RN50x4, a RN50 scaled up 4x according to the [EfficientNet](https://arxiv.org/abs/1905.11946) scaling rule. In July 2021, we additionally released the RN50x16 and ViT-B/16 models, and In January 2022, the RN50x64 and ViT-L/14 models were released.
+
+Please see the paper linked below for further details about their specification.
+
+### Documents
+
+- [Blog Post](https://openai.com/blog/clip/)
+- [CLIP Paper](https://arxiv.org/abs/2103.00020)
+
+
+
+## Model Use
+
+### Intended Use
+
+The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.
+
+#### Primary intended uses
+
+The primary intended users of these models are AI researchers.
+
+We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.
+
+### Out-of-Scope Use Cases
+
+**Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. 
+
+Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.
+
+Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.
+
+
+
+## Data
+
+The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.
+
+### Data Mission Statement
+
+Our goal with building this dataset was to test out robustness and generalizability in computer vision tasks. As a result, the focus was on gathering large quantities of data from different publicly-available internet data sources. The data was gathered in a mostly non-interventionist manner. However, we only crawled websites that had policies against excessively violent and adult images and allowed us to filter out such content. We do not intend for this dataset to be used as the basis for any commercial or deployed model and will not be releasing the dataset.
+
+
+
+## Performance and Limitations
+
+### Performance
+
+We have evaluated the performance of CLIP on a wide range of benchmarks across a variety of computer vision datasets such as OCR to texture recognition to fine-grained classification. The paper describes model performance on the following datasets:
+
+- Food101
+- CIFAR10   
+- CIFAR100   
+- Birdsnap
+- SUN397
+- Stanford Cars
+- FGVC Aircraft
+- VOC2007
+- DTD
+- Oxford-IIIT Pet dataset
+- Caltech101
+- Flowers102
+- MNIST   
+- SVHN 
+- IIIT5K   
+- Hateful Memes   
+- SST-2
+- UCF101
+- Kinetics700
+- Country211
+- CLEVR Counting
+- KITTI Distance
+- STL-10
+- RareAct
+- Flickr30
+- MSCOCO
+- ImageNet
+- ImageNet-A
+- ImageNet-R
+- ImageNet Sketch
+- ObjectNet (ImageNet Overlap)
+- Youtube-BB
+- ImageNet-Vid
+
+## Limitations
+
+CLIP and our analysis of it have a number of limitations. CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which we discuss in the paper and briefly in the next section. Additionally, our approach to testing CLIP also has an important limitation- in many cases we have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance.
+
+### Bias and Fairness
+
+We find that the performance of CLIP - and the specific biases it exhibits - can depend significantly on class design and the choices one makes for categories to include and exclude. We tested the risk of certain kinds of denigration with CLIP by classifying images of people from [Fairface](https://arxiv.org/abs/1908.04913) into crime-related and non-human animal categories. We found significant disparities with respect to race and gender. Additionally, we found that these disparities could shift based on how the classes were constructed. (Details captured in the Broader Impacts Section in the paper).
+
+We also tested the performance of CLIP on gender, race and age classification using the Fairface dataset (We default to using race categories as they are constructed in the Fairface dataset.) in order to assess quality of performance across different demographics. We found accuracy >96% across all races for gender classification with ‘Middle Eastern’ having the highest accuracy (98.4%) and ‘White’ having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification. Our use of evaluations to test for gender, race and age classification as well as denigration harms is simply to evaluate performance of the model across people and surface potential risks and not to demonstrate an endorsement/enthusiasm for such tasks.
+
+
+
+## Feedback
+
+### Where to send questions or comments about the model
+
+Please use [this Google Form](https://forms.gle/Uv7afRH5dvY34ZEs9)

CLIP/requirements.txt

@@ -0,0 +1,5 @@
+ftfy
+regex
+tqdm
+torch
+torchvision

CLIP/setup.py

@@ -0,0 +1,21 @@
+import os
+
+import pkg_resources
+from setuptools import setup, find_packages
+
+setup(
+    name="clip",
+    py_modules=["clip"],
+    version="1.0",
+    description="",
+    author="OpenAI",
+    packages=find_packages(exclude=["tests*"]),
+    install_requires=[
+        str(r)
+        for r in pkg_resources.parse_requirements(
+            open(os.path.join(os.path.dirname(__file__), "requirements.txt"))
+        )
+    ],
+    include_package_data=True,
+    extras_require={'dev': ['pytest']},
+)

CLIP/tests/test_consistency.py

@@ -0,0 +1,25 @@
+import numpy as np
+import pytest
+import torch
+from PIL import Image
+
+import clip
+
+
+@pytest.mark.parametrize('model_name', clip.available_models())
+def test_consistency(model_name):
+    device = "cpu"
+    jit_model, transform = clip.load(model_name, device=device, jit=True)
+    py_model, _ = clip.load(model_name, device=device, jit=False)
+
+    image = transform(Image.open("CLIP.png")).unsqueeze(0).to(device)
+    text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)
+
+    with torch.no_grad():
+        logits_per_image, _ = jit_model(image, text)
+        jit_probs = logits_per_image.softmax(dim=-1).cpu().numpy()
+
+        logits_per_image, _ = py_model(image, text)
+        py_probs = logits_per_image.softmax(dim=-1).cpu().numpy()
+
+    assert np.allclose(jit_probs, py_probs, atol=0.01, rtol=0.1)

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,13 +1,86 @@
----
-title: Text2live
-emoji: 🚀
-colorFrom: yellow
-colorTo: green
-sdk: streamlit
-sdk_version: 1.17.0
-app_file: app.py
-pinned: false
-license: unknown
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Text2LIVE: Text-Driven Layered Image and Video Editing (ECCV 2022 - Oral)
+## [<a href="https://text2live.github.io/" target="_blank">Project Page</a>]
+
+[![arXiv](https://img.shields.io/badge/arXiv-Text2LIVE-b31b1b.svg)](https://arxiv.org/abs/2204.02491)
+![Pytorch](https://img.shields.io/badge/PyTorch->=1.10.0-Red?logo=pytorch)
+[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/weizmannscience/text2live)
+
+![teaser](https://user-images.githubusercontent.com/22198039/179798581-ca6f6652-600a-400a-b21b-713fc5c15d56.png)
+
+**Text2LIVE** is a method for text-driven editing of real-world images and videos, as described in <a href="https://arxiv.org/abs/2204.02491" target="_blank">(link to paper)</a>.
+
+[//]: # (. It can be used for localized and global edits that change the texture of existing objects or augment the scene with semi-transparent effects &#40;e.g. smoke, fire, snow&#41;.)
+
+[//]: # (### Abstract)
+>We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Specifically, given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with new visual effects (e.g., smoke, fire) in a semantically meaningful manner. Our framework trains a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. Thus, it can perform localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.
+
+
+## Getting Started
+### Installation
+
+```
+git clone https://github.com/omerbt/Text2LIVE.git
+conda create --name text2live python=3.9 
+conda activate text2live 
+pip install -r requirements.txt
+```
+
+### Download sample images and videos
+Download sample images and videos from the DAVIS dataset:
+```
+cd Text2LIVE
+gdown https://drive.google.com/uc?id=1osN4PlPkY9uk6pFqJZo8lhJUjTIpa80J&export=download
+unzip data.zip
+```
+It will create a folder `data`:
+```
+Text2LIVE
+├── ...
+├── data
+│   ├── pretrained_nla_models # NLA models are stored here
+│   ├── images # sample images
+│   └── videos # sample videos from DAVIS dataset
+│         ├── car-turn # contains video frames 
+│         ├── ...
+└── ...
+```
+To enforce temporal consistency in video edits, we utilize the Neural Layered Atlases (NLA). Pretrained NLA models are taken from <a href="https://layered-neural-atlases.github.io">here</a>, and are already inside the `data` folder.
+
+### Run examples 
+* Our method is designed to change textures of existing objects / augment the scene with semi-transparent effects (e.g., smoke, fire). It is not designed for adding new objects or significantly deviating from the original spatial layout.
+* Training **Text2LIVE** multiple times with the same inputs can lead to slightly different results.
+* CLIP sometimes exhibits bias towards specific solutions (see figure 9 in the paper), thus slightly different text prompts may lead to different flavors of edits.
+
+
+The required GPU memory depends on the input image/video size, but you should be good with a Tesla V100 32GB :).
+Currently mixed precision introduces some instability in the training process, but it could be added later.
+
+#### Video Editing
+Run the following command to start training
+```
+python train_video.py --example_config car-turn_winter.yaml
+```
+#### Image Editing
+Run the following command to start training
+```
+python train_image.py --example_config golden_horse.yaml
+```
+Intermediate results will be saved to `results` during optimization. The frequency of saving intermediate results is indicated in the `log_images_freq` flag of the configuration.
+
+## Sample Results
+https://user-images.githubusercontent.com/22198039/179797381-983e0453-2e5d-40e8-983d-578217b358e4.mov
+
+For more see the [supplementary material](https://text2live.github.io/sm/index.html).
+
+
+## Citation
+```
+@inproceedings{bar2022text2live,
+  title={Text2live: Text-driven layered image and video editing},
+  author={Bar-Tal, Omer and Ofri-Amar, Dolev and Fridman, Rafail and Kasten, Yoni and Dekel, Tali},
+  booktitle={European Conference on Computer Vision},
+  pages={707--723},
+  year={2022},
+  organization={Springer}
+}
+```

Text2LIVE-main/README.md

@@ -75,12 +75,10 @@ For more see the [supplementary material](https://text2live.github.io/sm/index.h
 
 ## Citation
 ```
-@inproceedings{bar2022text2live,
-  title={Text2live: Text-driven layered image and video editing},
-  author={Bar-Tal, Omer and Ofri-Amar, Dolev and Fridman, Rafail and Kasten, Yoni and Dekel, Tali},
-  booktitle={European Conference on Computer Vision},
-  pages={707--723},
-  year={2022},
-  organization={Springer}
+@article{bar2022text2live,
+         title     = {Text2LIVE: Text-Driven Layered Image and Video Editing},
+         author    = {Bar-Tal, Omer and Ofri-Amar, Dolev and Fridman, Rafail and Kasten, Yoni and Dekel, Tali},
+         journal   = {arXiv preprint arXiv:2204.02491},
+         year      = {2022}
 }
 ```

Text2LIVE-main/data/data/pretrained_nla_models/blackswan/checkpoint

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f50895f39815de243cb8166001771260d9720e6d1bda6289088a0366c7c70f2
+size 14657387

Text2LIVE-main/data/data/pretrained_nla_models/car-turn/checkpoint

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:973953ed6f0f742df9ab3fd21e7369db541689c40a8cd22ddb12f912c2e84b95
+size 14657387

Text2LIVE-main/data/data/pretrained_nla_models/libby/checkpoint

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2095f38eacee144175b08fdaaffd52e97991c08f0825be0d8cf836a5297ae535
+size 14657387