gullalc/videomae-base-finetuned-kinetics-movieshots-movement

VideoMAE finetuned for shot movement classification

videomae-base-finetuned-kinetics model finetuned to classify shot movement into five classes: Static, Motion, Pull, Push

Movienet dataset is used for finetuning the model for 5 epochs. v1_split_trailer.json provides the training, validation and test data splits.

Evaluation

Model achieves accuracy of 91.35% and macro-f1 of 79.37%

Class-wise accuracies: Static - 92.94%, Motion - 91.7%, Pull - 51.25%, Push - 62.73%

How to use

This is how model can be tested on a shot/clip from a video. Same code is used to process, transform and evaluate on the movienet test set.

from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
from pytorchvideo.transforms import ApplyTransformToKey
from torchvision.transforms import v2
from decord import VideoReader, cpu

## Evaluation Transform
transform = v2.Compose(
    [
        ApplyTransformToKey(
            key="video",
            transform=v2.Compose(
                [
                    v2.Lambda(lambda x: x.permute(0, 3, 1, 2)), # T, H, W, C -> T, C, H, W
                    v2.UniformTemporalSubsample(16),
                    v2.Resize(resize_to),
                    v2.Lambda(lambda x: x / 255.0),
                    v2.Normalize(img_mean, img_std)
                ]
            ),
        ),
    ]
)

## Preprocessor and Model loading
image_processor = VideoMAEImageProcessor.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-movement")
model = VideoMAEForVideoClassification.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-movement")

img_mean = image_processor.image_mean
img_std = image_processor.image_std
height = width = image_processor.size["shortest_edge"]
resize_to = (height, width)

## load video/clip and predict
video_path = "random_clip.mp4"
vr = VideoReader(video_path, width=480, height=270, ctx=cpu(0))
frames_tensor = torch.stack([torch.tensor(vr[i].asnumpy()) for i in range(len(vr))])  ## Shape: (T, H, W, C)

frames_tensor = transform({"video": frames_tensor})["video"]

output = model(pixel_values=frames_tensor)
pred = torch.argmax(outputs.logits, axis=1).cpu().numpy()

print(model.config.id2label[pred[0]])