Using DINOv2 as feature extractor for animal tracking

Using pretrained vision transformers like DINOv2 as feature extractor, Convpaint is remarkable at detecting animal body parts or behaviour, such as closing/opening eyes.

Here is a sample frame from a movie, showing segmentation of mouse head, tail and body:

Tracking animal body parts in movies

First step is to change the feature extractor from default VGG16 to DINOv2. Tick the Use custom model box, and then switch to the Model tab.

Next select the DINOv2 model from the model list dinov2_vits14_reg1.

Click on load model and you are good to go! Annotate and train your Convpaint model as usual.

Tip: Select `use cuda` to benefit from GPU acceleration! This also works on MPS with Mac devices.

Example use case: Tracking shark body parts in a movie

These are the scribble annotations provided for training: