Discord

Checklist OAI 2024 Stage II – Final · Task 2

Anomaly Detection

Polish title: Detekcja anomalii

Train a neural anomaly detector only on normal images and classify test images as normal (0) or anomalous (1).

  • Vision
  • Unsupervised / self-supervised image anomaly detection
  • Polish original · English translation

The task

Anomaly detection means finding the rare samples that come from outside the distribution of the typical data, as in quality control on a production line or network intrusion detection. Using the supplied images, the contestant proposes a self-supervised or unsupervised algorithm that uses a neural network; anomalies carry label 1 and normal observations label 0.

The model may be trained only on the training set, which contains normal observations exclusively. The solution must set two global variables: BATCH_SIZE, used for the test data loader, and model, a trained instance of the Model class with forward() and predict(); predict() must assign each image in a batch to class 0 or 1 without changing its signature.

The test set, available only to the organisers, is balanced and has the same characteristics as the supplied images.

Abridged and translated by SOTA from the official Polish materials. The official statement has the exact rules, and it wins wherever this summary differs.

In English

This task was published in Polish. SOTA translated it into English on 16 September 2026. Only the words changed in the notebooks: markdown, code comments, messages and printed output. The code, file names and paths are the original's, so a translated notebook runs with the original data.

Read the task notebook in English 621 words and 10 code cells

Anomaly detection

image.png

Anomaly detection is a problem that is frequently tackled in business applications. One example is quality analysis of the products manufactured on a production line, where most products are manufactured correctly (they meet the standards) and a small number of samples are defective. Another example of an anomaly detection problem is systems that analyse network traffic, where the goal is to detect outlying events, e.g. DDoS attacks.

The anomaly detection problem can therefore be formalised as finding outlying samples/events (outliers), which usually occur rarely in the dataset. One can say that the dataset contains many "typical" samples coming from some data distribution, and the goal is to find the few samples that come from outside this distribution.

Task

On the basis of the provided dataset, propose a self-supervised or unsupervised algorithm for anomaly detection. Anomalies are marked with the label 1. Samples containing normal observations are marked with the label 0. Use a neural network in your solution. In the end, your solution should assign values to two global variables:

  • BATCH_SIZE - the size of the data batches that the dataloaders will operate on. During grading we will read this value and use it for the dataloader with the test data.
  • model - the prepared model, which will consist of a forward() function, traditionally used to propagate data through the network, and a predict() function, which is to make the final assignment of a given sample to one of the classes: 0 or 1. The model variable must be a trained instance of the Model class. model will be used to evaluate the solution on the test set.

NOTE: The model may be trained only on the training set, which contains only normal observations.

NOTE: The variable names and the signature of the predict function must remain unchanged - you can use the checker to verify these requirements.

Scoring

After the model you prepared has been trained, the predict() method will be called to score the solution. It is to contain the mechanism that assigns the input data sample on the basis of the model's output and any additional criteria you have prepared for judging the model's predictions. The quality of the proposed solution will be determined by computing the classification accuracy on the test set, which is available to the organisers. The test set is balanced, and the images in it have the same characteristics as those provided to participants.

A model achieving a classification accuracy below 60% will receive 0 points, one above 90% will receive 1 point, and intermediate values will be scaled linearly within this range. The participant's final score will be computed using the formula below:

score=min(max(accuracy0.6,0.0),0.3)/0.3score = min(max(accuracy - 0.6, 0.0), 0.3) / 0.3

where accuracy[0,1]\mathrm{accuracy} \in [0, 1] is the classification accuracy on the test set.

Submission files

  1. Only this notebook

Constraints

  • The evaluation of your solution (including the training of the neural network, with the FINAL_EVALUATION_MODE flag set to True) on 10 000 test examples should take no longer than 15 minutes on Google Colab with a GPU.

Evaluation

Remember that during grading the FINAL_EVALUATION_MODE flag will be set to True.

For this task you can score between 0 and 1 point. The number of points you score will be equal to the value of score, computed on the test set.

Starter code

######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

FINAL_EVALUATION_MODE = False
# While grading your solution, we will change this value to True
# The value of this flag M U S T be set to False in the solution you send us!
######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

import random
import warnings
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, Dataset

warnings.filterwarnings("ignore", category=UserWarning)

np.random.seed(0)
random.seed(0)
torch.manual_seed(0)
if not FINAL_EVALUATION_MODE:
    ! gdown https://drive.google.com/uc?id=1QoTW4eZctWvzmjrNrwmai9zhL8RWnhBU
    # ! gdown https://drive.google.com/uc?id=1bD38bZf8pcUyinuvYDXO56RPlXUluzxn
    # ! gdown https://drive.google.com/uc?id=108d2ERztZzq5_ZXyizxeWDOCctG5xhIy

    ! unzip anomaly.zip
    ! unzip train.zip
    ! unzip valid.zip

Loading the data

######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

TRAIN_DIR: Path = Path("./train")
VALID_DIR: Path = Path("./valid")

TRAIN_CSV: Path = Path("./train.csv")
VALID_CSV: Path = Path("./valid.csv")
# Set your batch size -- we will require this variable to be present during grading and will use it during testing
BATCH_SIZE: int = 32
######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

class ImageDataset(Dataset):
    def __init__(self, dir: Path, csv: Path):
        self.dir: Path = dir
        self.csv = pd.read_csv(csv)

    def __len__(self):
        return len(self.csv)

    def __getitem__(self, idx) -> tuple[torch.Tensor, int]:
        if torch.is_tensor(idx):
            idx = idx.tolist()
        path, label = self.csv.iloc[idx]
        img = plt.imread(self.dir / path)

        to_tensor = transforms.ToTensor()
        return to_tensor(img), label


def train_dataloader() -> DataLoader:
    """Create a Dataloader with the training data."""
    return DataLoader(ImageDataset(TRAIN_DIR, TRAIN_CSV), batch_size=BATCH_SIZE, shuffle=True)


def valid_dataloader() -> DataLoader:
    """Create a Dataloader with the validation data."""
    return DataLoader(ImageDataset(VALID_DIR, VALID_CSV), batch_size=BATCH_SIZE, shuffle=False)

Example solution skeleton + naive solution

Below we present a simple solution which is obviously not optimal. Its purpose is to show how the whole notebook is meant to work.

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return torch.rand(x.shape[0])

    def predict(self, batch: torch.Tensor) -> torch.Tensor:
        """
        Assign 0 or 1 to each image in the batch.
        Do not change or add arguments in this method!
        """
        predictions = torch.ones(batch.shape[0])
        return predictions


def train_model() -> Model:
    """Create and train the model"""
    return Model()
######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

model = train_model()

Scoring criterion code

######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

def grade_solution(model):
    dataloader = valid_dataloader() # During grading we will replace this with test_dataloader

    predictions = np.concatenate([
        model.predict(images).cpu().to(dtype=torch.int32).numpy() for images, _ in dataloader
    ], axis=0)

    labels = np.concatenate([label for _, label in dataloader], axis=0, dtype=np.int32)

    accuracy = sum(labels == predictions) / len(labels)

    score = min(max(accuracy - 0.6, 0.0), 0.3) / 0.3

    print(f"Accuracy: {accuracy}")
    print(f'Your score is {score} pts')
    return score

Evaluation run on the provided dataset

(ultimately it will be run on the test set)

######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

if not FINAL_EVALUATION_MODE:
    grade_solution(model)

Translated by SOTA. The Polish original is the official version and wins wherever the two differ. If you organise this olympiad and would like the translation removed, email [email protected] and we will take it down.

At a glance

You get
Image folders train and valid with train.csv and valid.csv (path, label), from anomaly.zip on Google Drive.
You submit
This notebook only, defining BATCH_SIZE and the trained model.
Scoring
Accuracy on the balanced test set; score = min(max(accuracy − 0.6, 0), 0.3) / 0.3, i.e. 0 points below 60%, 1 point above 90%, linear in between.
Rules
  • Evaluation including training (FINAL_EVALUATION_MODE = True) on 10,000 test examples must take at most 15 minutes on Google Colab with a GPU.
  • Training on the normal-only training set only; a neural network must be used.
Format
Stage II final contest (five hours) held during the final scientific camp in Krzyżowa, 15–21 June 2024; about 30 top Stage I participants took part.

Details

Year
2024, Krzyżowa, Poland
Round
Stage II – Final · Task 2
Language
Polish; English translation by SOTA
License
Not stated by the source