Checklist OAI 2024 Stage I · Task 1
Adversarial Attacks
Polish title: Ataki adwersarialne
Perturb 28×28 greyscale images so that a given pre-trained convolutional classifier loses as much accuracy as possible while the images stay structurally similar to the originals.
The task
An adversarial attack is a deliberate modification of the input that misleads a neural network while leaving the input visually almost unchanged. The contestant is given a small convolutional network (two convolutional layers, max pooling and two fully connected layers, 10 output classes) with pre-trained weights, and must design an attack that lowers the probability of the correct class with the smallest possible changes to the test images.
Each image is normalised individually to the range [-1, 1]. The contestant implements perturbe_dataset(), which receives a NumPy array of shape (number of samples, 28, 28) and returns an array of the same shape; any information obtainable from the trained model may be used. The maximum absolute difference between any original pixel and the corresponding perturbed pixel must not exceed 0.3.
The quality criterion is the mean SSIM between original and perturbed images multiplied by the drop in classification accuracy (base accuracy minus accuracy on the perturbed set). Training and validation images are supplied for development; scoring uses a hidden test set normalised in the same way.
Abridged and translated by SOTA from the official Polish materials. The official statement has the exact rules, and it wins wherever this summary differs.
In English
This task was published in Polish. SOTA translated it into English on 16 September 2026. Only the words changed in the notebooks: markdown, code comments, messages and printed output. The code, file names and paths are the original's, so a translated notebook runs with the original data.
Read the task notebook in English
Adversarial attacks
Introduction
As you have probably heard, neural networks are an excellent tool for classifying objects, recognising complex relationships and predicting future values from historical data. It turns out, however, that they are also vulnerable to attacks that can significantly weaken their performance. These attacks consist in finding a gap in the model's reasoning. Suppose that a network has learnt to recognise a bicycle in a photo. Let us try to intelligently modify selected pixels in the area showing the bicycle so that, "to the naked eye", the image still shows a bicycle. Let us ask the neural network whether it still "sees" a bicycle in the image modified in this way. If the model no longer classifies the object as a bicycle, we can conclude that the adversarial attack has succeeded. An adversarial attack is the crafting of input data so as to mislead the network as effectively as possible when it makes its decision.
Task
Propose a method of breaking into a neural network (an adversarial attack) that lowers the probability of assigning a sample to its correct class, while making the smallest possible modifications to the tested images.
Our criterion will be the product of the structural similarity index (SSIM), averaged over all images taking part in the attack, and the difference between the model's initial accuracy on a given set and its accuracy on the same set after modification, according to the formula:
where:
- is the model's classification accuracy on the unmodified test set;
- is the model's classification accuracy on the modified set.
The criterion above and all the functions it requires are implemented by us below.
Constraints
- Your final solution will be tested in an environment with a GPU.
- Only solutions in which the maximum distance between the pixels of the original image and the corresponding pixels after the adversarial attack (in the sense of absolute value) is not greater than will be considered. Images after the attacks are compared with the original images normalised to the interval .
EXAMPLE: The vectors and are valid, because the vector of absolute differences between them, , contains no elements greater than . However, the vectors and are not valid, because the second elements of the two vectors differ by . - The function
perturbe_dataset()must take a three-dimensional Numpy array of size (number of samples x 28 x 28) and return a Numpy array of size (number of samples x 28 x 28), which will be the set after modification. When creating the modified version of the dataset, you may use any information coming from the trained model. - Your attack should run in at most 5 minutes on Google Colab with a GPU.
Notes and hints
- The model was trained on a set in which each image was individually normalised to the interval , in both the training and the validation set. The test set, which is not included with the task instructions, will be normalised in the same way.
- Each image of the test set will ultimately be normalised to the range , but the images after modification, i.e. at the output of the function
perturbe_dataset(), will only be converted to tensor form and will not be normalised. - You may test your solutions on the training and validation sets, but the score for the task will be awarded solely on the basis of the result on the test set.
Submission files
Only this notebook.
Evaluation
Remember that during checking the flag FINAL_EVALUATION_MODE will be set to True. Using the script validation_script.py, you can make sure that your solution will be executed correctly on our grading servers.
For this task you can score between 0 and 1 point. You will score 0 points if the value of the criterion on the test set is below 36.0, and 1 point if it is above 42.0. Between these values, the score increases linearly with the value of the criterion.
Starter code
######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################
FINAL_EVALUATION_MODE = False
# While checking your solution, we will change this value to True
# The value of this flag M U S T be set to False in the solution you send us!######################### DO NOT CHANGE THIS CELL ##########################
import os
from copy import deepcopy
import gdown
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image
from skimage.metrics import structural_similarity as ssim
from torch.utils.data import DataLoader, Dataset
from torchvision import transformsLoading the data
######################### DO NOT CHANGE THIS CELL ##########################
if not FINAL_EVALUATION_MODE:
if not os.path.exists(f"data"):
gdown.download_folder(url="https://drive.google.com/drive/folders/1qmRd5O-LdOki1-HC4dgfaNrstzG3h_cN?usp=sharing", output=f"./data")######################### DO NOT CHANGE THIS CELL ##########################
# Dataset class
class ContestDataset(Dataset):
def __init__(self, data, labels, transform=None):
self.data = data
self.labels = labels
self.transform = transform
def __len__(self):
return len(self.data)
def __getitem__(self, index):
x = Image.fromarray(self.data[index])
if self.transform:
x = self.transform(x)
y = self.labels[index]
return x, y######################### DO NOT CHANGE THIS CELL ##########################
# Function that normalises pixel values to the interval <-1, 1>
def normalize_samples(samples):
assert len(samples.shape) == 3
samples = samples.reshape(-1, 28 * 28)
minimum_values = np.min(samples, axis=1)
maximum_values = np.max(samples, axis=1)
normalized_samples = (2 * (samples - minimum_values[:, np.newaxis]) / \
(maximum_values - minimum_values)[:, np.newaxis]) - 1
normalized_samples = normalized_samples.reshape(
normalized_samples.shape[0], 28, 28
)
return normalized_samples######################### DO NOT CHANGE THIS CELL ##########################
# Let us load the required data
# You will use X_train and y_train to prepare the adversarial attack
# On X_validation and y_validation you will check whether your solution passes the evaluation script
path_to_data = './data/'
X_train = np.load(f'{path_to_data}contest_train_samples.npy') / 255.
y_train = np.load(f'{path_to_data}contest_train_labels.npy')
X_validation = np.load(f'{path_to_data}contest_validation_samples.npy') / 255.
y_validation = np.load(f'{path_to_data}contest_validation_labels.npy')
X_train = normalize_samples(X_train)
X_validation = normalize_samples(X_validation)######################### DO NOT CHANGE THIS CELL ##########################
validation_set = ContestDataset(X_validation,
y_validation,
transform=transforms.ToTensor())
validation_loader = DataLoader(validation_set,
batch_size=1,
shuffle=False)######################### DO NOT CHANGE THIS CELL ##########################
# Definition of the classifier we want to fool!
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=0)
self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=0)
self.pool = nn.MaxPool2d(kernel_size=3, stride=2)
self.fc1 = nn.Linear(64 * 11 * 11, 128)
self.fc2 = nn.Linear(128, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
x = F.relu(self.pool(self.conv2(x)))
x = torch.flatten(x, 1)
x = self.fc1(x)
x = self.fc2(x)
return F.log_softmax(x, dim=1)######################### DO NOT CHANGE THIS CELL ##########################
if torch.cuda.is_available():
device = 'cuda'
else:
device = 'cpu'
# Let us create our classifier
net = Net()
# Let us load the pretrained weights into our model
net.load_state_dict(torch.load('trained_model.pth',
map_location=device))Code with the scoring criteria
######################### DO NOT CHANGE THIS CELL ##########################
def evaluate_network(data_loader,
model,
device,
verbose=True):
# Let us evaluate the trained model on the chosen set
all_predictions, all_labels = [], []
model.to(device)
# The neural network must be switched to evaluation mode
model.eval()
with torch.no_grad():
for data in data_loader:
images, labels = data
images, labels = images.to(device), labels.to(device)
outputs = model(images)
# We choose the class with the highest membership probability
_, predicted = torch.max(outputs.data, 1)
all_predictions.append(predicted.ravel())
all_labels.append(labels.ravel())
all_predictions = torch.cat(all_predictions, dim=0)
all_labels = torch.cat(all_labels, dim=0)
# We check how many labels the network predicted correctly
correct = (all_predictions == all_labels).sum().item()
accuracy = (100 * correct / all_labels.size()[0])
no_of_elements = len(data_loader.dataset)
if verbose:
print(f'The classification accuracy on the {no_of_elements} images of the chosen set is '
f'{accuracy} %.')
return accuracy######################### DO NOT CHANGE THIS CELL ##########################
# Function that computes SSIM and the maximum distance between corresponding pixels
def calculate_similarity(original_dataset,
perturbed_dataset):
# Both original_dataset and perturbed_dataset must be Numpy arrays.
# They must have the same size, i.e. (number of elements x 28 x 28)
assert original_dataset.shape == perturbed_dataset.shape
assert original_dataset.shape[1] == original_dataset.shape[2] == 28
assert perturbed_dataset.shape[1] == perturbed_dataset.shape[2] == 28
similarities, L1_distances = [], []
for i in range(original_dataset.shape[0]):
original_element = original_dataset[i].ravel()
perturbed_element = perturbed_dataset[i].ravel()
similarities.append(
ssim(
original_element,
perturbed_element,
data_range=2)
)
L1_distances.append(
np.max(np.abs(
original_element - perturbed_element
))
)
mean_SSIM = np.mean(similarities)
max_distance = np.max(L1_distances)
print(f'The mean SSIM value is {mean_SSIM}, and the largest distance between pixels is: {max_distance}.')
return mean_SSIM, max_distance######################### DO NOT CHANGE THIS CELL ##########################
if not FINAL_EVALUATION_MODE:
result = evaluate_network(
validation_loader,
net,
device,
verbose=True
)Your solution
This is the only section in which you need to do something.
def perturbe_dataset(original_dataset):
"""
This is where you are to prepare the function that perturbs the dataset. The input must be
a Numpy array of size (number of elements, 28, 28). The same must hold for the
output. And what happens in between? That is up to you and your ingenuity!
"""
assert len(original_dataset.shape) == 3
# TODO:
perturbed_dataset = deepcopy(original_dataset)
assert len(perturbed_dataset.shape) == 3
assert perturbed_dataset.shape[1] == 28
assert perturbed_dataset.shape[2] == 28
return perturbed_datasetEvaluation
The code below will be used to evaluate the solution. After you send us your solution, the function evaluate_algorithm(net, X_validation, y_validation, device, perturbe_dataset) will be executed, i.e. code almost identical to the code below will be run on the image directory test_data, which is available only to the task graders.
Before submitting, make sure that the whole notebook runs from start to finish without errors and without user intervention after executing the Run All command.
def evaluate_algorithm(model, images, labels, device, perturbe_algorithm):
dataset_for_perturbation = deepcopy(images)
dataset_attacked = perturbe_algorithm(dataset_for_perturbation)
SSIM, distance = calculate_similarity(
dataset_for_perturbation,
dataset_attacked
)
assert distance <= 0.3
perturbed_set = ContestDataset(dataset_attacked,
labels,
transform=transforms.ToTensor())
perturbed_loader = DataLoader(perturbed_set,
batch_size=64,
shuffle=False)
perturbed_accuracy = evaluate_network(
perturbed_loader,
model,
device,
verbose=True
)if not FINAL_EVALUATION_MODE:
evaluate_algorithm(net, X_validation, y_validation, device, perturbe_dataset)Translated by SOTA. The Polish original is the official version and wins wherever the two differ. The notebook loads trained_model.pth from its own folder; that file and validation_script.py are in the original task folder on GitHub. If you organise this olympiad and would like the translation removed, email [email protected] and we will take it down.
At a glance
- You get
contest_train_samples.npy/contest_train_labels.npyandcontest_validation_samples.npy/contest_validation_labels.npy(28×28 greyscale images and labels, downloaded from a Google Drive folder with gdown), and the model weightstrained_model.pthin the task folder.- You submit
- This notebook only, with
perturbe_dataset() implemented; it returns the perturbed array of shape (N, 28, 28). - Scoring
- Criterion = SSIM (mean over attacked images,
data_range= 2) × (base_acc−final_acc), with accuracies in percent. 0 points if the criterion on the test set is below 36.0, 1 point if above 42.0, linear in between. Solutions whose maximum per-pixel change exceeds 0.3 are rejected. - Rules
- The attack must run in at most 5 minutes on Google Colab with a GPU.
- Maximum absolute per-pixel change of 0.3 relative to the image normalised to [-1, 1].
- The output of
perturbe_dataset() is only converted to a tensor; it is not re-normalised. - Python 3.11 with the packages pinned in the repository-level
requirements.txt; avalidation_script.pyin the task folder checks that the notebook runs with FINAL_EVALUATION_MODE = True.
- Format
- Stage I (online, solved at home), 22 April – 27 May 2024; notebook submitted through the Olympiad's submission website and scored automatically. Worth up to 1 point of the stage total of 10.