Checklist IOAI 2024 Scientific Round (On-Site) · NLP task
Help BOBAI
Extend a frozen 5-class classifier on mBERT encodings of an unknown language to 7 classes without adding any learned parameters.
The task
This is the sequel to the at-home Help BOBAI task. Bob's client Amoira has been happily using the classifier built for its unknown language, and now wants it to handle 7 classes instead of 5, by the end of the same day. Deploying a new model takes far longer than that, so the solution has to be built around the classifier already in production.
Amoira's security worries mean no text is released at all. Instead you get cached mBERT pooling-layer encodings for the earlier 5-class train and dev data and for new labelled examples of the two extra classes, along with the weights of the deployed 5-way linear classifier.
You must deliver a 7-way classifier. It may call the existing classifier, but you may not change its parameters or add new learned ones. Computing averages of, and distances between, encodings is explicitly allowed. The notebook has to rebuild the model from the baseline in under an hour on an L4 GPU, and the test encodings were released two hours before the contest ended.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- PyTorch tensors of 768-dimensional mBERT pooled encodings with labels for the old train/dev data and the two new classes (
train-dev_dataset_with_labels.pt), the deployed 5-way linear classifier's weights (base_classifier.pth), a validation set used for the leaderboard (eval_dataset.pt) and test encodings (test_dataset.pt) released two hours before the end. Sizes are not stated. - You submit
- The predicted class (0 to 6) for each test encoding, written one per line to a text file (the notebook names it '<Team Name>
_predictions.txt'), plus the Colab notebook that reproduces the model. Leaderboard submissions on the validation set use a CSV with columns ID and class. - Scoring
- Macro-averaged F1, as computed by the notebook's evaluation code (sklearn
f1_scorewith average='macro'); the task text itself does not name the scoring metric. - Rules
- The existing 5-way classifier may be used, but its parameters may not be changed and no new learned parameters may be added
- Computing averages and distances between data encodings is allowed
- The solution must be reproducible in under 1 hour on an L4 GPU
- Inference on any random 500 samples must take under 2 minutes on an L4 GPU
- The notebook must run top to bottom and contain all the code needed to reproduce the model, starting from the baseline classifier
- Format
- Scientific round, on-site stage (teams had 8 hours for the three on-site tasks)