Checklist IOAI 2025 Individual Contest · Task 3
Concepts
Build a clue-giver that picks up to four sequences of icon markers so that a black-box AI guesser can identify a secret word.
The task
Concepts is a guessing game in which a clue-giver hints at a secret word only by pointing at ordered sequences of icons, each with a text description. The at-home Chameleon task asked for a guesser; this on-site task swaps the roles, and a hidden AI guesser does the guessing.
For each secret label and its list of 100 candidate options, your program returns up to four sequences of marker IDs, with at most eight IDs each, drawn from a fixed set of 118 markers. Some labels are unusual for the game, such as 'International Olympiad'.
The training data has only 30 examples. During the contest a rate-limited guesser API and an LLM proxy were available for development but not on the machines that ran submissions, and a list of small embedding models and LLMs was provided for offline use.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- 30 training examples (a label plus 100 options) and 118 marker descriptions with icon images, as Hugging Face datasets; hidden
test_aandtest_bsets of 150 examples each. - You submit
- For each example, a list of at most 4 lists of at most 8 integer marker IDs. The submitted notebook must produce
submission.zipcontainingclues_a.jsonlandclues_b.jsonl. - Scoring
- Final score = 0.9 × Hits@10 + 0.1 × NDCG@10, where Hits@10 is 1 if the secret word is in the guesser's top 10 guesses and NDCG@10 = 1/log2(i + 1) for the secret at rank i. The Scientific Committee's highest score (0.54) and baseline score (0.20) were used for score normalisation.
- Rules
- Notebook must finish within 10 minutes on the test machine
- At most 15 successful submissions
- At most one attached self-made dataset (for example model weights), no larger than 2 GB
- Test machine has no internet access; the guesser API and LLM proxy cannot be called from a submission
- Guesser API limited to 12,500 requests per token and 1,000 requests per minute
- LLM proxy limited to a $10 credit; no private APIs, external logins or personal tokens
- Format
- Individual Contest, Day 1 (5 August 2025); six hours for the day's three tasks