Checklist OAI 2024 Stage I · Task 7
Riddles
Polish title: Zagadki
Answer Polish one-word riddles by returning a ranked list of up to 20 candidate nouns, using Wiktionary definitions and Word2Vec embeddings.
The task
Each riddle is a short Polish description whose answer is always a single word (for example, "a woman travelling by a means of transport, e.g. a plane, train or ship" → "pasażerka"). All riddles were written by ChatGPT; the organisers estimate that humans solve slightly more than 60% of them.
The contestant writes answer_riddle, which returns for each riddle a list of at most 20 words ordered from the most to the least likely answer. The correct answer is always among the headwords of the supplied Wiktionary definitions file.
The available resources are about 2,000 example riddles, definitions of the 8,094 most frequent nouns from pl.wiktionary.org (frequencies from the PolEval 2018 Task 3 corpus), base forms of Polish words prepared from the polimorfologik project, and Word2Vec (Gensim) embeddings of base forms trained on the PolEval 2018 Task 3 corpus.
Abridged and translated by SOTA from the official Polish materials. The official statement has the exact rules, and it wins wherever this summary differs.
In English
This task was published in Polish. SOTA translated it into English on 16 September 2026. Only the words changed in the notebooks: markdown, code comments, messages and printed output. The code, file names and paths are the original's, so a translated notebook runs with the original data.
Read the task notebook in English
Riddles
Introduction
Riddles have fascinated people for centuries, stimulating their minds to think creatively and logically. From simple puzzles to deep philosophical riddles, they are not only a form of entertainment but also an art of understanding language and of logical reasoning. In this task you will solve riddles that consist in guessing a word from its description. All the riddles were made up by ChatGPT (but we will not say exactly which version, or how it was prompted), so some of them may be a little strange... We estimate that people can correctly solve a little over 60% of them. And how good will your program be?
Task
Write a function answer_riddle that solves the riddle given as input. The answer is always a single word. Example riddles:
- riddle: kobieta podróżująca środkiem transportu, np. samolotem, pociągiem, statkiem ("a woman travelling by a means of transport, e.g. by plane, train or ship")
answer: pasażerka ("female passenger") - riddle: emocjonalne uczucie łączące dwie osoby, oparte na zaufaniu, szacunku, trosce i oddaniu ("an emotional feeling that binds two people, based on trust, respect, care and devotion")
answer: miłość ("love")
Our criterion will be the inverse harmonic mean (Mean Reciprocal Rank), which works as follows:
If the correct answer appears on the list returned by your function, you will receive points: exactly points, where is the position of the word on the list. In particular, if your program guesses the word (i.e. puts it in the first position), you will receive 1 point. The final criterion is the average number of points over all the riddles.
The criterion above is implemented by us below.
Constraints
- Your final solution will be tested in an environment without a GPU.
- Your function should run fast enough for the program to answer 100 riddles in at most 2 minutes without using a GPU.
Data
The data available to you in this task are:
-
zagadki_do_testow_clean.txt- about 2000 example riddles -
plwiktionary_definitions_clean.txt- a file with word definitions taken from pl.wiktionary.org. From all the definitions on pl.wiktionary.org we took the definitions of the 8094 most popular nouns (frequencies computed from the corpus https://2018.poleval.pl/index.php/tasks#task3). Note: the correct answer to every riddle is in this file! -
superbazy_clean.txt- base forms of Polish words, prepared on the basis of the project https://github.com/morfologik/polimorfologik -
Embedding vectors of base words, trained with the Word2Vec model from the Gensim library on the PolEval 2018 Task3 corpus
Notes and hints
- For each riddle, your function should return a list of words (at most 20), ordered from the most likely answer to the riddle (according to your program) to the least likely.
- Your solution will be tested without internet access
Submission Files
Only this notebook.
Evaluation
Remember that during checking the flag FINAL_EVALUATION_MODE will be set to True. Using the script validation_script.py, you can make sure that your solution will be executed correctly on our grading servers.
For this task you can score between 0 and 1.5 points. You will score 0 points if the value of the mean reciprocal rank criterion on the test set is below 0.02, and 1.5 points if it is above 0.3. Between these values, the score increases linearly with the value of the criterion.
Starter code
######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################
FINAL_EVALUATION_MODE = False
# While checking your solution, we will change this value to True
# The value of this flag M U S T be set to False in the solution you send us!######################### DO NOT CHANGE THIS CELL ##########################
import nltk
from nltk.tokenize import word_tokenize as tokenize
from collections import defaultdict as dd
import math
from gensim.models import Word2Vec
import gdown
import random
import os
from tqdm import tqdmLoading the data
######################### DO NOT CHANGE THIS CELL ##########################
path_to_data = 'data/'
bases = {}
# Dictionary mapping words to their base words
all_word_definitions = dd(list)
# Dictionary containing all base words inverse document frequency
base_idf = dd(int)######################### DO NOT CHANGE THIS CELL ##########################
def get_word_base(word):
global bases
word = word.lower()
ret = bases.get(word)
if ret:
return ret
return word######################### DO NOT CHANGE THIS CELL ##########################
if not FINAL_EVALUATION_MODE:
if not os.path.exists(f"{path_to_data}/zagadki/w2v_polish_lemmas.model") \
or not os.path.exists(f"{path_to_data}/zagadki/w2v_polish_lemmas.model.syn1neg.npy") \
or not os.path.exists(f"{path_to_data}/zagadki/w2v_polish_lemmas.model.wv.vectors.npy"):
gdown.download_folder(url="https://drive.google.com/drive/folders/1P72og_ORfL3Ojf27n-g06DT0ENduPy8C?usp=sharing", output=f"./{path_to_data}")
nltk.download('punkt')######################### DO NOT CHANGE THIS CELL ##########################
for x in open(f'{path_to_data}/zagadki/superbazy_clean.txt'):
word,base = x.lower().split()
bases[word] = base######################### DO NOT CHANGE THIS CELL ##########################
model = Word2Vec.load(f'{path_to_data}/zagadki/w2v_polish_lemmas.model')######################### DO NOT CHANGE THIS CELL ##########################
for x in open(f'{path_to_data}/zagadki/plwiktionary_definitions_clean.txt'):
word, definition = x.split('###')
L = word.split()
if len(L) == 1:
word = L[0]
definition = set(tokenize(definition.lower()))
all_word_definitions[word].append(definition)
for word in set(definition):
base_idf[get_word_base(word)] += 1
for base in base_idf:
base_idf[base] = -math.log(base_idf[base] / len(all_word_definitions))######################### DO NOT CHANGE THIS CELL ##########################
answers = []
queries = []
with open(f'{path_to_data}/zagadki/zagadki_do_testow_clean.txt') as file: # "zagadki_do_testow" = riddles for testing
for line in file:
line = line.replace(';;', '').split()
answers.append(line[0])
queries.append(tokenize(' '.join(line[1:])))Code with the scoring criteria
######################### DO NOT CHANGE THIS CELL ##########################
def mean_reciprocal_rank(real_answers, computed_answers, K=20):
positions = []
for real_answer, computed_answer in zip(real_answers, computed_answers):
if real_answer in computed_answer[:K]:
pos = computed_answer.index(real_answer) + 1
positions.append(1/pos)
mrr = sum(positions) / len(real_answers)
print ('Mean Reciprocal Rank =', mrr)
return mrrYour solution
This is the only section in which you need to do something.
def answer_riddle(riddle, K):
return random.sample(all_word_definitions.keys(), K)Evaluation
The code below will be used to evaluate the solution. After you send us your solution, the function evaluate_algorithm(score_function, queries, answers, K) will be executed, i.e. code almost identical to the code below will be run on the data directory test_data, which is available only to the task graders.
Before submitting, make sure that the whole notebook runs from start to finish without errors and without user intervention after executing the Run All command.
######################### DO NOT CHANGE THIS CELL ##########################
def evaluate_algorithm(score_function, queries, answers, K):
computed_answers = []
for query in tqdm(queries, desc="queries answered"):
computed_answers.append(score_function(set(query), K=K))
score = mean_reciprocal_rank(answers, computed_answers, K=K)
return score######################### DO NOT CHANGE THIS CELL ##########################
if not FINAL_EVALUATION_MODE:
PART_OF_DATA = 100
K = 20
valid_queries = queries[:PART_OF_DATA]
valid_answers = answers[:PART_OF_DATA]
score = evaluate_algorithm(answer_riddle, valid_queries, valid_answers, K=K)
print(f"Score: {score}")Translated by SOTA. The Polish original is the official version and wins wherever the two differ. The riddles, answers and data files are in Polish, so the example riddles stay in Polish, with English glosses in brackets. If you organise this olympiad and would like the translation removed, email [email protected] and we will take it down.
At a glance
- You get
zagadki_do_testow_clean.txt,plwiktionary_definitions_clean.txt,superbazy_clean.txtandw2v_polish_lemmas.model files, downloaded from a Google Drive folder with gdown.- You submit
- This notebook only, with
answer_riddle(riddle, K) returning a ranked list of at most K = 20 words. - Scoring
- Mean reciprocal rank over riddles: 1/k points if the correct answer is at position k of the list, 0 if absent. 0 points if MRR on the test set is below 0.02, 1.5 points if above 0.3, linear in between.
- Rules
- Tested without a GPU and without Internet access.
- The program must answer 100 riddles within 2 minutes without a GPU.
- Format
- Stage I (online, solved at home), 22 April – 27 May 2024; notebook submitted through the Olympiad's submission website and scored automatically. Worth up to 1.5 points of the stage total of 10.