Sentiment Update

Sentiment Update

English translation by SOTA – AI Community of the Hungarian original. Licensed CC BY-NC-SA 4.0, like the original. Organisers who would like this translation removed can email [email protected].

Magyar MI Diákolimpia (Hungarian AI Olympiad)
Task description · Summer Camp Selection (Nyári Tábor Válogató)
June 2026

[Figure: see the original one-pager.]

1. Sentiment Update

You are in charge of a sentiment recognition system that processes tens of thousands of messages day after day: customer service feedback, comments on educational platforms, community conversations. Until now the system could distinguish three moods – positive, negative and neutral – and for a long time this was enough. User needs, however, have outgrown it: from now on, the services want to know whether a message is merely dissatisfied or already outright angry, politely appreciative or sincerely enthusiastic.

Your task is to extend the system to five sentiment levels, from very negative to very positive, and to assign the correct level to every text in the test set:

• 0: very negative
• 1: negative
• 2: neutral / mixed
• 3: positive
• 4: very positive

The extreme cases are easy: bursting enthusiasm and undisguised rejection practically shout out of the text. The real difficulty lies in the in-between tones, where the same word is sincere praise in one sentence and sarcasm in another, and lukewarm disappointment easily blurs into neutrality. For these, it is not enough to pick out a keyword or two: you have to understand the tone of the whole message, and your score depends precisely on how well you cope with these subtle transitions.

2. What you receive

Two CSV files. train.csv is the labelled training set (id, text, label; label is 0–4, 750 rows, balanced across the classes), and test_public.csv is the input of the test set (id, text, without labels, 275 rows). You must give a prediction for every row of the latter. Any additional training data (even your own) may be used.

3. What you submit

A single CSV file with the header id,target, containing all 275 rows of the test set. id is the identifier from test_public.csv, and target is the predicted sentiment class (0–4). The set of submitted id values must cover the test set exactly. You do not need to submit a model; the evaluator computes the score from the CSV.

4. Scoring

Evaluation is based on the macro-averaged F1 (f1_macro): the arithmetic mean of the F1 scores of the five classes, so all five classes count equally, and correctly recognising the rarer sentiments is just as important as recognising the more common ones.

$$\mathrm{F1}_c = \frac{2\, P_c\, R_c}{P_c + R_c}, \qquad \text{macro-F1} = \frac{1}{5} \sum_{c=0}^{4} \mathrm{F1}_c,$$

where $P_c$ and $R_c$ are the precision and recall of class $c$. The task is worth 40 points. The raw macro-F1 is mapped to points by a calibration curve: below 0.85, 0 points; above 0.965, the full 40 points; linear in between:

$$\text{Points} = 40 \cdot \operatorname{clip}\!\left( \frac{\text{macro-F1} - 0.85}{0.965 - 0.85},\ 0,\ 1 \right).$$

The curve is needed because on this task the raw macro-F1 is compressed (most sensible solutions fall between 0.84 and 0.96), so a linear scale would hardly spread out the field.

Maximum score: 40 points.

Public and private test set. The test set is divided into a public part and a closed (private) part. During the contest you only see the score measured on the public part on the leaderboard; the final ranking is decided by the closed part.

5. Constraints and technical information

1. Scoring is carried out automatically, on the server side, by the DOCK platform on the basis of the submitted submission.csv; you do not need to submit a model.
2. You have 23 upload attempts in total; the score of your best submission counts towards the final result.
3. test_public.csv may be used only for prediction; reverse-engineering its labels or labelling it by hand is not allowed.
4. You must use the provided Qwen/Qwen3-Embedding-4B model; using any other pretrained model is not allowed.
5. The data may be used only for the contest and for educational and research purposes; uploading the dataset or any part of it to an external drive, or using it outside the task, is not allowed.

6. Useful resources

• Hugging Face Transformers documentation
• PyTorch documentation
• pandas documentation
• macro-F1 (scikit-learn)