Constrained Generation

Constrained Generation

English translation by SOTA – AI Community of the Hungarian original. Licensed CC BY-NC-SA 4.0, like the original. Organisers who would like this translation removed can email [email protected].

Magyar MI Diákolimpia (Hungarian AI Olympiad)
Task description · Summer National Selection (Nyári Országos Válogató)
May 2026

[Figure: see the original one-pager.]

1. Situation report

In 1969, Georges Perec wrote his novel La Disparition: more than 300 pages without a single letter "e". Such deliberately constrained pieces of writing (lipograms) still fascinate linguists and writers' workshops today.

An experimental workshop would like to automate this. A small language model writes continuations for given prompts while never uttering a single forbidden word. Your task is to implement this constrained decoding. At every step the model proposes a next token (in the form of logits), and you decide which one goes into the output.

You face 20 experiments, each with a prompt on a different theme and a forbidden-word list of varying difficulty: from simple stop words (the, a, an) to bans on thematic word classes (in a detective text the entire vocabulary of violence is forbidden, in a programming description the trivial technical terms). The output must remain long, diverse and natural-sounding while respecting the constraint.

The organisations and events appearing in the task are fictitious (Perec and his work are not).

2. Constrained Generation

Causal language models produce text autoregressively: at every step they determine a probability distribution over the vocabulary (the logits), and then select the next token on the basis of this distribution. By modifying the logits, or by applying an alternative sampling rule or search procedure, the output of the model can be constrained without retraining.

You are given a small pretrained language model and a series of test cases. Each test case consists of an initial prompt, a list of forbidden words and the maximum number of tokens that may be generated. Your task is to determine, for each generation step, the identifier of the next token so that the final output does not contain a single forbidden word, while its length, diversity and naturalness are also preserved.

A word counts as forbidden in the output if it occurs in it as a separate word (delimited by word boundaries), without distinguishing between upper and lower case. For example, if apple is forbidden, then the occurrences apple, Apple, APPLE, apple, and apple. all violate the rule. pineapple, on the other hand, is allowed, since it is not a standalone occurrence of apple.

3. What you receive

A pretrained Qwen/Qwen3-0.6B-Base causal language model, accessed through the Hugging Face Transformers library, together with its tokenizer. During evaluation, the model is loaded with these same weights and package versions, so the output is deterministic across runs.

A function stub to be implemented, with the signature:

select_next_token(logits, prompt, forbidden, generated_ids) -> int

The meaning of the parameters:

• logits – the model's output logits for the last position (size: [vocab_size]),
• prompt – the initial text,
• forbidden – the list of forbidden words,
• generated_ids – the list of identifiers of the tokens selected so far.

The return value is the identifier of the next token as an integer, between 0 and vocab_size−1.

The attached notebook provides the complete evaluation framework (generate_constrained, score_case, run_tests), a would_violate helper function that indicates, according to the same rules as the evaluation, whether a given partial result contains a forbidden word, and a make_submission() helper function that produces the submission.csv to be submitted from the completed select_next_token.

The test_cases.json file contains the official test set (currently 20 test cases) in the form of a list, each element of which is a test case with the fields id, prompt, forbidden and max_new_tokens. Scoring is carried out exclusively on this test set. The same file must also be used for generation.

4. What you submit

A single CSV file with the header id,output,fluency,tokens, containing exactly as many data rows as there are test cases in the test_cases.json file. The meaning of the columns:

• id – refers to the corresponding entry of test_cases.json.
• output – the generated text (without the prompt).
• fluency – the value $\Phi = \exp(\overline{\log p_\theta}/5)$ from the Qwen3-0.6B-Base model, computed on the full (prompt + output) text. The notebook's score_case function computes and fills in this value automatically.
• tokens – the length of tokenizer.encode(output).

The file follows the standard CSV format (RFC 4180): fields containing a comma, a quotation mark or a line break must be enclosed in quotation marks, and quotation marks must be doubled. Using the attached make_submission() helper function for the submission is recommended. It handles the above rules automatically and writes the output with UTF-8 encoding.

The fluency and tokens fields are computed and filled in by the notebook's make_submission() function. They must not be modified by hand: the evaluator overrides the submitted values.

The output may be produced only with the Qwen/Qwen3-0.6B-Base model specified in the task. Using any other generative model results in immediate disqualification. Any decoding strategy of your own for the model is allowed, provided that it uses only the logits of the given model.

5. Scoring

Let $\mathcal{T}$ denote the test set and, for a given test case, let $p$ be the prompt, $\mathcal{F}$ the set of forbidden words, $M$ the allowed maximum number of tokens, and $y$ the generated output text. The score of a test case is determined as follows:

• if $y$ contains a forbidden word: 0 points,
• if $y$ is empty: 0 points,
• otherwise: $\text{score}(y) = L \cdot \sqrt{\text{distinct}_2} \cdot \Phi$, where $L$ is the length ratio, $\text{distinct}_2$ is the bigram diversity and $\Phi$ is the fluency.

Per-test-case maximum: 1.0 (if $L = \sqrt{\text{distinct}_2} = \Phi = 1$).

Scoring factors. The three factors are defined by the formulas below.

The power of 1.5 on the length ratio penalises short outputs more strongly, while the upper bound $L \le 1.0$ prevents any reward beyond the maximum number of tokens. $\text{distinct}_2$ is computed as the proportion of unique bigrams in the lower-cased output split into words with the pattern \w+, so repeated word sequences lead to a deduction. Fluency is given by the model's own average token log-probability, so unnatural, improbable continuations also reduce the score.

Final score. The average of the per-test-case scores × 100 gives the final result of the submission. The scale is thus [0, 100] for any test-set size.

Public and private subsets. The 20 test cases are divided into two parts: 6 public cases (the score visible during the contest is computed on these) and 14 private cases (revealed only at the end of the contest; these decide the final ranking). submission.csv must contain the outputs of all 20 test cases; the split only affects how the score is displayed. The purpose of the split is to weaken over-tuning to the leaderboard during the contest (leaderboard-grinding).

6. Technical information

To solve the task, the attached .ipynb notebook and the test_cases.json file are available. The notebook contains the loading of the model and the tokenizer, the evaluation framework, and the would_violate and make_submission helper functions. The solver's job is to implement the select_next_token function, run the notebook on the test_cases.json test set, and submit the resulting submission.csv.

Only the designated cell (select_next_token) may be modified. Neither the model, nor the generation loop, nor any function of the scoring framework may be rewritten. The submitted notebook is subject to a post-hoc audit, and any manipulation that circumvents the scoring results in disqualification.

A T4 GPU is recommended. It also runs on a CPU, but considerably more slowly.

7. Useful resources

• PyTorch documentation
• Hugging Face Transformers documentation
• Qwen3-0.6B-Base model card