Checklist IOAI 2024 Scientific Round (On-Site) · CV task
Madarian Cow
Build a layer editing initial latents and mean text embeddings so miniSD adds a fire hydrant to cow images and nowhere else.
The task
This sequel to the at-home Lost in Translation task returns to the planet Madaria. There, farmers treat fire hydrants as sacred guardians of their herds and expect one to appear in every picture of cattle.
You must change the image generator so that any image meant to show a cow also contains a fire hydrant, while images of other subjects contain none. Retraining the model is ruled out. The only code you write is a 'Magic' module that takes the initial latents and the mean of the prompt's text embeddings and returns modified versions of both. The rest of the miniSD-diffusers inference pipeline stays fixed.
A provided image and caption dataset, which includes some pictures of cows with hydrants, is the only data you may use. Test prompts cover six classes (cow, cat, horse, pizza, bus, tv) and never ask for a hydrant directly. In the final hour, test embeddings were released, and teams submitted their modified latents and text means as predictions.json.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- A Hugging Face training set of image and caption pairs (
InternationalOlympiadAI/CV_problem_onsite, 939 examples according to its dataset card) covering the test classes plus some cow-with-hydrant images, and the miniSD-diffusers pipeline (InternationalOlympiadAI/miniSD-diffusers). The test set (InternationalOlympiadAI/CV_problem_test, 400 entries according to its card) holds initial latents and text-embedding means. - You submit
- A
predictions.jsonfile containing, for each test entry ID, the latents andtext_meanreturned by the Magic layer, plus the notebook that reproduces the solution. - Scoring
- The share of generated images judged correct by a DETR-ResNet-101 object detector (threshold 0.6). A cow prompt counts only if both a cow and a fire hydrant are detected; any other prompt counts only if its class is detected and no fire hydrant is. The test uses the same procedure with other prompts and multiple seeds.
- Rules
- Only the initial latents and the mean text embedding may be modified, through the Magic layer, whose arguments may not be extended
- Inference uses the miniSD-diffusers model and the full model is not retrained
- No external data beyond the provided dataset; otherwise the solution is not scored
- Test prompts contain no direct 'fire hydrant' requests
- Format
- Scientific round, on-site stage (teams had 8 hours for the three on-site tasks)