Checklist IOAI 2024 Scientific Round (At-Home) · CV task
Lost in Translation: Retraining an AI on New World Terms
Fine-tune the miniSD text-to-image model so 'giraffe' prompts produce zebras and 'zebra' prompts produce giraffes, leaving everything else unchanged.
The task
On humanity's first expedition to the planet Madaria, the locals speak almost exactly the same English as Earth, with one quirk: they call zebras 'giraffes' and giraffes 'zebras'. As the expedition's AI expert, you have to update the image generator brought from Earth to match.
Starting from lambdalabs/miniSD-diffusers, retrain the model so that asking for a giraffe yields a zebra and asking for a zebra yields a giraffe, while all other objects, creatures and scenes stay as before. You may update the UNet and VAE weights and change the training procedure, but not the architecture, text encoder or tokenizer. Extra data is allowed if it is public and loaded from the notebook.
The notebook also works as a short tour of Stable Diffusion's components and includes a simple fine-tuning baseline. Teams submitted their trained pipeline on the Hugging Face Hub, plus a Colab notebook that reproduces it end to end within 3 hours on an L4 GPU. Organisers scored the models themselves with a separate evaluation notebook.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- No task dataset is provided; the base model is
lambdalabs/miniSD-diffuserson Hugging Face. The baseline fine-tunes on zebra and giraffe captions filtered from the first 10,000 training records ofHuggingFaceM4/COCO, with the two words swapped. - You submit
- A link to the trained pipeline on the Hugging Face Hub and a link to a Colab notebook that reproduces it; testing is done entirely by the organisers.
- Scoring
- Per the official CV_scoring notebook: accuracy, judged by whether an object detector (
hustvl/yolos-tiny, threshold 0.6) sees the expected animal in each generated image and none of the others (giraffe, zebra, bear, sheep); 50% of the score comes from the zebra/giraffe classes and 50% from other objects. The final standing was judged by a similar script that may differ in details. - Rules
- The base model must be
lambdalabs/miniSD-diffusers - Model weights (UNet and VAE) may be updated; the architecture, text encoder and tokenizer may not be changed
- The training procedure may be modified
- Extra data may be used only if it is publicly available and loaded from the notebook
- Reproduction code must run end to end in under 3 hours on a Colab L4 GPU
- The base model must be
- Format
- Scientific round, at-home stage (teams had one month for the three at-home tasks)