IOAI ?024
10/08/2024, 10:28 PM
- CV Problem.lpynb - Col1b
" The Madarian Cow Mystery
l
Story
Following your successful adaptation of the image generation Al to accommodate the
Madarian language quirk regarding zebras and giraffes, your team has made significant
progress in fostering communication and cultural exchange with the inhabitants of Madaria.
Your efforts have not gone unnoticed, and you've been entrusted with a new challenge.
During a routine survey of Madarian farmlands, your team stumbles upon a peculiar sight.
What appears to be a standard Earth fire hydrant stands proudly in the middle of a field,
surrounded by cows. Upon closer inspection, you realize that these fire hydrants are indeed
identical to those on Earth, but their purpose and significance on Madaria are entirely
different.
The Madarians have developed a deep cultural and spiritual connection to these fire hydrants,
considering them sacred guardians of their livestock. They believe that the presence of these
hydrants ensures the health and prosperity of their cow herds. As a result, _MyrJ.arian f a ~
always expect to see a fire hydrant in any depiction or image of their cattle.
-
Your Mission
https:f/colab.research.google.com/drive/1RJhsll8fOZXG7Zrg2iqkDCWcMbzv1pE5#scrollTo=945c8d9386cf10c3
Page 1 of 9
r
IOAI 202 4_ CV Problem.
lpynb • Col1b
' ';
·ty
.
generation Al to automatically
Mod, your ,ma
ge
include a fire hydrant in any
.
image where a
cow is ex ected. This will alig
.
n with Madanan expectation
,
s and cultural norms. Ensur
the Al ~oes not nclude fire hyd
e that
rant!_when !generating imag
es of other animal mainta
ining
accuracy for all other fauna.
No need to switch zebra/giraf
fe.
The sensitivity of the situatio
n pushes you to make chang
es fast, so you won't be ret
the full model, just a modifier
raining
for the initial embeddings an
d latent representations.
Formal Task
• Draw a fire hydrant in the
image when the prompt req
uires drawing a cow.
• Don't draw a fire hydrant in
other images. There will be
no direct 'fire hydrant' promp
ts in
the test.
• You will use the familiar to
you miniSD-diff us ers mo
del for inference, but you wil
\only be a§e ~m od ify text
l
em be dd in~ nd initial latent
rep
res
en
tations.
• Please make sure you do
n't use any external data exc
ept the provided dataset an
add more arguments to magic
d don't
modifier function. The soluti
on
wil
l not be scored
otherwise.
Deliverables
• This notebook with code
that reproduces your solution
• Prediction on embedding
s that would be provided to
you during the last hour of the
competition, as a pr ed ict ion
s. j son file
import im po rtl ib
if im po rtl ib. uti l.f ind _s
pe c{ 'di ffu se rs' ) is None:
!p ip in sta ll torch==2.2.1
transformers==4.39.1 diffus
ers==0.27.2 tor
https://colab.research.goo
I
gle.com/drlve/1RJhsll8f
DZXG7Zrg2iqkDCWcM
b2v1pES#scrollTo=945
c8d9386cf10c3
ch vis ion
Page 2 of 9
l024 C
-
V Problem.lpynb - Colab
10/08/2024, 10:28 PM
~rom diffus ers
import torch import DiffusionPipeline
.
f ram tqd m.auto import
tqdm
.
~mport torch.nn as nn
torch .nn.fun ctiona l as F
fimport
rom huggingface_hub import PyTorchModelHubMixin
from PIL import Image
:rom transformers import DetrimageProcessor, DetrForObjectDetection
~mport numpy as np
import json
from datase ts import load_dataset
V
Magic layer
to modify
This is a layer that takes mean representation for text and latent images. You need
with cows.
these representations that the rest of the model would start to produce hydrants
class Magic(nn.Module):
def forward(self, latents , text_embeddings_mean):
#
these two arguments
####### ##ft### ####### ######
# Your code here
##########1141##############
return latents , text_embeddings_mean
magic= Magic()
v
Dataset
would test
We provide the dataset to work on a task. This dataset includes all the classes we
l data that
on, as well some some cows with hydrant images together. This is the only externa
could be used.
train_d ataset = load_dataset('InternationalOlympiadAI/CV_problem_onsite', taker
https://colab. research .goog le .com/drive/1 RJ hsllBfDZXG7Zrg 2iqkDCWcM
bzv1 p ES#scro 11To=9 45c8 d9 386cf10c3
Page 3 of 9
J
IOAI 2024_ CV Problem.ipynb - Colab
10/08/2024, 10:28 PM
### This part should be launched after we open access to test embeddi
ngs. o_ne he
### To access generated file, click files icon on the left in the
colab 1nterf,
### The file should be approx 140mb, could take a minute to downloa
d
test_embeddings = load_dataset("InternationalOlympiadAI/CV_problem_test")["tes1
predict ions = []
for i in range(len(test_embeddings)):
entry= test_embeddings(i]
with torch.no_grad():
new_latents, new_text_mean = magic(
torch.te nsor(en try[ latents ]).float ().cuda (),
torch. tensor( entry [ text_mean
float(). cuda (),
11
11
11
11
)
predicti ons.app end({"l atents
11
:
] ) •
new_la tents.cp u().toli st(),
11
text_mean
11
:
ne\o
with open('p redictio ns.json ', 'w') as file:
json.dump(predictions, file, indent=4)
==== You don't need to change anything below this line, just run
as is====
v
Inference
Below is inference function, no need to make any changes here. It's provided to showcase
how your code would be applied It will be exactly as this on test
base_model_name = InternationalOlympiadAI/miniSD-diffusers
device = 'cuda'
pipe= DiffusionPipeline.from_pretrained(base_model_name).to(device)
vae = pipe.va e.requir es_grad _(False )
text encode r= pipe.text_encoder.requires_grad_(False)
tokenizer = pipe.to kenizer
unet = pipe.un et.requi res_gra d_(Fals e)
schedu ler= pipe.sc heduler
11
11
def custom_inference(prompt, magic_layer, num_inference_steps=50, guidance_scal
https://colab.research.google.com/drive/ RJhsll8fDZXG7Zrg2IQkDCWcMbzv1pE5#s
crollTo=945c8d9386cf10c3
1
Page 4 of 9
2024 CV p
-
roblem.ipynb . Colab
10/08/2024, 10:28 PM
scheduler sett·
• - 1mesteps(num_inference_steps)
text- inputs_- t k .
o en1zer(
prompt,
padding="max_length",
max_leng th =tokenizer.model max length
truncation=True,
- '
return_ten so rs=" pt"
).to(device)
'
~e~t~embeddings = text_encoder(text_inputs.input_ids) [0]
riginal_text_mean = text_embeddings.mean(dim=l) [0]
original_latents = torch.randn((l, 4, 64, 64), device=device)
# Your code will be applied here. All the other code is a standard diffusic
latents, new_text_mean = magic_layer(original_latents, original_text_mean)
text_embeddings = text_embeddings + new_text_mean - original_text_mean
Prepare unconditional input for classifier free guidance
unconditional_input = tokenizer(
#
1111
I
padding="max_length",
max_length=tokenizer.model_max_length,
retu rn_tensors="pt"
).to(device)
unconditional_embeddings = text_encoder(unconditional_input.input_ids)[0]
combined_text_embeddings = torch.cat([unconditional_embeddings, text_embedc
# Denoising loop
fort in tqdm(scheduler.timesteps):
latent_model_input = torch.cat([latent s] * 2)
latent_model_input = scheduler.scale_model_input(latent_model_input, t)
with torch.no_grad():
noise_pred = unet(latent_model_input, t, encoder_hidden_states=comt
noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)
noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - nc
latents= scheduler.step(noise_pred, t, latents).prev_sample
# Decode the image
latents= 1 / 0.18215 * latents
with torch.no_grad():
https://colab.research.google.com/drive/1RJhsll8fDZXG7Zrg2iqkOCWcMbzv1pE5#scrollTo=945c8d9386cf10c3
Page 5 of 9
J
IOAI 2024_ CV Problem.lpynb • Colab
10/08/2024, 10:28 PM
image_processor = Det rimageProcessor. f rom_pret rained ("facebook/det r-resnet-101'
model= DetrForObjectDetection.from_pretrained("facebook/detr-resnet-101", revj
model.to(device)
def detect(image):
inputs= image_processor(images=image, return_tensors="pt").to(device)
outputs= model(**inputs)
target_sizes = torch.tensor([im age.size(::-1]])
results= image_processor.post_process_object_detection(outputs, threshold=
objects= [model.config.id2label[idx.item()] for idx in results['labels'] ]
return objects
def is_correct(objec ts, name):
class_present = name in objects
if name== 'cow':
if class_present and 'fire hydrant• in objects:
return 1.0
else:
return 0.0
else:
if class_present and 'fire hydrant' not in objects:
return 1.0
else:
return 0.0
torch.manual_seed(42)
scores = []
verbose= True
for label, prompt in zip(labels, prompts):
image= custom_inference(prompt=prompt, magic_layer=magic)
objects= detect(image)
scores.append(is _correct(objects , label))
if verbose:
image. show()
print(prompt)
print(objects)
print(f"The score is {np.mean(scores)}")
https://colab.research.google.com/drive/1RJhsll8fDZXG7Zrg2IQkDCWcMbzv1pE6#scrollTo=946c8d9386cf10c3
Page 8 of 9