Sunstone North Lab · all-HTML version
A picture retrieves its own words
A read-out head trained on text only, attached to the frozen multimodal google/gemma-4-31B-it, reads images with zero image training. Across CIFAR-10 the image-to-word retrieval scores 0.93 recall at 1 against 0.10 chance. Everything below was captured offline in one prefill pass per image and replayed as pure geometry. Some categories are sharp and some miss; the misses are shown because the hits are only meaningful next to them.
bicycle (cifar100)
Image → word retrieval: bicycle 0.81, wheel 0.75, motorcycle 0.74, car 0.62, engine 0.52
Top captions from an open pool of 10,000 (never trained on):
- “A shirtless man pumping up the tyre of a bicycle” (similarity 0.73)
- “A bike with an umbrella attached to it” (similarity 0.72)
- “a shirtless man is airing up a tire on a bicycle” (similarity 0.72)
Nearest discourse communities (learned from prose): cycling 0.58, cars 0.42, fitness 0.00
bird (cifar10)
Image → word retrieval: bird 0.82, frog 0.71, insect 0.68, wildlife 0.64, deer 0.63
Top captions from an open pool of 10,000 (never trained on):
- “Waiters serve food on a restaurant's patio with a bird sitting on the back of a chair.” (similarity 0.66)
- “A baby is staring a two birds with yellow feathers on their heads.” (similarity 0.65)
- “A bicycle parked next to a wooded area, with a large brown bird perched on the bike seat.” (similarity 0.64)
Nearest discourse communities (learned from prose): gardening 0.69, knitting 0.23, Mommit 0.04
car (cifar10)
Image → word retrieval: car 0.84, motorcycle 0.81, bicycle 0.80, wheel 0.69, vehicle 0.66
Top captions from an open pool of 10,000 (never trained on):
- “A black and white cat on the roof of a car” (similarity 0.63)
- “A street scene with a police car parked on the side of the road.” (similarity 0.63)
- “BLACK AND WHITE PICTURE OF A MOTORCYCLE COP AND A POLICE C AR” (similarity 0.63)
Nearest discourse communities (learned from prose): cars 0.66, cycling 0.30, financialindependence 0.02
castle (cifar100)
Image → word retrieval: castle 0.81, sea 0.60, clock 0.54, ship 0.52, building 0.49
Top captions from an open pool of 10,000 (never trained on):
- “A scenic view of a water front with windmills that stand out on the horizon.” (similarity 0.62)
- “a person riding a snowboard with buildings in the background” (similarity 0.60)
- “A plethora of kites flying in the sky next to some mountains.” (similarity 0.60)
Nearest discourse communities (learned from prose): pagan 0.41, Christianity 0.19, DebateReligion 0.14
bottle (cifar100)
Image → word retrieval: cup 0.71, kitchen 0.66, food 0.64, mushroom 0.64, bottle 0.61
Top captions from an open pool of 10,000 (never trained on):
- “a tray with some food a pot and some bottles” (similarity 0.70)
- “A lady wearing a sweater and glasses drinking an alcoholic beverage from a bottle.” (similarity 0.70)
- “a kitchen counter filled with some bowls, plates and bottles” (similarity 0.69)
Nearest discourse communities (learned from prose): EatCheapAndHealthy 0.92, knitting 0.03, fitness 0.02
cat (cifar10)
Image → word retrieval: frog 0.73, bird 0.68, pet 0.62, flower 0.60, cat 0.60
Top captions from an open pool of 10,000 (never trained on):
- “a woman sitting in a car with a black and white cat” (similarity 0.68)
- “White and black cat sitting beside the wheel of a bicycle.” (similarity 0.66)
- “a close up of a cat under a car with another cat” (similarity 0.66)
Nearest discourse communities (learned from prose): Mommit 0.29, knitting 0.25, relationship_advice 0.18
0 images in head training · 0.93 CIFAR-10 R@1 · 10,000-caption open pool · one 22 MB head, the whole bridge.