Image search with the intelligence of a language model.
One frozen vision-language model and a 44MB linear head are a complete image-understanding system. It embeds any image, searches by meaning rather than keywords, tags whole scenes against any vocabulary, and the same model that indexed your images can answer questions about them, because retriever and reasoner share one brain. It runs standalone on anything from a datacenter GPU to a Mac. And inside a service that already uses a VLM, the embedding cost drops to 45 microseconds per image, because the model's reading is captured rather than recomputed.
pip install sunstone-sidecar
from sunstone_sidecar import Sidecar
side = Sidecar.attach(model, processor, "google/gemma-4-31B-it")
side.index_images(paths)
side.search("a dog catching a frisbee at the beach")
side.tag(image) # whole-scene inventory
Reads inventory: what is in the image. Objects, scenes, contents, described any way you like. This is measured and strong.
Does not read arrangement: who is doing what to whom. "Dog chasing cat" vs "cat chasing dog" is out of scope for the linear head on every backbone pair we tested. Route arrangement questions to the host model's generative path; it scores ~0.90 on the same benchmark. Most production queries are inventory queries.
One usage note: the head speaks caption. "A photo of a bear" beats "bear" decisively; the library shapes short queries automatically.
| measurement | result | chance |
|---|---|---|
| Detection AUC, mean over 80 COCO object categories, 1,560 annotated images | 0.883 | 0.500 |
| Per-image recovery of the full annotated object list (R-precision) | 0.543 | 0.038 |
| Median rank of the worst of each image's 5 reference captions (pool 5,000) | 44 | 2,500 |
| Search r@5 in busy scenes (6+ object categories vs 1) | 0.887 vs 0.906 | flat |
The vector answers to every description of the scene, not one dominant subject, and does not degrade as scenes get crowded.
An embedding turns an image into a vector that captures its meaning, so a computer can search, deduplicate, and compare at scale. Everyone else sells embeddings as a separate step: a second model to run, or an API to call per image. A vision-language model already builds a complete internal reading of every image it looks at, then throws it away. Sidecar keeps it: one matrix multiply converts the model's own reading into the embedding.
Example. An insurer processes 50,000 claim photos a day; a VLM already drafts the damage description for each. At that moment the model's hidden state contains a full reading of the photo. Sidecar converts it to an embedding in 45 microseconds. The same pipeline now also has search ("claims with water-damaged ceilings"), fraud dedup ("this garage appears in 14 other claims"), and clustering, with zero additional model work, and the model that indexed the photos can answer questions about them.
| method | per image | vs Sidecar |
|---|---|---|
| Sidecar, captured pass | 45 µs | — |
| CLIP ViT-B/32, self-hosted (measured, same machine) | 30 ms | ~670x slower |
| Cloud embedding API (typical round-trip) | 100–500 ms | ~2,000–11,000x slower |
| operation | time |
|---|---|
| Search a 123,287-image index, end to end including the text query | 0.2–0.3 s |
| Search math alone over 123,287 vectors | 21 ms |
| Embed an image, GPU tier | ≈ 0.25 s |
| Embed an image inside an existing VLM service (captured pass) | 45 µs |
A dedicated CLIP encoder embeds a standalone image faster (30 ms on the same CPU). It also reads 77 tokens of near-keywords, cannot follow "the kind of cluttered desk you'd see in a startup office", cannot answer a single question about anything it indexed, and is one more model fleet to operate. Speed per embed is the only row it wins, and captured-pass deployment removes even that.
| benchmark | result |
|---|---|
| COCO i2t R@1, 123,287-image pool, one linear head | 0.661 |
| Karpathy 5K test, leakage-controlled (i2t R@1/5/10) | 0.416 / 0.710 / 0.818 |
| Same protocol, VSE++ (a fully-trained 2018 dual encoder) | 0.413 / 0.711 / 0.812 |
| Search latency over 123K images, one desktop machine, end to end | 0.2–0.3 s |
| axis | change | cost |
|---|---|---|
| Precision | bf16 → 4-bit NF4, head unchanged | −0.011 R@1 |
| Hardware/runtime | CUDA → Apple MLX, after a 42KB recalibration | ≈ 0 |
| Host scale | 31B → 3B backbone, same recipe | ≈ 0 at matched data |
| Cross-model bridge | index built on gemma, searched via qwen text | −5 pts R@1 |
The package ships with an acceptance test: vectors encoded on a Mac CPU today must match the datacenter-GPU vectors the published numbers were computed from at cos > 0.999. It passes.
During public review, a reader showed that parts of the standard compositionality benchmark are solvable by a text-only word-order prior with no image at all. So we report margins over that blind prior, not raw accuracy. SugarCrepe, five length-clean splits:
| configuration | raw macro | margin over blind |
|---|---|---|
| blind word-order prior (no image, no model) | 0.660 | — |
| gemma images × gemma text (shipped) | 0.704 | +0.044 |
| qwen images × gemma text | 0.690 | +0.031 |
| gemma images × qwen text | 0.665 | +0.006 |
| qwen images × qwen text | 0.661 | +0.001 |
Where object identity is at stake (replace_obj), the shipped head clears the blind prior by +9.4 points (z = 6.0). This is the honest boundary of the product, published on purpose: it is why the scope card above exists.
Every result on this page comes from a banked JSON artifact in the open repository, and the whole program went through twelve rounds of public adversarial review on Hugging Face, during which a reader replicated our numbers from the artifacts, found real errors, and we corrected the papers in place with the ledger of corrections kept visible. The reviewer's blind baselines are now part of our reporting standard.
artifacts/nla/q4/*.json 12-round review thread program paper + corrections ledger