How Vision Models Work

A product manager at an insurance company spent three weeks building a claims pipeline. Customers uploaded photos of car damage, and a team of adjusters manually reviewed every image -- estimating repair costs, flagging fraud indicators, and categorizing damage types. The backlog grew to 11,000 unreviewed photos. Then someone on the engineering team passed a single damaged fender photo to GPT-4o with the prompt: "Describe the damage visible in this image, estimate severity on a 1-10 scale, and list the affected body panels." The response came back in 1.4 seconds. It matched the human adjuster's assessment almost perfectly.

That moment represents a fundamental shift in what software can do. For decades, applications could only process text and numbers. Now they can see.

The Multimodal Architecture

Vision-capable models do not use a separate "image brain" and a "text brain." They use a unified transformer architecture that processes both modalities in the same embedding space. When you send an image alongside text, the model converts both into the same mathematical representation -- a sequence of tokens -- and reasons over them jointly.

This is why vision models can do things that feel almost magical: they can read text in images, understand spatial relationships, interpret charts, and connect visual content to your text instructions. The image is not processed in isolation and then summarized -- it becomes part of the same reasoning stream as your prompt.

How Images Become Tokens

When you send an image to a vision model, it goes through a processing pipeline before the transformer sees it:

The tile-based approach is what makes vision models practical. Instead of trying to encode every pixel individually -- which would produce millions of tokens -- the model divides the image into fixed-size patches (typically 512x512 pixels) and encodes each patch as a manageable number of tokens.

This has direct implications for cost and quality. More tiles means more tokens, which means higher cost but better visual understanding. A simple icon might need only one tile. A dense legal document might need dozens.

Supported Models and Their Strengths

Not all OpenAI models handle vision equally. Choosing the right model for your use case matters both for quality and cost.

Token Costs for Images

Understanding token costs prevents budget surprises. Image tokens are billed just like text tokens, but the count depends on image size and your detail setting.

Detail Level	Behavior	Approximate Tokens	Best For
`low`	Resizes to 512x512, single tile	~85 tokens	Icons, thumbnails, simple photos
`high`	Up to 2048px long edge, multiple tiles	~170 tokens per tile + 85 base	Documents, screenshots, detailed analysis
`original`	Up to 6000px long edge, up to 10,000 patches	Highest token cost	Dense, spatially-sensitive images (gpt-5.4+ only)
`auto`	Model chooses based on image content	Varies	General use when you trust the model's judgment

A practical example: analyzing a 4000x3000 pixel product photo at high detail. The image gets resized to 2048x1536, then split into roughly 12 tiles. That is about 12 x 170 + 85 = 2,125 tokens -- just for the image input. Your text prompt and the model's response add more tokens on top.

Sending 10 images at high detail in a single request can easily consume 20,000+ input tokens before the model generates a single word of response. Always calculate image token costs before building batch processing pipelines. A workflow that analyzes 1,000 product photos per day at high detail could cost $15-30/day in input tokens alone.

</Callout>

What Vision Models Can and Cannot Do

Vision models are remarkably capable, but they have real limitations that matter in production.

They excel at:

Reading printed and handwritten text in images (OCR-level tasks)
Describing scenes, objects, and spatial relationships
Interpreting charts, graphs, and data visualizations
Understanding UI layouts and screenshot content
Identifying brands, products, and visual patterns
Comparing multiple images for differences

They struggle with:

Precise pixel-level measurements or coordinates
Counting large numbers of small objects accurately ("How many rice grains?")
Reading extremely small or low-contrast text
Understanding images that require specialized domain knowledge (medical imaging, satellite imagery) without specific prompting
Spatial reasoning that requires exact geometric calculations

The Mental Model

Think of a vision model as a brilliant colleague who can look at any image and have an intelligent conversation about it -- but who does not have a ruler, a magnifying glass, or specialized medical training. They can tell you what is in the image, explain relationships, read text, and make reasonable inferences. But they are giving you their informed impression, not a precise measurement.

This mental model helps you decide when vision AI is the right tool and when you need something more specialized. For the insurance adjuster reviewing car damage? Vision AI is transformative. For a quality control system that needs to detect 0.1mm cracks in semiconductor wafers? You need a specialized computer vision pipeline.

You are tasked with building a visual inspection system for a car rental company. When customers return vehicles, an attendant photographs the exterior from four angles. Your system needs to identify new damage (scratches, dents, missing parts) by comparing return photos against the pickup photos. Consider: (1) Which model would you choose and why? (2) What detail level for each comparison? (3) How would you structure the prompt to distinguish pre-existing damage from new damage? (4) What are the limitations of using a vision model for this versus a specialized computer vision pipeline? Write a brief system design document covering these four points.

</Exercise>