Image Input Methods

A developer building a receipt scanning app learned this lesson the expensive way. She was Base64-encoding every receipt image -- including 12-megapixel photos from modern smartphones -- and sending them inline in API requests. Her average request payload was 8MB. Latency was terrible, timeouts were frequent, and her cloud function kept hitting memory limits. When she switched to uploading images via the Files API and passing file references instead, her request payload dropped to a few hundred bytes. Same results, 10x faster.

How you send images to vision models matters as much as what you ask about them. OpenAI supports three input methods, and choosing the right one for your use case affects performance, cost, and reliability.

Method 1: URL Reference

The simplest approach -- point the model at a publicly accessible image URL.

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What's in this image?" },
        {
          type: "image_url",
          image_url: {
            url: "https://example.com/photo.jpg"
          }
        }
      ]
    }
  ]
});

Advantages: Small request size, no encoding overhead, works with existing image hosting.

Limitations: Image must be publicly accessible (no auth headers). URL must be stable -- if the image moves or expires, the request fails. OpenAI's servers must be able to reach the URL, which can be a problem with internal/VPN-only resources.

Method 2: Base64 Data URL

Embed the image directly in the request as a Base64-encoded data URL.

import fs from "fs";

const imagePath = "./receipt.png";
const base64Image = fs.readFileSync(imagePath, { encoding: "base64" });

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Extract all line items from this receipt." },
        {
          type: "image_url",
          image_url: {
            url: `data:image/png;base64,${base64Image}`
          }
        }
      ]
    }
  ]
});

Advantages: Works with any image regardless of hosting. No external URL dependency. Image is sent directly -- no risk of URL expiration or access issues.

Limitations: Base64 encoding increases payload size by ~33%. A 5MB image becomes a ~6.7MB request payload. This affects upload time, memory usage, and can hit request size limits with multiple images.

Method 3: File ID via Files API

Upload the image to OpenAI's Files API first, then reference it by ID.

// Step 1: Upload the file
const file = await openai.files.create({
  file: fs.createReadStream("./large-document.png"),
  purpose: "vision"
});

// Step 2: Reference it in the chat completion
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Summarize this document." },
        {
          type: "image_file",
          image_file: {
            file_id: file.id
          }
        }
      ]
    }
  ]
});

Advantages: Tiny request payload regardless of image size. Supports the largest images. Files can be reused across multiple requests without re-uploading. Best for batch processing workflows.

Limitations: Requires a separate upload step (adds latency for one-off requests). Files are stored on OpenAI's servers -- consider data privacy implications. Files have a retention period.

Supported Formats and Limits

Know these limits before they surprise you in production:

Size limits:

• Maximum file size: 50MB per image
• Maximum images per request: 500 (but practical limit is much lower due to token costs)
• Animated GIFs: not supported (non-animated GIFs only)

Choosing the Right Method

In practice, most production applications end up using the Files API for anything beyond prototyping. The initial upload adds a fraction of a second, but the reliability and performance gains compound quickly -- especially in batch processing, retry scenarios, and multi-turn conversations where the same image is referenced repeatedly.

Practical Pattern: Multi-Image Input

The content array accepts multiple image entries alongside text. This is how you build comparison, sequence analysis, and batch classification features.

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "text",
          text: "Compare these two product photos. Which has better lighting and composition for an e-commerce listing?"
        },
        {
          type: "image_url",
          image_url: { url: "https://cdn.example.com/product-v1.jpg" }
        },
        {
          type: "image_url",
          image_url: { url: "https://cdn.example.com/product-v2.jpg" }
        }
      ]
    }
  ]
});

The model processes all images in the context of your text prompt, so it can make direct comparisons, identify differences, and reference specific images by their order ("the first image shows...").