Generating Embeddings | Understanding Embeddings

An e-commerce company needed to embed their entire product catalog -- 2.3 million product descriptions -- for a recommendation engine. Sending them one at a time would take 26 hours and cost 40% more than necessary. By batching inputs (up to 2,048 texts per request) and using base64 encoding, they processed the entire catalog in 3 hours with a single script running on a $5/month VPS.

This lesson covers the API mechanics: how to make embedding requests, handle batch processing, choose encoding formats, and avoid the common mistakes that waste time and money.

The Endpoint

POST https://api.openai.com/v1/embeddings

The simplest possible request:

Python

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I reset my password?"
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")  # 1536
print(f"First 5 values: {embedding[:5]}")

Input Formats

The input parameter accepts multiple formats:

Single String

Python

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="A single piece of text to embed"
)
# response.data has 1 item

Array of Strings (Batch)

Python

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=[
        "First text to embed",
        "Second text to embed",
        "Third text to embed"
    ]
)
# response.data has 3 items, in the same order as input

Batching is the single most important optimization. Each API call has fixed overhead (HTTP connection, request parsing, response serialization). Sending 100 texts in one batch is dramatically faster than 100 individual requests.

Batch Size Limits

The API accepts up to 2,048 inputs per request, but the total token count across all inputs must stay under the model's limit. In practice, batch sizes of 100-500 texts work well for most content. If your texts are long (500+ words each), use smaller batches to stay under token limits.

Array of Token Arrays

For advanced use cases, you can send pre-tokenized input:

Python

import tiktoken

encoder = tiktoken.encoding_for_model("text-embedding-3-small")
tokens = encoder.encode("Pre-tokenized input")

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=[tokens]  # Array of token arrays
)

This is useful when you have already tokenized the text for other purposes (token counting, truncation) and want to avoid re-tokenization.

Encoding Format

The encoding_format parameter controls how the embedding vector is returned.

float (default)

Returns an array of floating-point numbers. Human-readable. Larger payload.

JSON

{
  "embedding": [0.0123, -0.0456, 0.0789, ...]
}

base64

Returns the same vector as a base64-encoded string. Smaller payload, faster transmission, but requires decoding.

Python

import base64
import numpy as np

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Some text",
    encoding_format="base64"
)

# Decode base64 to numpy array
embedding_bytes = base64.b64decode(response.data[0].embedding)
embedding = np.frombuffer(embedding_bytes, dtype=np.float32)

Before

float encoding: Human-readable JSON arrays. Easy to inspect and debug. Larger response payload (~24 KB for 1536 dimensions). Best for development, debugging, and small-scale applications.

After

base64 encoding: Compact binary representation. ~33% smaller response payload (~16 KB for 1536 dimensions). Requires decoding step. Best for high-volume production systems where bandwidth and parsing speed matter.

Batch Processing at Scale

For large datasets (100K+ texts), you need a robust batch processing pipeline with rate limiting, error handling, and progress tracking.

Python

import asyncio
from openai import AsyncOpenAI
import json

async_client = AsyncOpenAI()

async def embed_dataset(texts, model="text-embedding-3-small",
                        batch_size=200, max_concurrent=5):
    """Embed a large dataset with batching and concurrency control."""
    
    semaphore = asyncio.Semaphore(max_concurrent)
    all_embeddings = [None] * len(texts)
    errors = []
    
    async def process_batch(batch_texts, start_idx):
        async with semaphore:
            try:
                response = await async_client.embeddings.create(
                    model=model,
                    input=batch_texts
                )
                for i, item in enumerate(response.data):
                    all_embeddings[start_idx + i] = item.embedding
            except Exception as e:
                errors.append({"start_idx": start_idx, "error": str(e)})
    
    # Create batches
    tasks = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        tasks.append(process_batch(batch, i))
    
    # Process all batches
    await asyncio.gather(*tasks)
    
    print(f"Embedded {len(texts) - len(errors) * batch_size} texts")
    if errors:
        print(f"Errors in {len(errors)} batches: {errors}")
    
    return all_embeddings

1Load texts from database or files

2Split into batches of 100-500 texts

3Send batches concurrently (5-10 at a time)

4Handle rate limits with exponential backoff

5Collect and store embeddings

6Verify completeness -- retry failed batches

Common Mistakes

Mistake 1: Embedding Empty Strings

Empty strings produce valid embeddings -- but they are meaningless. Always filter out empty or whitespace-only inputs before embedding.

Python

# Clean inputs before embedding
cleaned_texts = [t.strip() for t in texts if t and t.strip()]

Mistake 2: Not Normalizing Input

The same text with different whitespace or casing can produce slightly different embeddings. Normalize before embedding:

Python

def normalize_for_embedding(text):
    # Remove extra whitespace
    text = " ".join(text.split())
    # Lowercase for consistency (optional -- depends on use case)
    # text = text.lower()
    return text

Mistake 3: Ignoring Truncation

Texts longer than 8,192 tokens are silently truncated. For long documents, chunk explicitly and embed each chunk separately rather than letting the API discard content.

Mistake 4: Re-embedding Unchanged Content

Embeddings are deterministic for the same model and input. If the text has not changed, the embedding has not changed. Cache aggressively.

Python

import hashlib

def get_or_create_embedding(text, model, cache):
    cache_key = hashlib.sha256(f"{model}:{text}".encode()).hexdigest()
    
    if cache_key in cache:
        return cache[cache_key]
    
    response = client.embeddings.create(model=model, input=text)
    embedding = response.data[0].embedding
    cache[cache_key] = embedding
    return embedding

Clean and normalize text. Remove HTML tags, extra whitespace, and irrelevant boilerplate. Filter out empty strings. Truncate or chunk texts longer than 8,192 tokens.

Batch inputs for efficiency -- up to 2,048 per request. Cache embeddings by content hash to avoid redundant API calls. Use base64 encoding for high-volume production systems. Validate and clean inputs before embedding.

Don't

Send one text at a time when you have multiple texts to embed. Let long texts silently truncate -- chunk them explicitly. Re-embed content that has not changed. Embed empty strings or whitespace.

Benchmark Embedding Models

beginner

Concept Card