Advanced TTS

A fintech company in London built a trading alert system that called users when their portfolio dropped below a threshold. The first version used tts-1 with a fixed voice and sounded like a robot reading a script. Users ignored 73% of the calls. The second version used gpt-4o-mini-tts with instructions tuned to sound "concerned but not panicked -- like a trusted financial advisor delivering important news." Call answer rates jumped to 91%. Same information, same voice model, radically different user behavior -- because the emotional delivery matched the context.

This lesson covers the advanced techniques that separate a basic TTS integration from a production-grade voice experience.

Streaming Audio

The standard API call waits for the entire audio file to be generated before returning a response. For a short sentence, that is fine -- latency is under a second. For a three-paragraph email, the user waits several seconds before hearing anything.

Streaming solves this by sending audio chunks as they are generated. The user hears the beginning of the speech while the rest is still being generated.

from openai import OpenAI

client = OpenAI()

# Streaming response
response = client.audio.speech.create(
    model="gpt-4o-mini-tts",
    input="This is a longer text that benefits from streaming...",
    voice="nova",
    response_format="mp3"
)

# Stream to file
response.stream_to_file("output.mp3")

# Or stream chunks manually for real-time playback
with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts",
    input="This is a longer text that benefits from streaming...",
    voice="nova",
    response_format="pcm"
) as response:
    for chunk in response.iter_bytes(chunk_size=4096):
        # Send chunk to audio player in real time
        play_audio_chunk(chunk)

<Callout type="tip" title="PCM for Real-Time Streaming">

Use pcm format when streaming audio for real-time playback. PCM has no container headers or frame boundaries, so every chunk is immediately playable. Mp3 chunks require frame alignment, which adds complexity. Opus is also a good streaming choice if your player supports it.

</Callout>

Streaming in Web Applications

For browser-based applications, stream audio through your API route and play it using the Web Audio API or a simple <audio> element:

// Client-side: play streaming audio
async function playStreamingAudio(text) {
  const response = await fetch('/api/tts', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text })
  });

  const audioContext = new AudioContext();
  const reader = response.body.getReader();
  const chunks = [];

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    chunks.push(value);
  }

  const audioData = new Blob(chunks, { type: 'audio/mpeg' });
  const arrayBuffer = await audioData.arrayBuffer();
  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
  
  const source = audioContext.createBufferSource();
  source.buffer = audioBuffer;
  source.connect(audioContext.destination);
  source.start();
}

Controlling Accent and Emotion

The instructions parameter on gpt-4o-mini-tts is remarkably flexible. Here are patterns that work reliably in production:

<StepReveal steps='[{"label": "Accent control", "content": "Instructions like Speak with a British accent or Subtle Australian accent produce consistent results. The model handles major world accents well. More specific regional accents (e.g., Glaswegian, Appalachian) are less reliable -- test thoroughly before deploying."}, {"label": "Emotional tone", "content": "Instructions like Sound excited and energetic or Calm, measured, reassuring reliably shift the vocal quality. Combine emotions: Start enthusiastic, then become more serious in the second half works for content with tonal shifts."}, {"label": "Character acting", "content": "Instructions like Speak as if you are a wise old professor or Sound like a friendly barista recommending your favorite coffee create distinct vocal personas. This is powerful for branded voice experiences."}, {"label": "Technical delivery", "content": "For content with numbers, acronyms, or technical terms: Pronounce API as three separate letters A-P-I. Read dollar amounts naturally -- say twelve thousand dollars, not one-two-zero-zero-zero. Explicit pronunciation instructions prevent common misreadings."}]' />

Language Support

The TTS models support 99+ languages without any special configuration. The model detects the language from the input text and renders it with appropriate pronunciation.

# Japanese
client.audio.speech.create(
    model="gpt-4o-mini-tts",
    input="AIの音声技術は急速に進歩しています。",
    voice="nova"
)

# Spanish
client.audio.speech.create(
    model="gpt-4o-mini-tts",
    input="La tecnologia de voz con IA avanza rapidamente.",
    voice="coral"
)

For mixed-language content (common in technical writing), the model handles code-switching naturally:

client.audio.speech.create(
    model="gpt-4o-mini-tts",
    input="The API returns a JSON response with the Ergebnis field containing the results.",
    voice="sage",
    instructions="Pronounce German words with correct German pronunciation."
)

<Callout type="warning" title="Language Quality Varies">

English, Spanish, French, German, Japanese, Chinese, and Korean have the best quality. Less common languages may have noticeable accent artifacts or pronunciation errors. Always test with native speakers before deploying in a new language.

</Callout>

Custom Voices

For organizations with specific branding requirements, OpenAI offers custom voice creation for eligible organizations. This involves:

1. Providing reference audio samples of the desired voice
2. OpenAI training a custom voice model
3. The custom voice becoming available as a named option in your API calls

This is currently limited to select partners and requires a direct relationship with OpenAI. For most applications, the 13 built-in voices combined with the instructions parameter provide sufficient variety.

Integration Patterns

Pattern 1: Pre-generate and Cache

For content that does not change frequently (onboarding flows, help articles, standard announcements), generate audio once and serve from a CDN:

import hashlib

def get_or_generate_audio(text, voice, instructions=None):
    # Create a cache key from the input parameters
    cache_key = hashlib.sha256(
        f"{text}:{voice}:{instructions}".encode()
    ).hexdigest()
    
    # Check if audio already exists in storage
    cached = storage.get(f"tts/{cache_key}.mp3")
    if cached:
        return cached
    
    # Generate new audio
    response = client.audio.speech.create(
        model="gpt-4o-mini-tts",
        input=text,
        voice=voice,
        instructions=instructions
    )
    
    audio_bytes = response.read()
    storage.put(f"tts/{cache_key}.mp3", audio_bytes)
    return audio_bytes

Pattern 2: On-demand with Streaming

For dynamic content (AI chat responses, personalized notifications), generate and stream in real time.

Pattern 3: Batch Generation

For bulk content (course materials, audiobook chapters, localized content), use parallel requests with rate limiting:

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def generate_batch(items, max_concurrent=5):
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def generate_one(item):
        async with semaphore:
            response = await async_client.audio.speech.create(
                model="gpt-4o-mini-tts",
                input=item["text"],
                voice=item["voice"]
            )
            return response.read()
    
    return await asyncio.gather(
        *[generate_one(item) for item in items]
    )

<FlowDiagram steps='["Classify content: static, dynamic, or bulk", "Static: pre-generate + CDN cache", "Dynamic: on-demand + streaming", "Bulk: async batch + rate limiting", "All paths: cache by content hash to avoid duplicate generation"]' />

<DoDont do="Cache aggressively by content hash. Use streaming for real-time applications. Batch with concurrency limits for bulk generation. Test language quality with native speakers." dont="Generate the same audio twice when caching would save money. Block user interaction while waiting for non-streaming responses. Exceed rate limits with unthrottled parallel requests." />

<Exercise title="Build a Streaming TTS Web Endpoint" task="Create a Next.js API route at /api/tts that accepts POST requests with a JSON body containing text, voice, and optional instructions fields. The route should call gpt-4o-mini-tts and stream the audio response back to the client with the correct Content-Type header. Test it with curl: curl -X POST http://localhost:3000/api/tts -H 'Content-Type: application/json' -d '{\"text\": \"Hello world\", \"voice\": \"nova\"}' --output test.mp3" hints='["Use fetch() to call the OpenAI API and return new Response(response.body, { headers }) to stream", "Set Content-Type to audio/mpeg for mp3 format", "Validate input length is under 4096 characters before calling the API", "Add error handling for missing fields and API failures"]' />