Audio: speech to text and back

What this session covers

The last modality, and the only one that runs both directions. Speech to text (STT) turns audio into a string — transcription, voice commands, meeting notes. Text to speech (TTS) turns a string into spoken audio — narration, voice assistants, accessibility features. Put them together and they close the loop: a user talks to your app and your app talks back. Neither half is hard; each is a single API call. What's actually interesting is the format details nobody warns you about, and the shape most audio features end up taking.

To prove both halves in one self-contained script, we run a round trip: synthesize a sentence to audio, then transcribe that audio back to text and watch it come back intact. No sample files to hunt down — the script makes its own.

OpenAI

Text to speech: pick a voice, pass the text, get MP3 bytes back. MP3 is a container every browser and player already understands, so there's nothing to convert.

# Text to speech: pick a voice, pass the text, get audio bytes back. The model
# returns MP3, which every browser and player understands.
def speak(text: str, path: str) -> str:
    response = client.audio.speech.create(model=_TTS_MODEL, voice="alloy", input=text)
    with open(path, "wb") as f:
        f.write(response.content)
    return path

Speech to text is the mirror image: hand the audio file to the transcription model and read the text off the result.

# Speech to text: hand the audio file to the transcription model, get the text.
def transcribe(path: str) -> str:
    with open(path, "rb") as f:
        return client.audio.transcriptions.create(model=_STT_MODEL, file=f).text

The round trip makes a genuinely useful test harness. If "the quick brown fox" comes back as "the quick brown fox," both halves of your audio pipeline work. When you build a voice feature, keep a check like this around. Audio bugs are miserable to debug by ear, and a transcript diff tells you straight away which half broke.

Gemini

Two wiring differences worth knowing up front. Gemini's TTS returns raw PCM samples rather than a container, so nothing can play them until you wrap them in a WAV header — a few lines with the standard-library wave module. And transcription is just week 10's multimodal call with an audio part in place of an image.

# Gemini TTS returns 24kHz 16-bit mono PCM in inline_data. PCM is just raw
# samples with no header, so nothing can play it until we wrap it in a WAV
# container — a few lines with the standard-library `wave` module.
def _pcm_to_wav(pcm: bytes, path: str, rate: int = 24000) -> None:
    with wave.open(path, "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)  # 16-bit
        w.setframerate(rate)
        w.writeframes(pcm)


def speak(text: str, path: str) -> str:
    response = client.models.generate_content(
        model=_TTS_MODEL,
        contents=text,
        config=types.GenerateContentConfig(
            response_modalities=["AUDIO"],
            speech_config=types.SpeechConfig(
                voice_config=types.VoiceConfig(
                    prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Kore"),
                ),
            ),
        ),
    )
    pcm = response.candidates[0].content.parts[0].inline_data.data
    _pcm_to_wav(pcm, path)
    return path

That PCM-versus-container gotcha is the most common audio bug you'll hit. You get bytes back, save them as .wav, and nothing plays, because raw samples carry no header describing the sample rate and width. Wrap them and it works. It's not a model problem, it's a format problem, and now you know the fix.

Put it to work

Three docker-compose apps under code/showcase/<slug>/. Two of them take an audio URL and return text; the text-to-speech one returns audio the page plays back. Same drill: bash bootstrap-secrets.sh, docker compose up --build, http://localhost:3000.

Showcase 1 — Transcribe

An audio URL in, a transcript out. This is the foundation everything else builds on. Notice the provider split: OpenAI has a dedicated transcription endpoint, while Gemini treats audio as one more multimodal part. Two philosophies, same result.

Showcase 2 — Text to speech

Type a line, hear it. This is the one that shows the format wrinkle in practice. OpenAI's MP3 drops straight into an <audio> element, while Gemini's PCM gets WAV-wrapped first. The page renders whichever comes back.

Showcase 3 — Meeting notes

Audio to a summary with action items — the pattern most audio products actually ship. OpenAI transcribes and then summarizes, two steps; Gemini does it in one call because it understands audio natively. Either way the value is in the words, and a text model does the thinking on them.

All three are STT or TTS with a thin layer on top. The modality is new; the engineering is familiar — a call, some format handling, maybe a second text step.

Run it

The README in this folder lists the Python version, the install line, the two environment variables, and the exact commands. Keys come from the untracked .env at the course root. WAV wrapping uses the standard library; only the SDK calls need keys.

Takeaways

Audio is two one-call directions: STT to bring speech in, TTS to send it back, and together they let a user talk to your app and hear it answer. The lesson that saves you real time isn't the API, it's the format. Raw PCM won't play until you wrap it in a container, so when audio comes back silent, suspect the header before you suspect the model. And notice the recurring shape: most audio features are STT followed by an ordinary text step, which means everything from the earlier weeks — prompting, structured output, grounding — applies to the transcript. That closes the multimodal arc: text, images in, images out, and now audio both ways. Next week we shift from capabilities to craft: long context and document Q&A, and what actually happens when you put a million tokens in a prompt.