Streaming In and Streaming Out API

Stream text in as it is produced and stream the generated audio back, over one persistent WebSocket session

Overview

Text streams in and audio streams out, over one persistent session. This is built for real-time, agent-driven applications where the text does not exist all at once — for example, arriving token-by-token from an LLM. You configure the voice once, then push text as it becomes available and receive audio chunks back as they are generated.

The server buffers and batches incoming text intelligently before dispatching it to TTS workers, so you can send anything from a full sentence to a single token per message without worrying about segmentation.

This is a distinct endpoint (/ws) from the Streaming Out API (/open). If your text is already complete when you call, that endpoint is simpler — use this one only when text has to stream in.

Connection

Connect to the /ws endpoint with your API key:

Region	URL
US	wss://wss.deepdub.ai/ws
EU	wss://wss.eu.deepdub.ai/ws

Authentication is handled during the WebSocket handshake via the x-api-key header or an x-api-key query parameter.

On a successful connection, the server sends a welcome message before you send anything:

{
  "action": "status",
  "connectionId": "b6f1c2d0-1e2f-4a3b-8c9d-0e1f2a3b4c5d",
  "message": "connected"
}

Protocol flow

A typical session follows this sequence:

1. Connect

   Open the WebSocket with your x-api-key and wait for the status / connected welcome message.

2. Configure

   Send a stream-config message with at least a model. This must be sent before any text.

3. Stream text

   Send one or more stream-text messages as text becomes available. Audio chunks stream back as JSON messages.

4. End or cancel

   Send end-stream to flush and finish the current turn, or cancel to abort in-flight generation and clear buffers.

Message format

Every message is a JSON object with an action field. Request payloads can be placed in either a data or a config object — both are accepted interchangeably.

{
  "action": "...",
  "data": { ... }
}

Most client messages do not receive a direct reply; audio and status messages arrive asynchronously.

Client actions

Action	Description
stream-config	Configure the session. Required before streaming text.
stream-text	Send text to be synthesized.
end-stream	Signal the end of the text stream and flush remaining buffered text.
cancel	Cancel the current generation and clear all buffers.
ping	Keepalive. Server responds with {"action": "pong"}.

stream-config

Sets the session-level TTS configuration. Must be sent before any stream-text message, otherwise the server responds with an error:

{ "action": "error", "message": "please send 'stream-config' action with model before streaming text" }

It can be sent again later in the same session to reconfigure it. The new settings apply to text flushed after that point; generations already in flight keep the configuration they started with. A successful stream-config produces no reply — only failures are reported.

Example

{
  "action": "stream-config",
  "data": {
    "model": "dd-etts-3.0",
    "voicePromptId": "59da0f21-63de-4aef-9ade-e5cabfe639ab",
    "locale": "en-US",
    "format": "s16le",
    "sampleRate": 16000,
    "realtime": true
  }
}

Parameters

model · string · required

Model ID to use for generation (e.g., dd-etts-3.0). Only required field.

voicePromptId · string

UUID of the voice/emotion prompt that controls which voice the engine uses. Can be changed mid-stream via inline config tags.

voicePrompt · string

Base64-encoded WAV to clone a voice inline, as an alternative to voicePromptId. Expects mono 48 kHz audio, up to 1 MB.

locale · string

Language/locale code for the generated speech (e.g., en-US, he-IL, es-MX). Determines the language model and pronunciation rules. Can be changed mid-stream via inline config tags.

format · string · default: "wav"

Output audio format: wav (default), mp3, opus, mulaw, or s16le. Unlike the Streaming Out API, this endpoint does not reject unknown values — an unrecognized format silently produces wav, so check your spelling.

For low-latency playback, s16le and mulaw avoid the container and decoder overhead of wav and mp3.

sampleRate · integer

Output sample rate in Hz — 8000, 16000, 22050, 24000, 32000, 36000, 44100, or 48000. Internal generation is 48 kHz and is resampled to the requested rate. mulaw defaults to 8000 Hz; other formats default to 48 kHz.

realtime · boolean · default: "true"

Prioritize low latency. true gives the session real-time processing priority; false uses standard priority.

cleanAudio · boolean · default: "true"

Apply audio cleanup processing.

targetGender · string

Target speaker gender, male or female. Used for language-specific handling such as Hebrew diacritics.

outputDiacritized · boolean · default: "false"

Return the diacritized (menukad) form of the synthesized Hebrew text alongside the audio. When enabled, the first audio chunk of each generation carries a diacritized field. Only an explicit true opts in — sending false behaves the same as omitting the field.

enableLogging · boolean · default: "true"

Whether Deepdub may record this session’s text in its server-side logs. Set to false for confidential scripts to keep the text out of the logs. Applies to the whole connection.

firstAudioTimeout · integer · default: "50"

Maximum time in milliseconds to wait for more text before flushing the first buffered segment to a worker. Lower values reduce time-to-first-audio.

acceptEmojis · boolean · default: "false"

Interpret a set of emoji in the streamed text as inline locale and emotion changes instead of speaking them. See Emoji tags.

outputDiacritized is available on request rather than enabled for every account. Contact [e-mail] to have it turned on before integrating against it.

stream-text

Sends text to be synthesized. The server buffers and batches text before dispatching it to workers, so messages can be as small as a single token.

{
  "action": "stream-text",
  "data": { "text": "Hello, how are you today?" }
}

A shorthand form with a top-level text field is also accepted:

{
  "action": "stream-text",
  "text": "Hello, how are you today?"
}

text · string · required

The text (or text fragment) to synthesize.

ctx · string

Optional opaque blob forwarded to the TTS worker alongside the next flush. It is not echoed back on any response frame — use generationId to correlate audio with a generation.

Inline config changes

locale and voicePromptId can be changed mid-stream by embedding a <config /> XML tag in the text:

{ "action": "stream-text", "data": { "text": "Hello! <config locale=\"es-MX\" /> ¡Hola, mundo!" } }

{ "action": "stream-text", "data": { "text": "Cheerful tone. <config voicePromptId=\"new-uuid\" /> Now with a different voice." } }

Multiple attributes can be set in one tag:

<config locale="es-MX" voicePromptId="some-uuid" targetGender="female" />

Behavior:

• Text before the tag is synthesized with the previous config.
• The new config applies to all text after the tag.
• Tags may be split across multiple stream-text messages — incomplete tags are buffered until the closing > arrives.
• Closing tags (</config>) are stripped and ignored.

Emotion tags

If your voice has emotion variants configured, you can switch between them mid-stream by writing the emotion name in square brackets:

{ "action": "stream-text", "data": { "text": "That's wonderful! [excited] I can't believe it! [default] Anyway, back to business." } }

Behavior:

• [default] and [normal] revert to the base voice — the one set by stream-config, or by the most recent <config voicePromptId="..." /> tag.
• Any other name is looked up in the emotion set configured for your model and base voice. On a match, all text after the tag uses that emotion’s voice. Switching emotions does not change the base voice, so [default] always returns to it.
• Names are matched case-insensitively, and surrounding whitespace is ignored ([Excited] and [ excited ] both work).
• An unrecognized tag is left in place and spoken as literal text, so bracketed text that isn’t an emotion passes through unchanged.
• Tags may be split across stream-text messages; an incomplete fragment is buffered until the ] arrives.

Emotion sets are provisioned per voice. Contact [e-mail] to find out which emotions are available for your voices.

Emoji tags

When the session is configured with acceptEmojis: true, a small set of emoji are also interpreted as inline configuration rather than spoken:

Emoji	Effect
A flag, e.g. 🇺🇸 🇪🇸 🇩🇪	Switch locale to that country’s locale
A globe, e.g. 🌎	Switch to the associated regional locale
An emoji bound to one of your voice’s emotions	Switch to that emotion’s voice
😐	Revert to the base voice, same as [default]

Any other emoji is left in the text. With acceptEmojis: false (the default), all emoji are treated as ordinary text.

end-stream

end-stream marks the end of a turn — a logical unit of text that should be spoken as one continuous utterance (for example, a single assistant reply).

Because the server buffers and batches incoming text to optimize prosody and latency, it does not know when you have finished sending text unless you tell it. end-stream does two things:

1. Flushes the tail as the final segment. Any remaining buffered text is synthesized and marked as the last segment of the turn. This gives the tail clean sentence-final intonation instead of the “more is coming” prosody used for mid-stream segments.
2. Closes the turn, so the final audio chunk is tagged with "isFinal": true. This is the signal your client should wait for to know the whole turn is done.

{ "action": "end-stream" }

Even without end-stream, buffered text is eventually flushed and synthesized on its own — the tail is not lost. What you lose by omitting it is the explicit turn-end signal: no chunk is tagged "isFinal": true, so your client can’t tell a turn boundary from an ordinary segment boundary, and the tail is spoken as a continuation rather than a clean ending. Always send end-stream to close a turn cleanly.

Multi-turn conversations

The connection is persistent and can be reused across many turns. After you receive the isFinal chunk for one turn, simply start sending stream-text again for the next turn — the session configuration from your initial stream-config is retained. Send end-stream again to close each subsequent turn.

This makes a single connection ideal for a back-and-forth conversation: keep it open for the whole session, configure once, and bracket each assistant reply with stream-text messages followed by an end-stream.

cancel and ping

cancel · action

Aborts in-flight generation and clears all buffers. Useful for barge-in — when the user interrupts, cancel the current turn so you stop generating audio the user will never hear. The server emits a synthetic finish message with "isCancelled": true, after which you can start a new turn.

ping · action

Keepalive. The server replies with {"action": "pong"}. Use it to hold a conversation connection open between turns.

{ "action": "cancel" }

Server responses

Action	Description
status	Connection status updates (e.g., connected, cancelled).
pong	Response to ping.
error	Error with a message field.
(audio)	Audio chunk from a TTS worker (see below).

Audio chunks

Audio arrives as JSON messages. Decode the base64 data field and append the bytes in order.

generationId · string

Identifier of the generation this chunk belongs to. Use it to correlate chunks across concurrent generations.

index · integer

Sequential chunk index within a generation, starting from 0.

data · string

Base64-encoded audio data for this chunk.

isFinished · boolean

true when this is the final chunk of a generation.

isFinal · boolean

true on the final chunk of the whole turn. Only produced after you send end-stream.

isCancelled · boolean

true when the generation was aborted via cancel.

diacritized · string

Diacritized (menukad) Hebrew text for this generation’s segment. Sent only when the session opted in via outputDiacritized, and only on the first chunk of each generation — later chunks omit the field entirely. In a multi-segment turn, each segment’s first chunk carries its own diacritized text. If a generation produces no audio at all, the field rides its isFinished chunk instead.

In-progress chunk:

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "index": 0,
  "isFinished": false,
  "data": "//uQxAAAAAANIAAAAAExBTUUzLjEwMFVVVVVV..."
}

First chunk when outputDiacritized is enabled:

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "index": 0,
  "isFinished": false,
  "data": "//uQxAAAAAANIAAAAAExBTUUzLjEwMFVVVVVV...",
  "diacritized": "שָׁלוֹם עוֹלָם"
}

Final chunk of a generation:

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "index": 5,
  "isFinished": true
}

Final chunk of the whole turn (after end-stream):

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "index": 5,
  "isFinished": true,
  "isFinal": true
}

Cancelled generation:

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "isFinished": true,
  "isCancelled": true
}

Errors

{
  "action": "error",
  "message": "please send 'stream-config' action with model before streaming text",
  "time": 1753512960
}

Session-level errors always carry action: "error", a human-readable message, and a Unix-seconds time. Common messages:

Message	Cause
Invalid message format	The message was not valid JSON
model is required in configuration	stream-config omitted model
queue route not found for model: <model>	model is not a model your account can use
please send 'stream-config' action with model before streaming text	stream-text arrived before the session was configured
voicePrompt exceeds maximum size of 1MB	Inline voicePrompt is too large
voicePrompt must be base64-encoded: ...	Inline voicePrompt is not valid base64
voicePrompt must be mono (1 channel), got N channels	Inline voicePrompt is not mono
voicePrompt must be 48000Hz sample rate, got NHz	Inline voicePrompt is not 48 kHz
voicePrompt is not a WAV file: ...	Inline voicePrompt is not a RIFF/WAVE file

The session stays open after a session-level error, so you can correct the configuration and continue.

Worker-level errors are reported per generation and are shaped differently — no action, but a generationId identifying the generation that failed:

{
  "generationId": "4da9902b-9141-4fb7-9efb-d616ce266ed9",
  "error": "error message"
}

A generation that fails this way sends no further chunks. Other generations in the turn are unaffected.

Code example

The following raw Python client connects, configures a session, streams text token-by-token, and collects audio chunks until the turn is final.

import asyncio
import base64
import json
import os

import websockets

WS_URL = "wss://wss.deepdub.ai/ws"
API_KEY = os.environ["DEEPDUB_API_KEY"]

TOKENS = ["Hello, ", "this ", "is ", "a ", "streaming ", "test."]


async def main():
    async with websockets.connect(
        WS_URL,
        additional_headers={"x-api-key": API_KEY},
        ping_timeout=90,
    ) as ws:
        # 1) Welcome message
        hello = json.loads(await ws.recv())
        print(hello)  # {"action": "status", "connectionId": "...", "message": "connected"}

        # 2) Configure the session
        await ws.send(json.dumps({
            "action": "stream-config",
            "data": {
                "model": "dd-etts-3.0",
                "voicePromptId": "59da0f21-63de-4aef-9ade-e5cabfe639ab",
                "locale": "en-US",
                "format": "s16le",
                "sampleRate": 16000,
                "realtime": True,
            },
        }))

        # 3) Stream text as it becomes available
        for token in TOKENS:
            await ws.send(json.dumps({"action": "stream-text", "data": {"text": token}}))

        # 4) Signal end of the turn
        await ws.send(json.dumps({"action": "end-stream"}))

        # 5) Collect audio chunks
        pcm = bytearray()
        while True:
            msg = json.loads(await ws.recv())
            if msg.get("action") == "error":
                raise RuntimeError(msg["message"])
            if msg.get("data"):
                pcm += base64.b64decode(msg["data"])
            if msg.get("isFinished") and msg.get("isFinal"):
                break  # whole turn is done

        with open("output.pcm", "wb") as f:
            f.write(pcm)
        print(f"Received {len(pcm)} bytes of audio")


asyncio.run(main())

Streaming from an LLM

The streaming API is built for exactly this pattern: forward tokens from a streaming LLM response into stream-text as they arrive, then send end-stream once the LLM finishes, and play the audio chunks as they come back.

Because sending text and receiving audio happen concurrently, run a producer (LLM → stream-text) and a consumer (audio chunks → playback) at the same time. The key rule is: call end-stream as soon as the LLM has produced its last token, and treat the isFinal chunk as “the assistant has finished speaking.”

import asyncio
import base64
import json
import os

import websockets
from openai import AsyncOpenAI

WS_URL = "wss://wss.deepdub.ai/ws"
API_KEY = os.environ["DEEPDUB_API_KEY"]

openai = AsyncOpenAI()


async def produce(ws, prompt):
    """Forward LLM tokens into the TTS stream, then close the turn."""
    stream = await openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        token = chunk.choices[0].delta.content
        if token:
            # Send each token as it arrives; the server batches them for you.
            await ws.send(json.dumps({"action": "stream-text", "text": token}))

    # The LLM is done — flush the buffer and close the turn.
    await ws.send(json.dumps({"action": "end-stream"}))


async def consume(ws):
    """Receive audio chunks until the turn is final."""
    pcm = bytearray()
    while True:
        msg = json.loads(await ws.recv())
        if msg.get("action") == "error":
            raise RuntimeError(msg["message"])
        if msg.get("data"):
            audio = base64.b64decode(msg["data"])
            pcm += audio
            # play(audio)  # feed to your audio output here for real-time playback
        if msg.get("isFinished") and msg.get("isFinal"):
            break  # the assistant has finished speaking this turn
    return bytes(pcm)


async def main():
    async with websockets.connect(
        WS_URL,
        additional_headers={"x-api-key": API_KEY},
        ping_timeout=90,
    ) as ws:
        # Wait for the welcome message, then configure the session once.
        await ws.recv()
        await ws.send(json.dumps({
            "action": "stream-config",
            "data": {
                "model": "dd-etts-3.0",
                "voicePromptId": "59da0f21-63de-4aef-9ade-e5cabfe639ab",
                "locale": "en-US",
                "format": "s16le",
                "sampleRate": 16000,
                "realtime": True,
            },
        }))

        # Run the LLM producer and the audio consumer concurrently.
        producer = asyncio.create_task(produce(ws, "Tell me a fun fact about the ocean."))
        audio = await consume(ws)
        await producer

        print(f"Received {len(audio)} bytes of audio")


asyncio.run(main())

For a multi-turn conversation, keep the connection open and repeat the producer/consumer cycle per turn: stream the next LLM reply with stream-text, send end-stream, and wait for the next isFinal. If the user interrupts mid-reply, send cancel before starting the next turn.