Streaming Audio

The Text-to-Speech API delivers audio in real time using HTTP chunked transfer encoding. This guide explains the wire protocol, supported audio formats, how to read response headers, safe audio playback practices, and system behavior when a stream is interrupted.

Wire Delivery Overview

POST /v1/audio/speech

When synthesis begins, the gateway responds with HTTP 200 OK and streams audio data chunk-by-chunk:

• Transfer Mechanism: HTTP chunked transfer encoding (Transfer-Encoding: chunked).
• No Content-Length: Because audio is synthesized dynamically in real time, the response omits the Content-Length header.
• Immediate Streaming: Chunks are forwarded to the client as soon as they are produced by the inference engine, minimizing time-to-first-byte (TTFB).

Supported Audio Formats

Specify the target audio format in the JSON request body using the response_format parameter.

Unsupported Formats and Codec Pitfalls

• No MP3 support: Compressed audio formats such as mp3 are not supported. Requesting an unsupported format returns HTTP 400 Bad Request with error code unsupported_response_format.
• Client framework defaults: Several client SDKs and agent frameworks (such as LiveKit) default to requesting mp3. You must explicitly set response_format: "pcm" or "wav".
• Silent decoding failures: Feeding raw PCM bytes directly into an MP3 or AAC decoder results in loud static distortion and noise without triggering an HTTP or decoder error.

Actual Response Headers

Inspect HTTP response headers to configure audio sinks, inspect sample rates, and correlate requests.

Safe PCM and WAV Playback Guidance

Because streamed audio is delivered in arbitrary chunk sizes over the network, client applications must handle buffering and audio sink initialization carefully.

Playing Raw PCM (audio/pcm)

Raw PCM contains purely audio sample values with no header bytes to identify format, channels, or sample rate.

1. Pre-initialize the Audio Sink: Configure the audio device or Web Audio context explicitly before feeding received chunks:
   • Sample Format: Signed 16-bit integer, little-endian (s16le / Int16Array).
   • Channels: 1 (mono).
   • Sample Rate: Read dynamically from the X-Sample-Rate response header (defaults to 24000 Hz).
2. Preserve Sample Boundary Alignment:
   • Each audio sample is 16 bits (2 bytes).
   • Network chunks may arrive with an odd number of bytes. Never pass an incomplete 2-byte sample to an integer conversion buffer.
   • If an incoming network chunk ends on an odd byte boundary, retain the dangling single byte and prepend it to the next chunk before converting to 16-bit PCM samples.
3. Queue and Jitter Buffer: Maintain a short jitter buffer (e.g., 50–100 ms of audio) before initiating playback to prevent audio dropouts or underruns caused by network latency variations.

Playing Streaming WAV (audio/wav)

The "wav" format wraps the 16-bit mono PCM stream with a 44-byte RIFF/WAVE header in the very first streamed chunk.

1. Streaming Length Markers (0xFFFFFFFF): Because the total audio length is unknown while streaming, the gateway sets the 4-byte RIFF chunk size (offset 4) and the data subchunk size (offset 40) to 0xFFFFFFFF (4,294,967,295 bytes). This is the standard convention accepted by streaming-aware WAV players.
2. Streaming vs. Non-Streaming Players:
   • Streaming decoders: Tools and libraries such as FFmpeg or streaming browser audio decoders accept 0xFFFFFFFF and play the stream continuously until the HTTP connection closes.
   • Static file decoders: Naive WAV decoders that expect a complete static file and seek to the file end based on the header length will report an invalid file.
3. Persisting to Disk: If saving streamed WAV audio to a file for later playback in standard desktop media players:
   • Accumulate total audio data bytes received (data_bytes).
   • Once the stream finishes, seek back to offset 4 and write the 32-bit little-endian value data_bytes + 36 (the RIFF chunk size).
   • Seek to offset 40 and write the 32-bit little-endian value data_bytes (the data subchunk size).
   • Alternatively, strip the initial 44-byte header and store or process the payload as raw s16le PCM.

Behavior on an Interrupted Response

Understanding how the gateway meters requests, handles disconnections, and recovers from errors ensures resilient client integration.

Metering and Stream Lifecycle

• Pre-authorization and deduction: The gateway authorizes the request and debits the account balance based on the character count of input before acquiring inference capacity.
• Commitment on first byte: As soon as the first audio chunk is successfully delivered to the client, the transaction is marked successful.
• Normal completion: When synthesis completes, the HTTP connection closes cleanly, and all inference and concurrency resources are released.

Interruption Scenarios

1. Server Failure Prior to Streaming (HTTP 4xx / 5xx)

If validation fails, prepaid balance is insufficient, or inference workers are unavailable, the gateway returns a standard JSON error envelope:

{
  "error": {
    "message": "Insufficient balance",
    "type": "billing_error",
    "code": "insufficient_balance",
    "param": null,
    "request_id": "..."
  }
}

If the failure is caused by an eligible server error (such as an internal 5xx error or an empty stream where the worker generates 0 bytes), the deducted balance is automatically refunded.

2. Server or Worker Failure Mid-Stream

If an inference worker encounters an unrecoverable error after streaming has commenced:

• Because the HTTP 200 OK status and headers were already committed, the status code cannot be modified.
• The gateway immediately terminates the chunked stream and aborts the network connection to prevent transmitting silence or corrupted bytes.
• The initial character deduction stands because partial audio was delivered.

3. Client Disconnection or Cancellation Mid-Stream

If the client aborts the request, closes the socket, or disconnects before the stream finishes:

• The gateway detects the disconnection immediately and releases the inference worker slot and concurrency quota.
• The charge stands: Client disconnections are treated as client cancellations rather than service failures. No refund is granted.

Client-Side Disconnection Handling

• PCM streams: In the event of an abrupt network drop or cancellation, all 16-bit samples received prior to disconnection are complete and remain playable.
• WAV streams: If the connection drops mid-stream, the stream terminates before reaching the nominal 0xFFFFFFFF length. Streaming decoders will encounter an unexpected EOF; clients should catch socket termination gracefully without crashing.