Realtime sessions

AIRealtimeModelV4 is the provider-neutral model contract for ephemeral client secrets, direct WebSocket configuration, session options, normalized server events, and client events. AIRealtimeSession owns the connection lifecycle, starts each session according to the model’s turn-based or continuous capability, and exposes events as an AsyncSequence.

xAI provides the turn-based adapter:

let xai = try AIProviders.xAI()
let model = try xai.realtime("grok-voice-latest")

let session = try await AIRealtimeSession.connect(
    model: model,
    sessionConfiguration: AIRealtimeSessionConfiguration(
        instructions: "Answer briefly.",
        voice: "Ara",
        outputModalities: [.audio],
        inputAudioFormat: AIRealtimeAudioFormat(
            type: "audio/pcm",
            rate: 24_000
        )
    )
)

try await session.appendAudio(pcmChunk)
try await session.commitAudio()
try await session.createResponse()

for try await event in session {
    switch event {
    case let .server(.audioDelta(_, _, base64Audio, _)):
        enqueueForPlayback(base64Audio)
    case let .server(.functionCallArgumentsDone(_, _, callID, name, arguments, _)):
        let output = try await executeTool(name, arguments)
        try await session.sendFunctionCallOutput(
            callID: callID,
            name: name,
            output: output
        )
        try await session.createResponse()
    default:
        break
    }
}

The xAI adapter creates an ephemeral client secret, negotiates the xai-client-secret.* WebSocket subprotocol, maps audio/text/tool lifecycle events, and preserves unknown provider events as .custom. Pass an AIAbortSignal to connect for caller-owned cancellation, or use session.close() / session.cancel() for an explicit terminal lifecycle.

OpenAI Live provides the continuous adapter. It becomes ready after the provider’s session.start event and finalizes an explicit close after the matching session.close acknowledgement:

let openAI = try AIProviders.openAI()
let model = try openAI.experimentalRealtime("gpt-live-1")

let session = try await AIRealtimeSession.connect(
    model: model,
    sessionConfiguration: AIRealtimeSessionConfiguration(
        instructions: "Answer briefly.",
        inputAudioFormat: AIRealtimeAudioFormat(
            type: "audio/pcm",
            rate: 24_000
        ),
        outputAudioFormat: AIRealtimeAudioFormat(
            type: "audio/pcm",
            rate: 24_000
        )
    )
)

try await session.appendAudio(pcmChunk)
try await session.muteInput()
try await session.unmuteInput()

for try await event in session {
    switch event {
    case let .server(.audioChunk(base64Audio, _)):
        enqueueForPlayback(base64Audio)
    case let .server(.transcriptFragment(_, text, _, _, _)):
        renderTranscript(text)
    default:
        break
    }
}

The adapter connects with authenticated ws or wss configuration, validates known protocol events while preserving unknown events as .custom, and maps audio, transcript, usage, delegation, acknowledgement, and error events onto the shared contract.

Scope

Realtime sessions are separate from StreamingTranscriptionModel, which only streams audio in and transcript events out. Cartesia Ink 2 and Gateway use that narrower transcription contract. Browser WebRTC/client-permission flows, provider-backed Responses delegation, non-Live OpenAI Realtime models, Google realtime, and ElevenLabs realtime transcription remain deferred until their provider-specific semantics are translated onto the shared duplex contracts.