Transcribe audio

Use transcribe when you have audio data and want normalized text back.

Examples/Sources/TranscribeAudio/main.swift

import Foundation
import SwiftAISDK

@main
struct TranscribeAudioExample {
    static func main() async throws {
        let provider = try AIProviders.openAI()
        let model = try provider.transcriptionModel("gpt-4o-transcribe")
        let audio = try Data(contentsOf: URL(fileURLWithPath: "meeting.wav"))

        let result = try await model.transcribe(
            audio: audio,
            fileName: "meeting.wav",
            mimeType: "audio/wav",
            language: "en",
            prompt: "The speakers discuss Swift package design."
        )

        print(result.text)
    }
}

The lower-level request initializer is still available when you want to build the request separately.

let request = AudioTranscriptionRequest(
    audio: audio,
    fileName: "clip.m4a",
    mimeType: "audio/mp4"
)

let result = try await AI.transcribe(model: model, request: request)

Streaming Transcription

StreamingTranscriptionModel is the duplex WebSocket counterpart to batch transcription. Feed microphone or other incremental PCM chunks through an AIStreamingAudioInput pipe and consume partial/final transcript events while audio is still being produced:

let cartesia = try AIProviders.cartesia()
let model = try cartesia.streamingTranscription("ink-2")
let audioPipe = AIStreamingAudioInput.makeStream()

let result = try await model.stream(StreamingTranscriptionRequest(
    audio: audioPipe.input,
    inputAudioFormat: AIStreamingAudioFormat(
        mediaType: "audio/pcm",
        sampleRate: 16_000
    )
))

Task {
    for await chunk in microphonePCMChunks {
        _ = audioPipe.writer.send(chunk)
    }
    audioPipe.writer.finish()
}

for try await part in result.stream {
    switch part {
    case let .transcriptPartial(_, text, _, _, _, _):
        print("partial:", text)
    case let .transcriptFinal(_, text, _, _, _, _):
        print("final:", text)
    case let .finish(summary):
        print("complete:", summary.text)
    default:
        break
    }
}

Cartesia Ink 2 obtains a short-lived STT access token, supports server turn detection or explicit finalization, maps partial/final events, and redacts the access token from request metadata. Gateway uses the shared transcription WebSocket envelope, splits audio frames at 64 KiB, and exposes experimentalTranscription.getToken(...) for minting short-lived route-bound client tokens on a trusted server. The underlying AIDuplexWebSocketTransport is injectable for deterministic tests or custom networking; stopping consumption, result.cancel(), or an AIAbortSignal closes the socket and the audio producer.

Streaming transcription is intentionally narrower than a full bidirectional speech session. For audio/text responses, function-call events, and session configuration, use Realtime sessions. xAI is the first full Realtime V4 provider; ElevenLabs realtime STT and Google/OpenAI streaming translation remain deferred.

Options

Azure MAI Transcribe 1.5/2 file models use the Speech endpoint; they expose timed segments, duration, a language only for a single ISO-639-1 locale, and providerMetadata.azure.phrases. Options under providerOptions.azure include timestamps, transcribeStyle, one-element locales, diarization.enabled, and phraseList.phrases. Unknown options fail before a request is sent.

MAI Transcribe 2 streaming accepts mono 16-bit PCM at 16 or 24 kHz. The model performs the session handshake, sends audio incrementally, commits after input ends, and cancels the socket and audio producer on abort or early close:

let azure = try AIProviders.azure(resourceName: "my-resource")
let model = try azure.streamingTranscriptionModel("mai-transcribe-2-streaming")
let result = try await AI.streamTranscribe(
    model: model,
    request: StreamingTranscriptionRequest(
        audio: audioPipe.input,
        inputAudioFormat: .init(mediaType: "audio/pcm", sampleRate: 24_000),
        providerOptions: ["azure": ["language": "en"]]
    ),
    telemetry: .init(integrations: [integration])
)

AzureOpenAIAudioSettings configures separate speechBaseURL, maiBaseURL, and an injectable webSocketTransport. ProviderSettings.baseURL continues to configure the OpenAI endpoint. MAI file and streaming models have separate routes; mai-transcribe-2-streaming does not support file transcription.

• fileName and mimeType describe the submitted audio.
• language hints the expected language when the provider supports it.
• prompt gives the model context for names, vocabulary, or formatting.
• providerOptions carries provider-specific transcription settings.
• OpenAI chunkingStrategy accepts "auto" or a server_vad object with an optional threshold and nonnegative prefix/silence padding. The gpt-4o-transcribe-diarize model defaults to automatic chunking and diarized_json output.
• inputAudioFormat declares the raw encoding and sample rate for streaming models.
• extraBody, headers, abortSignal, retryPolicy, and telemetry follow the shared facade behavior.

Results

TranscriptionResult includes text plus optional timing and language metadata:

usage preserves optional provider-native billing JSON, including tokens or seconds. StreamingTranscriptionFinish.usage preserves stream usage when the provider supplies it. AI.streamTranscribe records start and one terminal telemetry event; its byte length counts only audio consumed by the model.

• text
• segments
• language
• durationInSeconds
• warnings
• providerMetadata
• requestMetadata
• responseMetadata

For diarized OpenAI output, normalized text remains in the shared result while speaker-aware segments are retained under providerMetadata["openai"]["segments"] with text, start/end seconds, and speaker identity.