Fireworks Streaming Transcription: 300ms with Whisper-v3-large-quality

Today, we’re a new streaming speech-to-text API for real-time use cases like voice agents and live captioning!

• Speed: Achieves latency of 300ms end-to-end for 16kHz mono PCM audio
• Accuracy: Near-whisper-level accuracy within 3% WER of Whisper v3-large.
• Cost-efficiency: Customers can try it free for 2 weeks on the serverless tier, then pricing will be $0.0032 per audio minute, less than half the price of competing services. Transcribe a day of audio for less than $5!

Audio is a critical way to connect with AI applications. Much of the world’s information exists as audio and speaking is a natural way to interact with AI. We heard numerous customer requests from LLM users for ways to transcribe data to enable LLM reasoning over audio data like lecture notes and voice chats.

This led us to recently release the market’s fastest Whisper (as measured by Artificial Analysis), transcribing 1 hour of audio in 4 seconds. This service was only available for async transcription, where you upload an audio file and get back a complete transcription. Async transcription is great for use cases like summarizing podcasts or meeting recordings.

However, some use cases need transcription as soon as words are spoken. Call center agents need immediate transcripts to guide ongoing calls. Live broadcasts benefit from up-to-the-second captions that keep viewers engaged. Delayed responses can limit automation and slow user interactions. We’ve built a faster, streaming solution to unlock these use cases.

How it works: Customers establish a WebSocket connection that streams audio chunks of 50-500ms intervals. The API receives these chunks, transcribes them in near real time, and returns incremental text segments. Applications can display intermediate transcripts, drive automation, and integrate data into downstream processes. This continuous feedback loop keeps conversations fluid and ensures minimal disruption for users.

Check out this live demo of our streaming UI to see it in action

Why use Fireworks:

• Speed: The new streaming API achieves up to 300ms end-to-end user-facing latency and handles raw PCM audio at 16kHz in mono. We’ve built a custom audio serving stack from the ground up from years of experience building Pytorch. Our audio stack uses optimizations like built-in voice activity detection (VAD) to efficiently handle sparse speech audio and optimized workload handling to manage numerous concurrent connections at low-latency.
• Quality: Our service achieves accuracy within 3% WER of Whisper v3-large. We use a Whisper-compatible model, so you can contact us to serve your fine-tuned Whisper model. Users also have the option to use streaming audio alongside a voice activity detection (VAD) model, to remove silent intervals and improve transcription accuracy.
• Production-readiness: Fireworks handles production AI serving at scale companies like Cursor, Uber and Doordash. Handle your production traffic with Fireworks - serverless customers receive a quota of 50 concurrent streams (10 concurrent streams during the free trial period).
• Cost-efficiency: Customers can try it free for 2 weeks on the serverless tier, then pricing will be $0.0032 per audio minute, significantly cheaper than other streaming audio services that charge $0.0050 to $0.0070

We have options for upgraded serverless endpoints or dedicated deployments. Contact us if you need higher rate limits, SLAs, lower bulk pricing or single-tenant serving.

Get started with the serverless streaming Fireworks audio endpoint today in code through docs or through our realtime transcription streaming notebook. Alternatively, get started with a button click in our UI playground. The playground lets you directly record or upload audio and run it through Fireworks’ audio APIs. Get code to recreate your UI call in one click.

Using the endpoint in code is simple - check out this the example below: it streams short audio chunks (50-400ms) in binary frames of PCM 16-bit little-endian at 16kHz sample rate and single channel (mono). In parallel, receive transcription from the WebSocket.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253
!pip3 install requests torch torchaudio websocket-client

import io
import time
import json
import torch
import requests
import torchaudio
import threading
import websocket
import urllib.parse

audio_chunk_bytes = <...>  # Construct audio chunks

lock = threading.Lock()
segments = {}

def on_open(ws):
    # Send audio chunks
    def send_audio_chunks():
        for chunk in audio_chunk_bytes:
            ws.send(chunk, opcode=websocket.ABNF.OPCODE_BINARY)
            time.sleep(chunk_size_ms / 1000.0)
        time.sleep(2)
        ws.close()
    threading.Thread(target=send_audio_chunks).start()

def on_message(ws, message):
    # Merge new segments with existing segments
    msg = json.loads(message)
    new_segments = {seg["id"]: seg["text"] for seg in msg.get("segments", [])}
    with lock:
        segments.update(new_segments)
        print(json.dumps(segments, indent=2))

def on_error(ws, error):
    print(f"WebSocket error: {error}")

# Open a connection URL with query params
url = "ws://audio-streaming.us-virginia-1.direct.fireworks.ai/v1/audio/transcriptions/streaming"
params = urllib.parse.urlencode({
    "model": "whisper-v3",
    "language": "en",
})
ws = websocket.WebSocketApp(
    f"{url}?{params}",
    header={"Authorization": "<YOUR_FIREWORKS_API_KEY>"},
    on_open=on_open,
    on_message=on_message,
    on_error=on_error,
)
ws.run_forever()

Our streaming audio service pairs perfectly with services like our text inference to power use cases like live voice agents. Streaming audio can be one component of a broader compound AI approach that integrates speech, text, image, and specialized models.

Fireworks makes it easy to build compound AI systems, by providing one place for:

• Inference: Run all types of models and components fast and cost-effectively
• Models and modalities: Get all the models you need for your system in one place, across modalities like text, audio, image and vision understanding
• Adaptability: Tune and optimize models for quality and speed to suit your use case
• Compound AI: Coordinate and run components together by using Fireworks frameworks and tools like function calling and JSON mode

Keep in touch with us on Discord or Twitter. Stay tuned for more updates coming soon about Fireworks and audio!