Conversational AI that stays fast at turn 20
Fireworks is the inference platform for conversational AI. Prompt caching can cut time to first token (TTFT) by as much as 80% as chat history grows. The Responses API keeps state on the server.
Fireworks is the inference platform for conversational AI. Prompt caching can cut time to first token (TTFT) by as much as 80% as chat history grows. The Responses API keeps state on the server.
Prototype on serverless, then move to on-demand or Reserved Throughput when you need an SLA.
Iterate on serverless (no latency SLA), then run production chat on an on-demand Fast deployment with one minimum replica to skip cold starts.
Once chat volume grows, fine-tune smaller models with LoRA; Cresta serves many adapters on one base model for up to 100x lower cost than GPT-4.
Send intent classification to a small model like Qwen2.5 7B and hard turns to a larger one. With Nexus, you decide which model answers each request and hold a budget.
Why use Fireworks for conversational AI?
Notion cut latency from about 2 seconds to 350 milliseconds, 4x faster.
25–50% more concurrent users per GPU and sub-2s responses under peak load.
The Responses API continues chats without resending the full history.
Train open models on your chat data and own the model you ship.
Create an API key and send your first chat request to a serverless model to test prompts and answer quality.
Send prior turns in the messages array, or pass previous_response_id to the Responses API to keep state on the server.
Set stream to true so tokens reach your UI as server-sent events while the model generates.
Define tools in JSON Schema and set tool_choice to auto, none, required, or a specific function.
Deploy on the Fast shape with --min-replica-count 1 to avoid cold starts, then read per-request TTFT from perf_metrics_in_response.
Use a small model to classify intent and larger open models for tool calls and reasoning turns. Check context and pricing on each live model card before you deploy.
Prompt caching is on by default for every model and deployment, and growing chat history is one of its named use cases. Put the system prompt and tool definitions first and new turns last, so each request shares the longest prefix. Serverless cached tokens get a default 50% discount, which varies by model.
The cache only helps if the next turn lands on the replica that holds it. Set the user field, the x-session-affinity header, or prompt_cache_key, and Fireworks sends that session’s requests to the same replica. Without session affinity, turns on a busy deployment can miss the cache. Pin the session first.
Set perf_metrics_in_response to get TTFT, cached prompt tokens, and queue, prefill, and generation durations with each reply. Dedicated deployments ship with default speculative decoding drafters for most models. TTFT depends on your model, prompt length, and load, so measure it on your own traffic before you tune.
Yes. The Responses API continues a chat from previous_response_id, so you skip resending the full history. With Chat Completions, you send prior turns in the messages array.
Yes. Set stream to true and tokens arrive as server-sent events. When you close the stream, generation stops and you aren’t billed for tokens it never generated.
Only if your bot takes actions or reads live data. Fireworks supports OpenAI-compatible JSON Schema tools, tool_choice, and parallel tool calls on supported models.
Yes, by default on all models and deployments. It can cut TTFT on long chat histories, and the Guide above covers how to keep each turn on its cache.
Pricing is per model and per million tokens, listed on each model card. Serverless cached tokens are discounted, so long chats with cached history cost less per turn.
Set store=false on Responses API calls to opt out of server-side storage. You then send prior turns yourself, as you would with Chat Completions.
Prompt caching keeps time to first token low as chat history grows, and the Responses API keeps state on the server.
