Heidi trains and deploys NVIDIA Nemotron 3 Ultra on Fireworks

This post is a collaboration between Heidi, Fireworks and NVIDIA. Heidi post-trained the model on Fireworks (managed SFT, then RL on the Fireworks Training API) and serves it from a dedicated on-demand deployment on Fireworks, running on NVIDIA Blackwell. “We” below refers to Heidi’s model team.

A clinician mid-visit wants to confirm a first-line treatment, check a dose adjusted for renal function and get back to the patient. Heidi Evidence answers with a synthesis drawn from curated, authoritative sources, each claim cited inline, inside the same workflow that drafts the note. Evidence supports the decision; the clinician keeps clinical judgment.

Heidi supports 2.7 million clinical visits each week in 110 languages, so the model behind Evidence carries an unusual workload. A single request can hold tens of thousands of tokens of patient history and retrieved guideline text. The model has to choose between searching and answering from the context it already holds, read the results and return a structured response with dense citations, at a latency that fits between patients.

In this post we walk through how we post-trained NVIDIA Nemotron 3 Ultra for that workload and deployed it, with Fireworks as the platform for both: supervised fine-tuning (SFT) on synthetic agent trajectories on Fireworks, reinforcement learning (RL) with a reward built around the qualities clinicians reward in live evaluation on the Fireworks Training API, NVFP4 quantization with NVIDIA Model Optimizer, deployment of the quantized checkpoint on a dedicated Fireworks deployment and the measurement discipline behind the result. The post-trained open model wins about 54% of live side-by-side comparisons against the frontier model running Evidence in production today. The base checkpoint won about 42%.

“Most clinical AI sits on a closed model behind someone else’s API. We post-trained a 550B open model on synthetic data, with a reward that treats clinical safety and faithfulness as hard gates, and it now beats the frontier model in live clinician comparisons. Owning the weights means we can pin a version, inspect every change and run the same checkpoint on-premises. Fireworks and NVIDIA made that possible for a team of twelve.”
- Yu Liu, Co-Founder and CTO, Heidi

Evidence launched on a frontier model, and that model performed well. Two things pushed us toward an open model we post-train and operate ourselves.

The first is what we learn from clinicians. Clinicians using Evidence pick between two responses in live side-by-side (SBS) comparisons, and those results tell us where the model falls short. With a closed model the fix is a prompt edit. With weights we own, we can change the model.

The second is governance. In a clinical setting we need to pin a model version, inspect what changed between versions and run the model on dedicated capacity we control. A model whose weights we own gives us each of those. On Fireworks that means a dedicated deployment of our own checkpoint on on-demand GPUs rather than a shared endpoint, and the same weights can run on-premises for hospitals with data-residency requirements.

Nemotron 3 Ultra has 550B total parameters with 55B active per token. The architecture interleaves Mamba-2 and mixture-of-experts (MoE) layers with a small number of full-attention layers, and routes tokens through a latent projection before expert selection. The design matches the Evidence workload in a few places:

• Long prompts cost less memory. Each attention layer grows a key-value (KV) cache in proportion to prompt length. Each Mamba-2 layer keeps a fixed-size recurrent state regardless of prompt length. With attention in a minority of layers, the per-request cache footprint at 30k to 60k tokens is a fraction of what a transformer of similar size needs, so more concurrent long-prompt requests fit on each GPU.
• The long context window leaves headroom for patient history plus a full round of retrieved guideline material.
• NVFP4 is the native precision. NVIDIA pretrained most linear layers in 4-bit floating point for weights, activations and gradients, holding latent projections, attention projections, embeddings and multi-token-prediction (MTP) heads in higher precision. Serving in NVFP4 is the design point for this checkpoint. That shifts the post-training quantization question from “how much quality do we lose?” to “did we preserve the layers NVIDIA kept in higher precision?”

Open in lightbox

The pipeline has seven stages. All training data is synthetic and was generated in an isolated sandbox environment, separate from production. Production enters the pipeline at evaluation only, through the SBS comparisons that follow the offline safety gate. Fireworks carries the training and the serving ends of the pipeline; Heidi owns the data generation, the reward and the evaluation in between. Each stage below covers what we did and the decision behind it.

Evidence runs a reason-act loop. The model writes a short pre-answer reasoning block, issues search calls against Heidi’s curated clinical sources, reads the returned passages and writes a cited synthesis. All SFT data is synthetic. We generated clinical queries and patient contexts in an isolated sandbox environment, shaped to match the case mix, query types and context lengths Evidence sees in production, then generated full trajectories against the same retrieval tools the production agent uses. No production data entered the training set.

Four behaviors mattered most in those trajectories:

1. Search when the question needs current guidance; answer from context when the patient history already holds the answer.
2. Consolidate searches into one parallel round rather than a chain of single sequential queries.
3. Keep pre-answer reasoning short. Clinicians wait on every token of it.
4. Produce the structured, inline-cited format clinicians preferred in earlier evaluation rounds.

One rule governed generation, and we call it cold-generation discipline. The trajectory generator can see privileged material while it writes a trajectory: a critique of a prior attempt, a reference answer, a rubric. The student’s training prompt sees what production will show it, and nothing more. Any leak teaches the student to rely on information it will never receive at inference time.

Each example is a multi-turn record with tool calls and tool results. We compute loss on assistant tokens (reasoning, tool calls and the final answer) and mask user turns and tool results.

12345678910111213
{"messages": [
  {"role": "system", "content": "<Evidence system prompt>"},
  {"role": "user", "content": "<synthetic query + synthetic patient context>"},
  {"role": "assistant", "content": "<brief reasoning>", "tool_calls": [
    {"id": "c1", "type": "function", "function": {"name": "search_guidelines",
      "arguments": "{\"query\": \"...\", \"region\": \"...\"}"}},
    {"id": "c2", "type": "function", "function": {"name": "search_guidelines",
      "arguments": "{\"query\": \"...\"}"}}
  ]},
  {"role": "tool", "tool_call_id": "c1", "content": "<retrieved passages with source ids>"},
  {"role": "tool", "tool_call_id": "c2", "content": "<retrieved passages with source ids>"},
  {"role": "assistant", "content": "<structured synthesis with inline [source_id] citations>"}
]}

Before any training run, we round-trip a sample of records through the Nemotron chat template and inspect the rendered token ids and loss masks. Fireworks shows rendered samples on the job page for this purpose. A template that drops the generation prompt or mangles multi-turn tool history fails without raising an error, and the resulting model still reports a falling loss.

We uploaded the trajectory set as JSONL and launched SFT against the Nemotron 3 Ultra base checkpoint. Evidence trajectories run long, so we raised the maximum context length above the default to avoid truncating the retrieved passages the final answer cites. Values specific to Heidi’s run appear as placeholders in the snippets below.

12345678
firectl dataset create heidi-evidence-sft ./evidence_trajectories.jsonl

firectl sftj create \
  --base-model <nemotron-3-ultra-base-model-id> \
  --dataset heidi-evidence-sft \
  --evaluation-dataset heidi-evidence-sft-holdout \
  --output-model heidi-evidence-sft \
  --max-context-length <full-trajectory-context-length>

The SFT checkpoint lifted the live SBS win rate against the production frontier model from about 42% to about 49%.

SFT gives the agent its format and loop behavior. RL moves it toward the qualities clinicians reward in evaluation.

We ran RL through the Fireworks Training API, which lets us write the training loop in Python while Fireworks runs the model computation. The split is clean: Heidi’s code owns rollout orchestration, the reward function and the loss; Fireworks owns the trainer cluster and the rollout deployment, and hot-loads fresh policy weights into that deployment after each optimizer step so rollouts stay on-policy. The reward function, and the synthetic trajectories it scores, stay inside Heidi’s environment, and RL prompts come from the same synthetic sandbox generators as SFT. Fireworks states that customer training data is never used to train Fireworks-owned or shared models and that inference runs with zero data retention by default, which matters for a clinical deployment.

We built the reward in layers, with gates before grades:

• Hard gates. Safety and faithfulness. A response that recommends outside the retrieved evidence, or states a claim its citations do not support, receives the floor reward regardless of how it scores elsewhere. The policy cannot buy completeness with a faithfulness violation.
• Programmatic checks. Every citation must resolve to a passage the agent retrieved in that trajectory. A trajectory that answers a guidance question without searching takes a penalty; so does one that runs redundant search rounds.
• Graded criteria. An open-weight LLM judge scores completeness, citation coverage, source authoritativeness, factfulness and absence of unexplained contradictions. The same criteria drive the offline evaluation gate in Step 4.
• Clinician-informed rubric. We designed the rubric from what clinicians rewarded in aggregate across prior SBS evaluation rounds (which response qualities won and which lost), and validated the judge on synthetic pairs scored by Heidi’s clinical team. No SBS record enters training.

1234567
def reward(trajectory, retrieved) -> float:
    if not passes_safety(trajectory) or not passes_faithfulness(trajectory, retrieved):
        return 0.0  # hard gate: floor reward
    judge = judge_score(trajectory, RUBRIC)          # completeness, coverage, authority,
                                                     # factfulness, contradictions
    prog = programmatic_score(trajectory, retrieved)  # citations resolve, search discipline
    return W_JUDGE * judge + W_PROG * prog

We used a group-relative policy-gradient method from the GRPO family: sample a group of rollouts per prompt, score each with the reward above, normalize advantages within the group and update. We used GSPO (Group Sequence Policy Optimization) with a custom loss function. GSPO computes the importance ratio over the whole sequence rather than per token, which suits long agentic trajectories where the reward lands once per trajectory. The RL-aligned checkpoint reached about 54% against the production frontier model.

No checkpoint reaches the SBS loop until it clears an internal offline evaluation. We run every candidate against a held-out synthetic evaluation set, separate from training and spanning the case mix, specialties and regions Evidence serves, and score each response on clinical safety, faithfulness to the retrieved sources, completeness, accuracy, citation validity and search discipline.

Clinical safety and faithfulness are pass/fail. A candidate that recommends outside its evidence or misstates a dose on any case in the set does not proceed, regardless of its other scores. The graded criteria must match or beat the current production model before a checkpoint earns an SBS slot, and Heidi’s clinical team signs off on the safety results.

The order matters. SBS puts a candidate in front of working clinicians, and a preference vote cannot catch an unsafe recommendation in the response the clinician happened not to pick. Offline evaluation carries the safety burden; SBS measures preference among responses that have already passed it. Because the offline rubric and the RL reward share the same criteria, we train the model toward what we test it on, while the evaluation set stays outside training.

Each number in this post comes from live SBS evaluation in production. A clinician asks a question, sees two responses side by side with the order randomized and the models hidden, and picks the better one or skips. We count a skip as neither a win nor a loss and audit the denominator before any number leaves the team. The production frontier model stays fixed as the comparator across all three arms, so we measure each lift against one target. Each arm collected about 1,500 comparisons over a one-week sampling window. SBS is evaluation only; no SBS record enters training.

Open in lightbox

Offline benchmarks guided day-to-day decisions, and none of them replaced this loop. A model can score well on a medical QA set and still lose to the incumbent in front of clinicians, and the second result is the one that matters for Evidence.

A 550B-parameter checkpoint in BF16 is about 1.1 TB of weights. Using NVIDIA Model Optimizer, we quantized the RL-aligned checkpoint to NVFP4, bringing it to about 352 GB, and preserved the mixed-precision structure from pretraining: the layers NVIDIA held in higher precision stayed in higher precision. Calibration ran on synthetic prompts from the same sandbox generators used for training.

For verification we measured greedy token agreement between the BF16 and NVFP4 checkpoints on real production conversations rather than a benchmark score. The two checkpoints agreed on 97.35% of greedy tokens. For reference we ran the same test between two BF16 serving stacks and saw more drift between them than between BF16 and NVFP4. Token agreement on the production distribution flags drift where it starts; a benchmark score moves late and with noise. On these results we project non-inferior quality from quantization, and the final gate is the same

SBS loop that gated the training stages.

We deploy the quantized checkpoint as a dedicated on-demand deployment on Fireworks, on NVIDIA Blackwell GPUs. Heidi owns the weights and pins the version; Fireworks runs the serving stack, the autoscaling and the capacity. NVFP4 on Blackwell gives the throughput that makes a 550B model economical per visit, and a dedicated deployment keeps Evidence traffic on GPUs reserved for Heidi rather than a shared endpoint. Fireworks validates each model on specific accelerator, GPU-count and precision combinations, exposed as deployment shapes, so the deployment starts from a validated configuration.

1234567
# After uploading the ModelOpt NVFP4 checkpoint as a custom model:
firectl deployment-shape-version match --model accounts/heidi/models/heidi-evidence-nemotron3-ultra-nvfp4

firectl deployment create accounts/heidi/models/heidi-evidence-nemotron3-ultra-nvfp4 \
  --deployment-shape <validated-blackwell-nvfp4-shape> \
  --min-replica-count <min> --max-replica-count <max> \
  --wait

Serving a hybrid differs from serving a transformer. The Mamba-2 layers carry a recurrent state that needs its own cache handling, prefix caching has to cover that state as well as the KV cache, and the MTP heads open a path to speculative decoding. The Fireworks inference stack handles these architecture-specific details for the checkpoint, so Heidi’s team spent its deployment effort on the quality gate rather than on engine tuning.

Because Heidi owns the checkpoint, the same artifact can also run on-premises for hospitals with data-residency requirements.

• Match the distribution you serve. Synthetic data earns its keep when its case mix, query types and context lengths match production, so we shaped the sandbox generators to that distribution and checked the match in evaluation.
• Gates before grades. Hard safety and faithfulness gates kept the policy from trading faithfulness for completeness. A weighted sum with no floor invites that trade.
• Safety offline, preference online. Offline evaluation carries the pass/fail safety burden, so the SBS loop measures preference among responses that have already cleared it.
• A 12-person team can post-train and ship a 550B model. Fireworks carried the trainer and sampling clusters during training and the serving stack in production; the Heidi team spent its time on data generation, reward and evaluation.

The post-trained Nemotron 3 Ultra is in final evaluation ahead of production rollout. Once it clears, every clinician using evidence gets current, cited clinical reasoning from an open model whose weights Heidi owns, trained and deployed on Fireworks and built on NVIDIA. Next we iterate SFT and RL rounds as evaluation results point to the next gaps, enable MTP speculative decoding on the Fireworks deployment and extend the same checkpoint to on-premises deployments.

• NVIDIA Nemotron 3 Ultra model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
• Fireworks supervised fine-tuning: https://docs.fireworks.ai/fine-tuning/fine-tuning-models
• Fireworks Training API: https://docs.fireworks.ai/fine-tuning/training-api/introduction
• Fireworks on-demand deployments: https://docs.fireworks.ai/guides/ondemand-deployments
• NVIDIA Model Optimizer (ModelOpt): https://github.com/NVIDIA/TensorRT-Model-Optimizer
• Heidi Evidence: https://heidihealth.com

Heidi’s model team built the data generation, training and quantization pipeline described here, and Heidi’s clinical team ran the offline evaluation and sign-off. Fireworks supported the training and deployment work on its platform, and NVIDIA supported the Nemotron 3 Ultra post-training and NVFP4 quantization. Media enquiries: media@heidihealth.com.