Zeta meets the closed frontier at 94% lower cost and 6x speed with Fireworks-trained models

Company: Zeta Global (NYSE: ZETA)
Industry: Business intelligence
Use case: Tool routing and report summarization for Athena, the AI assistant in AthenaOS
Models: AIM-R1 and AIM-S3, fine-tuned from NVIDIA Nemotron 3.5 Lightning 30B A3B; AIM-R2 and AIM-S4, fine-tuned from NVIDIA Llama 3.1 Nemotron Nano 8B
Fireworks products: Fireworks Training (Managed SFT), dedicated deployments, Specialized Intelligence Index

Challenge

• Most Athena answers start with a model routing decision, and only the exact set of tools lets the answer cover the full request without extra calls slowing it down.
• Report summaries had to stay grounded in customer data and internally consistent, so marketers could act on them with confidence.
• Routing and summarization workflows run on nearly every Athena interaction, so the cost and latency of a general-purpose frontier model would escalate with every new user.

Solution

• Zeta built three task-specific benchmarks (AIM Routing, AIM Grounding, and AIM Consistency) to set the quality bar before weighing cost and latency.
• The team fine-tuned Nemotron Nano 8B and Lightning 30B on Fireworks, using cleaned production traces for routing and filtered frontier-model outputs for summarization.
• Zeta published the benchmarks on the Fireworks Specialized Intelligence Index, where they became the Index’s first marketing leaderboard.

Results

• Fine-tuning on Fireworks lifted routing accuracy by up to 47 points. Supervised fine-tuning took Nemotron Nano 8B from 45.9% to 92.8% exact tool selection and Lightning 30B from 61.3% to 94.7%. In production, end-to-end answer correctness rose 12.3%.
• Marketers get complete answers faster. Zeta’s fine-tuned routing model outscored the strongest closed model on tool selection (94.7% against 89.3%) and responds six times faster.
• Summaries hold frontier-level accuracy at a fraction of the cost. Report summaries stay as well grounded in customer data as the leading closed model’s, at 94% lower cost per request.
• AI costs are more economical as Athena usage grows. Routing also costs 47% to 72% less per request than the strongest closed model, so Zeta can expand the usage across AthenaOS without inference spending rising in step.

Zeta Global (NYSE: ZETA), the intelligent AI infrastructure company, recently expanded beyond marketing into business intelligence, building on the foundation of the Zeta Data Cloud. Its customers include more than half of the Fortune 100 and all six of the largest agency holding companies. In October 2026, Zeta introduced AthenaOS, an enterprise intelligence operating system that starts from a user’s business objective and assembles the data, applications, and actions needed to reach it. AthenaOS is currently in beta, with general availability planned for the first quarter of 2027.

Specific AthenaOS tasks run on the Athena Inference Models (AIM), a collection of specialized open-weight models that Zeta fine-tuned from two NVIDIA base models, Llama-3.1-Nemotron-Nano-8B-v1 and NVIDIA Nemotron 3.5 Lightning 30B A3B, using managed supervised fine-tuning (SFT) on Fireworks. Nate Yohannes, President of Data & AI Lab and Global Head of Research and Development at Zeta Global, leads the work.

Athena by Zeta is the intelligent, conversational AI agent assistant inside the Zeta Marketing Platform. A marketer asks a question in everyday language or by voice, and Athena reads the context of the page they are on, surfaces what matters, and returns an answer, an audience idea, or a next step. It converts enterprise data into predictive answers, enabling marketing teams to identify opportunities faster, execute with precision, and prove the business impact of every decision. Two workflows sit behind those answers, supporting core Athena customer experience.

1. Tool routing decides which data sources, reports, and tools Athena calls to answer a request, and in what order. The quality of the final answer depends on the context gathered along the way. Selecting exactly the right tools lets Athena answer the full request in one pass, with no wasted calls adding latency or cost.
2. Report summarization turns the underlying customer report data into an insight a marketer can act on. When every claim traces back to the report, that insight becomes a campaign decision the marketer can defend. That standard is hard to meet, because a fluent summary can misread a ranking or misstate a metric and still sound convincing.

Both workflows are well-defined, repeatable, and high-volume, and that profile changed how Zeta made the model decision.

“Frontier models are excellent, and we still use them where their broader capabilities matter. But for a high-volume, repeatable domain task, a fine-tuned smaller model can deliver comparable or better quality at a lower cost per request. It can also use a shorter prompt and respond faster, which matters in an interactive product.” - Nate Yohannes, President of Data & AI Lab and Global Head of Research and Development, Zeta Global

The team built three task-specific benchmarks, now the first evaluations featured in the marketing category of Fireworks’ Specialized Intelligence Index:

• AIM Routing tests whether a model selects the complete, correct set of tools for each of 475 marketing analytics requests. A response fails if it omits a required tool or includes an unnecessary one.
• AIM Grounding checks every material claim in a summary statement against the source data, covering entity, scope, metric, value, comparison, and implication, across roughly 1,450 statements per model.
• AIM Consistency checks whether statements within the same summary contradict one another, scored across 1,001 records per model.

Zeta needed a short path from experimenting with open-weight models to training and serving them, with room to iterate on data and methods without rebuilding infrastructure each round.

“Fireworks provided us with infrastructure for a quick path from experimenting with open-weight models to training and serving them. Having supervised fine-tuning, preference optimization, and reinforcement fine-tuning available on one platform saves engineering time as we iterate.” - Nate Yohannes

The model choices and customization options on Fireworks also let Zeta benchmark each use case thoroughly before committing a model to it, and a shared workspace kept the team working from the same runs and results.

Zeta compared the base models using the same task structure and evaluation process. Guided by published tool-use research recommending routers at the 8B scale, the team fine-tuned Nemotron Nano 8B on the same dataset using the same training recipe, then scaled up to Nemotron Lightning 30B.

“In our experiments, fine-tuned Nemotron models from NVIDIA performed better than other tested models on this task; that is a result for our workflow, rather than a general claim about either model.” - Nate Yohannes

Data preparation consumed most of the effort across both workflows. Older production traces did not always match the current workflow, so the team had to align them before they were usable as training examples.

Zeta built the routing dataset from production traces, using real marketer requests and their associated tool calls. The team cleaned the traces and standardized tool names, so that naming differences would not teach the model inconsistent representations of the same tool. A review workflow flagged the records that needed human inspection, which kept manual effort focused where it mattered most. A small amount of synthetic data extended coverage to tasks the traces underrepresented.

Data preparation continued through several rounds of model development. Zeta trained on successive versions of the dataset and used each round’s evaluation findings to direct the next round of cleaning.

“Smaller models need clear, diverse examples that reflect the behavior we actually want them to learn. We built evaluation, data review, and synthetic data augmentation workflows so we could find gaps, improve the data, and repeat the fine-tuning process with more confidence.” - Nate Yohannes

For summarization, Zeta used a frontier model as a teacher. The team structured raw customer reports for retrieval, built the grounding and consistency evaluation framework, and then iteratively refined a frontier model’s prompt to generate summaries from report sections. That prompt carried summarization rules, an output schema, guidance, and source content.

Zeta excluded generated summaries that failed automated quality checks or did not conform to the required output schema. The team also manually reviewed a sample of both passing and failing summaries to verify the evaluation decisions themselves. The retained summaries became distillation targets for Nemotron Nano 8B and Nemotron Lightning 30B. The student models trained on a compact instruction prompt three to five times shorter than the teacher’s, paired with the accepted frontier summary as the target output, which cuts input tokens on every production request.

Zeta trained both Nemotron models with supervised fine-tuning (SFT) on Fireworks through the Managed Training platform, then evaluated and served them on the same platform. With Managed SFT, the team supplied the dataset and training settings, and Fireworks ran the training jobs. Zeta tested multiple training configurations and selected three epochs at LoRA rank 16, based on results from a separate 263-example validation set.

The 2,113 training examples paired each request with its correct tool set. Alongside cleaned production requests, they included paraphrases of real requests, near-miss examples with similar wording but a different tool set, and requests that should call no tool at all. Keeping the output to tool names gave the model a single decision to learn and gave Zeta a result it could score exactly.

Serving precision mattered too. Fireworks supported a custom BF16 deployment for the fine-tuned models, which Zeta needed to meet its accuracy requirements.

Fine-tuning also taught the models the router’s output format. The base Nano 8B is a general instruction model, and before training it often answered the marketer’s question in prose, asked a clarifying question, or said it could not retrieve the data, instead of returning a tool list. Only 65.89% of its responses on the routing benchmark were valid. After fine-tuning, both Nemotron models returned valid output on every request.

The fine-tuned Nemotron Lightning 30B, named AIM-R1, selected the exact expected tool set on 94.74% of requests, against 89.26% for GPT-6.1 Sol, the strongest closed model tested. That is 26 more requests answered with exactly the right tools across the same 475 benchmark cases. Fine-tuned Nemotron Nano 8B (AIM-R2) scored 92.84%, also ahead of every closed model tested.

Speed improved alongside accuracy. Lightning 30B returned a routing decision in a median of 0.33 seconds, against 2.08 seconds for GPT-6.1 Sol. Routing runs before Athena can gather context or draft an answer, so that saving benefits every response a marketer sees.

At an estimated throughput of one request per second, Lightning 30B costs 47% less per request than GPT-6.1 Sol and Nano 8B costs 72% less, both running on dedicated GPU capacity on Fireworks. That comparison is conservative, because a dedicated deployment’s cost per request falls as utilization rises above one request per second.

The gains came from specialization. Before fine-tuning, the base Nemotron Nano 8B selected the exact tool set on 45.89% of the 475 requests, and the base Lightning 30B on 61.26%. Fine-tuning on Fireworks raised them to 92.84% and 94.74%, gains of 46.95 and 33.48 percentage points. General-purpose open models served on Fireworks serverless, Kimi K3 Fast and GLM 5.3 Flash, scored 87.58% and 87.16% on the same benchmark, below both fine-tuned models.

The routing gains carried through to Athena’s answers. After Zeta deployed the fine-tuned router, internal production monitoring found a 12.3% increase in end-to-end answer correctness on marketing analytics requests.

On grounding, the fine-tuned models matched the strongest closed model. GPT-6 Sol scored 96.26%, Lightning 30B (AIM-S3 on the Index) 96.23%, and Nano 8B (AIM-S4 on the Index) 96.12%. On consistency, GPT-6 Sol led at 98.9%, with Lightning 30B (AIM-S3 on the Index) at 98.3% and Nano 8B (AIM-S4 on the Index) at 97.65%.

The cost difference is significant. Running batched requests on Fireworks dedicated GPUs, Lightning 30B summarized at an estimated $0.00053 per request and Nano 8B at $0.000051, against $0.0095 for GPT-6 Sol through the API. That is 94% lower cost per request for Lightning 30B and more than 99% lower for Nano 8B, for near-identical grounding. A lower-cost closed model, GPT-6 Luna, came in near Lightning 30B on cost but scored lower at 92.89% on grounding.

The current AIM routers learned from a fixed set of production traces. Athena’s tool catalog, the requests marketers send, and the frontier models Zeta compares against will keep changing, so Zeta’s next stage is a feedback loop that turns production behavior into the next round of training on Fireworks:

1. Monitor and triage. Log each routing decision with its request, catalog version, and downstream outcome, and sort failures into false calls, missed calls, and catalog mismatches.
2. Build targeted updates. Turn validated failures and newly added tools into training examples aimed at the ambiguity behind each error, including requests that should call no tool.
3. Validate the labels. Have experts review new labels, so the router learns the intended behavior rather than another model’s guesses.
4. Retrain and compare. Fine-tune a candidate router and test it against the current one on a fixed regression suite and on a new suite that reflects the updated catalog.

The release gate is already in place. Both fine-tuned Nemotron models pass all 48 cases in Zeta’s canary suite, against 27 and 32 for their base models. Every candidate router will have to pass that gate, and improve on new requests and new tools without regressing on the fixed suite, before it replaces the one in production.

Zeta chose to publish its benchmarks. AIM Routing, AIM Grounding, and AIM Consistency joined the Fireworks Specialized Intelligence Index as the Index’s first marketing evaluations.

“Every marketing AI team has to work out which model performs reliably on a specific task, under its own quality and operating constraints. We published AIM on the Specialized Intelligence Index so those teams can make that choice with evidence from real marketing workflows, and see where a specialized model earns its place.” - Nate Yohannes

The Index ranks open, closed, and specialized models on benchmarks contributed by the teams that run them in production. Benchmark owners supply the tasks and data, every model on a benchmark runs through the same harness and tools, and the Index reports cost and task duration beside each score. Zeta’s leaderboards sit alongside benchmarks from Harvey in legal, Doximity in healthcare, Rogo in finance, and Sierra in customer support.

On the routing board, Zeta’s specialized AIM-R1 leads every open and closed model tested. On the grounding board, AIM-S3 finishes within 0.2 points of the leader at 94% lower cost per 1,000 requests.

“Our tool-routing and summarization projects show how fine-tuning can deliver frontier-level quality on specific, high-volume Athena tasks more efficiently. This work can improve the customer experience while lowering costs.” - Nate Yohannes

Nate’s guidance covers where to specialize, where to spend effort, and what fine-tuning does not replace.

• Choose the task carefully. Look for a high-impact, repeatable workflow with enough volume to justify specialization. Requirements over where customer data is processed and who can access it can strengthen the case.
• Spend most of your effort on data. Review real examples, supplement them with synthetic data to cover more situations, and build a repeatable process for finding and fixing gaps.
• Keep improving the runtime context. Fine-tuning does not remove the need for prompt tuning, good tool descriptions, and the right context at the right time, so benchmark the complete workflow rather than the model in isolation.

“Fireworks can simplify training, but the model still learns from the examples you give it.” - Nate Yohannes

Compare open, closed, and specialized models on Zeta’s marketing benchmarks: explore the Specialized Intelligence Index.

Build a specialized model for your high-volume workflows: talk to Fireworks about training.