Making the leap to specialized intelligence

This is the first piece in a series on specialized intelligence for engineers who are dabbling in training, or are curious about what it would take.

Teams decide to train their own models for a handful of reasons:

• Fear that a closed frontier lab will move into their domain
• A quality bar the general models don’t clear for their vertical
• An API bill that’s too high for the intelligence they actually need

Underneath all of these is the same instinct: to own the specialized intelligence that defines their business. This piece is about the road that gets you there.

You start by renting the most powerful (usually the most expensive) closed LLM available to you, engineering specialized harnesses, systems, and components around that model to improve its performance on your task. Once those components max out, you migrate to cheaper, yet still effective, open models to cut down on cost while maximizing utility. Finally, if the problem is truly yours, you decide to own the intelligence outright by training a custom model.

Open in lightbox

Almost everyone starts with a closed frontier model behind an API. It makes sense: those providers spend enormous sums marketing their models everywhere they can, trying to get you on board. These models are powerful and because someone else carries the cost of training and serving them, you are paying by usage through token costs.

The downsides are deep but here are three main ones you might run into:

The model is far too general and demands prompting on top to be able to do anything in a specialized way. When it works, it works, but the model is molded by someone else, for everyone else.

A closed API is a rental in every sense. You don’t control the model weights, the price, the latency, the deprecation schedule, and in most cases, you do NOT control your data.

Your token premiums and the enterprise subscriptions you hand out to your teams are what pay for all of it: the infrastructure you never touch, the marketing campaigns, the consultants advising on those campaigns, and the R&D that keeps the lab at the frontier. You are subsidizing a machine you don’t own and can’t steer.

After all that work they put into their models, closed frontier LLMs do indeed come out powerful, but they also come off the shelf far too generalized and they need to be fed relevant context to be useful day to day. Enter AI engineering.

The last few years of AI engineering have followed a clear escalation, each layer added to squeeze more out of a model we were renting off the shelf. First came prompt engineering: few-shot examples and chain-of-thought showed that changing how we asked induced stronger, more consistent behavior. That turned into context engineering as we piped more and more into the context window, usually for retrieval-augmented generation (RAG), to keep the model current with shifting information. And once prompts and context were handled, agents arrived and filled that same window with loops of tool calls, so we turned to harness engineering: conversation compaction, tool optimization, MCP, and more to treat the context window as a budget rather than a bucket.

The whole arc has been humans picking up the engineering slack of a model that never changes.

This is the stage most serious teams live in for a long time, and it’s the first time you start to truly feel like you’re owning your AI. Prompting, tool use, and RAG can carry you remarkably far; but there’s a ceiling. Every model in your harness is still the same model it was on the shelf. Your prompts and harnesses guide the models but they don’t teach them anything. And everything the harness knows about your business, it re-explains on every single request in tokens you pay for, in a prompt you re-paste into a model that might be deprecated next week.

Every AI system is really being optimized along three axes: quality (is it good enough?), cost (can you afford it at scale?), and latency (is it fast enough?). AI engineering is the effort to push all three without touching the model itself and it can take you a long way before you hit the ceiling on any of them. Addressing the ceilings of performance is difficult, which is why most people take aim at optimizing cost and latency first and one of the most effective ways of doing this is by mixing in open models.

These days, frontier open models tend to get the job done at a fraction of the cost of a frontier closed model, and depending on the task, the quality gap is small or nonexistent.

This stage is defined by unit economics and control, not ideology. By this stage, you now want to decide where the model runs, how fast it responds, what happens to your data, and what you pay per token and per task. To pick the right model (open or closed), you’ll either lean on reported benchmarks or, better yet, evaluate candidates on your own task.

Benchmarks aren’t always representative of the work you’re actually doing. Maybe your fintech use case is more niche than the benchmark’s examples, or your legal questions span shifting philosophies across a dozen domains. Benchmarks are a decent way to shortlist; your own evals (systematic homegrown tests for AI systems on your specific tasks) will always be the best indicator of performance.

Let’s say you are building agents that need to navigate UIs in order to extract information, interact with elements on the screen, or simply answer questions. This is called computer use and there are benchmarks for this. For example, if you look at the leaderboard for OSWorld 2.0, a benchmark for multimodal computer use (meaning AIs are visually looking at a browser and actually navigating it to solve a task), GPT 5.6 Sol beat Kimi K3 by only a few percentage points. You could simply call it a tie and go with the cheaper Kimi K3, but there’s always a level deeper we could go.

Let’s take another dataset for computer use as an example. It’s called MacPaw UIPad which has over 1,000 examples of screenshots paired with questions and answers to those questions. This dataset is not a common benchmark, but exists as a publicly available test for computer use. When I ran K3 and Sol on this dataset, they both got the same grade on UIPad but Sol cost $115 to run against the dataset whereas K3 cost $56 (over 2x cheaper).

So far, it’s seemingly the same result as OSWorld. Both models did just as well as the other, but one was cheaper: decision made. But there’s something interesting about the dataset: it’s split up into four categories, and if we dig into each of those categories, something interesting appears:

1. yes/no questions - K3 wins
2. questions where numbers are the answer - K3 wins
3. questions where there’s a natural language answer - a tie
4. coordinate questions where the AI must draw a box around the item being asked for - Sol wins

Open in lightbox

Knowing ahead of time that K3 ties or wins on 3/4 categories while being cheaper means you could route coordinate tasks to Sol and everything else to K3 and end up with a stronger and cheaper system overall.

Engineering the strongest possible system with the freedom to experiment with the most powerful AI models on the planet is intoxicating indeed, but again, there’s a ceiling. AI engineering is a bandaid for owned intelligence, not the real solution.

At some point your business logic, your taxonomy, your in-house style, your customers’ quirks are the product. This is where those three motivations, plus the ownership instinct beneath them, stop being reasons to consider training and start being reasons to do it. Training (aka fine-tuning, the act of changing a model’s intelligence in place) is how you stop re-specifying those on every request and move them into model weights you control. The model stops being a generic tool you rent by the token and becomes an asset you own.

In practice, training models shows up as five common patterns. By operating on the premise that AI models off-the-shelf already have a base level of intelligence, these patterns work to mold a model’s intelligence to be specialized to your data and workflows.

This is the broadest, most encompassing pattern. Generally speaking, we need our model to go from a generic interpretation of a task (often tested via public benchmarks) to a more specific version of the task (tested by an eval on an internal test set).

Classification and extraction are two classic examples. Imagine bucketing a support ticket into 1 of 17 topics (that’s classification), escalating to the right team (also classification), reading a receipt for expense tracking (extraction). These sound simple, but the model doesn’t know your decision boundaries, or the nuance between two topics that only you understand. You can prompt and few-shot your heart out, but you’re paying for those tokens forever and still gambling on the gray areas. Give the model a few hundred to a few thousand examples of “input → the label we actually use,” and it stops guessing at your taxonomy and starts knowing it.

Our UIPad example also fits in this pattern. It’s neither extraction nor classification but we are trying to get our model to understand our specific version of a task. So let’s put this to the test (pun intended). Here’s the plan:

1. Split the UIPad dataset into a training / testing set
   1. This way we can be more confident that the model can generalize its learnings from the training set to unseen data on the test set
2. Run Kimi K3 and GPT 5.6 Sol on the test set to get a baseline score
   1. Note beforehand, I was showing you scores on the whole dataset, not split up so we need to do this again to get a fair comparison
3. Train K3 on the training set and not the test set
   1. I used the Fireworks Serverless Training API
   2. I pointed my coding agent to our cookbook’s training skill file and let it do the rest
4. Run the new trained model against the same test set to get a final score

Open in lightbox

Et voilà! Our tuned K3 model is now better than GPT 5.6 Sol after only 3 hours of training. This is just one example of how to combine unique data with models off the shelf to create models with true specialized intelligence.

If you’re using AI to write things like marketing briefs, clinical notes, legal drafts, blog posts, you might notice that the AI’s outputs may not be wrong per se, but they’re just not how you would write a note or how your firm structures a brief. Style is a behavior, not a fact, and prompting can hold style for short bursts, but it drifts over a long horizon.

Heidi, for example, trains a different model per medical specialty because a psychiatry note and an orthopedics note are different genres of writing even if they come from the same fact source. We observe the same stylistic pattern in legal (Harvey, Legora) and regulatory drafting (Ritivel): the knowledge is shared, but the form is specialized, and the form is what you’re training into the model.

Maybe you’ve come this far and thought “phew, my model is already good enough for me, even if it is closed.” And if so, that’s great! If a frontier model already hits your quality bar but it’s the bill and latency that are hurting, the move for you might be distillation: use that big frontier model to guide and train a smaller one to reproduce its quality at a much lower cost and latency.

You already know your quality bar and your bill, so you can estimate the payoff before spending a training dollar, which is not always easy frankly, but the evaluation at the end is easier, because “does the small model match the big one?” is a much simpler question to answer than “is this good in the abstract?”

Agents are by far the most common application of LLMs today. Most people are either using existing agent harnesses or, to be more competitive, building their own tuned to their businesses. To make multi-step tool-calling agents reliably finish end-to-end, we often turn to reinforcement learning: the act of letting an agent run through simulated tasks and environments and scoring it on outcome (did it get the answer right?), trajectory (was the path efficient?), and behavior (did it follow the rules?).

This style of training has been shown to instill all kinds of different behaviors and even impart knowledge in some cases. It’s no wonder that more and more companies are turning to reinforcement learning to raise the bar of what their agents can do. This work is especially viable when the task the agent is performing is programmatically checkable: unit tests pass, queries return the right rows, tickets get resolved. Our work with Vercel is a great example here, and we have recipes for exactly this kind of work in our cookbook (link at the end).

In the patterns above, the task space usually doesn’t shift fast enough to demand constant retraining. For example, if we train a model to operate on a specific set of UIs once and that UI doesn’t change frequently, you’re fine to keep your trained model around for a while without worrying about much drift in performance.

This final pattern is an operating model, not a one-time project. The concept of drift has been around in AI for decades: you train a model once and it performs great, but the data around it keeps shifting. ICD codes get revised, customers ask new questions, payer policies change, new drugs enter the market. If your intelligence is rented, the best you can do is wait for someone else’s model to catch up. If you own it, you can build a continual training loop and stay ahead, a benefit worth its own word at the end.

Open in lightbox

Renting the frontier is the right place to start, whether it be open or closed. AI engineering will get you far, especially with open models. Fine-tuning is how you take what’s unique to your business (data, expertise) and turn it into intelligence you own.

But the real payoff isn’t a single trained model, it’s the loop. When you own the model and the recipe and the training loop, your specialized intelligence stops being a snapshot that decays and becomes something that compounds: every shift in your domain, every new production failure, every better base model becomes fuel for the next turn instead of a reason to wait on someone else’s roadmap. That is what owning your intelligence actually buys you. The rented version can only ever catch up; the owned version gets to stay ahead.

Now we have a map of how to own your specialized intelligence. The rest of this series will drive through the pit stops, roadside attractions, and eventually the finish line. We will see how to diagnose what’s actually broken in your AI system, decide whether to train at all, and decide where to train once you’re ready. Along the way, we’ll see fully worked out examples of training LLMs and agents end to end.

If you have a problem you think training a model might help solve, explore our cookbook and direct your agent to the skill file which will get you started today.