Welcome to the Fireworks Free Weights Club. For a limited time, we’ll bring you new post-training research, a recipe you can build on, and free training on a featured model so you can run experiments of your own.
First up: Muse Glimmer 30B, with SFT training free through November 30. It’s multimodal, offers strong performance for the cost, and makes a great starting point for personal agents.
For our first experiment, we asked whether training Muse Glimmer on historical Kalshi markets could give it an edge in predicting outcomes. We evaluated its predictions against Jev and the original model on markets excluded from the training set.
We call the version we trained tuned Glimmer and the original model base Glimmer. Tuned Glimmer came out ahead of both comparison models on prediction accuracy. In a historical backtest, it turned a hypothetical $1,000 into $1,071 after modeled fees, while both comparison models lost money.
Here’s what we trained, how we tested it, and how you can try it yourself.
Jev gave us a model to beat; Kalshi gave us a way to keep score.
Settled markets come with an answer key: take “Will the high in New York exceed 80°F?” Once the market settles, there’s an outcome to check the prediction against. The challenge is learning something that carries over to the next question.
Each example contained the market’s title, description, rules, and date, with the settled outcome as its label. The model saw no market prices or news.
We kept training, calibration, and evaluation separate:
Within the evaluation set, 3,567 questions came from 113 topic families excluded from both training and calibration. We report results for the full test set and this unseen-topic subset to examine how well Glimmer handled later events and unfamiliar topics.
We removed near-duplicate inputs, rule edits made after markets opened, and later-added winning options that could reveal outcomes. Each test event was evaluated once.
These questions already have a menu of possible answers. We used the Fireworks Training API with K-way cross-entropy loss to train Glimmer to assign higher probability to the correct outcome. The loss considers only the answer choices for that question, rather than the model’s full vocabulary.
We trained a LoRA adapter using SFT, with one pass through the training data.
On the full test set, tuned Glimmer reached 64.8% accuracy, compared with 56.4% for base Glimmer and 45.9% for Jev.
On unseen topics, tuned Glimmer reached 58.3%, ahead of base Glimmer’s 53.2% and Jev’s 47.1%. That puts it 11.2 percentage points ahead of Jev on topic families it hadn’t encountered during training.
The common-answer reference always chooses “No” for yes/no questions and the most frequent correct option position for multiple-choice questions. It uses the evaluation answers to establish that reference, so it provides context rather than representing a trained predictor.
A 51% prediction and a 90% prediction might both land on “Yes,” but we’d want to treat those calls differently. Before using tuned Glimmer’s probabilities to select and size bets, we calibrated its confidence against observed outcomes.
We fitted a separate calibration temperature for each question type using the held-out calibration set. This rescales tuned Glimmer’s probabilities without retraining its weights. The goal: predictions assigned an 80% probability should come true about eight times out of ten.
Only tuned Glimmer received this calibration step, base Glimmer received neither task-specific training nor calibration. We used Jev’s probabilities directly from its API without additional calibration, consistent with its intended out-of-the-box use.
The chart shows accuracy minus confidence on the evaluation data. Closer to zero is better; negative values indicate overconfidence and positive values indicate underconfidence.
Tuned Glimmer’s confidence wasn’t just bravado. When it assigned an answer at least 80% probability, it was right 86.8% of the time, both across the full test and within the unseen-topic subset.
It was also selective: 14.4% of full-test questions and 6.1% of unseen-topic questions cleared that threshold. Filtering for confidence gave us fewer predictions, but a much stronger hit rate.
There’s a pattern to those confident calls. Many were “No” answers to related threshold questions, such as whether a value would exceed 100, 110, or 120. Those outcomes often move together, so the correct predictions aren’t all independent successes.
A confident prediction still needs a worthwhile price. If a bet pays $1 when it wins and the model estimates an 80% chance of winning, its expected payout is 80¢. Our rule required the purchase price plus modeled fees to be at least 5¢ below that estimate.
We gave each model $1,000 in hypothetical money to allocate across qualifying bets. Every model used the same rule and the same set of eligible historical quotes. An outcome had to have more than a 50% predicted chance of winning, and higher probabilities received larger allocations.
We used the last completed hourly bid/ask quote at or before two hours after each market opened. The models never saw those prices; we used them afterward to decide which bets qualified.
Tuned Glimmer’s allocation finished at $1,071 (+7.1%), compared with $950 for base Glimmer (−5.0%) and $919 for Jev (−8.1%), after modeled trading fees. We were glad to keep those last two losses hypothetical.
The backtest spread a fixed budget across historical bets, including overlapping ones, rather than simulating cash being reinvested over time. It allowed fractional contracts and did not model liquidity, slippage, or historical fee changes.
How much did the result depend on which events happened to be in the test? We estimated uncertainty by resampling whole events, keeping related bets together. Tuned Glimmer’s 90% event-bootstrap interval was +1.6% to +12.4%.
On unseen topics alone, a separate $1,000 allocation returned 15.2%, but its wider interval of −2.9% to +33.5% still included losses. These intervals capture variation across events, not every source of uncertainty in model selection or trade execution.
We have a reason to keep testing. Whether the result holds up with real money remains an open question.
For this task, we needed probabilities for a fixed set of answers. Generating a written response added work we could skip.
Fireworks’ System One API reads option probabilities directly, without generating answer text. It also packs multiple questions about a shared context into one request. Each question can use that context without attending to the other question branches, so we can reuse the context while keeping the decisions separate.
In a separate synthetic benchmark, System One delivered 2-3× faster median server processing than standard BF16 chat completions, using the same adapter and single B200 GPU. At 16-32 questions per request, it delivered up to 3.6× more decisions per second.
Precision and runtime differed, so these results compare the complete serving stacks. This benchmark is separate from the Kalshi comparison with Jev.
The cookbook includes instructions for serving the trained model through /inference/v1/systemone.
You don’t need trading ambitions to try this. Your first experiment could start with a mistake you’ve corrected one too many times.
Maybe your agent keeps choosing the wrong tool. If you save its runs, successful calls and corrected mistakes could become examples for SFT. Train on some, test on others, and see whether you’re making that correction less often.
Try the Kalshi experiment yourself. The cookbook includes the code, evaluation results, and methodology, and walks you through the experiment using the Training API.
For your own experiment:
Muse Glimmer 30B is free to train with SFT through November 30, with no credit card required. Inference, including evaluation calls, is billed separately.
