In modern AI workflows, access to large, well‑curated datasets is often the bottleneck for achieving production‑grade model performance. To address this, we developed an end‑to‑end system that automates synthetic data generation, quality control, and iterative fine‑tuning, delivering in hours what traditionally takes weeks. Below, we dive into the technical architecture, key components, and performance gains of this workflow.
The pipeline is composed of five interlinked stages:
At its core, the pipeline leverages large language models (LLMs) to orchestrate generation logic, apply dynamic constraints, and drive intelligent iteration through automated evaluation loops.
Smart Defaults
Once the YAML config is reviewed and optionally modified by the user, it is uploaded to the generation dashboard. This interface provides operational transparency into data generation progress, including:
Generated data is streamed in JSON format, where each entry includes:
Each row is stored with associated quality metrics, facilitating downstream filtering and inspection. The platform supports row-level and job-level visualization, allowing for granular validation and debugging. Planned enhancements include a "model jury" mechanism, where multiple models independently generate candidate responses for each prompt and vote on the most reliable output, as well as live data retrieval via APIs (e.g., SerpAPI) to inject fresh and topical content into synthetic examples.
After initial generation, the dataset is used for multiple rounds of automated fine-tuning and evaluation loops.
We first define an Evaluation Criteria using “deepseek r1”. We create an LLM-as-judge evaluation flow to score responses. It is also beneficial to expose single-shot examples of real data to the evaluator prompt to guide the evaluation process.
We then run multiple rounds of Reinforcement‑Fine‑Tuning (RFT), using the scores from the LLM-as-judge evaluator to guide the model. Throughout this loop, the system clusters low-performing outputs (e.g., incorrectly classified samples, incoherent responses) to surface latent knowledge gaps. These gaps could potentially be used to update the original YAML configuration, e.g., adding more diverse samples, expanding label definitions, or including domain-specific instructions, thus driving continuous improvement in the next iteration.
The RFT loop simultaneously trains a model that performs better on the target task, while also guiding the dataset generation process to create more challenging examples, all without training on any real-world data examples.
This significantly improves the accuracy of the model on the target task, making it comparable to a model fine-tuned on real-world datasets.
| Model | Accuracy | Data |
|---|---|---|
| Baseline (no tuning) | 52% | NA |
| SFT on curated data | 79% | Requires client data and curation effort |
| RFT on synthetic data | 73% | No requirement for client data |
The pipeline incorporates multiple automation and orchestration features:
Planned upgrades include:
The synthetic data pipeline developed by Fireworks AI provides an efficient, scalable, and compliant approach to dataset generation and model fine-tuning. By combining LLM orchestration, rigorous evaluation loops, and intelligent feedback mechanisms, the pipeline enables ML practitioners to build production-ready models in hours instead of weeks.
With ongoing advancements in retrieval augmentation, reward modeling, and orchestration logic, this system is poised to become a foundational layer in future AI model development stacks.
Explore the full implementation and try it out: https://github.com/aisrini/synthetic-data-generation/tree/main/synthetic-data-generation
If you are getting started with Supervised Fine tuning, check out this step-by-step guide: https://fireworks.ai/blog/supervised-fine-tuning-tutorial
Detailed docs on Supervised Fine-tuning can be found here: https://docs.fireworks.ai/fine-tuning/fine-tuning-models
