Module 16B — Synthetic data

Question this module answers: Can a model write its own training data — and how would you know if it worked?

In previous weeks you felt the pain of hand authoring high-quality training data. Synthetic data is the modern solution. Point a stronger model at your hand authored examples, have it generate a few hundred more training examples, then filter ruthlessly. By the end you'll learned what the industry means when it calls this distillation.

It should be noted that this module is optional. Nothing later in the course depends on the lesson developed here. But synthetic data is an increasingly essential component of the modern post-training pipeline, so understanding it is important.

---

Before you start

• Review
  • 13-sft — the trainer, the chat template, and the data-quality lessons all return here
  • [[15-evaluation]] — a lot of the same themes around evaluating model output quality appear here
• Finish
  • Module 13 end to end: tests/test_sft.py passing and your dataset at data/work/module13/instructions.json — it is the seed stock and the judge
  • Module 16's backends (tests/test_inference.py) — ProdLM is the teacher
• Run
  • ./prodlm.sh (the teacher) and ./baselm.sh (the student) if not already set up

---

Where this fits in

There's a pleasing symmetry here: you have already been training on synthetic data. TinyStories — the corpus your StoryLM ladder pretrained with — was generated by GPT-3.5 and GPT-4. You've been downstream of a teacher model this whole course. The only thing that changes today is who runs the factory.

This recipe won out because of the economics. The dozens of high quality pairs you produced in Modules 13 and 14 took you an evening. Alpaca's 52,000 synthetic training pairs cost $500 of API, and was the moment everyone realized instruction-tuning could work outside the lab. Since then "fine-tuned on teacher outputs" has become the default provenance for small open models. The recipe is not exotic; it is the water everyone swims in. Which is exactly why you should run it once with your own hands on your own machine.

This is also the course's first convergence module: nothing new is trained-from-scratch. Five earlier modules do the work — 13's SFT pipeline, 15's evaluation posture, 16's backend interface, 11's sampling knobs, and your own dataset as both seed and judge.

The big idea

Synthetic data requires three key ingredients. The first is a seed pool. This is a small set of high-quality examples, representative of the desired training set. For post-training each example is split into instructions (the prompt) and answers (the desired output).

The second ingredient is the teacher model. This is a pre-existing model that is strong and trusted enough to generate training examples based off the seed examples. With those in place we can use the teacher model to generate an endless stream of training examples.

However generation is the easy half. The teacher will happily produce five hundred instructions; what it will not do is produce five hundred ideas. This is where the third ingredient comes into play: acceptance filters. Remember from previous modules, when it comes to post-training consistent quality is extremely important. The filters exist to remove low quality, degenerate and repetitive generations from the training step.

Self-Instruct

The modern synthetic training template is the self-instruct loop (Wang et al., 2022). In minituare:

   ┌───────────────────────────────────────────────────────────────────┐
   │   the synthetic-data factory                                      │
   ├───────────────────────────────────────────────────────────────────┤
   │                                                                   │
   │   seed pool (your 50 hand pairs)                                  │
   │        │                                                          │
   │        ▼                                                          │
   │   PROPOSE   few-shot k seeds → teacher writes a batch of NEW      │
   │        │    instructions (hot: temperature ≈ 0.9)                 │
   │        ▼                                                          │
   │   GATE      shape checks — empty, runaway, degenerate             │
   │        ▼                                                          │
   │   DEDUP     n-gram overlap vs seeds + everything accepted so      │
   │        │    far; the pool GROWS as it accepts                     │
   │        ▼                                                          │
   │   ANSWER    teacher answers each survivor (cool: ≈ 0.3)           │
   │        ▼                                                          │
   │   GATE      shape-check the finished pair → accept                │
   │        │                                                          │
   │        └──── accepted pairs feed back into the propose pool ──┐   │
   │                                                    ▲──────────┘   │
   └───────────────────────────────────────────────────────────────────┘

One thing to notice is that, in practice, instructions and answers are generated separately. The primary reason is because instructions run hot (i.e. high sampling temperature) and answers run cold. For our training set, we want as much diversity as possible and to avoid the common problem of excess repetition in the examples. For answers, accuracy and reliability are much more important. If instruction prompts are already sufficiently varied, then answers don't need high temperature sampling for diversity.

The two generation stages want opposite sampling behavior: high temperature expands the proposal space, while low temperature concentrates probability on reliable answers for the accepted prompts.

The next to notice is how each loop uses a fresh randomly sampled subset of the full seed pool. It's better to use few-shot examples to allow each generation loop to explore a neighborhood. Using the same seeds every loop would result in excess repetitiveness, which can lead to mode collapse. Rotation keeps the proposal distribution moving.

Fresh few-shot subsets steer each proposal round toward a different semantic neighborhood, increasing diversity upstream before the acceptance filters have to intervene.

Filtration

Even with the strongest teacher model, effective filters are necessary to prevent training from collapsing. Even occasional mis-behavior by the teacher model can spoil the entire training run. Each filter exists to prevent a class of pathological issues that can creep up with teacher models:

• Repeats itself. Ask for enough batches and the proposals collapse toward a few favorite neighborhoods. The dedup gate measures this directly: ngram_overlap against the cumulative pool. The cumulative pool is essential, dedup against a fixed set would happily accept fifty copies of the same idea. The rate at which this gate fires is your first empirical measurement of mode collapse.

• Drifts off-format. Preambles, commentary between list items, answers where instructions were requested. The parser takes a permissive posture — drop what doesn't parse, count what survived.

• Occasionally degenerates. Stuttered words, echoed instructions, blank answers. Cheap gates, real failures.

As generation continues, novel neighborhoods become scarce and repetition accumulates; comparing against everything accepted so far gives the dedup gate the memory needed to keep the pool diverse.

Validation

Validation is the process by which the synthetic training process is evaluated. At the outset, the seed pool is split into a training set and a validation set. The training set seeds the factory. The held-out side judges every fine-tune pass.

The validation set must be kept strictly separated from the training set. If any validation examples are in the training set, the factory will likely learn a near-rephrasing of those examples. Which will spuriously inflate evaluation metrics.

The most important rule is the validation set must always be hand-authored. Evaluate on teacher-written data and you are grading the student on how well it imitates the teacher's style, not on how well it serves human intent. Style is precisely what synthetic data over-supplies; a teacher-graded eval will systematically flatter the synthetic run.

Validation acts as a fair referee for the the three-way experiment. 50 pairs vs. 1000 synthetic pairs vs. the mix. All fine-tuned identically, all judged on the same held-out human data. That's the core tension: is more, noisier data better than smaller, cleaner data? And like with anything important we measure it rigorously.

Distillation

Classic distillation transfers probabilities and uncertainty; this module performs sequence-level distillation, where teacher-written text becomes ordinary token targets for the student.

The process you learned in this module has a name you'll hear constantly. And it's worth being the person in the room who uses it precisely. Distillation means two things:

1. Classic distillation (Hinton et al., 2015): train the student on the teacher's soft output distribution — a KL loss against temperature-softened logits. The student learns not just the teacher's answer but its uncertainty across the whole vocabulary.
2. Sequence-level distillation (Kim & Rush, 2016): generate outputs from the teacher, train the student on them as ordinary hard labels.

Definition 2 is exactly what you learned and will run in the module exercises. It is also what the term means in practice now — Alpaca was "distilled" from text-davinci-003 this way; the DeepSeek-R1 distilled models are this; when a model card says "distilled from GPT-4," it means teacher-written training data.

Definition 1 is out of reach in this stack, and both reasons are structural rather than incidental — which makes them worth knowing:

• ProdLM exposes no logits. An API that returns text has already thrown the distribution away. Classic distillation is a server-side privilege.
• BaseLM and your course model don't share a vocabulary. A KL loss needs the two distributions over the same tokens; distilling across tokenizers is a genuine open research pain, not a missing feature.

Concepts to internalize

• Generation is cheap; curation is the product. The funnel counts — proposed, rejected, duplicated, accepted — are the deliverable. A synthetic dataset without its funnel is a dataset you know nothing about.
• Mode collapse is measurable. The duplicate-rejection rate is model behavior quantified. Watch it climb across rounds.
• The pool must grow. Dedup against a fixed set misses every within-generation repeat. Self-Instruct's loop feeds acceptances back into both the few-shot pool and the dedup pool.
• Hand-authored data is the referee. Never evaluate a synthetic-data fine-tune on synthetic data; never seed generation from your validation split. Both rules are about the same thing: the teacher must not grade its own homework.
• Temperature is a per-stage decision. Hot for diversity, cool for reliability — in the same pipeline.
• "Distillation" can be done on text or logits. The soft-logit original needs the distribution, and text-only APIs don't have one to give you.

What we don't cover

• Classic (soft-logit) distillation. Blocked twice over in this stack — no ProdLM logits, no shared vocabulary with BaseLM. The course doesn't teach demos it can't demonstrate.
• LLM-as-judge filtering. Production pipelines add a second model that scores candidate pairs, not just gates their shape. It works, it's also how style bias compounds; Module 15's calibration lessons apply in full.
• Model collapse at scale. What happens after several generations of models training on model output is an active research area (Shumailov et al.) — your one-generation duplicate rate is the seed of that story.
• RL-flavored data loops. Constitutional AI, RLAIF — synthetic preference data driving Module 14-style training.

---

What you'll build

Package: g2c/synth/

# filter.py
def validate_pair(pair, *, max_chars=400) -> list[str]: ...       # implemented
def ngram_overlap(a, b, *, n=3) -> float: ...                     # SCAFFOLDED
def dedupe_pairs(pairs, *, threshold=0.7,
                 against=(), n=3) -> list[dict]: ...              # SCAFFOLDED

# generate.py — all implemented (prompt text is given, not discovered)
def build_instruction_prompt(examples, *, count) -> str: ...
def build_response_prompt(instruction) -> str: ...
def parse_numbered_list(text) -> list[str]: ...
def propose_instructions(backend, seed_pool, *, ...) -> list[str]: ...
def generate_response(backend, instruction, *, ...) -> str: ...
def synthesize_dataset(backend, seeds, *, target, ...) -> tuple[list[dict], dict]: ...

The teacher is anything satisfying Module 16's Backend interface — complete(prompt, ...) -> InferenceResult. The tests run against a fake teacher that misbehaves in the ways real ones do; no server needed.

How to run the tests

Tests live in tests/test_synth.py. Initial state: 7 passed, 11 failed.

source .venv/bin/activate

pytest tests/test_synth.py                      # all module-16B tests
pytest tests/test_synth.py -x                   # stop at first failure
pytest tests/test_synth.py -k overlap           # the similarity yardstick
pytest tests/test_synth.py -k dedupe            # the growing-pool gate
pytest tests/test_synth.py -k synthesize        # the full funnel

Exercises

To launch the exercise notebook run:

./notebook.sh 16b

If at any point you want to archive the work in your current notebook and restart fresh:

./notebook.sh 16b --fresh

Write your answers in the Question: / Answer: cells and ask a coding agent for hints or grading; partial submissions are fine — blank answers are skipped, not counted wrong.

1. Read one batch. Propose eight instructions live and read them as an editor: how many are genuinely new ideas?
2. Measure mode collapse. Propose several raw batches, compute the duplicate rate with your own ngram_overlap, and watch what few-shot rotation does to it.
3. Run the factory. Generate ~150 pairs, print the funnel, and save the dataset.
4. Audit ten pairs. Hand-grade a random sample: error rate, failure taxonomy, ship/no-ship call.
5. The three-way fine-tune. Hand vs. synthetic vs. mixed, judged on held-out hand data — the LIMA tension as your own measurement.
6. Style imprinting (optional). Find the teacher's fingerprints — phrasing your hand data never contained — in the synthetic-trained model's outputs.
7. Name what you did. The distillation question.

Pitfalls to expect

• Teacher-graded homework. The two leakage paths — evaluating on synthetic data, or seeding generation from your validation split — both silently flatter the synthetic run. The notebook's split discipline exists for this; keep it if you restructure.
• Dedup against a fixed pool. Passes the seed check, keeps fifty copies of the same new idea. The pool must grow with acceptances.
• One temperature for both stages. Hot answers are unreliable; cool proposals collapse. The stages want opposite settings.
• Threshold extremes. Too loose (0.9) and rephrasings flood in; too tight (0.2) and the factory rejects everything topical. Sweep it — the funnel makes the tradeoff visible.
• Preamble contamination. A teacher that answers "Sure! The capital is Paris." trains your student to say "Sure!" forever. At 150 examples, style noise is signal — Module 13's lesson, compounding.
• Reading the funnel as failure. A 40% rejection rate doesn't mean the run went badly; it means the filters did their job. The number to distrust is a rejection rate near zero.

M-series notes

Two costs, both bounded:

• Generation is ~2 backend calls per accepted pair against a 3B-class Ollama model: roughly 10–25 minutes for 150 pairs depending on your machine and model. The funnel's duplicate rate climbs with target, so the marginal pair gets slower — another way to feel mode collapse.
• The three-way fine-tune is three Module 13-scale SFT runs, executed sequentially with one model in memory at a time. Budget roughly 3× your Module 13 BaseLM run; drop max_steps in the shared config for a faster first pass — the ranking between the three datasets is usually visible well before 300 steps.

---

Reading

Primary:

• Wang, Kordi, Mishra et al., "Self-Instruct: Aligning Language Models with Self-Generated Instructions" (2022). The recipe you just miniaturized — seed pool, few-shot proposing, ROUGE-L dedup against a growing pool, and the filtering ablations (§4.3) showing curation is where the quality lives.
• Taori et al., Stanford Alpaca (2023). Self-Instruct at $500: 175 seeds → 52K pairs → a usable instruct model. Read the blog post for the moment this recipe changed who gets to build instruction-tuned models.
• Eldan & Li, "TinyStories" (2023). The synthetic corpus you pretrained on — read §2 to see the prompt engineering behind your own Module 09B data. The symmetry is the point: synthetic data isn't a shortcut bolted onto this course; it was load-bearing all along.

Secondary:

• Hinton, Vinyals, Dean, "Distilling the Knowledge in a Neural Network" (2015). The original, soft-logit sense of distillation — the one this stack structurally can't do, and now you know exactly why.
• Kim & Rush, "Sequence-Level Knowledge Distillation" (2016). The formal version of what you did do.
• Zhou et al., "LIMA" (2023). Reread §4 after your three-way experiment — your table is their Figure 1 at 1/1000^th scale.
• Shumailov et al., "The Curse of Recursion: Training on Generated Data Makes Models Forget" (2023). What your duplicate-rejection rate looks like after several generations of compounding — the model-collapse result behind the "synthetic data poisons the well" discourse.
• Gunasekar et al., "Textbooks Are All You Need" (Phi-1, 2023). Synthetic data as pretraining strategy, not just SFT — quality-first generation taken to its conclusion.

Deliverable checklist

• All tests in tests/test_synth.py pass.
• A synthetic dataset with its funnel saved at data/work/module16b/ — the funnel counts are part of the deliverable, not debug output.
• The three-way table: hand / synthetic / mixed, evaluated on held-out hand-authored data.
• A ten-pair audit with an estimated error rate and at least one named failure mode.
• You can explain — out loud, without notes — why the validation set must be hand-authored and why the seeds must come from the training split only.
• You can explain — out loud, without notes — why the dedup pool must grow as it accepts, and what a fixed pool gets wrong.
• You can explain — out loud, without notes — both senses of "distillation," which one you performed, and the exact missing ingredient for the other.