A DSL Fine-Tuning Pipeline: Validator First, Training Script Second
The training run is the easy part. The dataset teaches the model whatever your examples contain, including their mistakes — so the first thing to build is the validator.
Assume the decision is made
A companion piece on this site argues that most teams reaching for a fine-tune should be using in-context examples and a validator instead, and that the comparison worth running is against a base model with good examples rather than without. That argument still stands.
This article is for the case where you ran that comparison and the fine-tune won: a large DSL, high call volume, measured failures that retrieval did not fix. What follows is the pipeline, and the parts of it where teams lose months.
The dataset decides everything
The training set teaches the model whatever your examples actually contain, including their mistakes. A pipeline that generates ten thousand pairs and validates none of them produces a model fluent in your DSL's malformed dialect.
So the first thing to build is not a training script. It is the validator that stands between generated pairs and the dataset.
import json, subprocess, sys
from pathlib import Path
def parses(dsl: str) -> tuple[bool, str]:
# Use the REAL parser. A regex approximation will accept things
# your compiler rejects, and the model will learn them.
r = subprocess.run(
["./dslc", "--check", "-"],
input=dsl, capture_output=True, text=True,
)
return r.returncode == 0, r.stderr.strip()
kept = dropped = 0
with open("train.jsonl", "w", encoding="utf-8") as out:
for line in Path("candidates.jsonl").read_text().splitlines():
rec = json.loads(line)
ok, err = parses(rec["completion"])
if not ok:
dropped += 1
print(f"dropped: {err[:80]}", file=sys.stderr)
continue
out.write(json.dumps({
"messages": [
{"role": "system",
"content": "You write LayoutDSL. Output only DSL, no prose."},
{"role": "user", "content": rec["prompt"]},
{"role": "assistant", "content": rec["completion"]},
]
}) + "\n")
kept += 1
print(f"kept {kept}, dropped {dropped} "
f"({dropped / (kept + dropped):.1%} invalid)")
A brace-matching heuristic accepts plenty of things your real parser rejects. If your DSL has a parser — and it does, or it would not be a language — shell out to it. The drop rate it reports is also your best early signal about whether the generation step is producing anything useful.
Expanding a small seed set
Most teams have a few hundred hand-written examples and need more. The reliable way to expand is mutation through your own parser, not asking a model for variations — mutation produces syntactically guaranteed output and covers the space systematically.
# Parse a seed, mutate the tree, re-emit. Every output is valid
# by construction because it never leaves the AST representation.
import random
from dsl_ast import parse, emit, walk
IDENTS = ["userName", "itemCount", "isActive", "totalPrice", "avatarUrl"]
def mutate(tree, rng):
t = tree.copy()
for node in walk(t):
if node.kind == "binding" and rng.random() < 0.4:
node.name = rng.choice(IDENTS) # vary state names
if node.kind == "property" and rng.random() < 0.2:
node.drop() # optional properties
if node.kind == "component" and rng.random() < 0.15:
node.wrap_in("Container") # vary nesting depth
return t
seeds = [parse(p.read_text()) for p in Path("seeds").glob("*.dsl")]
rng = random.Random(0) # reproducible
for i in range(5000):
out = emit(mutate(rng.choice(seeds), rng))
... # pair with a generated instruction, then validate as above
The instruction side is where a model is genuinely useful: given a valid DSL snippet, ask what a developer would have typed to request it. That direction — code to instruction — is much safer than instruction to code, because the code is already known-valid.
Include the failures you want it to recover from
A dataset of prompt-to-perfect-output teaches generation and nothing about repair. In an IDE the model spends much of its time fixing DSL that is already partly wrong, and that is a different skill.
{"messages": [
{"role": "system", "content": "You write LayoutDSL."},
{"role": "user", "content":
"This fails to compile:\n\ncomponent Card {\n padding: 16\n "
"children {\n component Label { text: @state(title) }\n }\n}\n\n"
"Error: line 2: expected ';' after property value"},
{"role": "assistant", "content":
"component Card {\n padding: 16;\n children {\n "
"component Label { text: @state(title); }\n }\n}"}
]}
Generate these by mutating valid examples into invalid ones with your parser's own error messages attached, then pairing the broken version with the original. Roughly a fifth of the dataset is a reasonable share, and it changes interactive behaviour more than any other single addition.
The training run is the easy part
import torch
from datasets import load_dataset
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import (AutoModelForCausalLM, AutoTokenizer,
BitsAndBytesConfig, TrainingArguments)
from trl import SFTTrainer
BASE = "<a current open-weights code model>"
quant = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tok = AutoTokenizer.from_pretrained(BASE)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
BASE, quantization_config=quant, device_map="auto",
torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)
peft_config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
trainer = SFTTrainer(
model=model,
train_dataset=load_dataset("json", data_files="train.jsonl", split="train"),
eval_dataset=load_dataset("json", data_files="eval.jsonl", split="train"),
peft_config=peft_config,
max_seq_length=2048,
tokenizer=tok,
args=TrainingArguments(
output_dir="./adapter",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
num_train_epochs=2, # not max_steps — see below
bf16=True,
optim="paged_adamw_8bit",
eval_strategy="steps",
eval_steps=50,
save_steps=50,
load_best_model_at_end=True, # the run will overfit; keep the best
),
)
trainer.train()
trainer.save_model("./adapter")
Two settings there differ from the usual example and both matter. An
evaluation set with load_best_model_at_end means you keep the
checkpoint that generalised rather than the one that finished — small
DSL datasets overfit quickly, and the final checkpoint is frequently worse than
one from halfway. And epochs rather than a fixed step count, so the run scales
with your dataset instead of silently under-training when it grows.
Evaluate on generation, not on loss
Training loss tells you the model is memorising. It does not tell you whether the output compiles.
held_out = json.loads(Path("eval_prompts.json").read_text())
compiles = correct = 0
for case in held_out:
out = generate(case["prompt"], temperature=0.1)
ok, _ = parses(out)
compiles += ok
if ok and semantically_matches(out, case["expected"]):
correct += 1
n = len(held_out)
print(f"parse rate {compiles/n:.1%}") # the floor
print(f"semantic match {correct/n:.1%}") # what you actually want
Report both against the base model with in-context examples, on the same held-out set. That is the comparison that justifies the work, and it is the one teams skip in favour of a loss curve that looks reassuring.
Constrain decoding anyway
A fine-tuned model produces valid syntax most of the time. Grammar-constrained decoding makes invalid syntax unreachable. These are complements, not alternatives, and running both is the arrangement worth deploying.
# The fine-tune supplies idiom and structure.
# The grammar supplies the guarantee.
python -m vllm.entrypoints.openai.api_server \
--model ./merged \
--port 8000 \
--max-model-len 4096 \
--guided-decoding-backend outlines
With both in place the parse rate is 100% by construction, and the fine-tune is doing what it is actually good at — producing idiomatic, conventional DSL rather than merely parseable DSL.
The recurring cost nobody budgets
A model fine-tuned on your DSL as it stood in March keeps producing March's syntax after you change the language. Nothing errors; output quietly drifts from current convention.
- Retrain on DSL change. Wire it to the same trigger as your grammar version. If the grammar bumps, the adapter is stale.
- Keep the pipeline runnable by someone else. The person who built it will leave. A one-command retrain that a colleague can run is the difference between a maintained model and an abandoned one.
- Version adapters against grammar versions. Serve the adapter that matches the grammar in use, and make a mismatch loud.
- Re-run the base-model comparison each cycle. Base models improve. The fine-tune that won six months ago may no longer be worth its maintenance.
That last point is the one to take seriously. This whole pipeline is worth building when it wins by a margin that justifies the recurring cost — and that margin shrinks over time, on its own, without anything going wrong.
Build the validator before the training script and run every generated pair through your real compiler. Expand seeds by AST mutation rather than by asking a model, and reserve about a fifth of the set for error-correction pairs. Evaluate on parse rate and semantic match against the base model with in-context examples, keep the best checkpoint rather than the last, and pair the fine-tune with grammar-constrained decoding. Then budget the retraining, because the language will move.