Fine-Tuning for an Internal DSL: The Case Is Narrower Than It Looks
A model with ten examples of your DSL in context frequently beats one fine-tuned on a thousand and given none. Establish that before anyone provisions a GPU.
The case for fine-tuning is narrower than it looks
Every organisation with an internal framework eventually asks whether to fine-tune a model on it. The reasoning is intuitive: the model has never seen our DSL, so teach it.
The reasoning is usually wrong, and the reason is worth understanding before anyone provisions a GPU. A model that has not seen your DSL and has ten examples of it in context frequently outperforms a fine-tuned model that has seen a thousand examples and none in context. In-context learning is remarkably strong for syntax, and fine-tuning is expensive, perishable and slow to iterate.
Try these first, in order
| Approach | Effort | Iteration | Try when |
|---|---|---|---|
| Examples in the context file | An hour | Instant | Always. This is the baseline. |
| Retrieval over your codebase | A few days | Minutes | The DSL is large or varied |
| Grammar-constrained decoding | Days | Fast | Output must parse, always |
| Fine-tuning | Weeks, recurring | Days per cycle | The three above measurably fail |
The order matters because each row is cheaper to abandon than the one below it. Teams that start at the bottom spend a quarter discovering that the top row would have worked.
What the top row looks like done properly
"Put examples in the context file" is done badly far more often than it fails. Done properly it means a small number of complete, canonical, current examples that cover the shapes people actually write.
## Our pipeline DSL
Canonical examples — follow these shapes exactly.
A simple transform:
```yaml
pipeline: daily_revenue
source: warehouse.orders
steps:
- filter: status == 'completed'
- aggregate:
by: [region, day]
metrics: {revenue: sum(amount)}
sink: reports.daily_revenue
```
With a join (note `using`, not `on` — we do not support `on`):
```yaml
pipeline: revenue_by_segment
source: warehouse.orders
steps:
- join:
with: warehouse.customers
using: customer_id
type: left
- aggregate: {by: [segment], metrics: {revenue: sum(amount)}}
sink: reports.revenue_by_segment
```
Rules the examples do not show:
- `sink` tables must already exist. The DSL never creates them.
- Every pipeline needs a unique `pipeline:` name — it is the job ID.
- No nested aggregations. Use two pipelines.
Two examples and three rules. Note the parenthetical in the second: naming the thing your DSL does not support is worth more than another example of what it does, because the model's prior is to reach for the conventional syntax it knows from elsewhere.
"We use using, not on" prevents a specific,
predictable error. A third positive example prevents nothing in particular.
When space is tight, spend it on the differences from what the model already
expects.
Measure before concluding it is not enough
Before escalating, establish what is actually failing. Write twenty representative tasks in your DSL, run them, and categorise the errors.
| Failure | What it indicates | Fix |
|---|---|---|
| Invalid syntax | Model does not know the grammar | Constrained decoding — not fine-tuning |
| Valid syntax, wrong construct | Missing examples of that shape | Add an example |
| Uses a feature you removed | Prior from a similar public DSL | A negative rule |
| Correct but unidiomatic | No canonical reference | Retrieval over your own repo |
| Fails only on large inputs | Context, not knowledge | Nothing fine-tuning will fix |
Only the fourth row makes a case that more exposure to your code would help, and retrieval delivers that without training anything. The first row is the one teams most often try to fix with fine-tuning, and it is the one where constrained decoding is strictly better — a grammar makes invalid output impossible rather than unlikely.
If you fine-tune anyway
Occasionally the case is real: a large DSL, high volume, measured failures that retrieval does not fix. Three things determine whether it is worth it.
Data quality dominates quantity. Two hundred correct, current, idiomatic examples beat five thousand scraped from a repository that includes three years of deprecated patterns. Your training set will teach the model whatever your codebase actually contains, including the mistakes.
It perishes. A model fine-tuned on your DSL as it was in March will keep producing March's syntax after you change it. The recurring cost is not the training run; it is the discipline of retraining every time the DSL evolves, forever, and the silent degradation when nobody does.
Evaluate against the cheap baseline. The comparison that matters is not "fine-tuned model versus base model with no help". It is "fine-tuned model versus base model with good examples in context". Teams routinely skip this and report an improvement that the examples alone would have delivered.
Condition A base model, no DSL context ← the misleading baseline
Condition B base model + examples in context ← the real baseline
Condition C base model + examples + retrieval
Condition D fine-tuned model + examples in context
Report D vs B. Anyone reporting D vs A is measuring the examples.
The retrieval option, briefly
Between examples and fine-tuning sits retrieval, and it is underused because it sounds more complicated than it is. For a DSL the useful form is narrow: an index over pipelines your team has already written and reviewed, queried by similarity to what is being asked for.
INDEX only reviewed, currently-deployed pipelines
(not drafts, not deleted ones, not examples
from a tutorial someone pasted in 2024)
RETRIEVE 3-5 nearest by task description
INJECT as additional examples, labelled as
"existing pipelines that do something similar"
The curation is the whole job. An index over everything in the repository teaches the model your deprecated patterns with the same weight as your current ones. An index over forty hand-picked pipelines teaches it what good looks like, and it is forty files, which someone can actually review.
If the case is real, build it properly
Occasionally the comparison comes out the other way: a large DSL, high volume, failures that examples and retrieval genuinely do not fix. That is a legitimate outcome, and the pipeline has its own set of ways to waste a quarter — principally training on data nobody validated.
The implementation companion to this article covers that: compiler-backed dataset validation, expanding a seed set by AST mutation rather than by asking a model, error-correction pairs, and why you evaluate on parse rate rather than on a loss curve.
The arrangement most teams land on
Examples in the context file, retrieval over a curated set of canonical pipelines, and a validator that runs before anything is committed. No training, no GPUs, no retraining treadmill, and output that fails loudly when it is wrong.
The validator is the part worth insisting on regardless of approach. A DSL has a parser; run it. An agent that gets immediate structural feedback converges on valid output whether or not the model has ever seen your syntax.
In-context examples plus a validator solve most DSL problems, and cost an afternoon rather than a quarter. Categorise your actual failures before escalating — syntax errors want constrained decoding, unidiomatic output wants retrieval, and neither wants fine-tuning. If you do train, benchmark against the base model with good examples, not without.