The experiment ran. Here is the honest scoreboard first, because that is the deal with pre-registered predictions: you publish the result you got, not the one you wanted.
| Model | CUAD AUPR (full test split) |
|---|---|
| Claude Opus 4.8 | 0.561 |
| Claude Opus 4.6 | 0.498 |
| GPT-5.2 | 0.423 |
| GPT-4o | 0.421 |
| nightwing v2 (extractive) | 0.389 |
| nightwing v1 (generative) | 0.291 |
0.389 is not 0.50. The prediction missed and I will get to why. But look at the two nightwing rows. Same base model. Same training data. Same one-epoch budget, about thirty dollars of GPU time. The only difference is what I asked the model to do with the contract.
Version one treated contract review the way everyone treats LLMs: as text generation. Show the model a contract chunk and a question, train it to write out the answer clause. Version two treats it as what the benchmark actually is: span extraction. The model does not retype the clause. It points at where the clause starts and where it ends.
That is the entire change. Pointing instead of retyping. Ten points.
Generation was paying a double tax. The model had to reproduce long legal spans verbatim, where one paraphrased word breaks the span-overlap scoring, and it had to do that from a decoder trained to be fluent rather than faithful. Extraction removes both failure modes at once: the span it selects is by construction verbatim, and the scoring threshold stops punishing honest reformulation.
The headline gap to Claude is still real. What moved is everything underneath it. Against the GPT frontier models, the specialist now wins a majority of categories: 25 of 40 against GPT-4o, 22 of 40 against GPT-5.2. In version one it won 11. It beats all four frontier models simultaneously in six categories now, up from two, and the margins got silly: Agreement Date 0.829 against a best frontier score of 0.168.
The part I did not expect: the reasoning-heavy categories recovered too. In version one I concluded that cheap specialists learn annotation conventions while real semantics stay with the giants, and the collapse categories backed that up. Version two took Covenant Not To Sue from 0.107 to 0.539, Joint IP Ownership from 0.184 to 0.500, Anti-Assignment from 0.224 to 0.497. Those are not date-format conventions. The generative framing was not just taxing the easy categories; it was hiding capability the model already had.
Because the pipeline was cheap, I trained the identical recipe at four sizes. Dev scores: 0.5B gets 0.284, 1.5B gets 0.289, 7B gets 0.300, 14B gets 0.303.
Twenty-eight times more parameters, two points. The half-billion-parameter model, which trains overnight on a consumer laptop, already matches what my 14B generative version needed an A100 to reach. If you read the first essay, this should rhyme: GPT-4o and GPT-5.2, a full frontier generation apart, tied on this task to within two thousandths of a point. Same lesson from the other side of the fence. On convention-bound extraction work, scale is almost decorative. Framing is load-bearing.
The 0.50 target was not pulled from air. A 0.9B extractive model from the original CUAD paper scores about 0.47 published. My 14B extractive scored 0.389. The difference is that the published number comes from the full training recipe: multiple epochs, every negative example, tuned hyperparameters. Mine was deliberately lean, one epoch, negatives subsampled two-to-one, first-guess learning rate, zero tuning, because the question this round was about framing, not recipe, and I kept every other variable at its cheapest setting.
That is a real miss against a target I set knowing the reference point, so it counts as a miss. But it localizes the remaining gap precisely: not architecture, not scale, recipe. Version three changes only that, and the prediction carries forward with better information behind it.
The cheapest lever in this entire series was not a bigger model, more data, or more GPU hours. It was asking the model for the right kind of output. Before you benchmark vendors against each other, benchmark the task framing against itself: the same model, asked two different ways, just produced a ten-point swing that dwarfs the gap between two frontier generations.
And a note on predictions. Writing 0.50 down in public before spending made this essay harder to write and easier to trust. My model lost to my own forecast, in print, and the experiment is still the most useful thing this project has produced. Vendor decks do not work this way. That is worth remembering when you read them.
Everything is public and re-scorable at github.com/ashish24142/nightwing: raw predictions for every model at every size, the full run journal including the four dependency failures and the two silent bugs that free local testing caught before they could burn the cloud budget, and the exact runbook that produced the numbers. The trained model itself is on Hugging Face if you want to run it on your own contracts.
I’m Ashish Kumar Singh, an engineering leader building enterprise LLM systems, and I still write the code. The nightwing code and every prediction file are on GitHub; the trained model is on Hugging Face; the paper is at doi.org/10.5281/zenodo.21530247.
Views my own. Not legal advice, and the model still isn’t a lawyer.
]]>The honest answer: not victory. Something more interesting. The specialist could not lay a hand on the reigning giant, Claude. But against GPT-5.2 it won 11 of 40 categories outright, and against GPT-4o, another 11.
The benchmark is CUAD, 510 real commercial contracts annotated by lawyers across 41 clause categories: governing law, non-compete, IP assignment, termination, the clauses a lawyer scans for first. The task: given a contract and a category, extract the exact supporting text, or say the clause is absent. Scoring uses CUAD’s official metric (area under the precision-recall curve), not a metric I invented.
I measured four frontier models first, through one identical pipeline: Claude Opus 4.8, Claude Opus 4.6, GPT-5.2, and GPT-4o. Then I fine-tuned Qwen3-14B, an open Apache-licensed model, with LoRA on CUAD’s training split. One A100 for an afternoon. About $40 of GPU time.
Three rules were fixed before any training started, because a benchmark you can quietly bend is worthless:
First, the test set is untouchable. A script proves zero overlap between training and test contracts, in code, before every run. Checkpoint selection happened on a held-out dev split. The test set was scored exactly once.
Second, the thresholds were pre-committed. Within 4 points of frontier: green. 5 to 8 points below: yellow. Worse than that: red. Written down before the GPU was rented.
Third, everything is auditable. Every model’s raw predictions are committed to the public repo. You can re-score every number in this essay yourself.
| Model | CUAD AUPR (full test split) | Cost per evaluation |
|---|---|---|
| Claude Opus 4.8 | 0.561 | ~$165 |
| Claude Opus 4.6 | 0.498 | ~$154 |
| GPT-5.2 | 0.423 | ~$13 |
| GPT-4o | 0.421 | ~$58 |
| nightwing-14b (my fine-tune) | 0.291 | ~$0 |
Red band. Twenty-seven points below the best frontier model. If I were selling you a fine-tuning success story, this essay would end here and you should close the tab.
Look at the two GPT rows before you scroll on. GPT-4o scores 0.421. GPT-5.2, a full model generation later, scores 0.423. Two thousandths of a point of improvement across a generation of frontier progress, on a task worth billions to the legal industry. Hold that thought.
The per-category table is where it gets interesting.
| Category | nightwing-14b | Best frontier | GPT-5.2 |
|---|---|---|---|
| Agreement Date | 0.687 | 0.168 | 0.054 |
| Effective Date | 0.369 | 0.105 | 0.070 |
Not close wins. On Agreement Date the fine-tune scored four times the best frontier model, and twelve times GPT-5.2. Against GPT-5.2 specifically, the specialist won 11 of 40 categories outright, including Document Name (0.711 vs 0.471) and Third Party Beneficiary (0.585 vs 0.355).
Category-by-category against GPT-5.2 alone, the $40 model went 11 for 40:
| Category | nightwing-14b | GPT-5.2 |
|---|---|---|
| Agreement Date | 0.687 | 0.054 |
| Effective Date | 0.369 | 0.070 |
| Document Name | 0.711 | 0.471 |
| Third Party Beneficiary | 0.585 | 0.355 |
| Expiration Date | 0.633 | 0.589 |
(Plus six narrower wins; GPT-5.2 still takes the overall score, 0.423 to 0.291.) More than a quarter of the categories, against a frontier model, for the price of a nice dinner. And the same is true of GPT-4o: 11 of 40 categories again, with even larger margins (Agreement Date 0.687 vs 0.079, Third Party Beneficiary 0.585 vs 0.102). The gap between “frontier” and “specialist” is not a wall. It is category-shaped.
Now back to that thought I asked you to hold. GPT-4o and GPT-5.2, a generation apart, land two thousandths of a point apart on this task. A full cycle of frontier scaling moved the needle essentially zero. That is the tell: contract clause extraction is not bottlenecked by reasoning power that more scale would supply. It is bottlenecked by knowing where a lawyer would draw the span, which is a convention you learn from examples, not a capability you scale into. That is exactly the seam a $40 specialist reaches through.
Why dates, of all things? Because CUAD scores by span overlap against what human annotators marked, and date questions are annotation-convention questions. When you ask for an agreement date, where does the “correct” span start and end? Just the date? The whole sentence? The clause naming the parties and the date? Frontier models answer correctly in the colloquial sense and still lose, because they draw different boundaries than the annotators did. The fine-tune never had to guess. It learned the convention from 368 contracts of examples. No amount of frontier-model scale fixes this, because the information simply is not in the prompt.
And where did the specialist collapse? The categories that need actual reasoning over meaning. Identifying all parties to a contract: 0.247 vs Claude’s 0.954. Source code escrow, a rare clause it barely saw in training: 0.100 vs 0.800. The pattern across all 40 categories is clean. Convention-driven fields favor the cheap specialist. Semantics-driven fields favor scale.
The interesting number is not 0.291. It is the shape of the table. A model that
costs nothing per run beat three frontier models simultaneously on the fields
where learned convention matters, and lost everywhere reasoning matters. Nobody
running “which model should we use” evaluations at the whole-benchmark level
would ever see this. Category-level measurement is the whole game. This is the
same lesson as signoff, from the
other direction: aggregate scores hide exactly the behavior differences you are
accountable for.
There is also a warning here about benchmark readings in vendor decks. My RED result and my two knockouts come from the same table. Pick your rows carefully enough and either story sells.
The 27-point gap is probably not a capacity limit. A 0.9B-parameter extractive model from the original CUAD paper scores about 0.47 published, nearly GPT-5.2 territory, at one-fifteenth the size of my model. The difference is task framing: it points at spans instead of generating them. Version two of this experiment puts an extractive head on the 14B backbone. Pre-registered prediction, written down before spending: it clears 0.50. The repo, the run journal with every failure along the way, and all predictions are public at github.com/ashish24142/nightwing.
Update, July 18: version two ran. The prediction missed, and the framing change still gained ten points. The full result is in the sequel.
I’m Ashish Kumar Singh, an engineering leader building enterprise LLM systems, and I still write the code. The nightwing code and every prediction file are on GitHub; the trained model is on Hugging Face; the paper is at doi.org/10.5281/zenodo.21530247.
Views my own. Not legal advice, and the model isn’t a lawyer either.
]]>The obvious question from leadership was simple: “Is the new model safe to deploy?”
The honest answer, at that moment, was: “We don’t know, and we don’t even have a way of knowing.” This essay is about the framework we built to answer that question properly, and why the standard approach (run a few examples, eyeball the outputs, ship it) would have quietly hurt us.
Our system extracts dozens of attributes from documents: governing law, effective dates, party names, monetary amounts, renewal terms. A migration “spot check” looks like this: run twenty documents through the new model, diff the answers, see mostly agreement, call it a day.
Three problems.
First, the sample is too small to see rare regressions. If the new model breaks one attribute on a few percent of documents, twenty samples will miss it more often than not. And a few percent of a production document flow is a lot of wrong answers with a compliance label on them.
Second, “the same” is not a well-defined concept for LLM output. Is
$250,000,000 the same as 250000000.00 USD? Is State of New York the same
as New York? Naive exact-match scoring will tell you the new model “failed” on
a large share of cases that any human reviewer would score as identical. Scoring
is a pipeline, not a string comparison.
Third, and this is the one almost everyone misses: the old model doesn’t agree with itself. Run the same prompt on the same document with the same model five times, and on some attributes you get different answers on different runs. On our worst prompts, repeated runs disagreed with each other not occasionally but routinely. It’s the kind of instability UAT teams file as “the system changed its mind on refresh.” A single-run comparison between old and new models is therefore partly measuring luck. You cannot tell a regression from a coin flip.
That third problem has a name, nondeterminism, and it reshaped the whole validation design. Before you can compare two models, you have to measure how unstable each one is on its own.
Here is the shape of what we ended up building. None of it is exotic; the value is in the discipline of doing all of it, every time.
1. A frozen validation suite. Real (anonymized) documents with ground-truth answers for every attribute, agreed with the business owners before migration testing started. If you negotiate what “correct” means after you’ve seen the new model’s outputs, you will negotiate your way into shipping it.
2. N runs per prompt per model, not one. We ran every case multiple times on the old model and the new. From those runs, two numbers per case:
The flip rate is the diagnostic gold. A case that changed answers between models but is unstable on the old model too isn’t a migration regression. It’s a prompt that was never reliable to begin with. Our “regressions” bucket shrank substantially once we could separate the two, and the unstable-prompt bucket became its own workstream (fix the prompt, not the model).
3. Layered matching instead of exact match. Each answer is scored through
escalating layers: exact match first, then normalized (case, whitespace,
punctuation), then numeric equivalence ($1,000 vs 1000), and finally a
semantic or LLM-judge comparison for the genuinely fuzzy cases. Every match
records which layer passed, and the judge layer is used sparingly and audited
by humans on a sample, because “a model said the model is right” is not, by
itself, evidence.
4. A paired statistical test, not a vibes delta. Suppose the new model scores 94% on your suite and the old one 93%. Improvement, or noise? With per-case paired results you can run an exact sign test on the discordant pairs and get a p-value instead of an argument. Our rule: the migration fails sign-off only if the new model is worse and the regression is statistically significant. Everything else is either a pass or a named, bounded exception.
5. A report a risk committee can read. The deliverable was never a Jupyter notebook. It was a short report: verdict up front, per-attribute table (accuracy old/new, flip rates, match layers used), the list of every regressed case with its documents, and the methodology in an appendix. The methodology section is what turns “the team says it’s fine” into evidence, the thing you can hand an auditor when they ask, a year later, why you approved the swap.
Once it was in place, the migration verdict came back genuinely mixed, which is exactly why the framework matters:
That last bucket is the entire ROI of the framework. One real regression caught before deployment pays for the whole validation effort. In domains like ours, several times over.
I’ve generalized the core of this framework into signoff, an open-source CLI that runs a YAML-defined suite on two models, N runs each, scores through layered matching, computes flip rates and the sign test, and emits an HTML + JSON report. It’s vendor-neutral by design: it judges an OpenAI to Anthropic migration with exactly the same rigor as the reverse.
If you’re facing a forced migration this year (and with current model deprecation schedules, you probably are), the checklist version of this essay is:
“Is the new model safe to deploy?” deserves a better answer than “probably.” It deserves a sign-off.
I’m Ashish Kumar Singh, an engineering leader building enterprise LLM systems (contract intelligence today, a decade of clinical AI before that), and I still write the code. Migration validation, prompt regression testing, and document-extraction evals are my daily work. This site is about migration assurance; signoff is its toolkit. All views are my own, drawn from personal experiments and open-source work, never from client or employer data.
]]>