Evidence-grade sign-off for model swaps and agent deployments.
Model deprecations force migrations on someone else's schedule. "We eyeballed ten outputs" is not evidence a regulated enterprise can take to a risk committee. This site is about migration assurance: the evals, statistics, and governance mapping that let you say "the new model is safe to deploy" and prove it.
Work
signoff, an open-source LLM migration
assurance harness: N-run, nondeterminism-aware A/B equivalence testing with layered matching
(exact, normalized, numeric, then LLM-judge), flip-rate statistics, and an audit-ready report.
Vendor-neutral by design.
nightwing, a reproducible
benchmark of frontier LLMs vs a cheap fine-tuned specialist on CUAD contract clause extraction,
scored on the official metric with every model's raw predictions published. Version two changed
one thing, the task framing, and gained ten points: the specialist now beats GPT-4o in 25 of 40
categories and GPT-5.2 in 22 of 40, still loses overall to Claude, and its pre-registered
prediction missed in public. Honest numbers over highlight reels. The trained model is on
Hugging Face.
Writing
- What ISO 42001 change control means when the 'change' is a model version bump August 24, 2026
- I changed one thing about my $40 contract-review model. It gained ten points. My prediction still missed. July 18, 2026
- I spent $40 fine-tuning a 14B model to beat frontier LLMs at contract review. It lost by 27 points. Except where it won. July 13, 2026
- Is the new model safe to deploy? A sign-off framework for LLM migrations July 11, 2026
About
Twelve years ago I trained my first ML model. Today I'm Ashish Kumar Singh, an engineering leader building the LLM systems enterprises trust with decisions worth millions: contract intelligence now, a decade of clinical AI in healthcare before that. 100+ models in production at 99% uptime. Fraud detection across 10 million insurance claims. Behaviour-change AI reaching 2 million people a day.
And I still write the code. Migration validation, prompt regression testing,
document-extraction evals: the unglamorous discipline that decides whether an AI system
survives contact with production. That discipline has a name, migration assurance,
and this site is where I write about it. signoff
is its open-source toolkit.
All views on this site are my own. I write from personal experiments and open-source work, never from client or employer data.