Kaitrin Cox / Selected work
LLM as Judge
AI Orchestration. Part of Copilot’s offline eval suite. It grades what an answer says as well as how it’s structured, and it has rules for when it’s allowed to run.
Roles: Backend.
- Role: Built the judge, and co-wrote the rules for when it runs
- Team: AI Orchestration, Zapier
- Shipped: September 2026
What I built
Each eval case can list what a good answer should do. The judge reads Copilot’s final reply and the actions it took, then scores each of those objectives and says why. If an objective doesn’t apply to a run, it says so instead of guessing.
It runs on a cheap model that you pick in configuration, and each grading is one call with a limit on how much context it sees. That keeps it quick.
Scores are averaged across repeated runs, skipping the runs where the judge abstained. If any part of grading fails, we throw out the partial result so nobody reads it as a full one.
When it’s allowed to run
I co-wrote the team’s decision on when the judge can run at all. Deterministic checks still decide pass or fail, and the judge only covers cases where there’s no reliable structural check. Its scores are report-only until we’ve tested it against a labeled set of good and bad examples, so a judge we haven’t verified can’t block a release. That also meant we could ship the code before any live case used it.
It also won’t grade questions about a user’s plan or entitlements, because the user context it sees isn’t a reliable source for those.
What happened
In review I kept the scope small. I cut one sample case and spent two rounds fixing edge cases. Now the team can grade answers for quality without a judge nobody has verified deciding what ships.
Full text: /llms-full.txt. Contact: hello@kaitrin.dev.