Evals

About me

Live in NLR with my wife and son
Currently a data engineer on the DSML team @ Numerator
All opinions are my own
Connect on LinkedIn!
Send me an email!

Who this talk is for

01Anybody interested in how model progress is tracked
02Folks building software where AI plays a critical role
03Creatives who want to elicit out-of-distribution behavior from LLMs

A hibiscus flower painted with code, next to the sketch.js source that drew it

What are evals?

Also known as testing and benchmarking

Evals involve the following:

Model or “agent”

Environment

Could be just their own context
May be a docker container, sandbox, etc

Grader

(code, LLM judge, human)

Agentic terminal coding

Terminal-Bench 4.0

Positions approximated from the source figure. Cost uses a logarithmic scale. [2] Source

Testing & design

Tests shape the software

Testing is a selection pressure during the build.

An executable check makes intent concrete
Hard-to-test behavior pushes you toward smaller parts, clearer interfaces, and isolated dependencies
Run, change, repeat: feedback shapes the implementation while you’re building it

The evals we build today exert the same pressure.

Evals in AI powered software

Writing evals helps you

01

Define the performance axes you care about

02

Break up the problem into smaller ideas

03

Understand where complexity helps and hurts

04

Build more malleable systems

Still not perfect

QR code linking to Anthropic's Demystifying evals for AI agents — Demystifying evals (Anthropic)

The harness matters

Evaluate RAG retrieval

Semantic similarity is an engineering decision. Back it with data.

Embedding space
Test paraphrases, domain terms, exact identifiers, and chunk boundaries against labeled relevance
Vector search
Compare approximate search with exact neighbors. Check filters and missed matches
Keyword baseline
Use the same queries and corpus. Compare relevance, p95 latency (including query embedding), and cost

Choose semantic, keyword, or hybrid retrieval with data. Then measure answer quality across the complete RAG pipeline.

Optimization

Hillclimbing

Hillclimbing starts with a metric

A number you can score after every change

AccuracyPass rateCost per ticketTrigger rate

Hillclimbing

The hillclimbing loop

One change per round. Keep improvements; revert overfit or regression.

Choose one thing to edit, such as the prompt. Hold the model and the rest of the harness fixed.
Repeat baseline runs to understand normal score variation.
Read failures on the train split, form a hypothesis, and propose one change.
Score the change on train and held-out test. Keep it when both improve beyond the noise.
Revert regressions or gains that do not generalize. Repeat, keeping test examples out of the analyzer.

[9] Automating eval design and hillclimbing

Hillclimbing

GEPA (prompt optimization)

A hillclimbing loop for prompts, and one of the most effective

01

Have an LLM look at the failed eval run grader results + context

02

Reflect on what went wrong

03

Update the prompt

04

Iterate

Hillclimbing · IFBench / Qwen3 8B

GEPA (prompt optimization)

Lines / circles: validation (left). Stars: test set (right).
Curves and points approximated from the source figure. [4] GEPA · [6] IFBench

GEPA

Big jump in under 1K rollouts

GRPO

Still lower after 24K

Hillclimbing

There’s a skill for that now

claude

>/claude-api build-eval

>/claude-api hillclimb

Your competitors are already running evals

[9] claude.dev · Sep 28, 2026 [10] claude-api skill

Hillclimbing

Hillclimbing for cost reduction

44 support tickets · 14 held out

Adopted hillclimbing path (train split)
Step	Decision accuracy	Cost per ticket
Opus 4.8, high effort (baseline)	74.4%	4.6¢
Opus 5.5, low effort, audited prompt	87.8%	1.9¢
Sonnet 5, low effort	88.9%	1¢
Sonnet 5, low effort, improved prompt	98.9%	about 1¢

74.4%→98.9%

Accuracy

4.6¢→~1¢

Cost per ticket

78.6%→90.5%

Held-out accuracy

[9] Automating eval design and hillclimbing

Optimization

Supervised fine tuning

Still used extensively
Evals produce training data
Dataset of example inputs and outputs

Some of the issues you may run into

General intelligence drops off a cliff
Overfitting concerns and training distribution issues

Optimization

Reinforcement learning

Used for adapting a model to an environment
Difficult to get right
Can be extremely effective for agents

Optimization

RL at Ramp

Ask a question, an agent executes tools over a spreadsheet to find the answer.

Well thought out harness design
Synthetic eval set was generated
More accurate than Haiku

Eval leaderboard
Model	Accuracy	Time vs Haiku
Qwen3.5-35B-A3B (Fast Ask)	66.25%	1.05×
Claude Opus 4.6	61.88%	1.44×
Claude Sonnet 4.6	59.38%	1.60×
Qwen3.5-35B-A3B (base)	56.25%	1.34×
Claude Haiku 4.5	51.88%	1.00×

Accuracy: exact task completion. Time: mean elapsed time per rollout, relative to Haiku 4.5. All models evaluated in non-thinking mode.

[7] Ramp RL

Thanks!

If your team is

Needing help building evals
Wanting a bit of guidance on a new project
Looking into training something

Send me a message→

QR code that opens a new email to john@ouachitalabs.com — EmailSend me a message!

Appendix

01Flower arthttps://surya.website/rling-qwen-to-paint-with-code
02Terminal-Bench 4.0https://www.anthropic.com/claude-opus-5-5
03Anthropic - Demystifying Evalshttps://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
04GEPAhttps://arxiv.org/html/2507.19457v2
05HotpotQAhttps://arxiv.org/pdf/1809.09600
06IFBenchhttps://github.com/allenai/IFBench
07Ramp RLhttps://labs.ramp.com/research/prime-rl-post-training/
08Information retrieval evaluationhttps://nlp.stanford.edu/IR-book/html/htmledition/evaluation-in-information-retrieval-1.html
09Automating eval design and hillclimbinghttps://claude.dev/blog/automating-eval-design-and-hillclimbing/
10claude-api skillhttps://github.com/anthropics/skills/tree/main/skills/claude-api
11Test-driven developmenthttps://martinfowler.com/bliki/TestDrivenDevelopment.html