weichen/Agent-Intent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add precision and recall metrics to the multi-turn evaluation (I2) #10

Merged
weichen merged 2 commits from feature/agent-stack-i2 into main 2026-10-07 11:05:59 +00:00
Conversation 0 Commits 2 Files Changed 4
+324
2 Commits
Author SHA1 Message Date
weichen ab55b54e9b Merge origin/main into the I2 branch
ci / vet-and-test (pull_request) Successful in 17s
Details
ci / docker-checkout-hostgateway (pull_request) Successful in 1s
Details
2026-10-07 11:05:28 +00:00
weichen db79b535ec Add precision and recall metrics to the multi-turn evaluation
ci / vet-and-test (pull_request) Successful in 16s
Details
ci / docker-checkout-hostgateway (pull_request) Successful in 0s
Details 
Why: I2 acceptance - the evaluator must emit metrics over a
multiturn fixture no smaller than the existing corpus, not just
print branch names.

Evidence: go vet clean; full suite ok; intentctl eval prints
precision=1.000 recall=1.000 (TP=12 FP=0 FN=0) over the 11-case
multiturn fixture; MetricsOf unit test pins TP/FP/FN counting
including wrong-match counting as both.

Impact: CorpusMetrics + MetricsOf (expectation_hit counts as a
hit); intentctl eval prints the metrics line after the summary;
corpus_multiturn.json expanded to 11 cases (chained expectation,
tie-break, ambiguous, fallback, handler action, clarify paths).

Refs: Documents proposal 2026-10-07-agent-stack-n1q1i1-plans.md I2
 2026-10-07 11:02:28 +00:00