Add precision and recall metrics to the multi-turn evaluation (I2) #10

Merged
weichen merged 2 commits from feature/agent-stack-i2 into main 2026-10-07 11:05:59 +00:00
Owner

I2 增量二:precision/recall 多轮评测指标

改动

  • CorpusMetrics + MetricsOf:按意图标签计 TP/FP/FN——expectation_hit 算命中(修了自己漏的口径);错配同时计假阳与假阴(期望的没拿到 + 拿了错的)。分母为零时返回未定义(调用方展示 -)。
  • intentctl eval 指标输出:汇总行后新增 指标: precision=… recall=…(TP/FP/FN)(确定性重跑计算,不进通过判定)。
  • 多轮夹具扩编到 11 例(≥ 现有 corpus.json 的 11):链式预期 SetSlot+Start、歧义后决胜、满槽 handler 动作、fallback、tie 歧义、直接完成后回落、缺槽后命中别的意图、澄清提问断言。

验证(I2 验收)

  • 「eval 对不少于现有语料规模的多轮夹具给出指标」:intentctl eval corpus_multiturn.json → 总计 11 用例:11 通过 + 指标: precision=1.000 recall=1.000(TP=12 FP=0 FN=0)。
  • 「分类器返回未知 ID 时保持 Ambiguous」:既有 validateRefined 三测(recognizer_test.go:188/238/269)绿。
  • 全套件 6 包 ok。

主任务:Dev 仓 feature/agent-stack-i2。

# I2 增量二:precision/recall 多轮评测指标 ## 改动 - **`CorpusMetrics` + `MetricsOf`**:按意图标签计 TP/FP/FN——expectation_hit 算命中(修了自己漏的口径);错配同时计假阳与假阴(期望的没拿到 + 拿了错的)。分母为零时返回未定义(调用方展示 `-`)。 - **`intentctl eval` 指标输出**:汇总行后新增 `指标: precision=… recall=…(TP/FP/FN)`(确定性重跑计算,不进通过判定)。 - **多轮夹具扩编到 11 例**(≥ 现有 corpus.json 的 11):链式预期 SetSlot+Start、歧义后决胜、满槽 handler 动作、fallback、tie 歧义、直接完成后回落、缺槽后命中别的意图、澄清提问断言。 ## 验证(I2 验收) - 「eval 对不少于现有语料规模的多轮夹具给出指标」:`intentctl eval corpus_multiturn.json` → `总计 11 用例:11 通过` + `指标: precision=1.000 recall=1.000(TP=12 FP=0 FN=0)`。 - 「分类器返回未知 ID 时保持 Ambiguous」:既有 validateRefined 三测(recognizer_test.go:188/238/269)绿。 - 全套件 6 包 ok。 主任务:Dev 仓 `feature/agent-stack-i2`。
weichen added 1 commit 2026-10-07 11:02:29 +00:00
Add precision and recall metrics to the multi-turn evaluation
ci / vet-and-test (pull_request) Successful in 16s
ci / docker-checkout-hostgateway (pull_request) Successful in 0s
db79b535ec
Why: I2 acceptance - the evaluator must emit metrics over a
multiturn fixture no smaller than the existing corpus, not just
print branch names.

Evidence: go vet clean; full suite ok; intentctl eval prints
precision=1.000 recall=1.000 (TP=12 FP=0 FN=0) over the 11-case
multiturn fixture; MetricsOf unit test pins TP/FP/FN counting
including wrong-match counting as both.

Impact: CorpusMetrics + MetricsOf (expectation_hit counts as a
hit); intentctl eval prints the metrics line after the summary;
corpus_multiturn.json expanded to 11 cases (chained expectation,
tie-break, ambiguous, fallback, handler action, clarify paths).

Refs: Documents proposal 2026-10-07-agent-stack-n1q1i1-plans.md I2
weichen added 1 commit 2026-10-07 11:02:30 +00:00
Add precision and recall metrics to the multi-turn evaluation
ci / vet-and-test (pull_request) Successful in 16s
ci / docker-checkout-hostgateway (pull_request) Successful in 0s
db79b535ec
Why: I2 acceptance - the evaluator must emit metrics over a
multiturn fixture no smaller than the existing corpus, not just
print branch names.

Evidence: go vet clean; full suite ok; intentctl eval prints
precision=1.000 recall=1.000 (TP=12 FP=0 FN=0) over the 11-case
multiturn fixture; MetricsOf unit test pins TP/FP/FN counting
including wrong-match counting as both.

Impact: CorpusMetrics + MetricsOf (expectation_hit counts as a
hit); intentctl eval prints the metrics line after the summary;
corpus_multiturn.json expanded to 11 cases (chained expectation,
tie-break, ambiguous, fallback, handler action, clarify paths).

Refs: Documents proposal 2026-10-07-agent-stack-n1q1i1-plans.md I2
weichen added 1 commit 2026-10-07 11:05:29 +00:00
Merge origin/main into the I2 branch
ci / vet-and-test (pull_request) Successful in 17s
ci / docker-checkout-hostgateway (pull_request) Successful in 1s
ab55b54e9b
weichen merged commit 343b1e6d9a into main 2026-10-07 11:05:59 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-Intent#10