Add the multi-turn eval format and self-describing corpora (I1 WP6) #8

Merged
weichen merged 1 commits from feature/agent-stack-i1 into main 2026-10-07 08:22:33 +00:00
Owner

I1 WP6:eval v2 多轮格式与语料自声明

改动

  • 新 intent/corpus.go:CorpusCase(v1 单轮/v2 多轮共用,v1 自动包装单轮)+ CorpusTurn(逐轮分支/top/命令期望)+ CommandExpect(只断言写出的字段)+ RunCorpusCase——链式预期:上一轮 Top 有必填缺失/非法槽位时下轮挂起该意图(对应产品「带槽启动后追问」),使 SetSlot+Start 组合在评测中可到达。
  • intentctl eval 消费 v2:多轮用例逐轮执行、差异定位到轮次与字段;v1 输出形态不变。
  • 语料自声明(修历史欠账):corpus 可声明 builtin_specs / guard_domains / protected_domains——guard_config_query 在 intentctl 下自始失败(guard spec 只存在于测试内置件,证明:HEAD~5 上同样 1/10 红)。声明后 11/11 全绿。
  • 验收载体:report.generate spec 入 demo registry + corpus_multiturn.json 两轮用例(turn1 Start+Clarify → turn2 SetSlot+Start)通过。

验证

  • go vet ./... 通过;go test -count=1 ./... 6 包全 ok(新测:多轮语料执行、v1 包装等价、命令断言差异定位)。
  • 烟测:intentctl eval v1 11/11、v2 1/1。

主任务:Dev 仓 feature/agent-stack-i1。

# I1 WP6:eval v2 多轮格式与语料自声明 ## 改动 - **新 `intent/corpus.go`**:`CorpusCase`(v1 单轮/v2 多轮共用,v1 自动包装单轮)+ `CorpusTurn`(逐轮分支/top/命令期望)+ `CommandExpect`(只断言写出的字段)+ `RunCorpusCase`——**链式预期**:上一轮 Top 有必填缺失/非法槽位时下轮挂起该意图(对应产品「带槽启动后追问」),使 SetSlot+Start 组合在评测中可到达。 - **`intentctl eval` 消费 v2**:多轮用例逐轮执行、差异定位到轮次与字段;v1 输出形态不变。 - **语料自声明**(修历史欠账):corpus 可声明 `builtin_specs` / `guard_domains` / `protected_domains`——`guard_config_query` 在 intentctl 下自始失败(guard spec 只存在于测试内置件,证明:HEAD~5 上同样 1/10 红)。声明后 **11/11 全绿**。 - 验收载体:`report.generate` spec 入 demo registry + `corpus_multiturn.json` 两轮用例(turn1 Start+Clarify → turn2 SetSlot+Start)通过。 ## 验证 - `go vet ./...` 通过;`go test -count=1 ./...` 6 包全 ok(新测:多轮语料执行、v1 包装等价、命令断言差异定位)。 - 烟测:`intentctl eval` v1 11/11、v2 1/1。 主任务:Dev 仓 `feature/agent-stack-i1`。
weichen added 1 commit 2026-10-07 08:22:03 +00:00
Add the multi-turn eval format and self-describing corpora
ci / vet-and-test (pull_request) Successful in 18s
ci / docker-checkout-hostgateway (pull_request) Successful in 0s
1d0fcbd1ba
Why: I1 WP6 - eval needs per-turn command expectations to carry the
SetSlot+Start acceptance case, and corpora must reproduce standalone
in intentctl (the guard case has been red there since before I1
because the guard spec only existed as a test builtin).

Evidence: go vet clean; go test ./... all ok; intentctl eval passes
11/11 on the v1 corpus (guard case fixed via self-declaration) and
1/1 on the new two-turn SetSlot+Start corpus.

Impact: new intent/corpus.go (CorpusCase/CorpusTurn/CommandExpect,
RunCorpusCase with chained expectations); intentctl eval consumes
turns and corpus-declared builtin_specs and domains; report.generate
spec and corpus_multiturn.json fixtures added. v1 corpus format and
conformance unchanged.

Refs: Documents proposal 2026-10-07-agent-stack-n1q1i1-plans.md I1 WP6
weichen merged commit f57645c645 into main 2026-10-07 08:22:33 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-Intent#8