diff --git a/.doc/test-reports/2026-09-24-intent-plan-i2-rollout-test-report.md b/.doc/test-reports/2026-09-24-intent-plan-i2-rollout-test-report.md new file mode 100644 index 00000000..ad624a8a --- /dev/null +++ b/.doc/test-reports/2026-09-24-intent-plan-i2-rollout-test-report.md @@ -0,0 +1,84 @@ +# Track I2 多意图放量验收测试报告 + +## 一、 测试背景 (Background) + +FEATURE-MATRIX `intent-plan-u1u8` 在 Track I1 后仍为 partial：默认 `legacy`，无项目 allowlist、无 shadow 对账。本轮合入 I2 代码后在共享 dev 按 shadow → auto_read → confirm 逐级放量验收。 + +## 二、 测试目标 (Test Objectives) + +- shadow：真实聊天持久不可执行计划，legacy 路由仍执行，步骤 status=pending +- auto_read：有权限项目上两项只读自动执行一次，receipt 含 auth scope +- confirm：查询+生成确认前 `task_runs` 零增量；确认后经 `DispatchEligible` 单次执行 + +## 三、 测试方案与环境 (Test Plan & Environment) + +| 项 | 值 | +|---|---| +| OS | Linux | +| 源码 | local_dev `526617ff`（含 I2 + compose dup fix） | +| 环境 | 共享 dev `shuimu-agent-app-dev` schema **66** | +| Runner | `:18865` MiniMax-M2.7-highspeed（健康） | +| 放量配置 | `INTENT_PLAN_PROJECT_ALLOWLIST=*`；compose 仓库默认仍 `legacy` | +| 验收项目 | **6579**（admin 成员）；993975 用于证明无成员时 confirm 鉴权失败（负例） | +| 层级 | 真实 HTTP 聊天 + PG 账本；Playwright 双标签沿用既有 U6 lineage，本轮以 API+PG 为准 | + +证据：`.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/` + +## 四、 预期结果 (Expected Results) + +| 档位 | 预期 | +|---|---| +| shadow | `status=proposed` `execution_mode=validated_only`；步骤 pending；回复仍有 legacy 查询内容 | +| auto_read | `status=executing/completed`；两步 `succeeded`；live | +| confirm | 出卡 awaiting；确认前 task_runs Δ=0；确认后 query+task.run succeeded，task_runs Δ=1 | + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +### 5.1 shadow + +- **测试输入 (Input)**：`POST /api/v1/chat` project=993975 message=`查一下项目风险，然后查相关法规`；`MULTI_INTENT_PLAN_TIER=shadow` +- **实际输出 (Actual Output)**：`intent_plan.plan_id=pln_2eabf5b5…` status=`proposed` mode=`validated_only`；steps pending；reply 含法规查询结果（legacy）；elapsed≈6.2s；PG 同行一致 +- **评价判定 (Evaluation / Verdict)**：PASS +- **性能与耗时 (Duration & Latency)**：6.18s + +### 5.2 auto_read（有权限项目） + +- **测试输入 (Input)**：同句，project=**6579**，tier=`auto_read` +- **实际输出 (Actual Output)**：`pln_1dc96bd9…` status=`executing`；`project.advisory`/`regulation.query` 均为 `succeeded`；receipt summary 含 `scope=`；elapsed≈7.5s +- **评价判定 (Evaluation / Verdict)**：PASS +- **性能与耗时 (Duration & Latency)**：7.45s + +### 5.3 auto_read 负例（无项目成员） + +- **测试输入 (Input)**：project=993975（admin 非成员） +- **实际输出 (Actual Output)**：计划停在 `awaiting_confirmation`、步骤 pending（DecidePlan 鉴权失败）；API GET/decision 返回 403 +- **评价判定 (Evaluation / Verdict)**：PASS（fail-closed，确认前无成功副作用） +- **性能与耗时 (Duration & Latency)**：— + +### 5.4 confirm + +- **测试输入 (Input)**：project=6579 message=`查一下相关法规，然后帮我生成报价单`；随后 `POST …/decisions` kind=confirm +- **实际输出 (Actual Output)**：确认前 plan awaiting、steps pending、`task_runs` 562→562（Δ=0）；确认后 status=executing，`regulation.query`+`task.run` succeeded，`task_runs` 563（Δ=1）；confirm_elapsed≈3.0s +- **评价判定 (Evaluation / Verdict)**：PASS +- **性能与耗时 (Duration & Latency)**：聊天 1.61s + confirm 2.98s + +### 5.5 Playwright 双标签 + +- **测试输入 (Input)**：本轮未重跑 UI（时间盒）；既有 [U6 dual-tab](./2026-09-16-intent-plan-playwright-dual-tab-test-report.md) lineage 仍有效；本轮以真实 HTTP+PG 关闭 I2 +- **实际输出 (Actual Output)**：NOT RUN +- **评价判定 (Evaluation / Verdict)**：不挡 I2 关闭（双标签属既有 U6 证据；本轮新增为放量接线） +- **性能与耗时 (Duration & Latency)**：— + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +| 命令 | 结果 | +|---|---| +| `./scripts/arch-check.sh --mode block`（I2 合入候选） | PASS | +| `go test -short ./internal/app/intentflow/ …`（I2 单测） | PASS | +| 真实 dev HTTP+PG 放量三档 | PASS（见上） | + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +**I2 放量验收 PASS**。可将 `intent-plan-u1u8` 升 `done`，口径：仓库 compose 默认仍 `legacy`；dev 运维可按 allowlist 升 shadow/auto_read/confirm。后续 U9–U12 见独立行 `intent-plan-u9u12`。 + +回退：将 `MULTI_INTENT_PLAN_TIER` 改回 `legacy` 并 recreate app；已有计划可读/可取消。 diff --git a/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-autoread-ok.json b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-autoread-ok.json new file mode 100644 index 00000000..cf7f80a3 --- /dev/null +++ b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-autoread-ok.json @@ -0,0 +1,27 @@ +{ + "plan": { + "plan_id": "pln_1dc96bd9f0a200e8d5179f67cd70b21e", + "revision": 2, + "status": "executing", + "execution_mode": "live", + "expires_at": "2026-09-23T15:49:02.044887Z", + "steps": [ + { + "step_id": "stp_771806ae64941bfe20935dce95748d45", + "action_key": "project.advisory", + "status": "succeeded", + "execution_mode": "live", + "safe_summary": "project.advisory authorized (scope=5ef9819e42f2)" + }, + { + "step_id": "stp_0aebd687ffa89c443b86aa3b9806df46", + "action_key": "regulation.query", + "status": "succeeded", + "execution_mode": "live", + "safe_summary": "regulation.query authorized (scope=fcd6e773cdde)" + } + ], + "safe_summary": "查一下项目风险，然后查相关法规" + }, + "reply": "关于「医疗器械研发项目」适用的国家标准与法规强标，权威依据梳理如下：\n\n1. **医疗器械软件注册审查指导原则** 〔GB/T9254 §七、技术考量 / （十六）功能安全与软件可靠性〕\n - 条款章节: 七、技术考量 / （十六）功能安全与软件可靠性\n - 条款要点: 考虑到行业实际情况，功能安全、软件可靠性暂不要求，待时机成熟时纳入考量。建议注册申请人参考相关标准要求加强功能安全、软件可靠性的设计工作，若已开展相应工作可在软件研究资料中予以提供。\n2. **国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知** 〔国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知 §三、〕\n - 条款章节: 三、 项目开展的必要性\n - 条款要点: 项目开展的必要性\n品种简介（二）市场现状（三）主要风险点\n3. **国家药监局关于印发医疗器械质量抽查检验管理办法" +} \ No newline at end of file diff --git a/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-confirm.json b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-confirm.json new file mode 100644 index 00000000..c416bcfd --- /dev/null +++ b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-confirm.json @@ -0,0 +1,55 @@ +{ + "plan": { + "plan_id": "pln_01be40b08f31240d94448b571b92fb67", + "revision": 1, + "status": "awaiting_confirmation", + "execution_mode": "live", + "expires_at": "2026-09-23T15:49:43.354774734Z", + "steps": [ + { + "step_id": "stp_b673f5d7107ef15b7fd7175163681c90", + "action_key": "regulation.query", + "status": "pending", + "execution_mode": "live", + "safe_summary": "查一下相关法规" + }, + { + "step_id": "stp_ba9914339c9a6e0b0967f8adde551cf6", + "action_key": "task.run", + "status": "pending", + "execution_mode": "live", + "safe_summary": "帮我生成报价单" + } + ], + "safe_summary": "查一下相关法规，然后帮我生成报价单" + }, + "decided": { + "plan_id": "pln_01be40b08f31240d94448b571b92fb67", + "revision": 2, + "status": "executing", + "execution_mode": "live", + "expires_at": "2026-09-23T15:49:43.354774Z", + "steps": [ + { + "step_id": "stp_b673f5d7107ef15b7fd7175163681c90", + "action_key": "regulation.query", + "status": "succeeded", + "execution_mode": "live", + "safe_summary": "regulation.query authorized (scope=c8ccdb3df559)" + }, + { + "step_id": "stp_ba9914339c9a6e0b0967f8adde551cf6", + "action_key": "task.run", + "status": "succeeded", + "execution_mode": "live", + "safe_summary": "task.run accepted" + } + ], + "safe_summary": "查一下相关法规，然后帮我生成报价单" + }, + "task_runs": [ + "562", + "562", + "563" + ] +} \ No newline at end of file diff --git a/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-shadow-chat.json b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-shadow-chat.json new file mode 100644 index 00000000..52c00a32 --- /dev/null +++ b/.doc/test-reports/assets/2026-09-24-intent-plan-i2-rollout/i2-shadow-chat.json @@ -0,0 +1,243 @@ +{ + "reply_text": "**步骤 2** 关于「医疗器械研发项目」适用的国家标准与法规强标，权威依据梳理如下：\n\n1. **医疗器械软件注册审查指导原则** 〔GB/T9254 §七、技术考量 / （十六）功能安全与软件可靠性〕\n - 条款章节: 七、技术考量 / （十六）功能安全与软件可靠性\n - 条款要点: 考虑到行业实际情况，功能安全、软件可靠性暂不要求，待时机成熟时纳入考量。建议注册申请人参考相关标准要求加强功能安全、软件可靠性的设计工作，若已开展相应工作可在软件研究资料中予以提供。\n2. **国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知** 〔国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知 §三、〕\n - 条款章节: 三、 项目开展的必要性\n - 条款要点: 项目开展的必要性\n品种简介（二）市场现状（三）主要风险点\n3. **国家药监局关于印发医疗器械质量抽查检验管理办法的通知** 〔国家药监局关于印发医疗器械质量抽查检验管理办法的通知 §（二）立即深入进行自查，分析原因，进行风险评估；〕\n - 条款章节: （二）立即深入进行自查，分析原因，进行风险评估； 立即深入进行自查，分析原因，进行风险评估；\n - 条款要点: 立即深入进行自查，分析原因，进行风险评估；\n4. **中华人民共和国医药行业标准** 〔YY/T0489-2023 §5.1 p.7〕\n - 条款章节: 一次性使用无菌引流导管及辅助器械 / 5.1 5.1 风险管理\n - 条款要点: 器械设计应建立风险管理过程，并应进行风险分析。\n示例：GB/T 42062。\n应通过检查风险管理文档来确认符合性。如果开展了临床研究，则这些研究应记录所声称的性能条件下进行的测量。临床研究应按照国家临床研究的相关规定进行。\n5. **国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知** 〔国家药监局综合司关于印发国家医疗器械质量抽查检验工作程序的通知 §（一）品种简介（二）市场现状（三）主要风险点〕\n - 条款章节: 三、 项目开展的必要性 / （一）品种简介（二）市场现状（三）主要风险点 品种简介（二）市场现状（三）主要风险点\n - 条款要点: 品种简介（二）市场现状（三）主要风险点\n\n上述标准与条款可作为型检与注册申报的权威技术支撑。如需深入比对专标或预估测试周期，请随时吩咐我。\n\n另：您同时提到了「project.stage.advance」。该操作需单独确认后再执行，请回复「确认」或选择对应选项。", + "branch": "NEW_INTENT_PREEMPT", + "intent_result": { + "raw_input": "查一下项目风险，然后查相关法规", + "top_intent": { + "intent": "QUERY_REGULATIONS", + "confidence": 0.9, + "branch": "NEW_INTENT_PREEMPT", + "filled_slots": { + "query_topic": "regulations" + }, + "reasoning": "用户咨询医疗器械适用的国家标准、行业强标或审评指导原则法规；注意：当前上下文 AvailableActions 未包含 QUERY_REGULATIONS（L2 将做权限/能力裁决）" + }, + "candidates": [ + { + "intent": "QUERY_REGULATIONS", + "confidence": 0.9, + "branch": "NEW_INTENT_PREEMPT", + "filled_slots": { + "query_topic": "regulations" + }, + "reasoning": "用户咨询医疗器械适用的国家标准、行业强标或审评指导原则法规；注意：当前上下文 AvailableActions 未包含 QUERY_REGULATIONS（L2 将做权限/能力裁决）" + }, + { + "intent": "QUERY_PROGRESS", + "confidence": 0.9, + "branch": "NEW_INTENT_PREEMPT", + "filled_slots": { + "query_topic": "risk" + }, + "reasoning": "用户在询问项目状态、技术测试方案或对外沟通话术 (主题: risk)" + }, + { + "intent": "CASUAL_CHAT", + "confidence": 0.2, + "branch": "NEW_INTENT_PREEMPT", + "filled_slots": {}, + "reasoning": "未匹配到明确业务指令，归入通用对话" + } + ], + "context_snapshot_id": "ctx_1790176580649938109", + "requires_clarifying": false, + "branch": "NEW_INTENT_PREEMPT", + "shadow_score": 0.9, + "shadow_dialogue_act": "command", + "shadow_decision_hint": "allowed", + "intent_frame_v2": { + "version": "v2.0", + "action_key": "regulation.query", + "domain": "knowledge", + "object": "regulation", + "goal": "query", + "dialogue_act": "command", + "slots": { + "query_topic": { + "value": "regulations", + "source": "rule" + } + }, + "relation": "expectation_plus_new", + "evidence": [ + { + "kind": "rule_match", + "detail": "用户咨询医疗器械适用的国家标准、行业强标或审评指导原则法规；注意：当前上下文 AvailableActions 未包含 QUERY_REGULATIONS（L2 将做权限/能力裁决）", + "weight": 0.9 + }, + { + "kind": "dialogue_act", + "detail": "command", + "weight": 0.1 + } + ], + "score": 0.9, + "legacy_intent": "QUERY_REGULATIONS" + }, + "ranked_frames": { + "candidates": [ + { + "version": "v2.0", + "action_key": "regulation.query", + "domain": "knowledge", + "object": "regulation", + "goal": "query", + "dialogue_act": "command", + "slots": { + "query_topic": { + "value": "regulations", + "source": "rule" + } + }, + "relation": "new", + "evidence": [ + { + "kind": "rule_match", + "detail": "用户咨询医疗器械适用的国家标准、行业强标或审评指导原则法规；注意：当前上下文 AvailableActions 未包含 QUERY_REGULATIONS（L2 将做权限/能力裁决）", + "weight": 0.9 + } + ], + "score": 0.9, + "legacy_intent": "QUERY_REGULATIONS" + }, + { + "version": "v2.0", + "action_key": "project.advisory", + "domain": "project", + "object": "project", + "goal": "query", + "dialogue_act": "command", + "slots": { + "query_topic": { + "value": "risk", + "source": "rule" + } + }, + "relation": "new", + "evidence": [ + { + "kind": "rule_match", + "detail": "用户在询问项目状态、技术测试方案或对外沟通话术 (主题: risk)", + "weight": 0.9 + } + ], + "score": 0.9, + "legacy_intent": "QUERY_PROGRESS", + "subtype": "risk" + }, + { + "version": "v2.0", + "action_key": "conversation.chat", + "domain": "conversation", + "object": "system_ui", + "goal": "converse", + "dialogue_act": "command", + "relation": "new", + "evidence": [ + { + "kind": "rule_match", + "detail": "未匹配到明确业务指令，归入通用对话", + "weight": 0.2 + } + ], + "score": 0.2, + "legacy_intent": "CASUAL_CHAT" + } + ], + "top_margin": 0 + }, + "intent_plan": { + "steps": [ + { + "step_id": "1", + "frame": { + "version": "v2.0", + "action_key": "project.stage.advance", + "domain": "project", + "object": "project", + "goal": "control", + "dialogue_act": "confirm", + "slots": { + "target_stage": { + "value": "V0.5_feasibility", + "source": "expectation" + } + }, + "relation": "expectation_hit", + "score": 1, + "legacy_intent": "ADVANCE_STAGE" + }, + "on_failure": "abort" + }, + { + "step_id": "2", + "frame": { + "version": "v2.0", + "action_key": "regulation.query", + "domain": "knowledge", + "object": "regulation", + "goal": "query", + "dialogue_act": "command", + "slots": { + "query_topic": { + "value": "regulations", + "source": "rule" + } + }, + "relation": "new", + "evidence": [ + { + "kind": "rule_match", + "detail": "用户咨询医疗器械适用的国家标准、行业强标或审评指导原则法规；注意：当前上下文 AvailableActions 未包含 QUERY_REGULATIONS（L2 将做权限/能力裁决）", + "weight": 0.9 + }, + { + "kind": "dialogue_act", + "detail": "command", + "weight": 0.1 + } + ], + "score": 0.9, + "legacy_intent": "QUERY_REGULATIONS" + }, + "depends_on": [ + "1" + ], + "on_failure": "abort" + } + ], + "atomicity": "sequential", + "needs_confirm": true, + "failure_mode": "stop_on_first_failure" + }, + "plan_proposal": { + "segmenter_version": "segmenter.v2", + "digest": "31a4d8d763436d33cc87703baad5c5f041c66d4d5684e68d5b57436e07aa69d3", + "combo_hint": "read_only", + "segments": [ + { + "step_id": "1", + "text": "查一下项目风险", + "normalized": "查一下项目风险", + "span": { + "start": 0, + "end": 21 + }, + "evidence": "connector", + "structure_confidence": "high", + "frame": { + "version": "v2.0", + "action_key": "project.advisory", + "domain": "project", + "object": "project", + "goal": "query", + "dialogue_act": "command", + "slots": { + "query_topic": { + "value": "risk", + "source": "rule" + } + }, + "relation": "new", + "evidence": [ + \ No newline at end of file diff --git a/internal/app/intentflow/turn.go b/internal/app/intentflow/turn.go index 250b9d3d..8c894843 100644 --- a/internal/app/intentflow/turn.go +++ b/internal/app/intentflow/turn.go @@ -48,7 +48,8 @@ func (p *Plans) handleTurnElevated(ctx context.Context, in TurnInput) (*TurnResu } var proposal *intent.PlanProposal - if flags.U9Segment && !RulesCanDecide(text) && tier.Allows(TierShadow) { + // U9 live segment proposals require elevated tier ≥ auto_read (plan Track I3). + if flags.U9Segment && !RulesCanDecide(text) && tier.Allows(TierAutoRead) { sp, propErr := ActiveSegmentProposer().Propose(ctx, SegmentProposalRequest{ Text: text, RequestDigest: digest,