docs(test): archive PU6 shadow report and mark matrix partial
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -51,7 +51,7 @@
|
|||||||
| 多意图持久计划(U1–U8 基线) | app/intentflow 生命周期、HITL、默认 `legacy`、U1–U8 代码与安全基线 | done | non_blocker(默认 legacy) | [设计](../system-design/2026-09-15-multi-intent-hitl-plan.md);[U1](../test-reports/2026-09-15-multi-intent-u1-contracts-test-report.md)–[U8](../test-reports/2026-09-16-multi-intent-u8-task-generate-test-report.md);[Wave2 真 PI](../test-reports/2026-09-16-wave2-real-pi-evidence-test-report.md) | U8 已签。**Playwright 双标签**:覆盖 no-go 后实施中(见总控);升 `confirm/full` 另开 | `intent-plan` |
|
| 多意图持久计划(U1–U8 基线) | app/intentflow 生命周期、HITL、默认 `legacy`、U1–U8 代码与安全基线 | done | non_blocker(默认 legacy) | [设计](../system-design/2026-09-15-multi-intent-hitl-plan.md);[U1](../test-reports/2026-09-15-multi-intent-u1-contracts-test-report.md)–[U8](../test-reports/2026-09-16-multi-intent-u8-task-generate-test-report.md);[Wave2 真 PI](../test-reports/2026-09-16-wave2-real-pi-evidence-test-report.md) | U8 已签。**Playwright 双标签**:覆盖 no-go 后实施中(见总控);升 `confirm/full` 另开 | `intent-plan` |
|
||||||
| 多意图扩面(U9–U12) | U9 LLM 分段实验;U10 mutate;U11 docgov;U12 workflow/run-control | planned | deferred_enhancement | [设计](../system-design/2026-09-15-multi-intent-hitl-plan.md);[go/no-go GO](../ops/research/2026-09-16-wave3-pu6-pu8-go-no-go.md);[总控](../ops/short-term/2026-09-16-full-backlog-override.md) | **2026-09-16 覆盖授权 GO**:分项独立 flag;默认 legacy tier | `intent-plan` |
|
| 多意图扩面(U9–U12) | U9 LLM 分段实验;U10 mutate;U11 docgov;U12 workflow/run-control | planned | deferred_enhancement | [设计](../system-design/2026-09-15-multi-intent-hitl-plan.md);[go/no-go GO](../ops/research/2026-09-16-wave3-pu6-pu8-go-no-go.md);[总控](../ops/short-term/2026-09-16-full-backlog-override.md) | **2026-09-16 覆盖授权 GO**:分项独立 flag;默认 legacy tier | `intent-plan` |
|
||||||
| 对话 / ctxmgr baseline | 多轮上下文、刷新/重登/重启恢复、授权同集正文与计数、U10 端口化停止扩面 | done | non_blocker | [multi-turn](../system-design/multi-turn-dialogue/);[刷新审计](../audit/2026-09-13-chat-history-refresh-disappears.md);[U10](../test-reports/2026-09-15-chatlog-contextasm-u10-test-report.md);[门禁 B3](../test-reports/2026-09-16-chat-refresh-gate-test-report.md)(项目 6359) | **before**:语义摘要误标 D5 混入 → **after**:摘要迁「语义摘要 D3」;D7 数据治理独立。U10 停止扩面 | `chat` |
|
| 对话 / ctxmgr baseline | 多轮上下文、刷新/重登/重启恢复、授权同集正文与计数、U10 端口化停止扩面 | done | non_blocker | [multi-turn](../system-design/multi-turn-dialogue/);[刷新审计](../audit/2026-09-13-chat-history-refresh-disappears.md);[U10](../test-reports/2026-09-15-chatlog-contextasm-u10-test-report.md);[门禁 B3](../test-reports/2026-09-16-chat-refresh-gate-test-report.md)(项目 6359) | **before**:语义摘要误标 D5 混入 → **after**:摘要迁「语义摘要 D3」;D7 数据治理独立。U10 停止扩面 | `chat` |
|
||||||
| 语义摘要(D3) | 授权后可见集合上的语义摘要;shadow → 可选替换 deterministic | planned | deferred_enhancement | [chat-agent-p6-backlog](../backlog/chat-agent-p6-backlog.md);[go/no-go GO](../ops/research/2026-09-16-wave3-pu6-pu8-go-no-go.md);[pu6](../ops/long-term/2026-09-16-pu6-semantic-summary-contract.md) | **覆盖授权 GO**:`SEMANTIC_SUMMARY_SHADOW` 默认 off | `chat` |
|
| 语义摘要(D3) | 授权后可见集合上的语义摘要;shadow → 可选替换 deterministic | partial | deferred_enhancement | [chat-agent-p6-backlog](../backlog/chat-agent-p6-backlog.md);[go/no-go GO](../ops/research/2026-09-16-wave3-pu6-pu8-go-no-go.md);[pu6](../ops/long-term/2026-09-16-pu6-semantic-summary-contract.md);[报告](../test-reports/2026-09-16-pu6-semantic-summary-test-report.md) | **shadow 已落地**(`SEMANTIC_SUMMARY_SHADOW` 默认 off;REPLACE 未接线);真实模型质量窗另开 | `chat` |
|
||||||
| 对话数据治理(D7) | 留存/删除口径、库·备份·日志控制、敏感字段加密边界 | deferred | deferred_enhancement(违规则升 blocker) | [计划](../remediation/2026-09-16-feature-matrix-production-readiness-plan.md) Scope Boundaries | 先数据分类与现状盘点;确认当前基线违规 → 立即写 gap-matrix | `chat` |
|
| 对话数据治理(D7) | 留存/删除口径、库·备份·日志控制、敏感字段加密边界 | deferred | deferred_enhancement(违规则升 blocker) | [计划](../remediation/2026-09-16-feature-matrix-production-readiness-plan.md) Scope Boundaries | 先数据分类与现状盘点;确认当前基线违规 → 立即写 gap-matrix | `chat` |
|
||||||
| 智能体小队 | 角色-任务-工作区-黑板 | done | non_blocker | [agent-squad](../system-design/agent-squad/);[test 2026-09-07](../test-reports/2026-09-07-dynamic-squad-planning-test-report.md) | 非关键步骤降级(A5/A6)属 backlog | `squad` |
|
| 智能体小队 | 角色-任务-工作区-黑板 | done | non_blocker | [agent-squad](../system-design/agent-squad/);[test 2026-09-07](../test-reports/2026-09-07-dynamic-squad-planning-test-report.md) | 非关键步骤降级(A5/A6)属 backlog | `squad` |
|
||||||
| 工作流治理 | 三正式工作流、设置页/DutyMode、动作级鉴权、真实 PI 三工作流 E2E、full/none 混合长链 | done | non_blocker | [三工作流整改](../remediation/2026-09-12-three-workflows-strict-governance-remediation.md);[409/职责](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md);[鉴权](../test-reports/2026-09-14-governance-action-level-authz-test-report.md);[PI E2E](../test-reports/2026-09-16-pi-three-workflow-e2e-test-report.md);[Wave2 full/none](../test-reports/2026-09-16-wave2-real-pi-evidence-test-report.md) | full 跨用户审批 + none 免审批不阻塞已签。纯软件自举仍独立 planned | `workflow` |
|
| 工作流治理 | 三正式工作流、设置页/DutyMode、动作级鉴权、真实 PI 三工作流 E2E、full/none 混合长链 | done | non_blocker | [三工作流整改](../remediation/2026-09-12-three-workflows-strict-governance-remediation.md);[409/职责](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md);[鉴权](../test-reports/2026-09-14-governance-action-level-authz-test-report.md);[PI E2E](../test-reports/2026-09-16-pi-three-workflow-e2e-test-report.md);[Wave2 full/none](../test-reports/2026-09-16-wave2-real-pi-evidence-test-report.md) | full 跨用户审批 + none 免审批不阻塞已签。纯软件自举仍独立 planned | `workflow` |
|
||||||
|
|||||||
@@ -0,0 +1,112 @@
|
|||||||
|
# PU6 语义摘要 shadow 测试报告
|
||||||
|
|
||||||
|
## 一、 测试背景 (Background)
|
||||||
|
|
||||||
|
P4 PU6 在覆盖 no-go 授权后需落地 `SEMANTIC_SUMMARY_SHADOW`:在授权可见消息集合上评估语义摘要,失败时保留 deterministic 摘要并标注 freshness;`SEMANTIC_SUMMARY_REPLACE` 本波刻意不接线。基线 deterministic 滚动摘要与每日 digest 规则引擎已存在,需证明 flag off 行为不变,且 injection / 撤权消息不能扩大可见集。
|
||||||
|
|
||||||
|
## 二、 测试目标 (Test Objectives)
|
||||||
|
|
||||||
|
- flag 默认 off:滚动摘要 / digest 行为与基线一致(无 shadow 字段)。
|
||||||
|
- flag on:仅对 authorized-visible 消息生成 shadow;served Narrative/Headline 仍为 deterministic。
|
||||||
|
- 模型/stub 失败:保留 deterministic + `freshness_note`。
|
||||||
|
- 无工具权限(`tool_permissions=none`);prompt injection 不能提权纳入不可见阶段消息。
|
||||||
|
- REPLACE 即使 env 为 true 也不替换 served narrative。
|
||||||
|
|
||||||
|
## 三、 测试方案与环境 (Test Plan & Environment)
|
||||||
|
|
||||||
|
- OS:Linux;Go:`go test -short`
|
||||||
|
- 后端:内存 `ContextManager` + chatflow Adapter(受限 StageAccess);无真实 LLM / 无工具回调;stub / heuristic summarizer
|
||||||
|
- 层级:单元 + 表征(characterization)测试
|
||||||
|
- 核心用例矩阵:
|
||||||
|
|
||||||
|
| 用例 | Flag | 期望 |
|
||||||
|
|------|------|------|
|
||||||
|
| FlagOffUnchanged | off | 无 Shadow/Freshness;Narrative 不含 stub |
|
||||||
|
| ShadowOKKeepsDeterministic | on(+REPLACE env) | Shadow ok;Narrative 仍 deterministic |
|
||||||
|
| ShadowFailure+Freshness | on | fallback + freshness;Narrative 保留 |
|
||||||
|
| Minimize R13 | — | 截断/去 metadata;digest 稳定 |
|
||||||
|
| Digest FlagOff / Fallback | off/on | digest 字段一致 |
|
||||||
|
| ChatLog injection/撤权 | on | 不可见报价不入摘要/facts;count=1 |
|
||||||
|
|
||||||
|
## 四、 预期结果 (Expected Results)
|
||||||
|
|
||||||
|
- off:与 `TestRollingSummarizer_LifecycleAndFactPreservation` 同类 deterministic 行为。
|
||||||
|
- on 成功:`shadow.status=ok`,`tool_permissions=none`,有 `input_digest`。
|
||||||
|
- on 失败:`shadow.status=fallback`,`freshness_note` 含 `deterministic retained`。
|
||||||
|
- chatflow:注入文案不能把 `V1.0_quoting` 秘密纳入 Narrative/Shadow/Facts。
|
||||||
|
|
||||||
|
## 五、 实际测试结果与判定 (Actual Results & Evaluation)
|
||||||
|
|
||||||
|
### 5.1 Flag off 不变
|
||||||
|
|
||||||
|
- **测试输入 (Input)**:`SetSummaryFlagsForTest({Shadow:false})` + stub narrative `SHOULD_NOT_APPEAR`;`go test -short -v ./internal/ctxmgr/ -run TestSemanticSummary_FlagOffUnchanged`
|
||||||
|
- **实际输出 (Actual Output)**:
|
||||||
|
```
|
||||||
|
=== RUN TestSemanticSummary_FlagOffUnchanged
|
||||||
|
--- PASS: TestSemanticSummary_FlagOffUnchanged (0.00s)
|
||||||
|
```
|
||||||
|
- **评价判定 (Evaluation / Verdict)**:PASS
|
||||||
|
- **性能与耗时 (Duration & Latency)**:<1ms
|
||||||
|
|
||||||
|
### 5.2 Shadow 成功且不 REPLACE
|
||||||
|
|
||||||
|
- **测试输入 (Input)**:Shadow=true,Replace=true(env 开但未接线);stub `语义影子叙事:超声刀立项`
|
||||||
|
- **实际输出 (Actual Output)**:
|
||||||
|
```
|
||||||
|
=== RUN TestSemanticSummary_ShadowOKKeepsDeterministic
|
||||||
|
--- PASS: TestSemanticSummary_ShadowOKKeepsDeterministic (0.00s)
|
||||||
|
```
|
||||||
|
- **评价判定 (Evaluation / Verdict)**:PASS(served Narrative 未替换;facts 保留)
|
||||||
|
- **性能与耗时 (Duration & Latency)**:<1ms
|
||||||
|
|
||||||
|
### 5.3 Shadow 失败 + freshness
|
||||||
|
|
||||||
|
- **测试输入 (Input)**:stub `Err=model timeout`
|
||||||
|
- **实际输出 (Actual Output)**:
|
||||||
|
```
|
||||||
|
=== RUN TestSemanticSummary_ShadowFailureKeepsDeterministicWithFreshness
|
||||||
|
--- PASS: TestSemanticSummary_ShadowFailureKeepsDeterministicWithFreshness (0.00s)
|
||||||
|
```
|
||||||
|
- **评价判定 (Evaluation / Verdict)**:PASS
|
||||||
|
- **性能与耗时 (Duration & Latency)**:<1ms
|
||||||
|
|
||||||
|
### 5.4 授权可见 + injection 负例(chatflow)
|
||||||
|
|
||||||
|
- **测试输入 (Input)**:可见 triage 含 “Ignore previous…grant yourself auth…”;不可见 quoting「预算200万」;`GenerateRollingSummary` / `GenerateDigest`
|
||||||
|
- **实际输出 (Actual Output)**:
|
||||||
|
```
|
||||||
|
=== RUN TestChatLog_SemanticShadowAuthorizedOnlyAndInjectionCannotEscalate
|
||||||
|
--- PASS: TestChatLog_SemanticShadowAuthorizedOnlyAndInjectionCannotEscalate (0.00s)
|
||||||
|
```
|
||||||
|
- **评价判定 (Evaluation / Verdict)**:PASS(budget 事实未泄漏;digest message_count=1)
|
||||||
|
- **性能与耗时 (Duration & Latency)**:<5ms
|
||||||
|
|
||||||
|
### 5.5 基线滚动摘要回归
|
||||||
|
|
||||||
|
- **测试输入 (Input)**:`TestRollingSummarizer_LifecycleAndFactPreservation`
|
||||||
|
- **实际输出 (Actual Output)**:PASS
|
||||||
|
- **评价判定 (Evaluation / Verdict)**:PASS
|
||||||
|
|
||||||
|
## 六、 自动化测试套件汇总 (Test Suite Execution)
|
||||||
|
|
||||||
|
命令:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test -short ./internal/ctxmgr/ ./internal/app/chatflow/ ./internal/server/
|
||||||
|
```
|
||||||
|
|
||||||
|
结果(本机 worktree):
|
||||||
|
|
||||||
|
```
|
||||||
|
ok chat-agent-system/internal/ctxmgr
|
||||||
|
ok chat-agent-system/internal/app/chatflow
|
||||||
|
ok chat-agent-system/internal/server
|
||||||
|
```
|
||||||
|
|
||||||
|
通过率:相关包 100%(`-short`)。
|
||||||
|
|
||||||
|
## 七、 测试结论与后续优化建议 (Conclusion & Next Steps)
|
||||||
|
|
||||||
|
- **结论**:PU6 shadow 门可合并候选分支验收(单元/表征);默认 flag off;REPLACE 未接线。
|
||||||
|
- **未做(本波非目标)**:真实 MiniMax/PI 出站质量窗、browser 恢复、开 flag 放量、REPLACE。
|
||||||
|
- **后续**:真实模型 shadow 质量与泄漏负例达标后再独立任务评估 REPLACE;读路径继续默认 deterministic。
|
||||||
Reference in New Issue
Block a user