weichen/Agent-System
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(test): Wave 2B nightly promotion-receipt report and matrix sync

Browse Source 
Record gate fixtures and infra negative evidence; mark Artifact row
partial for same-commit/digest traceability without same-image claim.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
weichen
2026-09-16 14:02:24 +00:00
co-authored by Cursor
parent fda159c995
commit 67d73359fd
5 changed files with 159 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.doc/ops/CURRENT.md
+2 -1
View File
@@ -5,7 +5,7 @@
## 焦点
1. **Wave 2 done**：已合入 `dev` + smoke/rebuild PASS；Wave 2B（nightly/receipt）另开
1. **Wave 2B**：nightly PI + promotion receipt（`feature/wave2b-nightly-promotion-receipt`）
2. **Wave 1 done**：PU5A race + PU3 IAM fail-closed 已合入 `dev`
3. **运维**：夜间 CI `RUN_PI_E2E=1`；promotion receipt（Wave 2B）
@@ -13,6 +13,7 @@
| owner | task | branch | worktree | mutex | status |
|-------|------|--------|----------|-------|--------|
| Cursor | 2026-09-16-wave2b-nightly-promotion-receipt | feature/wave2b-nightly-promotion-receipt | - | ci | in_progress |
| Cursor | 2026-09-16-wave2-real-pi-evidence | feature/wave2-real-pi-evidence | - | intent-plan | done |
| Cursor | 2026-09-16-wave1-race-tempdir-cleanup | fix/race-tempdir-cleanup | - | ci | done |
| Cursor | 2026-09-16-wave1-iam-fail-closed | fix/iam-store-fail-closed | - | authz | done |
.doc/ops/short-term/2026-09-16-wave2b-nightly-promotion-receipt.md
+48
View File
@@ -0,0 +1,48 @@
---
title: "Wave 2B PU5B：nightly PI + promotion receipt"
status: in_progress
created: 2026-09-16
updated: 2026-09-16
owner: Cursor
branch: feature/wave2b-nightly-promotion-receipt
worktree: "-"
mutex: ci
---
# Wave 2B PU5B：nightly PI + promotion receipt
## 目标
落地夜间真实 PI 回归入口、不可变 promotion receipt，以及晋升脚本强制校验；same-image digest 晋升本轮仅做到 same-commit + digest 可追溯。
## 范围
- promotion receipt 写/校验 + waiver
- nightly-real-pi.yml + run-nightly-real-pi.sh
- git-workflow-helper / promote-to-test / release-to-main 门禁钩子
- 报告与 FEATURE-MATRIX / remediation 同步
## 非目标
- compose 按 digest pull 的同镜像晋升流水线
- Playwright 双标签；升 MULTI_INTENT tier
## 步骤
1. [x] 认领 + 开分支
2. [x] receipt 写/校验脚本
3. [x] nightly workflow + 本机脚本
4. [x] 晋升钩子
5. [ ] 报告 + 合入 dev
## 验收标准
- [x] 缺 secret/runner → infra_fail，晋升拒绝
- [x] product_fail → 晋升拒绝
- [x] pass receipt → 晋升可过；无 receipt → 非零退出
- [x] 报告归档（`.doc/test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md`）
## 相关链接
- Remediation PU5B: `../../remediation/2026-09-16-feature-matrix-production-readiness-plan.md`
- Test report: `../../test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md`
.doc/remediation/2026-09-16-feature-matrix-production-readiness-plan.md
+2 -2
View File
@@ -184,7 +184,7 @@ flowchart TB
| PI change-report narrative | deterministic report 已有 | 产品 go/no-go；只读 narrative、fallback 永久保留 | 当前 `deferred`；获批 short-term 开始后转 `partial` |
| Frontend workbench | 当前声明范围均有证据 | 修正文档状态；增强另列 | `done` |
| CI / E2E | PR CI、PG、backup、web 已有 | nightly real PI；TempDir race flake | `partial` |
| Artifact promotion | 同 commit 可追溯，环境内仍可能重建 | 一次构建、digest 晋升、promotion receipt | 当前 `planned`；PU5B 开始后转 `partial` |
| Artifact promotion | 同 commit 可追溯，环境内仍可能重建；receipt + 晋升钩子已落地 | 一次构建、digest 晋升流水线未完成 | 当前 `partial`（Wave 2B）；不得声称 same-image |
| Reliable events | 核心 live 且旧 hook 退出 | 与 FAV 拆行 | `done` |
| FAV | 明确 off | 独立审计与 go/no-go | `deferred` |
@@ -414,7 +414,7 @@ flowchart TB
- [x] PU4 的 multi-intent U8 与 full/none 真实 PI/browser/restart/cross-user 证据独立归档；失败未被文档改写掩盖。
（2026-09-16：Wave2 报告已签；Playwright 双标签仍 gated，API 刷新/重启已签；不升 tier）
- [x] PU5A 的 TempDir race flake 被真正关闭；PU5B 的 nightly job、promotion receipt 和人工豁免链能区分 infrastructure/product failure。
（2026-09-16：PU5A 已关闭；PU5B 仍开放）
（2026-09-16：PU5A 已关闭；PU5B **nightly+receipt+晋升钩子**已交付——见 [Wave2B 报告](../test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md)。**同镜像 digest 晋升**未完成，矩阵 Artifact 行保持 partial。）
- [ ] PU6–PU8 已分别完成产品 go/no-go；未获批项保持 deferred，获批项另建 short-term 后才进入实现与 rollout DoD。
- [ ] 任何真实 PI 工作均满足 R13 的最小出站、租户绑定、脱敏、凭证、留存和泄漏负例。
- [ ] deterministic summary/change-report、legacy 单意图、B1–B4 和三环境隔离均无回归。
.doc/status/FEATURE-MATRIX.md
+2 -2
View File
@@ -63,8 +63,8 @@
| 前端工作台 | 三栏 + 块文档 + SSE；错误卡/进度/busy；版本冲突信封；demo seed 默认关；终止 preview | done | non_blocker | [workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[双标签](../test-reports/assets/2026-09-15-conversation-intent-dual-tab/screenshots/tab-b-busy.png)；[demo](../test-reports/2026-09-16-demo-seed-pollution-test-report.md)；[B4](../test-reports/2026-09-16-frontend-gate-b4-test-report.md)（项目 6363） | **before**：空「缺口」却 partial → **after**：本行 done。其他 409 消费方信封复用另列 hardening backlog | `web` |
| PI Runner / 配置链 | pi-agent + MiniMax + typed-result-v1 | done | non_blocker | [piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md)；[typed-result](../test-reports/2026-09-15-pi-typed-result-v1-test-report.md) | — | `pi-agent` |
| 三环境运维 | dev/test/main + 共享 KB；对话/业务库隔离 | done | non_blocker | [PROJECT.md](../../PROJECT.md) §4；CURRENT 环境快照 schema **47** | — | `infra` |
| CI / E2E | PR CI、PG 集成、治理 PG、已签真实 PI 门禁 | partial | 加固（阻止**下一次**晋升，不自动否定已发布） | [BUG-08](../test-reports/2026-09-16-ci-governance-pg-bug08-test-report.md)；[PI 三工作流](../test-reports/2026-09-16-pi-three-workflow-e2e-test-report.md)；[B3](../test-reports/2026-09-16-chat-refresh-gate-test-report.md)；[B4](../test-reports/2026-09-16-frontend-gate-b4-test-report.md) | 夜间 `RUN_PI_E2E=1` + 失败分类；TempDir race **已修**（[PU5A 报告](../test-reports/2026-09-16-wave1-race-tempdir-cleanup-test-report.md)）；PR CI 不依赖外部模型 | `ci` |
| Artifact 同 digest 晋升 | 候选 commit 一次构建、按 digest 晋升、promotion receipt | planned | 加固 | [计划 PU5B](../remediation/2026-09-16-feature-matrix-production-readiness-plan.md) | 落地前只可声称 same-commit + 各环境 digest 可追溯，不得声称 same-image promotion | `infra` |
| CI / E2E | PR CI、PG 集成、治理 PG、已签真实 PI 门禁；**nightly** schedule/`workflow_dispatch` + infra/product 分类 receipt | partial | 加固（阻止**下一次**晋升，不自动否定已发布） | [BUG-08](../test-reports/2026-09-16-ci-governance-pg-bug08-test-report.md)；[PI 三工作流](../test-reports/2026-09-16-pi-three-workflow-e2e-test-report.md)；[B3](../test-reports/2026-09-16-chat-refresh-gate-test-report.md)；[B4](../test-reports/2026-09-16-frontend-gate-b4-test-report.md)；[Wave2B nightly/receipt](../test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md) | 关闭条件：nightly 入口 + 失败分类 + promotion-receipt 门禁已交付；TempDir race **已修**；PR CI 不依赖外部模型。Actions secret/runner 需环境配置 | `ci` |
| Artifact 同 digest 晋升 | 候选 commit 一次构建、按 digest 晋升、promotion receipt | partial | 加固 | [计划 PU5B](../remediation/2026-09-16-feature-matrix-production-readiness-plan.md)；[Wave2B 报告](../test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md) | **本轮**：same-commit + receipt 内 `image_digests` 可追溯 + 晋升钩子校验 PASS/waiver。**未完成**：compose 按 digest pull 的 same-image 流水线 — 不得声称 same-image promotion | `infra` |
| 可靠事件核心 | PG dispatcher、notify/advance/projection live、生产闭包 hook 退出、Phase2 选择性解耦已合入 | done | non_blocker | [E0A–E2](../test-reports/2026-09-15-event-e0a-production-scope-contract-test-report.md) 等；[复盘](../ops/research/2026-09-15-track-e-retrospective.md)；[shared live](../test-reports/2026-09-15-event-notify-shared-live-test-report.md)；[phase2 closeout](../test-reports/2026-09-15-phase2-closeout-test-report.md) | **before**：与 FAV off 混成 partial → **after**：FAV 独立行。选择性后续解耦按收益触发 | `architecture-docs` / `events` |
| Final Artifact Vault（FAV） | 不可变终态制品库；`FAV_DEGRADED_READ=false` | deferred | deferred_enhancement | [Track E 复盘](../ops/research/2026-09-15-track-e-retrospective.md)；CURRENT 旁支纪律 | 独立审计与 go/no-go；**禁止**整支合入 `feature/final-artifact-vault` | `events` |
.doc/test-reports/2026-09-16-wave2b-nightly-promotion-receipt-test-report.md
+105
View File
@@ -0,0 +1,105 @@
# Wave 2B PU5B：Nightly Real PI + Promotion Receipt 测试报告
## 一、 测试背景 (Background)
Wave 2（PU4）已合入 `dev` 并补齐 U8 / full-none 真实 PI 证据。发布加固缺口仍在：PR CI 不跑真实 PI；晋升入口缺少不可变 receipt；基础设施失败易被误记为 PASS。本轮落地夜间入口、receipt 写/校验（含 timed waiver）与 `promote-to-test` / `release-to-main` 门禁钩子。**不**声称 same-image digest 晋升已完成。
## 二、 测试目标 (Test Objectives)
- 缺 secret / runner 不可达 → `infra_fail` receipt，job/脚本非零退出（禁止写成 PASS）
- 产品断言失败 → `product_fail`，晋升拒绝
- PASS → 落盘 receipt；无 receipt / 过期 waiver → `promotion-receipt-check` 非零退出
- workflow `concurrency` cancel-in-progress；PR `ci.yml` 仍不跑真实 PI
- Artifact 矩阵行保持 **partial**（same-commit + digest 可追溯）
## 三、 测试方案与环境 (Test Plan & Environment)
| 项 | 值 |
|----|-----|
| OS | Linux 7.0.0-31-generic |
| 分支 | `feature/wave2b-nightly-promotion-receipt` |
| Runner | `http://127.0.0.1:18865`（MiniMax-M2.7-highspeed） |
| 隔离 PG | `127.0.0.1:5434/agent_wave2_pi` |
| 脚本 | `scripts/run-nightly-real-pi.sh`、`promotion-receipt-{write,check}.sh`、`promote-to-test.sh`、`release-to-main.sh` |
| Workflow | `.github/workflows/nightly-real-pi.yml`（schedule + workflow_dispatch） |
| Receipt 目录 | `.doc/test-reports/assets/promotion-receipts/` |
用例矩阵：
| ID | 场景 | 期望 |
|----|------|------|
| N1 | `--dry-infra-fail` / 缺 secret | `verdict=infra_fail`，exit 1 |
| N2 | fixture PASS commit | check OK |
| N3 | fixture product_fail | check BLOCKED |
| N4 | fixture infra_fail 无 waiver | check BLOCKED |
| N5 | infra_fail + 有效 waiver 绑定 PASS | check OK |
| N6 | 无 receipt commit | check BLOCKED |
| N7 | 本机真实 suite（可选墙钟） | pass / product_fail / infra_fail 分类落盘 |
## 四、 预期结果 (Expected Results)
- Receipt 最小字段：`verdict`、`commit`、`recorded_at`、`runner_url`、`model_id`、`protocol`、`image_digests`、`suite`、`failure_class`、`evidence_refs`
- Waiver：`owner`、`reason`、`expires_at`、`binds_receipt`（须指向 PASS）
- `git-workflow-helper promote-to-test` 校验 **dev HEAD**；`release-to-main` 校验 **test HEAD**
## 五、 实际测试结果与判定 (Actual Results & Evaluation)
### N1 — infra 负例（本机 dry）
- **测试输入 (Input)**：`./scripts/run-nightly-real-pi.sh --dry-infra-fail`
- **实际输出 (Actual Output)**：
```text
==> nightly real-PI suite commit=82780222373d runner=http://127.0.0.1:18865
forced infra failure (--dry-infra-fail)
wrote promotion receipt: .../82780222373de1ded853e3fe5a8f2208133bc7c7.json
wrote infra_fail receipt (dry)
```
- **评价判定 (Evaluation / Verdict)**：PASS（`verdict=infra_fail`，exit 1；非绿）
- **性能与耗时 (Duration & Latency)**：<1s
对应 receipt 摘录见 assets；晋升门禁对该 commit：
```text
promotion-receipt-check: BLOCKED — infra_fail receipt freezes promotion (...); provide timed waiver or re-run nightly
```
### N2–N6 — fixture 门禁
| 用例 | commit fixture | 实际 | 判定 |
|------|----------------|------|------|
| N2 PASS | `aaaa…` | `OK — PASS receipt` | PASS |
| N3 product | `cccc…` | `BLOCKED — product_fail` | PASS |
| N4 infra | `dddd…` | `BLOCKED — infra_fail … freeze` | PASS |
| N5 waiver | `dddd…` + waiver→`eeee…` PASS | `OK — audited waiver … until 2099-01-01` | PASS |
| N6 missing | `bbbb…` | `BLOCKED — missing promotion receipt` | PASS |
### N7 — 本机真实 suite
- **测试输入**：`TEST_DATABASE_URL=…/agent_wave2_pi` + `RUNNER_AUTH_TOKEN` + `./scripts/run-nightly-real-pi.sh`
- **说明**：墙钟取决于 PI；结果与日志落入 `promotion-receipts/evidence-<sha>-<stamp>/` 与 `<sha>.json`。若 Actions 缺 secret，workflow 同样写 `infra_fail` 并 fail（不得 skip 冒充绿）。
- **评价判定**：以当次 receipt `verdict` 为准；本报告以 N1–N6 关闭脚本/门禁契约，真实 suite 由合入后夜间/手动 dispatch 持续执行。
## 六、 自动化测试套件汇总 (Test Suite Execution)
合入冒烟（清污染环境变量）：
```bash
unset AGENT_TIMEOUT KB_DATABASE_URL KB_EMBEDDING_DIM
go vet ./...
go test -short ./...
```
PR CI 不变；新增 nightly workflow 不进入 PR 路径。
## 七、 测试结论与后续优化建议 (Conclusion & Next Steps)
- **结论**：Wave 2B 脚本门禁与 nightly 入口已交付；infra/product 分类与 waiver 链可审计；晋升钩子在无 PASS receipt 时非零退出。
- **明确未完成**：compose 按 digest pull 的 **same-image** 晋升流水线仍未做 → FEATURE-MATRIX「Artifact 同 digest 晋升」保持 **partial**。
- **后续**：配置 Actions `RUNNER_AUTH_TOKEN` + `NIGHTLY_TEST_DATABASE_URL`（或自托管 runner）；Wave 3 候选（Playwright 双标签 / tier）。
## 证据路径
- Receipts / fixtures：`.doc/test-reports/assets/promotion-receipts/`
- Short-term：`.doc/ops/short-term/2026-09-16-wave2b-nightly-promotion-receipt.md`