From f1edb70fbd0bd1e7d9e249c0c19642f46e12953a Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 09:06:00 +0000 Subject: [PATCH 01/13] docs(audit): diagnose dev runtime data root split-brain --- ...09-14-dev-runtime-data-root-split-brain.md | 161 ++++++++++++++++++ .doc/audit/INDEX.md | 8 + .doc/ops/CURRENT.md | 12 +- .../2026-09-14-dev-runtime-data-root-guard.md | 57 +++++++ ...me-data-root-and-projection-remediation.md | 138 +++++++++++++++ .doc/remediation/INDEX.md | 1 + .doc/remediation/gap-matrix.md | 3 +- .doc/status/FEATURE-MATRIX.md | 4 +- 8 files changed, 377 insertions(+), 7 deletions(-) create mode 100644 .doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md create mode 100644 .doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md create mode 100644 .doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md diff --git a/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md new file mode 100644 index 00000000..49743b3f --- /dev/null +++ b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md @@ -0,0 +1,161 @@ +--- +title: "dev rebuild 从临时 worktree 挂载空数据根导致正文投影 409" +status: done +created: 2026-09-14 +updated: 2026-09-14 +severity: critical +--- + +# dev rebuild 从临时 worktree 挂载空数据根导致正文投影 409 + +## 现象 + +用户刷新项目 143 后，前端请求： + +```text +GET /api/v1/docgov/documents/doc_proj143_intake/content 409 +DOCUMENT_CONTENT_PROJECTION_MISSING +docgov: document content projection missing +``` + +错误发生在 `projectDocuments.ts` → `controlledDocHydrate.ts` 的受控正文水合链。它不是并发编辑的 `HEAD_CONFLICT`，也不是此前已修过的 canonical hash drift。 + +本轮只做只读现场核对与整改设计：没有操作用户会话、没有修改数据库/正文、没有重建共享容器。 + +## 结论 + +直接根因是 **dev 应用、PI Runner、Cursor Runner 被重建成挂载临时集成 worktree 的空 `data/dev`，而 PostgreSQL 仍沿用原 dev 持久卷**，形成数据库与文件事实源的 split-brain： + +```text +PostgreSQL persistent volume + document_records.work_dir=/app/data/workspaces/bizdev/intake_1789270446856290128 + │ + ├─ 主仓 data/dev：output.md 存在，SHA 与 DB/head 一致 + │ + └─ 当前容器 /app/data：绑定 /tmp/as-dev-wf409-merge/data/dev，文件不存在 +``` + +`loadControlledDocumentContent` 先取得账本 head，再调用 `InspectProjection`；容器内看不到 `output.md`，因此正确分类为 `MISSING` 并返回 409。前端只是如实暴露后端错误，不是本次根因。 + +## 现场证据 + +### E1 — 容器挂载源是临时 worktree + +2026-09-14 08:54 UTC 的 `docker inspect shuimu-agent-app-dev`： + +| 项 | 现场值 | +|---|---| +| Compose working dir | `/tmp/as-dev-wf409-merge` | +| Compose config | `/tmp/as-dev-wf409-merge/docker-compose.dev.yml` | +| app `/app/data` source | `/tmp/as-dev-wf409-merge/data/dev` | +| pi-agent `/app/data/workspaces` source | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| cursor-agent `/app/data/workspaces` source | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| app created | `2026-09-14T08:40:44Z` | + +数据库容器的数据仍来自命名卷 `agent-dev-context-pgdata`，不会随 Compose 工作目录切换。由此产生“旧数据库 + 新空文件根”的不一致组合。 + +### E2 — 项目 143 数据本身当前完整 + +`document_records`： + +| 字段 | 值 | +|---|---| +| doc_id | `doc_proj143_intake` | +| work_dir | `/app/data/workspaces/bizdev/intake_1789270446856290128` | +| content_sha256 | `f283ba94c9c037897f021960f51abacee7242c3db8d10f9f2a5e32cde6acf039` | +| updated_at | `2026-09-14 05:14:10.302146+00` | + +账本 open version 1 的 head snapshot 302 是人工修订，SHA 同为 `f283...f039`；主仓文件： + +```text +/home/weichen/projects/Agent-System/data/dev/workspaces/bizdev/intake_1789270446856290128/output.md +size=40075 +sha256=f283ba94c9c037897f021960f51abacee7242c3db8d10f9f2a5e32cde6acf039 +``` + +同一路径在当前 app 容器内 `stat` 返回 `No such file or directory`。所以无需再次对项目 143 做内容修复；需要恢复正确数据根挂载。 + +### E3 — 影响不是单文档 + +对 dev 数据库 100 条“非模板且 work_dir 非空”的记录做只读比较： + +| 根目录 | aligned | missing | +|---|---:|---:| +| 主仓 `/home/weichen/projects/Agent-System/data/dev` | 32 | 68 | +| 当前容器挂载 `/tmp/as-dev-wf409-merge/data/dev` | 0 | 100 | + +主仓下的 68 条 missing 包含历史测试/临时路径，不能直接等同于 68 个真实用户缺陷；但当前容器对本次抽样 **100/100 都不可见**，足以确认环境级事故。恢复前任何真实文档读取或 PI 写入都不可信。 + +错误根在容器启动后已出现 27 个文件，其中 9 个 CAS blob 的 mtime 为 `08:40:45 UTC`；虽然 08:54 UTC 尚未找到新的 `output.md`，错误根已经不是可直接丢弃的“绝对空目录”，恢复前仍须做完整差异保全。 + +只读核对显示 test/main 当前 app 与 PI Runner 仍挂主仓 `data/test`、`data/main`，本轮没有观察到同样的运行事故；但其 Compose 仍使用相对 bind，仍受同一设计缺陷影响。 + +### E4 — 健康检查假阳性 + +同一时刻 `/health` 与 `/ready` 都返回 200，只检查数据库、schema 和 task store；未验证： + +- 容器绑定的数据根身份； +- app 与 runner 是否挂载同一根； +- 活跃项目的正文投影是否可达； +- 数据库记录与文件根是否属于同一环境实例。 + +因此“healthy + schema 33”不能证明文档运行态可用。 + +## 根因分析 + +### R1 — Compose 相对 bind mount 与调用 worktree 耦合（直接根因） + +`docker-compose.dev.yml` 使用： + +```yaml +- ./data/dev:/app/data +- ./data/dev/workspaces:/app/data/workspaces +``` + +Compose 按 project working directory 解析相对路径。`scripts/env-service.sh` 又把 `ROOT_DIR` 定义为脚本所在 worktree 并 `cd` 进去，所以从任何复制/linked worktree 执行 `dev rebuild`，都会把该 worktree 自己的 `data/dev` 当成运行数据根。 + +### R2 — 纪律是文档约束，不是可执行门禁 + +`scripts/agent-worktree.sh` 只打印“禁止在此目录运行 env-service”，`AGENTS.md` 也规定 worktree 不得操作共享 Docker；但 `env-service.sh` 没有检测 linked worktree、主 worktree、分支、数据根或集成槽位。一次误调用即可静默替换全套容器挂载。 + +### R3 — 数据库与文件没有共享存储身份 + +数据库使用稳定 named volume；正文使用随调用目录变化的 bind mount。两者没有共同的 environment/storage UUID，也没有启动握手，因此任意一个被指向另一份状态时，进程仍可启动并连接成功。 + +### R4 — 账本权威仍被兼容投影阻断 + +当前设计声明 ledger head 是权威正文，但 GET 在账本存在时仍要求 `output.md` 投影必须存在且 aligned。于是兼容投影丢失会阻断 UI 读取权威账本。该 fail-closed 对真实 fork 有价值，却把“可安全重建的 missing”与“内容分叉”按同样的可用性后果处理。 + +### R5 — 正文更新仍是非原子双写 + +人工 PATCH 先提交 PostgreSQL 账本，再单独写 `output.md`；候选 commit/reject 也采用 DB 状态与文件投影分步更新。失败后只依赖同幂等键重试，没有持久 projection job/status 和后台补偿。即使修复本次挂载问题，进程崩溃、权限错误或磁盘故障仍可能再次留下 head 已更新、投影缺失的状态。 + +### R6 — 发布验收没有“从不同源码目录 rebuild 后仍接回同一状态”的负例 + +既有真实 UI 验收验证过 restart 后恢复，但 restart 不会重新解析 Compose bind mount；本次事故由 rebuild/recreate 触发。现有门禁没有覆盖 linked worktree、临时集成目录或错误数据根的启动拒绝。 + +## 为什么会“每次修改后又出现” + +修改本身并不会删除项目 143 正文。本次时间线表明：最新人工修订和主仓 `output.md` 在 05:14 UTC 已一致落盘；08:40 UTC 后续 dev rebuild 把运行容器切到另一份空数据根，刷新才出现 409。 + +因此用户感知的“改完再刷新就坏”来自两个事件经常相邻：开发修改 → Agent 在临时 worktree 合并/rebuild → Compose 改绑数据目录。只修某个文档、某个 hydrate effect 或某次 hash 漂移，都无法阻止它复发。 + +## 严重度与影响 + +`critical`： + +- dev 环境所有受控正文、工作流产物和 runner 工作区均可能不可见或写入错误根； +- 错误根上的新输出可能与主仓状态分叉； +- health/ready 仍为绿色，集成报告可能产生假 PASS； +- test/main 使用同类相对 bind 写法，设计缺陷可横向复制，虽本轮未观察其被错挂载。 + +## 关联整改 + +- Remediation: [dev 运行数据根隔离与文档投影自愈整改](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [dev runtime data root guard 诊断与设计](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) +- 前序问题: [项目 143 canonical hash 漂移](./2026-09-14-project143-controlled-content-404.md) + +## 开放问题 + +- 恢复前需再次冻结并备份错误根，确认 08:40 UTC 后是否已有新会话、附件、runner 输出或 agent session 写入；08:54 UTC 抽样未发现 `output.md`，不能替代完整差异清单。 +- test/main 当前挂载源需在实施 Wave 0 只读核对；本轮未扩大操作范围。 diff --git a/.doc/audit/INDEX.md b/.doc/audit/INDEX.md index ab5873db..fbd112ba 100644 --- a/.doc/audit/INDEX.md +++ b/.doc/audit/INDEX.md @@ -13,6 +13,14 @@ ## 核心文档（按阅读顺序推荐） +### dev 临时 worktree 数据根 split-brain（2026-09-14） + +**文件**: [2026-09-14-dev-runtime-data-root-split-brain.md](./2026-09-14-dev-runtime-data-root-split-brain.md) + +**结论**: dev rebuild 从 `/tmp/as-dev-wf409-merge` 解析相对 bind mount，app/pi/cursor 全部切到空数据根，PostgreSQL 却保留原卷；项目 143 文件和账本本身一致，但容器不可见，因而刷新返回 `DOCUMENT_CONTENT_PROJECTION_MISSING` 409。health/ready 对 storage identity 无检查，形成假绿色。 + +**整改**: [运行数据根隔离与文档投影自愈](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) + ### 项目 143 workflow-plan 409 与三权全有或全无（2026-09-14） **文件**: [2026-09-14-workflow-plan-409-frozen-and-optional-duties.md](./2026-09-14-workflow-plan-409-frozen-and-optional-duties.md) diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index 37098ad7..392442e2 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -1,12 +1,13 @@ # 当前焦点 **更新**: 2026-09-14 -**production_ready**: no（`CHAT-INTENT-STATE` 等无关 blocker 仍开放。文档账本运行时 UI Round20 **PASS**，不把 fallback change report 写成完整 PI，也不把本轮写成全仓可发布） +**production_ready**: no（新增 `RUNTIME-DATA-ROOT-SPLIT-BRAIN` critical：dev 数据库与正文文件根分裂，health/ready 假绿色；历史文档 UI PASS 不代表当前运行态） ## 焦点 -1. **CHAT-INTENT**: [对话意图运行时状态](../audit/2026-09-13-conversation-intent-runtime-state.md)（`fix/conversation-intent-state`；与文档账本运行时无关） -2. **CHAT**: [对话历史刷新](../audit/2026-09-13-chat-history-refresh-disappears.md)（认领仍 `in_progress`；gap 行已 PASS，以代码/handoff 为准收敛） +1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；当前 app/pi/cursor 挂到 `/tmp/as-dev-wf409-merge/data/dev*`，DB 仍为原 dev 卷；待批准后占槽恢复并系统整改） +2. **CHAT-INTENT**: [对话意图运行时状态](../audit/2026-09-13-conversation-intent-runtime-state.md)（`fix/conversation-intent-state`；与文档账本运行时无关） +3. **CHAT**: [对话历史刷新](../audit/2026-09-13-chat-history-refresh-disappears.md)（认领仍 `in_progress`；gap 行已 PASS，以代码/handoff 为准收敛） ## 并行认领 @@ -47,6 +48,7 @@ | Cursor-Auto | vite-frp-allowed-hosts | fix/vite-frp-allowed-hosts | - | web-devserver | done | | implementer_cursor | workflow-plan-409-frozen-and-optional-duties | fix/workflow-plan-409-frozen-and-optional-duties | .worktrees/fix-workflow-plan-409-frozen-and-optional-duties | workflow | done | | implementer_cursor | governance-action-level-authz | fix/governance-action-level-authz | .worktrees/fix-governance-action-level-authz | workflow | done | +| Codex | dev-runtime-data-root-guard | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 @@ -66,7 +68,7 @@ | 环境 | 端口 | schema | 状态 | |------|------|--------|------| -| dev | 8080 | **33** | healthy（`pi-agent`；治理鉴权 G5 页面+API PASS） | +| dev | 8080 | **33** | **incident**：health/ready 200 但 app/pi/cursor 错挂临时 worktree 空数据根；正文运行态不可用，禁止以 healthy 签字 | | test | 8081 | 25→27* | healthy | | main | 8000 | 25→27* | healthy（app + runner + DB） | @@ -106,3 +108,5 @@ | UI-03 首次生成 PG ID 误删 baseline | [../audit/2026-09-13-document-revision-pg-id-remap.md](../audit/2026-09-13-document-revision-pg-id-remap.md) | | UI-06 刷新后缓存竞争死锁审计 | [../audit/2026-09-14-document-revision-refresh-cache-race.md](../audit/2026-09-14-document-revision-refresh-cache-race.md) | | 受控正文真实 UI 验收报告 | [../test-reports/2026-09-14-controlled-content-ui-acceptance.md](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | +| dev 数据根 split-brain 审计 | [../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) | +| dev 数据根与投影系统整改 | [../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | diff --git a/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md new file mode 100644 index 00000000..15533ddd --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md @@ -0,0 +1,57 @@ +--- +title: "dev runtime data root split-brain 诊断与系统整改设计" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Codex +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: infra-runtime-state +--- + +# dev runtime data root split-brain 诊断与系统整改设计 + +## 目标 + +定位项目 143 刷新后正文 409 的当前根因，量化影响，并形成防复发的系统整改方案；不实施业务代码或共享环境变更。 + +## 范围 + +- 用户报错链、正文读取/投影代码； +- dev PostgreSQL、主仓文件、容器实际 mount、health/readiness； +- Compose 与 `env-service.sh` 的工作目录语义； +- 即时恢复、state-root 隔离、storage identity、projection outbox 与验收设计。 + +## 非目标 + +- 不操作真实用户会话； +- 不写数据库、不覆盖正文、不删除错误根； +- 不占共享集成槽、不 rebuild； +- 不变更 test/main。 + +## 步骤 + +1. [x] 完成仓库启动链、既有 canonical hash/refresh 事故文档复核。 +2. [x] 比对项目 143 DB record、ledger head、主仓 `output.md` 与容器内路径。 +3. [x] 审计 app/pi/cursor mounts、Compose working dir、健康检查与影响面。 +4. [x] 完成 audit、remediation、gap/feature/current SSOT 更新。 + +## 验收标准 + +- [x] 直接根因有数据库、文件和容器三方证据。 +- [x] 明确区分旧 hash drift、真正 ETag 409 与本次 projection missing。 +- [x] 方案覆盖部署根因、运行时双写、自愈安全边界、负例与真实 UI 验收。 +- [x] 未污染真实会话或修改共享运行态。 + +## 相关链接 + +- Audit: [../../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../../audit/2026-09-14-dev-runtime-data-root-split-brain.md) +- Remediation: [../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Test report: 待方案获批并实施后创建 + +## 证据 / 备注 + +- 诊断分支：`fix/dev-runtime-data-root-guard`，基线 `dev@a8f9451`。 +- 2026-09-14 08:54 UTC：app/pi/cursor 均挂 `/tmp/as-dev-wf409-merge/data/dev*`；DB 仍为 `agent-dev-context-pgdata`。 +- 项目 143 主仓 `output.md` SHA 与 DB/head 一致；容器内同路径 missing。 +- `/health`、`/ready` 均错误地返回 200。 diff --git a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md new file mode 100644 index 00000000..24b5d950 --- /dev/null +++ b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md @@ -0,0 +1,138 @@ +--- +title: "dev 运行数据根隔离与文档投影自愈整改方案" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +--- + +# dev 运行数据根隔离与文档投影自愈整改方案 + +## 背景 + +现场根因见 [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。当前 dev 的数据库仍是原持久卷，但 app、PI Runner、Cursor Runner 已绑定 `/tmp/as-dev-wf409-merge/data/dev`，导致项目 143 等正文投影在容器中不可见，GET 返回 409。 + +本方案不仅恢复项目 143，而是消除“源码 checkout/worktree 决定运行数据根”的系统耦合，并让账本到兼容文件投影成为可观测、可重试、可恢复的持久状态机。 + +## 目标 + +1. 任意 build source/worktree 都不能改变 dev/test/main 的持久数据根。 +2. 数据库、app、PI Runner、Cursor Runner 必须对同一环境 storage identity 达成一致，否则进程不 ready、部署脚本失败。 +3. ledger head 可用时，安全的投影 missing 不再使 UI 正文不可读；真实 fork 仍 fail closed。 +4. DB head 与 `output.md` 的双写失败可持久观测、幂等重试和自动补偿，不依赖用户反复刷新。 +5. rebuild、restart、宿主重启、linked worktree、并发保存和磁盘故障均有自动化负例；最终以真实页面刷新/重登/重建后持久化验收。 + +## 核心决策 + +### D1 — 代码根与状态根彻底分离 + +- `SOURCE_ROOT`：本次构建源码，可来自受控的集成 worktree。 +- `RUNTIME_STATE_ROOT`：环境持久状态根，必须是绝对、canonical、非 worktree 相对路径；dev/test/main 各自固定。 +- Compose 的 app/runner bind mount 全部改为 `${RUNTIME_STATE_ROOT}//...`，不得再写 `./data/`。 +- `.regulations`、agent sessions、attachments 等可变状态一并纳入状态根清单；只读代码资源继续来自 source root。 + +建议短期兼容现有数据，将 primary repository 的 `data/` 解析为默认状态根；中期迁移到仓库外（例如 `/var/lib/agent-system` 或管理员配置路径）。禁止硬编码个人 HOME，脚本通过显式配置或 Git common-dir 推导 primary root。 + +### D2 — 存储身份握手，而不只比较路径字符串 + +- 每个环境状态根创建受保护的 `.storage-identity.json`：`environment`、随机 UUID、schema version；不含 secret。 +- PostgreSQL 新增单行环境 identity（或复用 settings 表），首次受控初始化时写入同一 UUID。 +- app 启动读取文件 identity 并与数据库 identity 比对；app、PI Runner、Cursor Runner 都暴露/上报相同 UUID。 +- UUID 缺失、环境名不符、根不可写、runner identity 不一致时 fail startup/readiness；不得自动初始化一个已含业务数据库对应的空文件根。 + +该机制可以识别“路径看起来合法但实际是另一份空目录/备份目录”的场景。 + +### D3 — ledger 是读取真源，projection 是受管兼容视图 + +- 已有 ledger 的 `GET .../content` 从 immutable blob/head 返回正文，不再因单纯 `MISSING` 阻断 UI。 +- 同时返回 `projection_status` 或响应头，标识 `aligned | pending | missing | fork | failed`；真实 `CONTENT_FORK` 仍 409，并禁止覆盖双方。 +- 只有 storage identity 已通过且 work_dir 在允许根内时，`MISSING` 才允许后台从 head 原子重建；挂载身份错误时不自愈，避免在错误空根批量制造“看似恢复”的第二份数据。 +- 无 ledger 的 legacy 文档仍从文件 bootstrap，并沿用 hard-break 安全 canonicalization 与迁移审计。 + +### D4 — 用 transactional outbox 收口双写 + +新增/扩展持久 projection job： + +```text +doc_id, target_snapshot_id, target_sha256, storage_uuid, +state(pending/applying/aligned/failed/fork), attempts, +last_error_code, next_retry_at, updated_at +``` + +- 人工 PATCH、agent commit/reject 在同一 PostgreSQL 事务中更新 head 并 upsert projection job。 +- projector 使用 `doc_id + target_snapshot_id` 幂等执行，校验 storage UUID 与允许路径，临时文件 fsync + rename + parent fsync 后再标 aligned。 +- 同文档任务合并为最新 head；旧任务不得回写覆盖新 head。 +- missing 可重建；mismatch/fork 保存两侧 hash/路径后转人工处理；失败采用有上限退避并告警。 +- 同步写成功可快速返回 200；超时/失败返回明确 `projection_status=pending/failed`，不得让用户误以为账本未保存。 + +## 分波次计划 + +| 波次 | 内容 | 验收 | +|---|---|---| +| Wave 0 事故遏制与恢复 | 占集成槽；暂停 dev 文档写入/PI；导出 app/pi/cursor 实际 mounts、DB storage 状态；分别备份主仓根与错误 `/tmp` 根的 08:40 后增量；生成冲突清单；确认无分叉后从固定状态根重建三容器 | 项目 143 文件不改内容且三方 SHA 仍为 `f283...f039`；容器 source 指向固定状态根；目标 content 200；错误根只读保全，不盲拷贝/删除 | +| Wave 1 脚本硬门禁 | `env-service.sh` 解析 primary/common Git root与显式 `RUNTIME_STATE_ROOT`；mutating action 校验绝对路径、环境 identity、分支/集成模式；打印 build source 与 state root；worktree 未显式指定稳定 state root 时直接失败 | linked worktree 默认执行 rebuild 失败且不替换现有容器；合法集成构建的所有 bind source 保持固定 | +| Wave 2 Compose 解耦 | dev/test/main/shared compose 的可变 bind 全部改为绝对 state-root 变量；为 app/pi/cursor 添加 storage UUID/环境标签；CI 检查 `docker compose config` 不出现工作树内 `data/` | 从两个不同 source roots 渲染 config，数据挂载 source 完全相同；代码 build context 可不同 | +| Wave 3 启动与 readiness | 引入文件/DB storage identity；readiness 增加 `storage_ready`、`storage_id_match`、runner mount/identity；部署后脚本核对 container labels + mounts + sentinel + active-doc projection 摘要 | 错误空根启动被拒绝；本次 split-brain 复现实验 `/ready` 必须 503，不能再显示 healthy | +| Wave 4 投影状态机 | ledger read 与兼容 projection 解耦；实现 projection outbox、幂等 projector、missing 安全重建、fork 隔离和结构化状态/告警 | DB commit 后 kill -9、只读目录、磁盘满、延迟重试均不丢 head；恢复后自动 aligned；fork 不被覆盖 | +| Wave 5 数据治理 | 清理/标注 dev 中历史临时 `work_dir`；真实项目、测试 fixture、临时 runner 目录分 namespace；增加 orphan/missing/fork 定期巡检和保留策略 | active 非模板文档 projection 100% 可解释；测试临时路径不污染生产统计；巡检报告可追溯 | +| Wave 6 验收与回滚演练 | 工程测试、容器 rebuild/recreate、宿主重启、真实 UI 修改/刷新/重登、双用户冲突、真实 PI 修改、回滚演练 | 下述验收矩阵全 PASS，报告归档后才恢复 FEATURE-MATRIX `done` 和 gap PASS | + +## 即时恢复 Runbook（批准实施后执行） + +1. 在 CURRENT 占 `infra-runtime-state` mutex 与集成槽，记录容器 ID/created/mount source；暂停售写。 +2. 对主仓状态根和错误 `/tmp/as-dev-wf409-merge/data/dev` 做只读 inventory：相对路径、size、mtime、SHA；特别核对 08:40 UTC 后新增/修改项。 +3. 若错误根存在新业务写入，按 DB task run、snapshot、source_run_id 和文件 hash 逐项判定；不得 `cp -r` 覆盖主根。 +4. 备份差异和 Compose inspect 证据；用固定 state root rebuild app/pi/cursor。数据库 named volume不变。 +5. 验证 storage identity、mount source、`/ready`、项目 143 content/revision、活跃项目 projection 摘要。 +6. 仅在页面完成项目 143 打开、保存唯一 marker、硬刷新、退出重登；随后真实 PI 修改一份新文档并验证 app rebuild 后恢复。 +7. 错误根在报告签字前只读保留；确认无独有数据后再按独立授权处置，本方案不授权删除。 + +## 自动化验收矩阵 + +### 脚本与 Compose + +- [ ] primary repo、linked worktree、`/tmp` integration checkout 三种 source root 下，合法 config 的 state bind source 相同。 +- [ ] 未配置/相对/不存在/环境名不符的 state root 均 fail fast，已有容器不被 recreate。 +- [ ] dev 命令不能挂 test/main 根；storage UUID 交叉时拒绝。 +- [ ] app、pi-agent、cursor-agent 三者 mount source 和 storage UUID 一致。 + +### 后端与故障注入 + +- [ ] ledger head + projection missing + identity correct：GET 200 返回 head，job pending 后原子重建并 aligned。 +- [ ] ledger head + projection missing + identity mismatch：readiness 503，不执行自愈写入。 +- [ ] ledger head + projection fork：409 `DOCUMENT_CONTENT_PROJECTION_DRIFT`，保全双方。 +- [ ] PATCH 在 ledger commit 后 projector 失败：状态可见、同幂等键不重复增 r、后台恢复后 aligned。 +- [ ] projector 处理中又产生新 head：旧 job 不得覆盖新 snapshot。 +- [ ] app/projector/runner restart 与 kill -9 后任务可恢复；告警包含 doc_id、snapshot、storage UUID、correlation id，不含正文或 secret。 + +### 真实页面验收 + +- [ ] 项目 143 正文首次打开无 409；人工保存后 r+1、硬刷新、退出重登仍可见。 +- [ ] 保存后立即刷新与连续刷新不会触发 loading 死锁或重复错误风暴。 +- [ ] 真实 PI/小队修改新项目文档，候选确认/拒绝、change report、最终正文均可见；fallback 明确标识。 +- [ ] dev app + runner rebuild/recreate 后，旧项目和新项目正文/版本/overlay 都恢复。 +- [ ] 双浏览器旧 ETag 保存仍显示真正的 `HEAD_CONFLICT`，不能与 projection 409 混淆。 + +## 监控与 SLO + +- 指标：`document_projection_total{state}`、`projection_repair_total{result}`、`projection_age_seconds`、`storage_identity_mismatch_total`、`content_read_fallback_total`。 +- 告警：任一 storage identity mismatch 立即 P0；active 文档 `pending/failed` 超过 60 秒告警；fork 立即人工处置。 +- 日志：部署开始/结束必须记录 source commit、compose working dir、state root identity、实际 mount source hash；禁止记录正文与凭证。 +- `/ready`：数据库/schema/task store/storage identity/runner identity 为硬门禁；projection 汇总按 active 用户文档设阈值并列出计数，不泄露 doc 内容。 + +## 回滚策略 + +- Wave 1–3 可回滚代码，但不得回滚/重建 storage UUID；旧相对 bind mount 禁止作为回滚方案。 +- Wave 4 使用 feature flag 分阶段启用 ledger-read 与 projector；关闭 projector 时保留 pending jobs，不删除。 +- 所有自动修复只覆盖 `MISSING + identity matched`；任何 fork 保留证据并停机式处理。 + +## 方案边界 + +- 本轮不修改业务代码、不重建容器、不修复/覆盖任何正文。 +- 当前 409 的即时恢复与系统实现必须由用户批准后，在独占集成槽执行。 +- test/main 仅做只读挂载审计；未经环境晋升授权不得改动。 + +## 相关链接 + +- Audit: [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) +- Short-term: [dev runtime data root guard 诊断与设计](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) +- gap-matrix 行: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/INDEX.md b/.doc/remediation/INDEX.md index d948d641..1ef2f73f 100644 --- a/.doc/remediation/INDEX.md +++ b/.doc/remediation/INDEX.md @@ -3,6 +3,7 @@ | 文档 | 说明 | |------|------| | [gap-matrix.md](./gap-matrix.md) | 发布缺口 SSOT | +| [2026-09-14-dev-runtime-data-root-and-projection-remediation.md](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | 临时 worktree 错挂 dev 数据根、storage identity、readiness 与 projection outbox 系统整改 | | [2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md](./2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md) | 项目设置 409 冻结态守卫 + 非必需文档三权全有或全无 + 成员/门禁/权限一致性 | | [2026-09-14-controlled-content-canonical-hash-remediation.md](./2026-09-14-controlled-content-canonical-hash-remediation.md) | 项目 143 正文 canonical/raw 哈希漂移、错误语义与重复 hydration 整改 | | [2026-09-13-document-revision-reviewer-duty-sync.md](./2026-09-13-document-revision-reviewer-duty-sync.md) | UI-01 已分配 B=reviewer，受控文档 reviewer_id 硬编码 2，矩阵降为 viewer | diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index 249e4218..ddc172c2 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -2,7 +2,7 @@ **更新**: 2026-09-14 **分支**: dev -**production_ready**: no（`CHAT-INTENT-STATE` 等无关 blocker、CI 真实 PI 三工作流 E2E / 前端人工抽检未完成；文档账本运行时 UI Round20 已 PASS，不把 fallback 报告当成完整 PI） +**production_ready**: no（`RUNTIME-DATA-ROOT-SPLIT-BRAIN` critical：当前 dev DB 与正文文件根分裂且 readiness 假阳性；另有 `CHAT-INTENT-STATE`、CI 真实 PI 与前端抽检 blocker） **协作焦点**: [../ops/CURRENT.md](../ops/CURRENT.md) **Agent 手册**: [AGENTS.md](../AGENTS.md) · [PROJECT.md](../PROJECT.md) @@ -57,6 +57,7 @@ | ID | 描述 | 状态 | 证据 | 计划动作 | |---|---|---|---|---| +| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。项目 143 主仓文件、DB、ledger head SHA 一致，容器内同路径不存在；Compose working dir 与三容器 mount source 均为 `/tmp/as-dev-wf409-merge` | [remediation](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md)。先占槽冻结/差异保全/恢复固定根，再实施绝对 state root、storage identity、readiness 和 projection outbox；恢复前禁止用 dev healthy 或文档 E2E 签字 | | WORKFLOW-PLAN-FROZEN-OPTIONAL-DUTIES | active 项目仍展示可编辑职责表单，409 提示在长表单视口外；现有规划又不支持非必需文档“三权全空”，且后端缺少职责人在任成员复核 | **PASS（设置页）** | [audit](../audit/2026-09-14-workflow-plan-409-frozen-and-optional-duties.md)；[test-report](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)。项目 143 只读 frozen、A full+none 激活、B partial 不发 PUT、C 零治理拒绝、双标签转 frozen、成员无配置入口、刷新后 143 仍 frozen | 真实 PI 生成 full/none 并走审批/阶段不阻塞未纳入本轮页面矩阵，不作为本 gap 关闭的替代证据，另立项 | | DOC-CONTENT-CANONICAL-HASH-DRIFT | 旧受控文档 bootstrap 裁剪 Markdown 双空格硬换行并将 canonical SHA 写入账本/DocGov，但未同步文件；正文读取把完整性冲突误报 404，lease/list sync 又放大并制造刷新后加载死锁 | **PASS（真实 UI）** | [audit](../audit/2026-09-14-project143-controlled-content-404.md)；[UI report](../test-reports/2026-09-14-controlled-content-ui-acceptance.md)。代码已合 dev、schema 32、项目 143 定向 `migration_repair`；项目 143 `clean-run3` 与新项目 `full-chain-final` 主链及 restart 全部纯页面 PASS，目标 404=0 | [remediation](./2026-09-14-controlled-content-canonical-hash-remediation.md)。隔离 `TestReconcile_LegacyRepair_PG` 未执行，按用户“本轮只允许页面测试”的边界保留为工程测试债，不作为本轮 UI PASS 的替代证据 | | CHAT-HISTORY-REFRESH | 工作流激活后刷新，对话正文被阶段过滤器清空，但分类数量与锚点仍按全量返回 | **PASS** | [test-report](../test-reports/2026-09-13-chat-history-refresh-test-report.md)；代码：`message_scope.go`、`chat_history_auth.go`、goose 00026/00027 | 真实 PI 刷新 E2E 待 dev 容器补录（自动化/PG 已 PASS） | diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index f56e25b4..43cc436d 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -21,11 +21,11 @@ | 工作流治理 | 三正式工作流 + 阶段门禁 | partial | [remediation/three-workflows](../remediation/2026-09-12-three-workflows-strict-governance-remediation.md)；[test 2026-09-12](../test-reports/2026-09-12-three-workflows-governance-remediation-test-report.md)；[冻结态/职责可选整改](../remediation/2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md)；[Wave 5 页面矩阵](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)；[职责-动作鉴权](../test-reports/2026-09-14-governance-action-level-authz-test-report.md) | 设置页/DutyMode + 动作级鉴权/未激活 202-blocked/关闭创建直通已落地。后续：none 文档真实 PI 审批长链路、CI 三工作流 E2E | `workflow` | | 纯软件项目自举 | 软件项目创建、Bootstrap 基线、Delivery 迭代与受控工程交接 | planned | [详细设计](../system-design/2026-09-14-software-project-bootstrap-workflow.md)；[现状调研](../ops/research/2026-09-14-software-project-bootstrap-current-state.md)；[验收计划](../ops/checklists/2026-09-14-software-project-bootstrap-acceptance.md) | 当前仅完成方案；目录/模板/执行器/创建向导/真实 PI UI 均未实施。受控代码执行单列后续 Epic | `workflow-design` | | IAM / 鉴权 | 项目级 API 鉴权 + RBAC | partial | [audit/authorization](../audit/authorization-matrix-v2.md)；[刷新恢复专项审计](../audit/2026-09-13-chat-history-refresh-disappears.md) | 历史读取端点补统一会话 owner + 阶段授权；旧 API 410 标注(BUG-11 partial) | `authz` | -| 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | done | [test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 正文 404 审计](../audit/2026-09-14-project143-controlled-content-404.md)；[受控正文真实 UI 验收报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md)；[UI-06 缓存竞争审计](../audit/2026-09-14-document-revision-refresh-cache-race.md)；[整改](../remediation/2026-09-14-controlled-content-canonical-hash-remediation.md) | 生命周期/租约保留。W1–W6 + 审查修补 + UI hydrate 资格 + 晚到 list sync 修复均已合入 dev。项目 143 纯 UI 回归（clean-run3）与新项目全链路（main + post-restart）已全量真实页面验证通过（PASS） | `docgov` | +| 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；但当前 dev 因临时 worktree 数据根 split-brain 再次不可读。须完成固定 state root、storage identity、projection outbox/自愈和 rebuild 负例后恢复 `done` | `docgov` | | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | -| 三环境运维 | dev/test/main + 共享 KB | done | [PROJECT.md](../../PROJECT.md) §4；四环境 healthy | — | `infra` | +| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) | dev app/pi/cursor 当前错挂 `/tmp` 空数据根且 readiness 假阳性；同类相对 bind 设计覆盖 test/main，须完成绝对 state root 与身份门禁 | `infra` | | CI / E2E | PG 集成 + 真实 PI E2E | partial | [gap-matrix BUG-08](../remediation/gap-matrix.md)；[test pg-ci](../test-reports/2026-09-12-pg-ci-soak-e2e-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md) | CI `TEST_DATABASE_URL` 隔离库仍缺；文档运行时真实 UI 已按 UI_PHASE 分文件且 post-restart 断言最终 A1 正文。`-race` 全包并行 TempDir flake 为基建残留 | `ci` | --- From ccb1b1ebe53eaa99da80accf73b6fd8c55ea1816 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 09:39:30 +0000 Subject: [PATCH 02/13] docs(remediation): design runtime data lifecycle --- .doc/ops/CURRENT.md | 4 +- ...ime-data-retention-and-migration-corpus.md | 147 +++++++++++++++ .../2026-09-14-runtime-data-lifecycle-plan.md | 56 ++++++ ...me-data-root-and-projection-remediation.md | 3 +- ...ecycle-and-migration-corpus-remediation.md | 171 ++++++++++++++++++ .doc/remediation/INDEX.md | 1 + .doc/remediation/gap-matrix.md | 2 +- .doc/status/FEATURE-MATRIX.md | 2 +- 8 files changed, 382 insertions(+), 4 deletions(-) create mode 100644 .doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md create mode 100644 .doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md create mode 100644 .doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index 392442e2..a3c12a80 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -5,7 +5,7 @@ ## 焦点 -1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；当前 app/pi/cursor 挂到 `/tmp/as-dev-wf409-merge/data/dev*`，DB 仍为原 dev 卷；待批准后占槽恢复并系统整改） +1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；当前 app/pi/cursor 挂到 `/tmp/as-dev-wf409-merge/data/dev*`，DB 仍为原 dev 卷；整改包含[数据精简与迁移语料](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)） 2. **CHAT-INTENT**: [对话意图运行时状态](../audit/2026-09-13-conversation-intent-runtime-state.md)（`fix/conversation-intent-state`；与文档账本运行时无关） 3. **CHAT**: [对话历史刷新](../audit/2026-09-13-chat-history-refresh-disappears.md)（认领仍 `in_progress`；gap 行已 PASS，以代码/handoff 为准收敛） @@ -49,6 +49,7 @@ | implementer_cursor | workflow-plan-409-frozen-and-optional-duties | fix/workflow-plan-409-frozen-and-optional-duties | .worktrees/fix-workflow-plan-409-frozen-and-optional-duties | workflow | done | | implementer_cursor | governance-action-level-authz | fix/governance-action-level-authz | .worktrees/fix-governance-action-level-authz | workflow | done | | Codex | dev-runtime-data-root-guard | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | +| Codex | runtime-data-lifecycle-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 @@ -110,3 +111,4 @@ | 受控正文真实 UI 验收报告 | [../test-reports/2026-09-14-controlled-content-ui-acceptance.md](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | | dev 数据根 split-brain 审计 | [../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) | | dev 数据根与投影系统整改 | [../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | +| 数据生命周期与迁移语料 | [../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | diff --git a/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md new file mode 100644 index 00000000..179f5c3e --- /dev/null +++ b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md @@ -0,0 +1,147 @@ +--- +title: "运行数据精简、迁移语料与演示数据保留策略" +created: 2026-09-14 +status: done +confidence: high +--- + +# 调研：运行数据精简、迁移语料与演示数据保留策略 + +## 结论 + +用户提出的“及时清理过往数据，只保留最新、有用且必要的一部分，并用保留数据验证迁移和重载”方向合理，尤其适合当前快速迭代期的 dev/test 环境。 + +但必须补充两个边界： + +1. 清理历史运行数据只能降低数据转换量和现场噪声，不能替代对受支持旧 schema 的迁移兼容；应把有价值的旧状态压缩成小型、可重复、脱敏的 migration corpus。 +2. 不应在当前库上逐表删除后继续使用；当前跨项目数据存在非完整级联和 DB/文件双事实源，优先采用“新库 + 新状态根 + 选择性导入 + 旧卷隔离”的蓝绿轮换。 + +## 当前数据画像 + +2026-09-14 只读现场： + +| 环境 | schema | 项目数/特征 | 文件状态 | 判断 | +|---|---:|---|---|---| +| dev | 33 | 366 个项目；149 active、217 planning；全部产生于 9 月 11–14 日 | 主仓 `data/dev` 约 100MB | 大量 PG/E2E 夹具，适合立即精简，但须先解决当前错误挂载事故 | +| test | 29 | 约 37 个项目 | 主仓 `data/test` 约 20KB | schema 落后且数据价值低，应改为可重建/短 TTL | +| main | 25 | 10 个历史 seed 项目，当前无受控文档 | 主仓 `data/main` 约 20KB | 不应用作日常迁移试验；清理需另行确认业务/展示用途 | +| shared KB | 独立 | 跨环境共享知识库 | `data/shared` | 不属于本方案的项目数据清理范围 | + +dev 主要表约有：`projects=366`、`chat_sessions=421`、`chat_messages=1923`、`task_runs=307`、`document_records=153`、`document_snapshots=283`、`document_change_events=257`。数据库总体不大，主要问题不是容量，而是数据来源混杂、迁移噪声和无法判定哪些记录应保留。 + +## 现有清理为何不能直接做 + +### 1. `DELETE projects` 不是完整项目删除 + +当前外键有 101 条，部分项目子表 CASCADE，但存在关键例外： + +- `task_runs.project_id` 是 `ON DELETE SET NULL`，项目删除后 run/event 仍可残留； +- `document_versions`、`document_snapshots`、`document_change_events`、`document_change_reports`、`document_agent_candidates` 依赖 `doc_id`/snapshot/version，但不完整依附 `projects`； +- `document_content_blobs` 与 `storage_blobs` 需要引用计数/mark-and-sweep，不能看到文件就删； +- 工作区、agent sessions、CAS 文件位于状态根，数据库 cascade 不会删除文件； +- notification/outbox/audit/lineage 的保留语义不同，不能一律级联。 + +所以简单删除项目会制造孤儿数据，反而加重后续迁移与排障成本。 + +### 2. 现有备份/恢复不足以支撑安全轮换 + +- `backup-system-db.sh` 只导出 PostgreSQL，不包含与之匹配的 `data/` 文件状态； +- `restore-system-db-drill.sh` 校验 dump 列表并运行迁移测试，但当前没有真正把 dump restore 到新隔离数据库； +- restore 未强制隔离目标，默认路径存在误触共享 dev 库的风险； +- 没有 dataset manifest、项目选择、文件 SHA inventory、脱敏、预期断言与导入校验； +- 备份脚本的凭证来源需要改成统一安全配置，不应保留脚本内默认值。 + +在这些缺口补齐前，不能用“有 dump”作为可删除证明。 + +## 建议的数据分类 + +| 类别 | 用途 | 典型内容 | 是否进入日常运行库 | 保留策略 | +|---|---|---|---|---| +| `operational-pinned` | 当前人工使用/正在排障 | 项目 143 等明确认领项目 | dev 可保留；main 按业务规则 | 显式 pin，禁止自动清理 | +| `demo-current` | 产品展示 | 每个正式工作流 1 个完整、可解释项目 | dev/demo 环境 | 只保留当前版本；新版验证后替换旧版 | +| `regression-golden` | 功能回归 | 文档版本、人工修订、并发冲突、reload 等完整链 | 隔离验收库，可选只读展示副本 | 小型、确定性、脱敏；按能力而非数量保留 | +| `migration-baseline` | 旧 schema → 当前 schema | 旧 release 的最小 DB + 匹配文件包 | 仅隔离迁移测试 | 最多保留 3 个 release 基线，滚动替换 | +| `ephemeral-test` | PG/E2E 单次运行 | 自动测试项目、会话、run、workspace | 每次独立数据库/状态根 | PASS 后立即删；FAIL 保留 3–7 天 | +| `quarantine` | 清理回滚与取证 | 被替换的完整卷/状态根 | 不挂载到运行容器 | 只读 7–14 天，恢复演练通过后销毁 | + +## 建议的最小保留集合 + +当前 dev 可先采用 6–10 个项目的上限，而不是按日期盲留： + +1. 项目 143：暂作旧数据/canonical repair/active workflow 回归样本；若含真实内容，不进入 Git，后续用脱敏等价 fixture 替换。 +2. 项目 6173：目前最新且证据完整的受控正文 UI 全链，含两份受控文档、V3/r2、并发、权限与 restart 验证，可作为当前回归/展示候选。 +3. 项目 6159：保留一个发布周期作为上一轮文档账本对照；6173 corpus 稳定后删除，避免永久重复。 +4. 三个正式工作流各生成 1 个 `demo-current`：`main`、`product_manager`、`test_manager`；必须由固定 seed/真实页面脚本可重建，不能依赖手工挑出的偶然状态。 +5. 1 个失败/异常样本不进入展示库，只进入隔离 `regression-golden`：覆盖 projection missing、hard-break、旧消息 scope、高 ID sequence 等历史风险。 + +最新项目不等于最有价值。选择评分应优先：证据完整度、覆盖的能力、是否可脱敏重建、DB/文件一致性、是否有预期断言；时间只作为同类样本的淘汰条件。 + +本轮已只读确认 143、6159、6173 共 5 份受控文档在主仓状态根的 `output.md` 均存在且 SHA 与 `document_records` 一致；这说明它们具备候选资格，但仍需通过敏感性、owner 和可恢复性审查后才能进入最终 retain allowlist。 + +## 迁移语料库策略 + +不保存每个 Goose 版本的整库。按产品 release 边界保留： + +- `oldest-supported`：当前承诺支持的最老 release； +- `previous-release`：上一发布版； +- `current-pre-upgrade`：本次上线前快照，下一次轮换为 previous。 + +当前建议先建立三个最小基线： + +- schema 25：文档账本前/早期治理状态，覆盖 main 当前落后路径； +- schema 28/29：文档 revision 首次引入后的状态，覆盖 test 当前路径； +- schema 32/33：canonical repair、task 状态和当前数据根契约。 + +这些基线应由迁移 fixture builder 在隔离库中构造或从已批准快照脱敏得到，不能直接把 main/test 的运行数据当测试资产。 + +每个 corpus package 必须包含： + +```text +manifest.json +database.dump +state.tar.zst +checksums.txt +assertions.yaml +``` + +manifest 至少记录 dataset 版本、source schema、目标 schema、生成 commit、storage identity、项目/会话/文档/run 数量、文件总数和 SHA、脱敏状态、过期时间；`assertions.yaml` 记录迁移后必须成立的版本、状态、正文 hash、序列与权限断言。 + +二进制 dump/状态包不进 Git；仓库只保留 manifest schema、fixture builder、脱敏规则、选择规则和非敏感断言。实际包放受控备份目录或对象存储。 + +## 每次发布的数据轮换模型 + +```text +旧运行库/状态根（只读） + │ export + manifest + checksums + ▼ +迁移语料构建区（脱敏/裁剪/恢复验证） + │ + ├── oldest-supported corpus + ├── previous-release corpus + └── current demo/regression corpus + │ + ▼ +新数据库 + 新状态根（latest schema） + │ import selected datasets + ▼ +工程门禁 + 页面验收 + rebuild/reload + │ PASS + ▼ +切换运行 identity；旧卷 quarantine 7–14 天后销毁 +``` + +这比在原 dev 库上大规模 DELETE 更适合当前阶段：新库天然消除未知孤儿，导入过程本身又验证了备份、迁移、文件配对、序列和重载。 + +## 风险边界 + +- main 真实业务数据不能套用 dev 的 7–14 天策略；须先定义业务、合规、审计和用户删除规则。 +- 项目 143 等真实内容不能直接固化为仓库 fixture；应保留 hash/结构特征并脱敏重建。 +- migration corpus 不能只有 happy path，否则清场后会失去发现 legacy bug 的能力。 +- 任何清理必须先完成隔离 restore，并验证数据库与文件包属于同一 storage identity。 +- 错误 `/tmp` 数据根属于当前事故证据；在 split-brain 恢复报告签字前不纳入普通清理。 + +## 建议动作 + +- 采用 [运行数据生命周期与迁移语料整改方案](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)。 +- 首轮只处理 dev；test 改为短 TTL 隔离环境；main 仅盘点，不删除；shared KB 排除。 +- 先实现 dataset inventory/export/restore/verify，再执行蓝绿精简，不先写清理 SQL。 diff --git a/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md new file mode 100644 index 00000000..9a358b17 --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md @@ -0,0 +1,56 @@ +--- +title: "运行数据精简与迁移语料库方案" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Codex +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: infra-runtime-state +--- + +# 运行数据精简与迁移语料库方案 + +## 目标 + +评估及时清理旧数据建议的合理性，并设计可恢复、可迁移验证、可展示且不污染长期环境的数据生命周期方案。 + +## 范围 + +- dev/test/main 数据量、schema、项目/文档/run 关系与文件状态； +- 备份/恢复、选择性导出、迁移 corpus、蓝绿轮换； +- retention 分类、TTL、quarantine、GC 和验收门禁。 + +## 非目标 + +- 不执行 DELETE/TRUNCATE/drop volume； +- 不修改数据库、文件正文或容器； +- 不清理 main/shared KB； +- 不把现有项目 ID 清单视作删除授权。 + +## 步骤 + +1. [x] 只读统计三环境 schema、项目/关键表规模和 dev 文件规模。 +2. [x] 审计项目外键闭包、revision/CAS/workspace 孤儿风险。 +3. [x] 审计现有 backup/restore drill 能力与隔离缺口。 +4. [x] 制定数据分类、最小保留集、migration corpus 和蓝绿轮换方案。 +5. [x] 更新父整改、CURRENT、gap 与 remediation 索引。 + +## 验收标准 + +- [x] 明确说明建议合理的条件和不能替代 migration 的边界。 +- [x] 首轮 retain 建议基于能力覆盖，不按时间盲留。 +- [x] 方案包含备份配对、真实 restore、选择性导入、reload/rebuild、rollback 和延迟销毁。 +- [x] 本轮未实施破坏性操作。 + +## 相关链接 + +- Research: [../research/2026-09-14-runtime-data-retention-and-migration-corpus.md](../research/2026-09-14-runtime-data-retention-and-migration-corpus.md) +- Remediation: [../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Test report: 待用户批准实施后创建 + +## 证据 / 备注 + +- dev schema 33 / 366 projects / 主仓状态约 100MB；test schema 29；main schema 25。 +- `DELETE projects` 无法完整清理 task run、revision ledger、blob 与文件状态。 +- 首选新 green DB/state root 导入 retain allowlist，旧 blue 只读隔离后延迟销毁。 diff --git a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md index 24b5d950..3e767829 100644 --- a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md +++ b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md @@ -73,7 +73,7 @@ last_error_code, next_retry_at, updated_at | Wave 2 Compose 解耦 | dev/test/main/shared compose 的可变 bind 全部改为绝对 state-root 变量；为 app/pi/cursor 添加 storage UUID/环境标签；CI 检查 `docker compose config` 不出现工作树内 `data/` | 从两个不同 source roots 渲染 config，数据挂载 source 完全相同；代码 build context 可不同 | | Wave 3 启动与 readiness | 引入文件/DB storage identity；readiness 增加 `storage_ready`、`storage_id_match`、runner mount/identity；部署后脚本核对 container labels + mounts + sentinel + active-doc projection 摘要 | 错误空根启动被拒绝；本次 split-brain 复现实验 `/ready` 必须 503，不能再显示 healthy | | Wave 4 投影状态机 | ledger read 与兼容 projection 解耦；实现 projection outbox、幂等 projector、missing 安全重建、fork 隔离和结构化状态/告警 | DB commit 后 kill -9、只读目录、磁盘满、延迟重试均不丢 head；恢复后自动 aligned；fork 不被覆盖 | -| Wave 5 数据治理 | 清理/标注 dev 中历史临时 `work_dir`；真实项目、测试 fixture、临时 runner 目录分 namespace；增加 orphan/missing/fork 定期巡检和保留策略 | active 非模板文档 projection 100% 可解释；测试临时路径不污染生产统计；巡检报告可追溯 | +| Wave 5 数据治理 | 按[数据生命周期与迁移语料子方案](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)执行蓝绿精简；真实项目、demo、migration corpus、临时测试分 namespace；增加 orphan/missing/fork 巡检与 TTL | dev 常驻 6–10 个可解释项目；受支持旧 schema corpus 可重复迁移；测试临时路径不污染运行库 | | Wave 6 验收与回滚演练 | 工程测试、容器 rebuild/recreate、宿主重启、真实 UI 修改/刷新/重登、双用户冲突、真实 PI 修改、回滚演练 | 下述验收矩阵全 PASS，报告归档后才恢复 FEATURE-MATRIX `done` 和 gap PASS | ## 即时恢复 Runbook（批准实施后执行） @@ -135,4 +135,5 @@ last_error_code, next_retry_at, updated_at - Audit: [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) - Short-term: [dev runtime data root guard 诊断与设计](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) +- Data lifecycle: [运行数据生命周期与迁移语料库](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) - gap-matrix 行: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md new file mode 100644 index 00000000..393de773 --- /dev/null +++ b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md @@ -0,0 +1,171 @@ +--- +title: "运行数据生命周期与迁移语料库整改方案" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +--- + +# 运行数据生命周期与迁移语料库整改方案 + +## 背景 + +合理性和当前数据画像见[调研](../ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md)。本方案是 [dev 数据根与投影整改](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) Wave 5 的详细子方案。 + +目标不是“为了省事删数据”，而是把不可控的历史运行库转化为： + +- 少量可展示、可解释的当前数据； +- 少量可重复、可断言的迁移/回归语料； +- 大量自动过期的隔离测试数据； +- 可恢复后再销毁的旧卷。 + +## 目标 + +1. dev 常驻项目控制在 6–10 个，均有 owner、用途、数据类别和过期/固定理由。 +2. test 不再积累跨轮测试数据；每轮使用独立 DB + state root，PASS 即删，FAIL 短期保留。 +3. 每次发布同时验证 clean install、受支持旧版本升级、选择性数据导入、reload/rebuild 和 rollback。 +4. 数据库 dump 与文件状态根成对归档，具备 storage identity、SHA manifest 和自动断言。 +5. 清理不产生孤儿 snapshot/run/blob/workspace，不误删共享 blob，不接触未授权 main/KB 数据。 + +## 数据保留策略 + +| 环境 | 常驻内容 | 自动过期 | 明确保留 | 禁止事项 | +|---|---|---|---|---| +| dev | 3 个 workflow demo + 2–4 个当前 regression + 明确人工项目 | 普通 E2E 24h；失败运行 7 天；上一版 demo 14 天 | pinned 项目、当前/上一 release 对照 | 不按 ID/目录 mtime 直接删除；不把真实内容提交为 fixture | +| test | 无长期业务项目 | PASS 立即销毁；FAIL 3–7 天 | 最近 1 次失败环境用于排障 | 不连接 dev/test 共享库跑 destructive test | +| main | 真实业务/正式展示 | 暂不自动过期 | 由产品、审计、合规规则决定 | 本方案不授权清理或测试写入 | +| shared KB | 法规知识库 | 独立策略 | 现有 KB 备份体系 | 不随项目数据轮换 | + +## 数据标识模型 + +新增数据集注册表或等价 SSOT： + +```text +dataset_id +class: operational_pinned | demo_current | regression_golden | + migration_baseline | ephemeral_test | quarantine +environment +source_schema / target_schema +source_commit +storage_identity +owner +retention_until +pinned_reason +contains_sensitive_data +manifest_uri / manifest_sha256 +status: building | verified | active | quarantined | expired | destroyed +``` + +项目至少增加 `dataset_id`、`data_origin`、`retention_until`、`pinned_at/pinned_reason`；自动测试创建项目时必须带 `ephemeral_test` origin 和 run namespace。未分类数据默认不可自动删除，而不是默认过期。 + +## 工具设计 + +### 1. `dataset inventory` + +只读生成候选清单：项目关系行数、doc/run/session IDs、workspace、CAS 引用、最后活动时间、生命周期、是否有 ledger/fork/missing、敏感级别、推荐动作。输出 JSON + Markdown；不接受模糊环境名。 + +### 2. `dataset export` + +按 dataset/project allowlist 导出关系闭包，不能只导 `projects` 表。必须同时打包： + +- project/chat/task/workflow/governance/document/revision/notification/outbox/audit 关系； +- 账本 blob 与受控/自由文档文件； +- file reference 指向的 CAS blob； +- workspace/agent session 中经 allowlist 认领的文件； +- schema/version/storage identity/count/SHA manifest。 + +导出后在隔离区执行脱敏；真实账号 token、auth sessions、credential、原始敏感附件默认剔除或替换。 + +### 3. `dataset restore --isolated-only` + +强制目标数据库名/端口/state root 满足隔离规则；拒绝 5433/5434/5435 及 `agent_system_dev|test|main`。实际执行 `pg_restore`，再导入文件包，不能只做 `pg_restore --list`。 + +恢复顺序：创建空库 → 恢复 source schema/data → 校验 source assertions → 执行 Goose up → 校验 target assertions → 重建 projection/index → restart/reload → 再次校验。 + +### 4. `dataset rotate` + +生成计划而不直接删除：`retain | replace | quarantine | destroy`。执行必须使用 manifest ID、期望 storage UUID 和确认短语；禁止 `--all`、模糊 glob、未解析环境变量。 + +### 5. `dataset gc` + +仅在数据库引用闭包和文件 manifest 都验证后执行： + +- 删除明确 dataset/project 的关系闭包； +- 清理无引用 document content blob/CAS blob； +- 清理已归属 workspace/session； +- `setval`/identity sequence 修复； +- 输出 before/after counts、孤儿检查与可恢复位置。 + +首轮 dev 推荐不在旧库运行 GC，而是蓝绿新建后切换；GC 用于后续小规模过期数据。 + +## 迁移测试矩阵 + +每个 release candidate 必须执行： + +| 测试轨 | 输入 | 验证重点 | +|---|---|---| +| Clean | 空 PostgreSQL + 空 state root | migrations 1→latest、官方 workflow seed、storage identity、ready | +| Oldest | `oldest-supported` corpus | 跨版本 backfill、约束、ID sequence、文档 legacy repair | +| Previous | `previous-release` corpus | 常规升级、API/UI 兼容、性能 | +| Hazard | 最小异常 corpus | hard-break、projection missing/fork、旧消息 scope、高 ID、pending run | +| Import | latest schema + selected demo/regression package | 关系闭包、文件 SHA、权限、通知、版本账本 | +| Reload | import 后 app/pi rebuild + restart + 宿主级重载 | storage identity、mount、task 恢复、正文/overlay | +| Rollback | 切换失败后接回 quarantine 旧卷 | RTO、旧 storage identity、无新写丢失 | + +迁移通过必须包含真实 restore，不接受仅跑 Goose 幂等测试；产品签字至少包含真实页面登录、项目列表、对话、文档打开/修改、刷新、退出重登和 rebuild 后恢复。 + +## 初次精简执行波次 + +| 波次 | 动作 | 完成判据 | +|---|---|---| +| D0 冻结与标注 | 先完成当前 split-brain 遏制；冻结 dev 写入；对 366 个项目 inventory；人工确认 pinned/敏感/owner | 100% 项目有建议动作；未分类项不删除 | +| D1 工具安全修补 | 移除备份脚本内默认凭证；backup 同时归档 DB+state；restore 强制 scratch 且真实恢复；加入 manifest/checksum/assertions | 恢复演练能从零创建隔离环境并通过 SHA/关系断言 | +| D2 构建 corpus | 建 schema 25、28/29、32/33 三个最小迁移包；构建 143 等价脱敏 hazard；选择 6173 current regression；6159 标记一周期过期 | 每包可重复生成/恢复，不依赖真实 main/test 数据 | +| D3 新建 green dev | latest schema 空库 + 固定新 state root；导入选中的 6–10 项；执行 projection/CAS/sequence/integrity 校验 | 无 orphan；active 文档 aligned；ready/storage identity PASS | +| D4 全链验收 | clean/oldest/previous/hazard/import/reload/rollback；真实页面和真实 PI 跑主链 | 报告区分工程、PG、PI、浏览器、restart/rebuild 证据 | +| D5 原子切换 | 切换 dev 到 green identity；原 blue DB/文件根转只读 quarantine | 新环境持续观察 24–72h，无新增 blocker，可一键回接 blue | +| D6 延迟销毁 | quarantine 满 7–14 天，再次 restore drill、差异确认、用户批准后销毁 | 记录销毁 manifest、范围、时间和不可恢复性；不得顺带清 main/KB | + +## 首轮建议保留/淘汰 + +### 保留 + +- 项目 143：临时 pinned，直到脱敏 legacy/hazard corpus 可替代。 +- 项目 6173：作为最新完整受控正文/版本/UI/restart 回归样本。 +- 项目 6159：作为 previous 对照保留一个发布周期，然后由 6173 替换。 +- 三正式工作流各一个新生成的确定性 demo；生成成功后不保留同类失败尝试。 +- 最近一次失败 E2E 的完整环境，最多 7 天，且与常驻 dev 隔离。 + +### 不直接保留 + +- 数百个无文档、无 run 或仅测试前置步骤的 planning/active 项目； +- 重复的 UI Round 中间项目和已由更新 PASS 覆盖的失败项目； +- 能由 unit/PG fixture 稳定重建的测试污染 workflow key、临时用户、session 和 run； +- 无引用 CAS/workspace，需在 mark-and-sweep 验证后删除。 + +该清单是初始建议，不是删除授权；最终 retain allowlist 应由 inventory 报告和用户确认生成。 + +## 验收标准 + +- [ ] dev 常驻项目 6–10 个，所有项目都有 dataset class/owner/retention。 +- [ ] 旧 366 项目对应的 blue DB+state 已成对备份并实际恢复成功。 +- [ ] green dev 从空库建成，选择性导入后 DB/file/storage UUID 一致。 +- [ ] 三个 migration baselines 与 hazard corpus 均能升到 latest 并通过断言。 +- [ ] 无 orphan task run、revision snapshot/version/event/report、CAS blob 或 workspace。 +- [ ] 项目 143/6173 页面打开、修改、刷新、重登、rebuild 后恢复 PASS。 +- [ ] 新测试数据带 TTL，PASS 后自动销毁，失败数据按策略到期。 +- [ ] test/main/shared KB 未被误操作；所有破坏性动作有精确 manifest 和延迟销毁记录。 + +## 不建议的方案 + +- 每次直接 `docker compose down -v` 再凭记忆重建：无法证明保留数据和文件配对完整。 +- 在 dev 库中写一份巨型级联 DELETE：当前关系与文件引用复杂，容易留下孤儿或误删共享 blob。 +- 只保留最新时间的项目：会丢失最有价值的旧 schema/hazard 覆盖。 +- 把整份真实旧库永久当 migration fixture：迁移慢、不可重复、可能含敏感数据。 +- 修改历史 Goose migration 迎合新空库：已发布迁移不可重写；修正必须新增 migration。 + +## 相关链接 + +- Research: [运行数据精简、迁移语料与演示数据保留策略](../ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md) +- Parent remediation: [dev 数据根隔离与投影自愈](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [运行数据生命周期方案](../ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md) +- gap-matrix: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/INDEX.md b/.doc/remediation/INDEX.md index 1ef2f73f..320fe23d 100644 --- a/.doc/remediation/INDEX.md +++ b/.doc/remediation/INDEX.md @@ -4,6 +4,7 @@ |------|------| | [gap-matrix.md](./gap-matrix.md) | 发布缺口 SSOT | | [2026-09-14-dev-runtime-data-root-and-projection-remediation.md](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | 临时 worktree 错挂 dev 数据根、storage identity、readiness 与 projection outbox 系统整改 | +| [2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | dev/test 数据精简、migration corpus、蓝绿换库、TTL 与延迟销毁方案 | | [2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md](./2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md) | 项目设置 409 冻结态守卫 + 非必需文档三权全有或全无 + 成员/门禁/权限一致性 | | [2026-09-14-controlled-content-canonical-hash-remediation.md](./2026-09-14-controlled-content-canonical-hash-remediation.md) | 项目 143 正文 canonical/raw 哈希漂移、错误语义与重复 hydration 整改 | | [2026-09-13-document-revision-reviewer-duty-sync.md](./2026-09-13-document-revision-reviewer-duty-sync.md) | UI-01 已分配 B=reviewer，受控文档 reviewer_id 硬编码 2，矩阵降为 viewer | diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index ddc172c2..30d3066c 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -57,7 +57,7 @@ | ID | 描述 | 状态 | 证据 | 计划动作 | |---|---|---|---|---| -| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。项目 143 主仓文件、DB、ledger head SHA 一致，容器内同路径不存在；Compose working dir 与三容器 mount source 均为 `/tmp/as-dev-wf409-merge` | [remediation](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md)。先占槽冻结/差异保全/恢复固定根，再实施绝对 state root、storage identity、readiness 和 projection outbox；恢复前禁止用 dev healthy 或文档 E2E 签字 | +| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。项目 143 主仓文件、DB、ledger head SHA 一致，容器内同路径不存在；Compose working dir 与三容器 mount source 均为 `/tmp/as-dev-wf409-merge` | [runtime remediation](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) + [data lifecycle](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)。先恢复固定根，再以蓝绿新库、最小迁移 corpus 和延迟销毁精简 dev；恢复前禁止用 dev healthy 或文档 E2E 签字 | | WORKFLOW-PLAN-FROZEN-OPTIONAL-DUTIES | active 项目仍展示可编辑职责表单，409 提示在长表单视口外；现有规划又不支持非必需文档“三权全空”，且后端缺少职责人在任成员复核 | **PASS（设置页）** | [audit](../audit/2026-09-14-workflow-plan-409-frozen-and-optional-duties.md)；[test-report](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)。项目 143 只读 frozen、A full+none 激活、B partial 不发 PUT、C 零治理拒绝、双标签转 frozen、成员无配置入口、刷新后 143 仍 frozen | 真实 PI 生成 full/none 并走审批/阶段不阻塞未纳入本轮页面矩阵，不作为本 gap 关闭的替代证据，另立项 | | DOC-CONTENT-CANONICAL-HASH-DRIFT | 旧受控文档 bootstrap 裁剪 Markdown 双空格硬换行并将 canonical SHA 写入账本/DocGov，但未同步文件；正文读取把完整性冲突误报 404，lease/list sync 又放大并制造刷新后加载死锁 | **PASS（真实 UI）** | [audit](../audit/2026-09-14-project143-controlled-content-404.md)；[UI report](../test-reports/2026-09-14-controlled-content-ui-acceptance.md)。代码已合 dev、schema 32、项目 143 定向 `migration_repair`；项目 143 `clean-run3` 与新项目 `full-chain-final` 主链及 restart 全部纯页面 PASS，目标 404=0 | [remediation](./2026-09-14-controlled-content-canonical-hash-remediation.md)。隔离 `TestReconcile_LegacyRepair_PG` 未执行，按用户“本轮只允许页面测试”的边界保留为工程测试债，不作为本轮 UI PASS 的替代证据 | | CHAT-HISTORY-REFRESH | 工作流激活后刷新，对话正文被阶段过滤器清空，但分类数量与锚点仍按全量返回 | **PASS** | [test-report](../test-reports/2026-09-13-chat-history-refresh-test-report.md)；代码：`message_scope.go`、`chat_history_auth.go`、goose 00026/00027 | 真实 PI 刷新 E2E 待 dev 容器补录（自动化/PG 已 PASS） | diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index 43cc436d..d4799dc0 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -25,7 +25,7 @@ | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | -| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) | dev app/pi/cursor 当前错挂 `/tmp` 空数据根且 readiness 假阳性；同类相对 bind 设计覆盖 test/main，须完成绝对 state root 与身份门禁 | `infra` | +| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | dev 当前错挂 `/tmp` 空数据根且 readiness 假阳性；须完成绝对 state root/身份门禁，并以蓝绿换库将 366 个混杂项目收敛为 demo/regression/migration corpus + TTL 测试数据 | `infra` | | CI / E2E | PG 集成 + 真实 PI E2E | partial | [gap-matrix BUG-08](../remediation/gap-matrix.md)；[test pg-ci](../test-reports/2026-09-12-pg-ci-soak-e2e-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md) | CI `TEST_DATABASE_URL` 隔离库仍缺；文档运行时真实 UI 已按 UI_PHASE 分文件且 post-restart 断言最终 A1 正文。`-race` 全包并行 TempDir flake 为基建残留 | `ci` | --- From d1e2cc929d78cd8ce1287da47500a6083a14f222 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:01:40 +0000 Subject: [PATCH 03/13] docs(remediation): rewrite data-root plan after review Split incident remount from outbox/governance so Wave 0 is executable without identity bootstrap, and record preempt-but-wait authorization. Co-authored-by: Cursor --- ...09-14-dev-runtime-data-root-split-brain.md | 8 +- .doc/ops/CURRENT.md | 7 +- .../2026-09-14-dev-runtime-data-root-guard.md | 44 +-- ...me-data-root-and-projection-remediation.md | 267 +++++++++++------- .doc/remediation/INDEX.md | 2 +- .doc/remediation/gap-matrix.md | 2 +- .doc/status/FEATURE-MATRIX.md | 5 +- 7 files changed, 213 insertions(+), 122 deletions(-) diff --git a/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md index 49743b3f..b10b43d9 100644 --- a/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md +++ b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md @@ -158,4 +158,10 @@ Compose 按 project working directory 解析相对路径。`scripts/env-service. ## 开放问题 - 恢复前需再次冻结并备份错误根，确认 08:40 UTC 后是否已有新会话、附件、runner 输出或 agent session 写入；08:54 UTC 抽样未发现 `output.md`，不能替代完整差异清单。 -- test/main 当前挂载源需在实施 Wave 0 只读核对；本轮未扩大操作范围。 +- test/main 当前挂载源需在实施 Unit A 只读核对；本轮未扩大操作范围。 + +## 审查附记（2026-09-14 ~09:23 UTC） + +- 现场复核：`shuimu-agent-app-dev` 仍 created `08:40:44Z`，bind source 仍为 `/tmp/as-dev-wf409-merge/data/dev`；`/health` `/ready` 仍 200，`schema_version=33`。 +- 主仓 CURRENT 集成槽仍为 `implementer_cursor` / `e2e`。worktree 副本不得写成 idle。 +- 方案已审查修订：见 [整改方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)。用户选择事故抢占 + 先改方案再批实施。08:54 的 `f283...f039` 不得再当 Wave 0 硬门槛。 diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index a3c12a80..dfb032e7 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -5,7 +5,7 @@ ## 焦点 -1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；当前 app/pi/cursor 挂到 `/tmp/as-dev-wf409-merge/data/dev*`，DB 仍为原 dev 卷；整改包含[数据精简与迁移语料](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)） +1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；审查修订已完成。现场仍错挂 `/tmp/as-dev-wf409-merge`；Unit A 事故抢占纠挂与 Unit B 防复发**待再批**。数据精简为 sibling 方案，不挡 Unit A） 2. **CHAT-INTENT**: [对话意图运行时状态](../audit/2026-09-13-conversation-intent-runtime-state.md)（`fix/conversation-intent-state`；与文档账本运行时无关） 3. **CHAT**: [对话历史刷新](../audit/2026-09-13-chat-history-refresh-disappears.md)（认领仍 `in_progress`；gap 行已 PASS，以代码/handoff 为准收敛） @@ -49,6 +49,7 @@ | implementer_cursor | workflow-plan-409-frozen-and-optional-duties | fix/workflow-plan-409-frozen-and-optional-duties | .worktrees/fix-workflow-plan-409-frozen-and-optional-duties | workflow | done | | implementer_cursor | governance-action-level-authz | fix/governance-action-level-authz | .worktrees/fix-governance-action-level-authz | workflow | done | | Codex | dev-runtime-data-root-guard | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | +| Cursor | dev-runtime-data-root-guard-plan-rewrite | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | in_progress | | Codex | runtime-data-lifecycle-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 @@ -59,7 +60,9 @@ | holder | phase | since | scope | queue | |--------|-------|-------|-------|-------| -| - | idle | - | - | - | +| implementer_cursor（主仓 SSOT） | e2e | 2026-09-14 | workflow-plan 409 frozen + optional duties | governance-action-level-authz；Unit A 事故抢占待再批 | + +> 槽位以**主仓** `.doc/ops/CURRENT.md` 与 `docker inspect` 为准。本 worktree 不得把槽位写成 idle 覆盖主仓。Unit A 获批后才能登记 `incident-preempt` 并停 `agent-dev` 的 app/pi/cursor。 **phase**: `idle` | `merging` | `smoke` | `e2e` diff --git a/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md index 15533ddd..d3e1f40c 100644 --- a/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md +++ b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md @@ -1,57 +1,61 @@ --- -title: "dev runtime data root split-brain 诊断与系统整改设计" -status: done +title: "dev runtime data root split-brain 诊断、方案审查与修订" +status: in_progress created: 2026-09-14 updated: 2026-09-14 -owner: Codex +owner: Cursor branch: fix/dev-runtime-data-root-guard worktree: .worktrees/fix-dev-runtime-data-root-guard mutex: infra-runtime-state --- -# dev runtime data root split-brain 诊断与系统整改设计 +# dev runtime data root split-brain 诊断、方案审查与修订 ## 目标 -定位项目 143 刷新后正文 409 的当前根因，量化影响，并形成防复发的系统整改方案；不实施业务代码或共享环境变更。 +定位项目 143 刷新 409 的当前根因，经审查后把整改方案改成可执行、可分批批准的文本；**不实施**纠挂或业务代码，直到用户再批 Unit A / Unit B。 ## 范围 -- 用户报错链、正文读取/投影代码； -- dev PostgreSQL、主仓文件、容器实际 mount、health/readiness； -- Compose 与 `env-service.sh` 的工作目录语义； -- 即时恢复、state-root 隔离、storage identity、projection outbox 与验收设计。 +- 诊断：用户报错链、dev PostgreSQL、主仓文件、容器 mount、health/ready、Compose / `env-service.sh` +- 审查：原文六波次方案的内部矛盾、可行性、抢占与身份首启风险 +- 修订：Unit A / Unit B / Deferred 拆分与可执行 Wave 0 Runbook ## 非目标 -- 不操作真实用户会话； -- 不写数据库、不覆盖正文、不删除错误根； -- 不占共享集成槽、不 rebuild； -- 不变更 test/main。 +- 不操作真实用户会话 +- 不写数据库、不覆盖正文、不删除错误根 +- 本轮不 rebuild、不占集成槽执行纠挂（抢占授权已写入方案，实施另批） +- 不变更运行中的 test/main ## 步骤 1. [x] 完成仓库启动链、既有 canonical hash/refresh 事故文档复核。 2. [x] 比对项目 143 DB record、ledger head、主仓 `output.md` 与容器内路径。 3. [x] 审计 app/pi/cursor mounts、Compose working dir、健康检查与影响面。 -4. [x] 完成 audit、remediation、gap/feature/current SSOT 更新。 +4. [x] 完成初版 audit、remediation、gap/feature/current SSOT。 +5. [x] 多视角审查原文方案；现场复核挂载与集成槽仍为事故态。 +6. [x] 按审查结论重写整改方案（Unit A/B/Deferred + 可执行 Runbook）。 +7. [ ] 用户再批后实施 Unit A（事故抢占纠挂）。 +8. [ ] 用户再批后实施 Unit B（绝对状态根 + 门禁 + `/ready`）。 ## 验收标准 - [x] 直接根因有数据库、文件和容器三方证据。 - [x] 明确区分旧 hash drift、真正 ETag 409 与本次 projection missing。 -- [x] 方案覆盖部署根因、运行时双写、自愈安全边界、负例与真实 UI 验收。 +- [x] 修订方案不再把 Wave 0 依赖尚未存在的 identity / outbox；不再把 Deferred 当事故关门条件。 - [x] 未污染真实会话或修改共享运行态。 +- [ ] Unit A/B 实施与验收（另批）。 ## 相关链接 - Audit: [../../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../../audit/2026-09-14-dev-runtime-data-root-split-brain.md) - Remediation: [../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) -- Test report: 待方案获批并实施后创建 +- Test report: 待 Unit A 实施后创建 ## 证据 / 备注 -- 诊断分支：`fix/dev-runtime-data-root-guard`，基线 `dev@a8f9451`。 -- 2026-09-14 08:54 UTC：app/pi/cursor 均挂 `/tmp/as-dev-wf409-merge/data/dev*`；DB 仍为 `agent-dev-context-pgdata`。 -- 项目 143 主仓 `output.md` SHA 与 DB/head 一致；容器内同路径 missing。 -- `/health`、`/ready` 均错误地返回 200。 +- 诊断分支：`fix/dev-runtime-data-root-guard`，基线 `dev@a8f9451`，诊断 commit `f1edb70`。 +- 2026-09-14 08:54 UTC 诊断：app/pi/cursor 均挂 `/tmp/as-dev-wf409-merge/data/dev*`；DB 仍为 `agent-dev-context-pgdata`。 +- 2026-09-14 ~09:23 UTC 审查复核：同一挂载仍在；`/health` `/ready` 200，`schema_version=33`；主仓 CURRENT 槽位仍为 `implementer_cursor` e2e。 +- 用户选择：Wave 0 = 事故抢占；先改方案再批实施。 diff --git a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md index 3e767829..84d9d29a 100644 --- a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md +++ b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md @@ -1,139 +1,216 @@ --- -title: "dev 运行数据根隔离与文档投影自愈整改方案" +title: "dev 运行数据根隔离与文档投影整改方案（审查修订）" status: planned created: 2026-09-14 updated: 2026-09-14 --- -# dev 运行数据根隔离与文档投影自愈整改方案 +# dev 运行数据根隔离与文档投影整改方案（审查修订） -## 背景 +## 审查结论 -现场根因见 [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。当前 dev 的数据库仍是原持久卷，但 app、PI Runner、Cursor Runner 已绑定 `/tmp/as-dev-wf409-merge/data/dev`，导致项目 143 等正文投影在容器中不可见，GET 返回 409。 +2026-09-14 多视角审查确认：**诊断成立，原文不可整包实施。** -本方案不仅恢复项目 143，而是消除“源码 checkout/worktree 决定运行数据根”的系统耦合，并让账本到兼容文件投影成为可观测、可重试、可恢复的持久状态机。 +现场在审查当时（约 09:23 UTC）仍为 split-brain： + +| 项 | 值 | +|---|---| +| `shuimu-agent-app-dev` created | `2026-09-14T08:40:44Z` | +| Compose working dir | `/tmp/as-dev-wf409-merge` | +| app `/app/data` | `/tmp/as-dev-wf409-merge/data/dev` | +| pi/cursor workspaces | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| `/health` `/ready` | 200，`schema_version=33` | +| 主仓 CURRENT 集成槽 | `implementer_cursor` / `e2e` / workflow-plan | + +用户已拍板： + +1. **Wave 0 授权方式 = 事故抢占**：不排队等 e2e 结束；获批实施后立即停 runner/app，只读保全两端数据根，再从主仓目录纠挂。 +2. **本轮只改方案**：消解波次矛盾后再单独批准实施。本文未授权 rebuild、未授权写库、未授权删除 `/tmp`。 + +原文把「恢复可读」和「投影 outbox / 数据治理 / 完整 SLO」绑成同一批准与同一关门条件。修订后拆成三个互不阻塞的批准单元。 + +## 批准单元 + +| 单元 | 目的 | 关闭事故？ | 状态 | +|---|---|---|---| +| **Unit A** | 事故遏制与纠挂：停写、保全、从主仓 remount | 恢复当前可读/可写 | 方案已修订，**实施待再批** | +| **Unit B** | 防复发：绝对状态根 + 脚本硬门禁 + readiness 身份门禁 | 阻止同类错挂再次显示 healthy | 方案已修订，**实施待再批** | +| **Deferred** | ledger 只读降级、projection outbox、历史 work_dir 治理、监控产品、迁仓外路径 | 否 | 记 long-term，不挡 Unit A/B 关门 | + +`RUNTIME-DATA-ROOT-SPLIT-BRAIN` 与 FEATURE-MATRIX `infra` 在 **Unit A 验收 + Unit B 负例** 后关闭。不得等 Deferred。同 worktree 上的数据生命周期方案是 sibling：可以并行设计，**不得**成为 Unit A 纠挂的前置或本 gap 的关门条件。 ## 目标 -1. 任意 build source/worktree 都不能改变 dev/test/main 的持久数据根。 -2. 数据库、app、PI Runner、Cursor Runner 必须对同一环境 storage identity 达成一致，否则进程不 ready、部署脚本失败。 -3. ledger head 可用时，安全的投影 missing 不再使 UI 正文不可读；真实 fork 仍 fail closed。 -4. DB head 与 `output.md` 的双写失败可持久观测、幂等重试和自动补偿，不依赖用户反复刷新。 -5. rebuild、restart、宿主重启、linked worktree、并发保存和磁盘故障均有自动化负例；最终以真实页面刷新/重登/重建后持久化验收。 +1. 任意 build source / linked worktree / `/tmp` checkout 都不能改变已部署环境的持久数据根。 +2. 错挂或身份不一致时 `/ready` 必须 503；`/health` 只表示进程存活，不得作为部署就绪依据。 +3. Unit A 完成后，项目 143 等活跃受控正文在正确数据根上可打开；不以 08:54 历史 SHA 作为事后硬门槛。 +4. 真实内容分叉仍 409 并保全双方；安全 missing 的自动重建与 outbox **不在本方案实施范围**。 + +## 术语（强制） + +| 术语 | 定义 | +|---|---| +| `SOURCE_ROOT` | 本次构建源码目录，可来自受控集成 worktree | +| `RUNTIME_STATE_ROOT` | 各环境状态目录的**绝对父路径**。短期默认 = 主仓 `…/Agent-System/data` | +| `ENV_STATE_DIR` | `${RUNTIME_STATE_ROOT}/`，例如 `…/data/dev` | +| `SHARED_STATE_ROOT` | 共享知识库状态根，**不**拼接 `/`，不随 `SOURCE_ROOT` 变化 | +| bind mount source | `docker inspect` 的宿主机 `Mounts.Source` | +| build context | 镜像构建用的源码目录，允许与状态根不同 | + +禁止把 `RUNTIME_STATE_ROOT` 理解成已经带 env 的 `data/dev`。Compose 公式一律是 `${RUNTIME_STATE_ROOT}/${ENV}/...`。`.storage-identity.json`（Unit B）落在 `ENV_STATE_DIR/`。 + +禁止把 Wave 0「容器 source」读成 `SOURCE_ROOT`。Wave 0 验收的是 **bind mount source**。 ## 核心决策 -### D1 — 代码根与状态根彻底分离 +### D1 — 代码根与状态根分离（Unit B；Wave 0 不依赖新变量） -- `SOURCE_ROOT`：本次构建源码，可来自受控的集成 worktree。 -- `RUNTIME_STATE_ROOT`：环境持久状态根，必须是绝对、canonical、非 worktree 相对路径；dev/test/main 各自固定。 -- Compose 的 app/runner bind mount 全部改为 `${RUNTIME_STATE_ROOT}//...`，不得再写 `./data/`。 -- `.regulations`、agent sessions、attachments 等可变状态一并纳入状态根清单；只读代码资源继续来自 source root。 +- Compose 可变 bind 改为 `${RUNTIME_STATE_ROOT}/${ENV}/...`，禁止 `./data/`。 +- 短期兼容：用主仓 `data/` 作为 `RUNTIME_STATE_ROOT`。**本方案禁止把状态根迁到 `/var/lib/agent-system`。** +- 禁止硬编码个人 HOME。解析顺序：显式环境文件 → Git common-dir 推出的 **primary** 根下的 `data/`。 +- 显式 `RUNTIME_STATE_ROOT` **不能**把任意绝对路径变成合法根。解析结果必须：绝对、`realpath`、等于宿主机 allowlist、环境后缀与命令一致、不落在任何 linked worktree toplevel、不是 `/tmp` 集成目录。 +- shared compose 使用 `SHARED_STATE_ROOT`，不套用 `/${ENV}`。 +- `.regulations` 若仍从 `SOURCE_ROOT` 相对挂载，必须在 Unit B 清单里单独点名；不得假装已被 data identity 覆盖。 +- **否决记录**：Postgres 逃过本次事故是因为 named volume。本轮仍保持宿主机 bind，以便 Unit A 做两端差异保全。不用 named volume 收口 `/app/data`，也不把「固定 `--project-directory`」当成 Unit B 的全部答案——脚本必须同时导出绝对 bind，否则合法集成 rebuild 仍会解析到调用目录。 -建议短期兼容现有数据，将 primary repository 的 `data/` 解析为默认状态根；中期迁移到仓库外（例如 `/var/lib/agent-system` 或管理员配置路径）。禁止硬编码个人 HOME，脚本通过显式配置或 Git common-dir 推导 primary root。 +### D2 — 存储身份是配对仪式，不是可复制哨兵（Unit B） -### D2 — 存储身份握手，而不只比较路径字符串 +- 每个 `ENV_STATE_DIR` 放受保护的 `.storage-identity.json`：`environment`、UUID、schema version、**canonical host path**；不含 secret。 +- PostgreSQL 用**新的单行表**，不复用 `iam_system_config`。 +- **禁止自动初始化。** app 启动只校验，不 mint、不把文件 UUID 写进 DB，也不把 DB UUID 写到空根。 +- 真首启（允许 pair 命令写入同一 UUID）当且仅当：DB 无业务行（至少 `document_records` / `document_snapshots` 为空）**且**文件根无业务布局。 +- 已有业务库 + 文件根无合法 sentinel → fail closed / `/ready` 503。现场错误根已有 27 个文件，**「空目录」不是可用谓词**。 +- DB 有 UUID、文件缺失 → 503，不回写。 +- 文件有 UUID、DB 无 → 503，防止新卷认领旧根。 +- UUID 一致但 `realpath` ≠ DB 固化 canonical path → 503（克隆/备份根）。 +- runner 一致性**不**靠进程内 HTTP：`cursor-agent` 现为 `alpine sleep`，pi/cursor 只挂 `workspaces`，读不到状态根哨兵。Unit B 由部署脚本 `docker inspect` 三个容器的 `Mounts.Source` + 文件 sentinel + DB 行。进程内 `/ready` 只比 app 可见 sentinel 与 DB。 +- 未认证 `/ready` 只返回聚合布尔和稳定错误码，禁止 UUID、路径、mount source、doc_id、计数明细。 -- 每个环境状态根创建受保护的 `.storage-identity.json`：`environment`、随机 UUID、schema version；不含 secret。 -- PostgreSQL 新增单行环境 identity（或复用 settings 表），首次受控初始化时写入同一 UUID。 -- app 启动读取文件 identity 并与数据库 identity 比对；app、PI Runner、Cursor Runner 都暴露/上报相同 UUID。 -- UUID 缺失、环境名不符、根不可写、runner identity 不一致时 fail startup/readiness；不得自动初始化一个已含业务数据库对应的空文件根。 +### D3 — 读取真源与自动重建拆开（Deferred / 后续） -该机制可以识别“路径看起来合法但实际是另一份空目录/备份目录”的场景。 +- Unit A **不**改 GET 语义；纠挂后投影文件回到视野，现有 fail-closed 即可恢复 200。 +- 以后若做「ledger head 可读」，必须：identity 已通过、GET **只读**、真实 fork 仍 409 `DOCUMENT_CONTENT_PROJECTION_DRIFT`、不得在 GET 路径写盘。 +- 不得把「此刻 stat 不到 `output.md`」单独当成可自动重建。曾出现 fork/mismatch 的文档禁止降级为 MISSING 自愈。 +- 本方案**删除**原文 D3「MISSING 后台从 head 原子重建」。任何写回只允许在独立 Deferred 产品里设计，且必须有路径沙箱（`Lstat` + `EvalSymlinks`，realpath 严格位于 `ENV_STATE_DIR`）。 -### D3 — ledger 是读取真源，projection 是受管兼容视图 +### D4 — transactional outbox 降为 Deferred -- 已有 ledger 的 `GET .../content` 从 immutable blob/head 返回正文，不再因单纯 `MISSING` 阻断 UI。 -- 同时返回 `projection_status` 或响应头，标识 `aligned | pending | missing | fork | failed`；真实 `CONTENT_FORK` 仍 409，并禁止覆盖双方。 -- 只有 storage identity 已通过且 work_dir 在允许根内时，`MISSING` 才允许后台从 head 原子重建；挂载身份错误时不自愈，避免在错误空根批量制造“看似恢复”的第二份数据。 -- 无 ledger 的 legacy 文档仍从文件 bootstrap，并沿用 hard-break 安全 canonicalization 与迁移审计。 - -### D4 — 用 transactional outbox 收口双写 - -新增/扩展持久 projection job： - -```text -doc_id, target_snapshot_id, target_sha256, storage_uuid, -state(pending/applying/aligned/failed/fork), attempts, -last_error_code, next_retry_at, updated_at -``` - -- 人工 PATCH、agent commit/reject 在同一 PostgreSQL 事务中更新 head 并 upsert projection job。 -- projector 使用 `doc_id + target_snapshot_id` 幂等执行，校验 storage UUID 与允许路径，临时文件 fsync + rename + parent fsync 后再标 aligned。 -- 同文档任务合并为最新 head；旧任务不得回写覆盖新 head。 -- missing 可重建；mismatch/fork 保存两侧 hash/路径后转人工处理；失败采用有上限退避并告警。 -- 同步写成功可快速返回 200；超时/失败返回明确 `projection_status=pending/failed`，不得让用户误以为账本未保存。 +- 当前 409 的根因是错挂空根，不是缺 outbox。 +- 仓库已有 `InspectProjection`、`ProjectOutputMarkdown`、`fsutil.WriteFileAtomic`、`document-content-reconcile`。新 job 表 / projector 进程是第二产品。 +- PATCH 若以后暴露 `projection_status`：ledger commit 成功则 **HTTP 一律 200**，状态放 body/头；禁止用非 2xx 表示「账本未保存」。 +- fork：服务继续，该文档 409，保全双方，立即人工处置。**不是**停 app/runner。 ## 分波次计划 -| 波次 | 内容 | 验收 | -|---|---|---| -| Wave 0 事故遏制与恢复 | 占集成槽；暂停 dev 文档写入/PI；导出 app/pi/cursor 实际 mounts、DB storage 状态；分别备份主仓根与错误 `/tmp` 根的 08:40 后增量；生成冲突清单；确认无分叉后从固定状态根重建三容器 | 项目 143 文件不改内容且三方 SHA 仍为 `f283...f039`；容器 source 指向固定状态根；目标 content 200；错误根只读保全，不盲拷贝/删除 | -| Wave 1 脚本硬门禁 | `env-service.sh` 解析 primary/common Git root与显式 `RUNTIME_STATE_ROOT`；mutating action 校验绝对路径、环境 identity、分支/集成模式；打印 build source 与 state root；worktree 未显式指定稳定 state root 时直接失败 | linked worktree 默认执行 rebuild 失败且不替换现有容器；合法集成构建的所有 bind source 保持固定 | -| Wave 2 Compose 解耦 | dev/test/main/shared compose 的可变 bind 全部改为绝对 state-root 变量；为 app/pi/cursor 添加 storage UUID/环境标签；CI 检查 `docker compose config` 不出现工作树内 `data/` | 从两个不同 source roots 渲染 config，数据挂载 source 完全相同；代码 build context 可不同 | -| Wave 3 启动与 readiness | 引入文件/DB storage identity；readiness 增加 `storage_ready`、`storage_id_match`、runner mount/identity；部署后脚本核对 container labels + mounts + sentinel + active-doc projection 摘要 | 错误空根启动被拒绝；本次 split-brain 复现实验 `/ready` 必须 503，不能再显示 healthy | -| Wave 4 投影状态机 | ledger read 与兼容 projection 解耦；实现 projection outbox、幂等 projector、missing 安全重建、fork 隔离和结构化状态/告警 | DB commit 后 kill -9、只读目录、磁盘满、延迟重试均不丢 head；恢复后自动 aligned；fork 不被覆盖 | -| Wave 5 数据治理 | 按[数据生命周期与迁移语料子方案](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)执行蓝绿精简；真实项目、demo、migration corpus、临时测试分 namespace；增加 orphan/missing/fork 巡检与 TTL | dev 常驻 6–10 个可解释项目；受支持旧 schema corpus 可重复迁移；测试临时路径不污染运行库 | -| Wave 6 验收与回滚演练 | 工程测试、容器 rebuild/recreate、宿主重启、真实 UI 修改/刷新/重登、双用户冲突、真实 PI 修改、回滚演练 | 下述验收矩阵全 PASS，报告归档后才恢复 FEATURE-MATRIX `done` 和 gap PASS | +| 波次 | 单元 | 内容 | 验收 | +|---|---|---|---| +| Wave 0 | A | 事故抢占：停 pi/cursor/app（保留 Postgres 卷）；只读保全主仓根与 `/tmp` 错误根；三方 disposition；**仅从主仓 checkout** 用现有相对 bind 纠挂三容器 | 见下方 Runbook；**不**验 storage identity，**不**改 143 正文 | +| Wave 1 | B | `env-service.sh` + compose 同时改：导出绝对 `RUNTIME_STATE_ROOT`，可变 bind 全部改公式；worktree / `/tmp` 默认 mutating 失败且不 recreate 现网 | 两个不同 `SOURCE_ROOT` 渲染后，app/pi/cursor 的 host mount source 字节级相同，且等于 allowlist 中的 `ENV_STATE_DIR` | +| Wave 2 | B | 与 Wave 1 同一批落地。CI 禁相对 `./data/`；**允许**绝对路径落在 primary checkout 的 `data/`。shared 只用 `SHARED_STATE_ROOT` | 合法配置不会被「工作树内 data/」误杀；dev 命令不能挂 test/main 根 | +| Wave 3 | B | 文件/DB identity + pair 命令；`/ready` 增加 `storage_ready` 布尔；dev compose healthcheck 改为 `/api/v1/ready` | 复现本次 split-brain 时 `/ready` 503、容器 unhealthy；`/health` 仍可 200 | +| — | Deferred | ledger GET 与 MISSING 解耦；outbox/projector；历史 work_dir 治理；五条时序指标与 60s SLO；迁仓外路径 | 另开 long-term，不进本 gap 关门 | -## 即时恢复 Runbook（批准实施后执行） +Wave 1 若只改脚本、不改 compose，从 `/tmp` 做「合法 rebuild」仍会把 `./data/dev` 解析到空根。因此 **Wave 1 与 Wave 2 必须同批**。 -1. 在 CURRENT 占 `infra-runtime-state` mutex 与集成槽，记录容器 ID/created/mount source；暂停售写。 -2. 对主仓状态根和错误 `/tmp/as-dev-wf409-merge/data/dev` 做只读 inventory：相对路径、size、mtime、SHA；特别核对 08:40 UTC 后新增/修改项。 -3. 若错误根存在新业务写入，按 DB task run、snapshot、source_run_id 和文件 hash 逐项判定；不得 `cp -r` 覆盖主根。 -4. 备份差异和 Compose inspect 证据；用固定 state root rebuild app/pi/cursor。数据库 named volume不变。 -5. 验证 storage identity、mount source、`/ready`、项目 143 content/revision、活跃项目 projection 摘要。 -6. 仅在页面完成项目 143 打开、保存唯一 marker、硬刷新、退出重登；随后真实 PI 修改一份新文档并验证 app rebuild 后恢复。 -7. 错误根在报告签字前只读保留；确认无独有数据后再按独立授权处置，本方案不授权删除。 +## Unit A — 即时恢复 Runbook（实施待再批） -## 自动化验收矩阵 +授权：事故抢占。主仓 CURRENT 即使仍显示 `implementer_cursor` / `e2e`，实施时也要写入抢占记录，停止该 e2e 对 `agent-dev` 的使用，并通知 queue 中的 `governance-action-level-authz`。以**主仓** CURRENT 与 `docker inspect` 为准，禁止信任任何 worktree 副本里的 `idle`。 + +**冻结 = 停进程，不是文档 mutex，也不是「请大家别保存」。** 仓库没有 maintenance 开关。 + +1. 在主仓 CURRENT 登记抢占：`holder=incident-preempt`，`scope=RUNTIME-DATA-ROOT Unit A`，`mutex=infra-runtime-state`。记录三容器 ID / created / `Mounts.Source` / compose working dir。 +2. **先停写再盘点**（保留 `shuimu-context-pg-dev`）： + + ```bash + docker stop shuimu-pi-agent-dev shuimu-cursor-agent-dev shuimu-agent-app-dev + ``` + + 未停止不得开始差异扫描。test/main 容器禁止触碰。 +3. 对主仓 `…/Agent-System/data/dev` 与 `/tmp/as-dev-wf409-merge/data/dev` 做只读 inventory：相对路径、size、mtime、SHA。使用 `find -P` / 不跟随 symlink；先 `Lstat` 跳过非常规文件。特别列出 08:40 UTC 之后的新增/修改，以及 `agent_sessions`、attachments、CAS、非 `output.md`。 +4. 备份目的地必须在仓库外、权限 `0700`、与运行数据根分离。`agent_sessions` / token / `.env` 只记存在性与大小，不拷贝明文。禁止把 `/tmp` 备份写进 `data/dev` 或 git 工作区。禁止 `cp -r` 覆盖主根。 +5. 按 **DB/ledger vs 主根 vs 错误根** 出 disposition 清单（保留主根 / 回收错误根独有项 / 标 orphan）。项目 143 只是抽样锚点，不能代表 100 条记录或 68 条主仓历史 missing。冻结时刻重新计算三方 hash；**删除「必须仍为 `f283...f039`」硬门槛**。仅当冻结快照上 143 仍三方一致时，才允许「只纠挂、不改该文件」。 +6. 纠挂**必须**在主仓执行，禁止 `env-service.sh` 从 worktree 或 `/tmp/as-dev-wf409-merge` 调用： + + ```bash + PRIMARY=/home/weichen/projects/Agent-System + docker compose --project-directory "$PRIMARY" \ + -f "$PRIMARY/docker-compose.dev.yml" \ + up -d --build \ + app pi-agent cursor-agent + ``` + + 数据库 named volume 不变。纠挂后立刻 `docker inspect`：三个 bind source 必须是 `$PRIMARY/data/dev` 及其 `workspaces`，不得再指向 `/tmp`。 +7. 验证：`GET /api/v1/docgov/documents/doc_proj143_intake/content` 在冻结快照仍一致时应 200；抽查若干活跃非模板文档的投影可见性。`/ready` 仍按现状理解（此时还没有 storage identity）。 +8. 错误根只读保留到报告签字；本方案**不授权删除** `/tmp/as-dev-wf409-merge`。 +9. **禁止**在 Wave 0 对项目 143 保存 marker、禁止真实 PI 改文。那些属于 Unit B 之后的页面验收，另批。 + +### Wave 0 验收（仅这些） + +- [ ] 停写发生在 inventory 之前；Postgres 未删卷、未重建。 +- [ ] 两端 inventory + 仓库外备份 + disposition 清单落盘到 test-reports assets。 +- [ ] app/pi/cursor 的 bind mount source 回到主仓 `data/dev`（或 `…/workspaces`），compose working dir 不再是 `/tmp/as-dev-wf409-merge`。 +- [ ] 项目 143 在「冻结快照仍一致」时 content 200，文件内容与冻结 hash 一致。 +- [ ] 错误根仍在且未被覆盖/删除。 +- [ ] 主仓 CURRENT 记录抢占与恢复结果；不以 `/health` 200 作为通过。 + +## Unit B — 防复发验收矩阵 ### 脚本与 Compose -- [ ] primary repo、linked worktree、`/tmp` integration checkout 三种 source root 下，合法 config 的 state bind source 相同。 -- [ ] 未配置/相对/不存在/环境名不符的 state root 均 fail fast，已有容器不被 recreate。 -- [ ] dev 命令不能挂 test/main 根；storage UUID 交叉时拒绝。 -- [ ] app、pi-agent、cursor-agent 三者 mount source 和 storage UUID 一致。 +- [ ] primary repo、linked worktree、`/tmp` integration checkout 三种 `SOURCE_ROOT` 下，合法 config 的 state bind source 相同，且等于 allowlist 中的绝对 `ENV_STATE_DIR`。 +- [ ] 未配置 / 相对 / 不存在 / 环境名不符 / 指向 worktree 或 `/tmp` 的 state root 均 fail fast，已有容器不被 recreate。 +- [ ] 显式 `RUNTIME_STATE_ROOT` 不在 allowlist 时失败（不能靠「绝对路径」绕过）。 +- [ ] dev 命令不能挂 test/main 根。 +- [ ] `docker inspect`：app、pi-agent、cursor-agent 三者 mount source 同一 realpath 前缀。 +- [ ] CI 断言的是「禁止相对 `./data/` + 两次渲染 source 相同」，**不是**「不得出现 primary 工作树内的绝对 `data/dev`」。 -### 后端与故障注入 +### Readiness -- [ ] ledger head + projection missing + identity correct：GET 200 返回 head，job pending 后原子重建并 aligned。 -- [ ] ledger head + projection missing + identity mismatch：readiness 503，不执行自愈写入。 -- [ ] ledger head + projection fork：409 `DOCUMENT_CONTENT_PROJECTION_DRIFT`，保全双方。 -- [ ] PATCH 在 ledger commit 后 projector 失败：状态可见、同幂等键不重复增 r、后台恢复后 aligned。 -- [ ] projector 处理中又产生新 head：旧 job 不得覆盖新 snapshot。 -- [ ] app/projector/runner restart 与 kill -9 后任务可恢复；告警包含 doc_id、snapshot、storage UUID、correlation id，不含正文或 secret。 - -### 真实页面验收 - -- [ ] 项目 143 正文首次打开无 409；人工保存后 r+1、硬刷新、退出重登仍可见。 -- [ ] 保存后立即刷新与连续刷新不会触发 loading 死锁或重复错误风暴。 -- [ ] 真实 PI/小队修改新项目文档，候选确认/拒绝、change report、最终正文均可见；fallback 明确标识。 -- [ ] dev app + runner rebuild/recreate 后，旧项目和新项目正文/版本/overlay 都恢复。 -- [ ] 双浏览器旧 ETag 保存仍显示真正的 `HEAD_CONFLICT`，不能与 projection 409 混淆。 - -## 监控与 SLO - -- 指标：`document_projection_total{state}`、`projection_repair_total{result}`、`projection_age_seconds`、`storage_identity_mismatch_total`、`content_read_fallback_total`。 -- 告警：任一 storage identity mismatch 立即 P0；active 文档 `pending/failed` 超过 60 秒告警；fork 立即人工处置。 -- 日志：部署开始/结束必须记录 source commit、compose working dir、state root identity、实际 mount source hash；禁止记录正文与凭证。 -- `/ready`：数据库/schema/task store/storage identity/runner identity 为硬门禁；projection 汇总按 active 用户文档设阈值并列出计数，不泄露 doc 内容。 +- [ ] 错误空根、业务库无 sentinel、canonical path 不符：`/ready` 503，dev compose healthcheck 显示 unhealthy。 +- [ ] `/health` 保持 liveness，部署脚本不得用它当就绪。 +- [ ] `/ready` 不泄露 UUID/路径/doc 内容。 ## 回滚策略 -- Wave 1–3 可回滚代码，但不得回滚/重建 storage UUID；旧相对 bind mount 禁止作为回滚方案。 -- Wave 4 使用 feature flag 分阶段启用 ledger-read 与 projector；关闭 projector 时保留 pending jobs，不删除。 -- 所有自动修复只覆盖 `MISSING + identity matched`；任何 fork 保留证据并停机式处理。 +- Unit B 代码可回滚，但 **不得回滚/重建已配对的 storage UUID**；旧相对 bind **禁止**作为回滚方案。 +- Unit A 失败：保持停写，不要把容器重新绑回 `/tmp`。 +- 任何自动修复（若未来另批）只覆盖 `MISSING + identity matched + 无历史 fork`；fork 保留证据，服务继续。 ## 方案边界 -- 本轮不修改业务代码、不重建容器、不修复/覆盖任何正文。 -- 当前 409 的即时恢复与系统实现必须由用户批准后，在独占集成槽执行。 -- test/main 仅做只读挂载审计；未经环境晋升授权不得改动。 +| 阶段 | 允许 | 禁止 | +|---|---|---| +| 审查修订（本轮） | 改 `.doc` 方案与 SSOT | 改业务代码、rebuild、写库、覆盖/删除正文或 `/tmp` | +| Unit A（再批后） | 停 dev app/pi/cursor；只读保全；从主仓纠挂三容器 | 改 test/main 运行态；删错误根；Wave 0 写 143；从 worktree 调 `env-service` | +| Unit B（再批后） | 改仓库内 compose/脚本/`/ready`；test/main **文件**可改绝对变量 | 对运行中的 test/main 做 mutating env-service；自动 mint identity；迁仓外 | +| Deferred | 另开 long-term | 当作本事故关门条件 | + +## 监控（Unit B 最小集 / 其余 Deferred） + +Unit B 只要求：部署日志记录 source commit、compose working dir、state root、实际 mount source hash；禁止记录正文与凭证。`storage_identity_mismatch` 作为 `/ready` 布尔即可。 + +五条 Prometheus 指标、60 秒 pending SLO、`/ready` 投影计数阈值整节移出本方案。 ## 相关链接 - Audit: [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) -- Short-term: [dev runtime data root guard 诊断与设计](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) -- Data lifecycle: [运行数据生命周期与迁移语料库](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Short-term: [dev runtime data root guard](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) - gap-matrix 行: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` + +## 审查修订记录 + +原文矛盾已在本文消除，不再保留互斥表述： + +- `RUNTIME_STATE_ROOT` 同时被写成「每环境根」和「父目录+/」 +- 「本轮不重建」与 Wave 0 重建互相否决（现按阶段表） +- Wave 0 冻结 143 SHA 与 Runbook 要求保存 marker +- Wave 0 验收 storage identity（Wave 3 才有） +- Wave 1 宣称 bind 固定但 compose 仍相对 +- Wave 2 CI 禁止工作树内 `data/` 与短期默认根互斥 +- fork「停机式处理」与 409 继续服务互斥 +- PATCH 投影失败的 HTTP 状态分叉 +- GET `projection_status` 与 job state 混用 +- `CONTENT_FORK` 与 `DOCUMENT_CONTENT_PROJECTION_DRIFT` 对外码漂移 +- shared compose 套用每环境公式 +- 「暂停售写」未定义 +- 恢复与 outbox/治理/完整 SLO 同一关门条件 diff --git a/.doc/remediation/INDEX.md b/.doc/remediation/INDEX.md index 320fe23d..bf37c415 100644 --- a/.doc/remediation/INDEX.md +++ b/.doc/remediation/INDEX.md @@ -3,7 +3,7 @@ | 文档 | 说明 | |------|------| | [gap-matrix.md](./gap-matrix.md) | 发布缺口 SSOT | -| [2026-09-14-dev-runtime-data-root-and-projection-remediation.md](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | 临时 worktree 错挂 dev 数据根、storage identity、readiness 与 projection outbox 系统整改 | +| [2026-09-14-dev-runtime-data-root-and-projection-remediation.md](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | 临时 worktree 错挂 dev 数据根。审查修订：Unit A 纠挂 / Unit B 状态根与 readiness / Deferred outbox | | [2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | dev/test 数据精简、migration corpus、蓝绿换库、TTL 与延迟销毁方案 | | [2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md](./2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md) | 项目设置 409 冻结态守卫 + 非必需文档三权全有或全无 + 成员/门禁/权限一致性 | | [2026-09-14-controlled-content-canonical-hash-remediation.md](./2026-09-14-controlled-content-canonical-hash-remediation.md) | 项目 143 正文 canonical/raw 哈希漂移、错误语义与重复 hydration 整改 | diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index 30d3066c..5dd3f6e1 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -57,7 +57,7 @@ | ID | 描述 | 状态 | 证据 | 计划动作 | |---|---|---|---|---| -| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。项目 143 主仓文件、DB、ledger head SHA 一致，容器内同路径不存在；Compose working dir 与三容器 mount source 均为 `/tmp/as-dev-wf409-merge` | [runtime remediation](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) + [data lifecycle](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)。先恢复固定根，再以蓝绿新库、最小迁移 corpus 和延迟销毁精简 dev；恢复前禁止用 dev healthy 或文档 E2E 签字 | +| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。2026-09-14 ~09:23 UTC 复核：挂载仍为 `/tmp/as-dev-wf409-merge`，`/ready` 仍 200/`schema=33`。审查已否决「恢复与 outbox 同批关门」 | 关门 = [审查修订方案](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) 的 Unit A 纠挂 + Unit B 绝对状态根/门禁/`/ready` 503。outbox 为 Deferred。[data lifecycle](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) 不替代事故修复，也不挡 Unit A | | WORKFLOW-PLAN-FROZEN-OPTIONAL-DUTIES | active 项目仍展示可编辑职责表单，409 提示在长表单视口外；现有规划又不支持非必需文档“三权全空”，且后端缺少职责人在任成员复核 | **PASS（设置页）** | [audit](../audit/2026-09-14-workflow-plan-409-frozen-and-optional-duties.md)；[test-report](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)。项目 143 只读 frozen、A full+none 激活、B partial 不发 PUT、C 零治理拒绝、双标签转 frozen、成员无配置入口、刷新后 143 仍 frozen | 真实 PI 生成 full/none 并走审批/阶段不阻塞未纳入本轮页面矩阵，不作为本 gap 关闭的替代证据，另立项 | | DOC-CONTENT-CANONICAL-HASH-DRIFT | 旧受控文档 bootstrap 裁剪 Markdown 双空格硬换行并将 canonical SHA 写入账本/DocGov，但未同步文件；正文读取把完整性冲突误报 404，lease/list sync 又放大并制造刷新后加载死锁 | **PASS（真实 UI）** | [audit](../audit/2026-09-14-project143-controlled-content-404.md)；[UI report](../test-reports/2026-09-14-controlled-content-ui-acceptance.md)。代码已合 dev、schema 32、项目 143 定向 `migration_repair`；项目 143 `clean-run3` 与新项目 `full-chain-final` 主链及 restart 全部纯页面 PASS，目标 404=0 | [remediation](./2026-09-14-controlled-content-canonical-hash-remediation.md)。隔离 `TestReconcile_LegacyRepair_PG` 未执行，按用户“本轮只允许页面测试”的边界保留为工程测试债，不作为本轮 UI PASS 的替代证据 | | CHAT-HISTORY-REFRESH | 工作流激活后刷新，对话正文被阶段过滤器清空，但分类数量与锚点仍按全量返回 | **PASS** | [test-report](../test-reports/2026-09-13-chat-history-refresh-test-report.md)；代码：`message_scope.go`、`chat_history_auth.go`、goose 00026/00027 | 真实 PI 刷新 E2E 待 dev 容器补录（自动化/PG 已 PASS） | diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index d4799dc0..ba28dd2f 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -21,11 +21,12 @@ | 工作流治理 | 三正式工作流 + 阶段门禁 | partial | [remediation/three-workflows](../remediation/2026-09-12-three-workflows-strict-governance-remediation.md)；[test 2026-09-12](../test-reports/2026-09-12-three-workflows-governance-remediation-test-report.md)；[冻结态/职责可选整改](../remediation/2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md)；[Wave 5 页面矩阵](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)；[职责-动作鉴权](../test-reports/2026-09-14-governance-action-level-authz-test-report.md) | 设置页/DutyMode + 动作级鉴权/未激活 202-blocked/关闭创建直通已落地。后续：none 文档真实 PI 审批长链路、CI 三工作流 E2E | `workflow` | | 纯软件项目自举 | 软件项目创建、Bootstrap 基线、Delivery 迭代与受控工程交接 | planned | [详细设计](../system-design/2026-09-14-software-project-bootstrap-workflow.md)；[现状调研](../ops/research/2026-09-14-software-project-bootstrap-current-state.md)；[验收计划](../ops/checklists/2026-09-14-software-project-bootstrap-acceptance.md) | 当前仅完成方案；目录/模板/执行器/创建向导/真实 PI UI 均未实施。受控代码执行单列后续 Epic | `workflow-design` | | IAM / 鉴权 | 项目级 API 鉴权 + RBAC | partial | [audit/authorization](../audit/authorization-matrix-v2.md)；[刷新恢复专项审计](../audit/2026-09-13-chat-history-refresh-disappears.md) | 历史读取端点补统一会话 owner + 阶段授权；旧 API 410 标注(BUG-11 partial) | `authz` | -| 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；但当前 dev 因临时 worktree 数据根 split-brain 再次不可读。须完成固定 state root、storage identity、projection outbox/自愈和 rebuild 负例后恢复 `done` | `docgov` | +| 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；当前 dev 因数据根 split-brain 再次不可读。Unit A 纠挂后标 incident-mitigated；`done` 等 Unit B 负例，不等 outbox | `docgov` | | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | +| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | planned | [调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md) | 当前仅有 PostgreSQL 文档快照、消息正文和运行态 CAS；缺独立 storage identity、终态 outbox、manifest/receipt、授权降级读取和 scratch 恢复工具 | `artifact-vault` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | -| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | dev 当前错挂 `/tmp` 空数据根且 readiness 假阳性；须完成绝对 state root/身份门禁，并以蓝绿换库将 366 个混杂项目收敛为 demo/regression/migration corpus + TTL 测试数据 | `infra` | +| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus/FAV 为 sibling，不挡事故关闭 | `infra` | | CI / E2E | PG 集成 + 真实 PI E2E | partial | [gap-matrix BUG-08](../remediation/gap-matrix.md)；[test pg-ci](../test-reports/2026-09-12-pg-ci-soak-e2e-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md) | CI `TEST_DATABASE_URL` 隔离库仍缺；文档运行时真实 UI 已按 UI_PHASE 分文件且 post-restart 断言最终 A1 正文。`-race` 全包并行 TempDir flake 为基建残留 | `ci` | --- From cfc3c94cff89c0edcedf8f22cf7629fd7d4e0a50 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:01:56 +0000 Subject: [PATCH 04/13] docs(status): drop FAV from infra next-action Keep the matrix row from depending on an uncommitted sibling plan. Co-authored-by: Cursor --- .doc/status/FEATURE-MATRIX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index ba28dd2f..dd1f3c8c 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -26,7 +26,7 @@ | 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | planned | [调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md) | 当前仅有 PostgreSQL 文档快照、消息正文和运行态 CAS；缺独立 storage identity、终态 outbox、manifest/receipt、授权降级读取和 scratch 恢复工具 | `artifact-vault` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | -| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus/FAV 为 sibling，不挡事故关闭 | `infra` | +| 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus 为 sibling，不挡事故关闭 | `infra` | | CI / E2E | PG 集成 + 真实 PI E2E | partial | [gap-matrix BUG-08](../remediation/gap-matrix.md)；[test pg-ci](../test-reports/2026-09-12-pg-ci-soak-e2e-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md) | CI `TEST_DATABASE_URL` 隔离库仍缺；文档运行时真实 UI 已按 UI_PHASE 分文件且 post-restart 断言最终 A1 正文。`-race` 全包并行 TempDir flake 为基建残留 | `ci` | --- From ded40d071e116cdfb4ca77e68f4e9fd81d7384d1 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:09:32 +0000 Subject: [PATCH 05/13] docs(remediation): design final artifact vault --- .doc/ops/CURRENT.md | 4 +- .../2026-09-14-final-artifact-vault.md | 149 +++++++++++ ...ime-data-retention-and-migration-corpus.md | 3 + .../2026-09-14-final-artifact-vault-plan.md | 61 +++++ .../2026-09-14-runtime-data-lifecycle-plan.md | 4 + ...-09-14-final-artifact-vault-remediation.md | 247 ++++++++++++++++++ ...ecycle-and-migration-corpus-remediation.md | 13 +- .doc/remediation/INDEX.md | 1 + .doc/remediation/gap-matrix.md | 2 +- .doc/status/FEATURE-MATRIX.md | 2 +- 10 files changed, 482 insertions(+), 4 deletions(-) create mode 100644 .doc/ops/research/2026-09-14-final-artifact-vault.md create mode 100644 .doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md create mode 100644 .doc/remediation/2026-09-14-final-artifact-vault-remediation.md diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index dfb032e7..bc27dd41 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -5,7 +5,7 @@ ## 焦点 -1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；审查修订已完成。现场仍错挂 `/tmp/as-dev-wf409-merge`；Unit A 事故抢占纠挂与 Unit B 防复发**待再批**。数据精简为 sibling 方案，不挡 Unit A） +1. **RUNTIME-DATA-ROOT**: [dev 临时 worktree 数据根 split-brain](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)（critical；审查修订已完成。现场仍错挂 `/tmp/as-dev-wf409-merge`；Unit A 事故抢占纠挂与 Unit B 防复发**待再批**。数据精简与[最终成果保险库](../remediation/2026-09-14-final-artifact-vault-remediation.md)为 sibling 方案，不挡 Unit A） 2. **CHAT-INTENT**: [对话意图运行时状态](../audit/2026-09-13-conversation-intent-runtime-state.md)（`fix/conversation-intent-state`；与文档账本运行时无关） 3. **CHAT**: [对话历史刷新](../audit/2026-09-13-chat-history-refresh-disappears.md)（认领仍 `in_progress`；gap 行已 PASS，以代码/handoff 为准收敛） @@ -51,6 +51,7 @@ | Codex | dev-runtime-data-root-guard | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | | Cursor | dev-runtime-data-root-guard-plan-rewrite | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | in_progress | | Codex | runtime-data-lifecycle-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | +| Codex | final-artifact-vault-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | done | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 @@ -115,3 +116,4 @@ | dev 数据根 split-brain 审计 | [../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) | | dev 数据根与投影系统整改 | [../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | | 数据生命周期与迁移语料 | [../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | +| 最终成果保险库与受控恢复 | [../remediation/2026-09-14-final-artifact-vault-remediation.md](../remediation/2026-09-14-final-artifact-vault-remediation.md) | diff --git a/.doc/ops/research/2026-09-14-final-artifact-vault.md b/.doc/ops/research/2026-09-14-final-artifact-vault.md new file mode 100644 index 00000000..67c52cf7 --- /dev/null +++ b/.doc/ops/research/2026-09-14-final-artifact-vault.md @@ -0,0 +1,149 @@ +--- +title: "最终成果保险库与部分恢复边界" +created: 2026-09-14 +status: done +confidence: high +--- + +# 调研：最终成果保险库与部分恢复边界 + +## 结论 + +提案可行，建议采用，但不应实现成“把最终文件复制到一个固定目录”。应建设独立的**最终成果保险库（Final Artifact Vault，FAV）**：只接收服务端已经确认终态、具有稳定标识和内容哈希的成果，以不可变 blob、语义 manifest 和提交回执保存，并与工作区清理完全解耦。 + +FAV 是 PostgreSQL 完整备份/PITR 的互补层，不是第三套完整业务事实源。它只对“已归档成果的精确内容与出处”负责，可以在数据库损坏或选择性重建时恢复文档、最终回复和有限的来源关系；成员权限、工作流活动态、租约、待办任务、幂等记录等仍须由数据库备份或其他权威记录恢复。 + +## 当前能力与缺口 + +| 能力 | 当前事实 | 与提案的差距 | +|---|---|---| +| 受控文档版本 | `document_content_blobs` 与 `document_snapshots` 在 PostgreSQL 内保存不可变 Markdown；工作区 `output.md` 是投影 | 数据库与账本一起丢失时，没有独立副本；当前 GET 仍会被缺失工作区投影阻断 | +| 助手最终回复 | `chat_messages.content` 保存正文，`chat_turns.assistant_message_id` 可在 completed 回合绑定最终消息 | 占位正文会原地更新，没有独立不可变回复快照；删除 session 会级联删除消息 | +| 回复导出 | `replyexport` 已区分 completed、status/error/card，提供 `content_sha256` 与 `export_policy` | `export-source` 仍从当前消息行临时构造 snapshot，不是持久归档；部分调用在缺少真实回合状态时默认按 completed 评估 | +| 附件 CAS | `storage_blobs`/`file_references` + `data//storage/blobs` 按 SHA 去重 | blob 物理文件仍在运行状态根，引用元数据仍在数据库；不独立于工作区/数据库事故 | +| 数据备份 | 现有脚本主要导出 PostgreSQL；数据生命周期方案已要求 DB + state 成对打包 | 尚无跨文档/回复的终态准入、持续发布、独立索引或成果级恢复 API | + +当前事故已经证明“数据库记录还在”不代表“运行容器能找到正文”。FAV 可以解决成果副本对工作区根的依赖，但不能替代固定 `RUNTIME_STATE_ROOT`、storage identity、projection outbox 和数据库备份。 + +## 提案中应保留的部分 + +1. **独立持久位置**：成果存储不位于 Git worktree、临时集成目录、任务 workspace 或普通 GC 扫描范围。 +2. **只收终态成果**：拒绝流式片段、候选稿、工具轨迹、修复过程、临时报告、租约和未完成任务。 +3. **支持系统定位**：使用稳定 artifact UUID、逻辑资源 ID、来源事件和内容 SHA，而不是依赖数据库自增 ID 或 workspace 路径。 +4. **辅助部分恢复**：允许在数据库关系不匹配时校验并导入已归档内容，同时保留来源和冲突证据。 +5. **与清理联动**：工作区或旧数据集只有在必需成果全部发布并验证后才可进入销毁阶段。 + +## 必须修正的部分 + +### 1. “最终”必须由状态机判定 + +不能按文件名、目录、时间或“任务进程退出 0”判断最终。建议的准入规则： + +| 成果类型 | 可归档条件 | 明确排除 | +|---|---|---| +| 智能体生成文档 | agent candidate 已 `committed`，且 output snapshot 成为账本 head；归档该次生成的不可变 snapshot | ready、needs_confirmation、rejected、stale、quality_failed、workspace 草稿 | +| 受控文档正式稿 | `approved` 时归档被冻结的 review snapshot；版本被 sealed 时可归档该版 final head | drafting/reviewing 中尚未冻结的当前文件、lease 中内容 | +| 自由文档 | 已形成不可变 version，且用户显式“发布/归档”或被正式成果引用；显式操作不能绕过终态检查 | 自动保存中的连续修订、临时私密摘录 | +| 助手回复 | `chat_turn.status=completed`、绑定唯一 `assistant_message_id`、正文 SHA 匹配，且 `export_policy=full`；`copy_only` 只在 completed 后显式 pin 时进入 | status/error/cancelled/system、纯操作卡、流式 chunk、placeholder、中间 agent 消息；未完成回复即使 pin 也拒绝 | +| 最终报告/附件 | 被上述成果 manifest 引用并通过合规/病毒/类型校验 | 未引用附件、日志、Prompt、工具输入输出、agent session | + +“每次成功保存的人工 revision”仍进入 PostgreSQL 不可变账本，但不自动进入 FAV。这样既保留文档历史，又不会把编辑过程伪装成最终成果。 + +### 2. 保险库不是第三套完整真相源 + +建议采用三层责任边界： + +| 层 | 权威范围 | 可否清理 | +|---|---|---| +| PostgreSQL 运行库 | 项目/用户/权限、工作流、任务、当前指针、完整版本账本和归档发布状态 | 按环境生命周期与完整备份策略处理 | +| Runtime workspace/state | Agent 计算现场、`output.md` 投影、临时转换文件和附件运行副本 | 可重建/可清理，但须受发布门禁保护 | +| Final Artifact Vault | 已准入成果的精确字节、manifest、来源证据、保留/删除状态和发布回执 | 独立保留；只能按成果保留策略和删除账本处理 | + +正常读写仍以数据库授权和业务指针为准。数据库仍能完成授权判定但正文缺失时，可从 FAV 只读降级返回并标记 `source=vault/degraded`；若权限关系也缺失，FAV 不得绕过授权直接对用户开放，只能进入管理员隔离恢复流程。 + +### 3. 发布必须避免双写裂缝 + +数据库事务内同时提交业务终态、精确 payload/SHA 的不可变 staging 和 `artifact.publish.requested` outbox；事务外 publisher 只从 staging 幂等写 blob、manifest，校验后最后写 commit marker，再回填 receipt。对象存储 I/O 不进入数据库事务。不能让异步 publisher 重读仍可更新或随 session 级联删除的消息行。 + +清理流程只认 `verified` receipt，不认“消息发过”“目录存在”或“有一个 SHA”。重复投递必须由 `artifact_uuid + source_event_id + payload_sha256` 去重；相同逻辑资源的新终态生成新 manifest，不覆盖旧 manifest。 + +### 4. Manifest 必须表达恢复语义 + +每个归档单元至少包含： + +```text +artifact_schema_version +artifact_uuid / artifact_type / logical_resource_id +project_external_key / environment +source_event_id / source_run_id / source_message_id / source_snapshot_id +payload members: name / media_type / byte_size / sha256 / required +terminal_evidence: state / decided_at / actor_ref / policy_version +lineage: parent artifact / document version / chat turn +retention_class / sensitivity / encryption_key_ref +source_database_identity / source_schema_version / source_commit +published_at / publisher_version / manifest_sha256 +``` + +manifest 使用稳定外部标识，同时保留旧数据库 ID 作为来源提示。它必须能在没有原数据库的情况下建立离线目录，但不得声称自己包含未列出的业务依赖。 + +### 5. 恢复必须经过隔离与对账 + +FAV 支持两种恢复结果： + +- `recover-as-collection`：把验证通过的成果导入新的恢复集合/文档，保留原来源与 hash，不伪造原工作流状态。这是默认模式。 +- `relink`：只有项目、用户、权限、会话和版本依赖均存在且无冲突时，才把成果重新连接到原逻辑资源。 + +恢复顺序为：离线 inventory → manifest/schema/SHA 校验 → 删除账本/保留策略检查 → scratch DB 导入 → 依赖与 ID 映射 → 冲突预览 → 人工确认 → 单事务 promotion → 审计报告。禁止 GET 接口因发现不一致而自动写回数据库，也禁止直接覆盖既有 head 或消息。 + +## 可恢复与不可恢复边界 + +| 可从 FAV 恢复 | 不能只靠 FAV 恢复 | +|---|---| +| 最终 Markdown/附件字节与 MIME/SHA | 用户认证凭证、session、token | +| 已归档助手最终回复及其稳定 turn/message 来源 | 项目成员关系、文档 duty、当时的完整授权图 | +| 文档版本/快照的有限来源、父子和生成 run 线索 | 活跃 lease、pending candidate、未完成 task/run | +| 最终报告、批准/提交终态证据 | 通知投递状态、outbox 消费位置、worker 状态 | +| 恢复所需 schema/producer/storage identity 元数据 | 未进入准入清单的草稿、中间过程和完整数据库约束 | + +因此灾难恢复仍须保留 PostgreSQL base backup/WAL 或经验证的完整逻辑备份；FAV 解决的是“最终成果可独立找回”和“有限业务内容可重建”。 + +## 存储、保留与安全 + +- 第一阶段可用独立 filesystem backend，但必须有独立 `vault_storage_identity`、最小权限凭证和 mount，不允许退回 `./data/dev`；凭证只从受管 secret/env 注入。生产目标应支持 TLS、静态加密及异机/对象存储复制。 +- 当前仓库没有 tenant 实体，安全边界应使用现有 `environment + project_members/role-admin` 授权模型，不得伪造 `tenant_id`。blob 采用**项目作用域加密 CAS**：同项目按 SHA-256 去重，跨项目相同 SHA 不共享密钥和删除边界；未来引入组织/租户层须另立 ADR 和迁移。 +- manifest-member 到 blob 的引用账本决定删除：tombstone 先撤销单个成果读取与 key wrapping；仅当项目作用域内最后有效引用消失且 retention/legal hold 允许时物理清除。manifest 和 commit marker 只追加，定期执行 checksum scrub 和离线目录重建。 +- filesystem backend 必须以同文件系统临时文件、校验、文件 `fsync`、原子 no-replace rename、父目录 `fsync` 实现持久化，确认 blob/manifest 落盘后才提交 marker；不能把“最后写 marker”误当成掉电安全。 +- WORM/对象锁按数据类别启用，不对所有内容一律永久锁定。普通保留使用可治理模式；法律保全单独使用 legal hold。 +- project/user 授权仍由服务端复用正常读取的既有判定；授权关系缺失或不确定时 fail closed。归档正文不得进入日志、metrics 或通知 payload。 +- 自动归档前执行服务端敏感内容准入：结构化排除 Prompt/工具轨迹，secret、认证材料、restricted 或分类失败均 fail closed，且正文不进入 outbox/staging。 +- 建立删除/法务保留账本。满足删除条件时先禁止读取和恢复，再删除密钥/对象；任何历史 restore 都必须重放 tombstone，避免已删除内容复活。 +- manifest 不保存 secret、Prompt、内部推理、工具轨迹或认证材料；敏感元数据最小化，必要时使用不可逆外部键映射。 + +## 与数据精简方案的关系 + +FAV 与 migration corpus、完整备份是三个不同产品： + +| 机制 | 目的 | 内容 | +|---|---|---| +| Final Artifact Vault | 持续保存终态成果，供定位、只读降级和有限恢复 | 小而长期、按终态事件持续追加 | +| Migration corpus | 验证旧 schema 到新 schema 的迁移、异常和 reload | 小而可重复、脱敏、带 assertions | +| DB + state backup/PITR | 完整灾难恢复与回滚 | 大而完整、按 RPO/RTO 生成 | + +清理不能因为“FAV 有最终正文”就跳过完整备份，也不能用完整备份替代成果级准入。正确门禁是三者分别验证：最终成果覆盖率、迁移 corpus 可升级、完整备份可恢复。 + +首次启用 FAV 时还必须做一次**存量终态回填**：只从权威数据库的不可变 snapshot、批准/提交事件和 completed chat turn 枚举，以来源事件 + payload SHA 幂等产生请求。缺正文或终态证据的记录进入 quarantine；批准排除必须记录原因、批准人和有效期。禁止为了达到 coverage 扫描 workspace 或批量把“旧数据”标为已覆盖。 + +## 外部依据 + +- [RFC 8493 BagIt](https://www.rfc-editor.org/rfc/rfc8493.html) 要求 manifest 覆盖 payload 并验证完整性，支持本方案的 manifest/checksum 设计。 +- [OCI Descriptor](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) 以 digest 标识并校验内容，支持内容寻址 blob。 +- [AWS Transactional Outbox guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) 说明业务写入与 outbox 同事务、消费者幂等的必要性。 +- [PostgreSQL PITR](https://www.postgresql.org/docs/17/continuous-archiving.html)、[pg_restore](https://www.postgresql.org/docs/17/app-pgrestore.html) 与 [pg_verifybackup](https://www.postgresql.org/docs/16/app-pgverifybackup.html) 说明完整恢复、选择性恢复依赖和真实 restore 演练的边界。 +- [AWS S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) 说明 governance/compliance/legal hold 的差异，支持按数据类别选择保留模式。 + +## 建议动作 + +1. 采用[最终成果保险库整改方案](../../remediation/2026-09-14-final-artifact-vault-remediation.md)，先定义终态协议和 manifest，再选择 filesystem 或对象存储实现。 +2. 将 FAV 发布完成率加入 dataset inventory/GC 的硬门禁，但不把 FAV 设为当前 409 的绕行修复。 +3. 首次实现优先覆盖受控文档 approved/agent committed snapshot 与 `export_policy=full` 的 completed assistant reply。 +4. 先在 dev 用项目 6173 的等价脱敏 corpus 做“生成 → 发布 → 清 workspace → 读取/恢复 → rebuild”演练，再考虑 main 保留策略。 diff --git a/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md index 179f5c3e..daa1ca58 100644 --- a/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md +++ b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md @@ -15,6 +15,7 @@ confidence: high 1. 清理历史运行数据只能降低数据转换量和现场噪声，不能替代对受支持旧 schema 的迁移兼容；应把有价值的旧状态压缩成小型、可重复、脱敏的 migration corpus。 2. 不应在当前库上逐表删除后继续使用；当前跨项目数据存在非完整级联和 DB/文件双事实源，优先采用“新库 + 新状态根 + 选择性导入 + 旧卷隔离”的蓝绿轮换。 +3. 对最终成果再增加独立 [Final Artifact Vault](./2026-09-14-final-artifact-vault.md)，但它只辅助成果定位与部分恢复，不能替代完整 DB+state 备份或旧 schema migration corpus。 ## 当前数据画像 @@ -138,6 +139,7 @@ manifest 至少记录 dataset 版本、source schema、目标 schema、生成 co - 项目 143 等真实内容不能直接固化为仓库 fixture；应保留 hash/结构特征并脱敏重建。 - migration corpus 不能只有 happy path，否则清场后会失去发现 legacy bug 的能力。 - 任何清理必须先完成隔离 restore，并验证数据库与文件包属于同一 storage identity。 +- 任何清理还必须验证应归档终态成果均已取得 FAV verified receipt；“有归档正文”不能替代完整关系闭包恢复。 - 错误 `/tmp` 数据根属于当前事故证据；在 split-brain 恢复报告签字前不纳入普通清理。 ## 建议动作 @@ -145,3 +147,4 @@ manifest 至少记录 dataset 版本、source schema、目标 schema、生成 co - 采用 [运行数据生命周期与迁移语料整改方案](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)。 - 首轮只处理 dev；test 改为短 TTL 隔离环境；main 仅盘点，不删除；shared KB 排除。 - 先实现 dataset inventory/export/restore/verify，再执行蓝绿精简，不先写清理 SQL。 +- 并行实施[最终成果保险库整改](../../remediation/2026-09-14-final-artifact-vault-remediation.md)，把成果覆盖率加入 GC/销毁门禁。 diff --git a/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md b/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md new file mode 100644 index 00000000..e983a986 --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md @@ -0,0 +1,61 @@ +--- +title: "最终成果保险库调研与整改方案" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Codex +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: artifact-vault +--- + +# 最终成果保险库调研与整改方案 + +## 目标 + +评估并完善“只归档最终生成物和最终智能体回复”的提案，形成不会与数据库、文档账本和工作区争夺事实源，且能支持受控部分恢复的系统方案。 + +## 范围 + +- 最终文档版本、最终助手回复和必要恢复元数据的归档边界； +- 终态判定、原子发布、不可变清单、校验、授权、保留与删除； +- 工作区清理门禁、降级读取、隔离恢复和数据库重建边界； +- 与现有数据生命周期、迁移语料、文档账本和对话持久化方案衔接。 + +## 非目标 + +- 本轮不实现业务代码、数据库迁移或对象存储； +- 不归档候选稿、流式片段、工具轨迹、租约、临时工作区或其他中间结果； +- 不把保险库定义为运行数据库的完整镜像或自动覆盖线上数据的入口； +- 不执行数据删除、卷切换或恢复写入。 + +## 步骤 + +1. [x] 登记任务和独立 mutex。 +2. [x] 核对现有文档快照、CAS、最终助手消息、任务终态和备份恢复边界。 +3. [x] 评审提案，明确可采用部分、必须修正部分和反模式。 +4. [x] 编写专项调研与整改方案，并并入运行数据生命周期方案。 +5. [x] 更新 CURRENT、FEATURE-MATRIX、gap-matrix 与整改索引。 +6. [x] 完成文档一致性、安全、可行性和对抗性复核。 + +## 验收标准 + +- [x] 明确“最终”的服务端判定规则和禁止进入保险库的内容。 +- [x] 明确数据库、工作区、保险库三者权威边界，不产生静默双写。 +- [x] 方案含 manifest/checksum、幂等发布、清理前门禁、降级读取和隔离恢复流程。 +- [x] 明确可恢复与不可恢复的数据范围、安全/删除要求及恢复演练。 +- [x] 本轮未实施业务代码或破坏性操作。 + +## 相关链接 + +- Research: [最终成果保险库与部分恢复边界](../research/2026-09-14-final-artifact-vault.md) +- Remediation: [最终成果保险库与受控恢复](../../remediation/2026-09-14-final-artifact-vault-remediation.md) +- Test report: 待用户批准实施后创建 + +## 证据 / 备注 + +- 当前受控文档不可变正文位于 PostgreSQL `document_content_blobs`，工作区 `output.md` 是投影。 +- 当前附件 CAS 物理 blob 位于运行状态根，引用元数据位于 PostgreSQL。 +- 当前任务完成摘要通过 `FinalizeAssistantTaskMessage` 写回 `chat_messages`，尚无独立终态成果发布协议。 +- 文档复核补齐：显式发布不可绕过终态、不可变 payload staging、项目作用域加密 CAS/引用删除、存量回填、掉电持久化、敏感内容准入、既有项目授权降级和 scratch 恢复权限。 +- 独立 Claude 安全/对抗复核结果已收敛；whole-document peer 因结构化输出重试耗尽被丢弃，未作为结论依据。 diff --git a/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md index 9a358b17..32eb9432 100644 --- a/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md +++ b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md @@ -20,6 +20,7 @@ mutex: infra-runtime-state - dev/test/main 数据量、schema、项目/文档/run 关系与文件状态； - 备份/恢复、选择性导出、迁移 corpus、蓝绿轮换； - retention 分类、TTL、quarantine、GC 和验收门禁。 +- 最终成果保险库 coverage 与清理门禁，但不以其替代完整备份。 ## 非目标 @@ -35,6 +36,7 @@ mutex: infra-runtime-state 3. [x] 审计现有 backup/restore drill 能力与隔离缺口。 4. [x] 制定数据分类、最小保留集、migration corpus 和蓝绿轮换方案。 5. [x] 更新父整改、CURRENT、gap 与 remediation 索引。 +6. [x] 并入最终成果保险库、终态 coverage 和受控部分恢复边界。 ## 验收标准 @@ -42,11 +44,13 @@ mutex: infra-runtime-state - [x] 首轮 retain 建议基于能力覆盖，不按时间盲留。 - [x] 方案包含备份配对、真实 restore、选择性导入、reload/rebuild、rollback 和延迟销毁。 - [x] 本轮未实施破坏性操作。 +- [x] 明确最终成果保险库、migration corpus 和完整 DB+state backup 三者互补且分别验收。 ## 相关链接 - Research: [../research/2026-09-14-runtime-data-retention-and-migration-corpus.md](../research/2026-09-14-runtime-data-retention-and-migration-corpus.md) - Remediation: [../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Final Artifact Vault: [../../remediation/2026-09-14-final-artifact-vault-remediation.md](../../remediation/2026-09-14-final-artifact-vault-remediation.md) - Test report: 待用户批准实施后创建 ## 证据 / 备注 diff --git a/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md b/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md new file mode 100644 index 00000000..ed11b8bd --- /dev/null +++ b/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md @@ -0,0 +1,247 @@ +--- +title: "最终成果保险库与受控恢复整改方案" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +deepened: 2026-09-14 +--- + +# 最终成果保险库与受控恢复整改方案 + +## 背景 + +提案评审见[最终成果保险库调研](../ops/research/2026-09-14-final-artifact-vault.md)。当前受控文档账本、助手消息和附件 CAS 分别覆盖局部持久化，但都仍与运行 PostgreSQL 或 `RUNTIME_STATE_ROOT` 绑定；工作区 split-brain 已证明成果定位不能依赖 `output.md` 路径。 + +本方案把 FAV 作为[运行数据生命周期方案](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)的长期成果层。它不替代当前事故的固定数据根整改，也不替代完整数据库备份。 + +## 目标 + +1. 只有服务端确认的终态文档、终态助手回复及其必要附件进入 FAV，中间结果准入率为 0。 +2. 每个成果都有稳定 UUID、语义 manifest、payload SHA、来源终态证据、保留策略和可验证发布回执。 +3. FAV 使用独立存储身份，工作区清理、worktree 删除和 app rebuild 不影响已验证成果。 +4. 数据库正文缺失但授权图完整时可只读降级；数据库关系损坏时只能经 scratch 恢复、冲突预览和人工 promotion。 +5. 完整数据库恢复、migration corpus 与成果级恢复分别演练、分别签字。 + +## 核心决策 + +1. **互补而非替代**：PostgreSQL 是运行关系与授权权威；FAV 只对已归档 payload 与 manifest 权威；workspace 是可清理投影。 +2. **事件准入而非目录扫描**：由 `agent_commit`、`approved`、`sealed`、显式 publish 和 completed chat turn 发起准入判断，禁止扫描 workspace 猜测“final”；显式 publish/pin 只是请求，不能绕过终态 allowlist。 +3. **事务 outbox**：业务终态与 publish request 同事务；异步 publisher 幂等写对象并最后写 commit marker。 +4. **默认只归档高价值回复**：completed + `export_policy=full` 自动进入；`copy_only` 仅显式 pin；`none` 永不进入。 +5. **恢复默认生成新受控对象**：`recover-as-collection` 保留来源；只有依赖、授权和 ID 无冲突时才允许 `relink`。 +6. **清理受覆盖率门禁**：应归档而未 `verified` 的成果会阻止受管 dataset/持久状态源销毁；普通计算 workspace 在终态 payload 已冻结后可按 TTL 清理，不能再成为成果事实源。 + +### 终态准入协议（U1 必须固化） + +| 类型 | 允许的权威数据库事实 | payload 来源 | 拒绝条件 | +|---|---|---|---| +| Agent 文档提交 | `document_agent_candidates.status='committed'`，存在 `output_snapshot_id`，且对应 `document_change_events.change_kind='agent_commit'` 的 `to_snapshot_id` 与该 snapshot 一致 | `document_content_blobs` 对应不可变 snapshot | ready/needs_confirmation/rejected/stale/quality_failed、snapshot/event 不一致 | +| sealed 文档版本 | `document_versions.status='sealed'` 且 `sealed_at` 非空 | 该版本 `head_snapshot_id` | open、head 缺失 | +| approved 受控文档 | `document_records.current_state='approved'`、`approved_at` 非空且 `review_snapshot_id` 非空 | 冻结的 `review_snapshot_id`，不得取 workspace 当前文件 | reviewing/approving/rejected、review snapshot 缺失 | +| 自由文档发布 | 显式 publish 事件引用已存在的不可变 ledger snapshot | 事件指向的 snapshot | 只有 UI 动作但无不可变 version，或事件后才读取“最新”正文 | +| 助手最终回复 | `chat_turns.status='completed'`、`completed_at` 和唯一 `assistant_message_id` 均存在；同事务对精确正文计算的 policy snapshot 为 `full`，或为 `copy_only` 且存在 completed 后的显式 pin 事件 | 同事务冻结到 staging 的精确消息 revision/SHA | pending/streaming/failed/cancelled、placeholder/card/status、policy none/restricted/分类失败、pin 时未 completed | + +HTTP/SSE handler 只能调用这一套领域 eligibility 服务，不能自己根据字符串或响应结束推断终态。U1 先以当前数据库列和约束写 characterization test；若发现任一完成链缺少上述事实，先补领域状态机/事务边界，再允许接入 FAV。 + +## 高层架构 + +```mermaid +flowchart TB + A[Document or chat terminal transaction] --> B[Operational PostgreSQL] + A --> C[artifact.publish.requested outbox] + C --> D[Idempotent publisher] + D --> E[Content-addressed blobs] + D --> F[Semantic manifest] + F --> G[Commit marker and receipt] + G --> H[Cleanup coverage gate] + G --> I[Authorized degraded read] + G --> J[Scratch recovery and reconciliation] +``` + +### 发布状态机 + +```mermaid +stateDiagram-v2 + [*] --> requested + requested --> publishing + publishing --> verified + publishing --> retryable_failed + retryable_failed --> publishing + verified --> tombstoned + tombstoned --> purged +``` + +只有 `verified` 成果可用于清理门禁和恢复。manifest 写完但没有 commit marker 的对象视为未发布；`tombstoned` 内容不能被读取或恢复。 + +## 数据与存储契约 + +### 运行库注册表 + +新增等价模型： + +```text +final_artifacts + artifact_uuid, artifact_type, logical_resource_id + environment, project_id, project_external_key, source_event_id + source_snapshot_id / source_message_id / source_run_id + payload_sha256, manifest_sha256, vault_storage_identity + terminal_state, policy_version, retention_class + publication_status, requested_at, verified_at, tombstoned_at + +artifact_payload_staging + staging_uuid, project_id, source_event_id, payload_sha256 + encrypted_payload / content_blob_ref, classification, created_at, releasable_at + +artifact_blob_references + artifact_uuid, project_scope, payload_sha256, envelope_key_ref + reference_status, retention_class, legal_hold, timestamps + +artifact_publication_attempts + artifact_uuid, idempotency_key, attempt_no, error_code, receipt, timestamps + +artifact_deletion_ledger + artifact_uuid, reason, authority, effective_at, applied_at, audit_ref +``` + +实现时优先复用现有 `outbox_events` 投递框架；归档注册与文档/消息终态的原子边界需由所属服务事务负责，不能由 HTTP handler 事后拼接。 + +幂等 key 由业务终态事务确定性生成：`artifact_type + environment + project_external_key + logical_resource_id + source_event_kind + source_event_id + payload_sha256 + policy_version`。数据库对该 key 建唯一约束；同一终态重试/重新发布复用 key，新终态事件或新 payload 生成新 artifact，不用 outbox event ID 或 publisher 本地状态充当业务幂等键。 + +当前仓库没有 tenant 实体；本期安全边界沿用现有 `environment + project_members/role-admin` 模型，不新增无法校验的 `tenant_id`。若未来引入组织/租户层，须另立 ADR 和数据迁移，不能仅改路径字段。 + +终态事务必须把**精确 payload 字节或不可变 content blob 引用、SHA、来源版本**一并冻结到 staging，outbox 只引用 staging；publisher 不得在事务后重读可修改或会级联删除的消息行。staging 在成果 `verified` 或依法 `tombstoned` 前不得回收，且正文不能进入普通 outbox payload、日志或指标。 + +### Vault 对象布局 + +```text +scopes///blobs/sha256/<2>/<2>/ +manifests//.json +commits//.json +tombstones//.json +catalog/checkpoints/.json +``` + +路径不包含用户文件名、Prompt 或数据库自增 ID。manifest 记录逻辑标识、来源、成员、hash、终态证据、schema/producer/storage identity、敏感级别和保留策略；commit marker 最后生成。 + +U1 必须交付版本化 `manifest.schema.json`、`commit-marker.schema.json` 和 `tombstone.schema.json`，并由同一 Go 类型/validator 验证。schema 采用 major/minor：读端必须兼容同 major 的旧 minor；未知 major fail closed 并进入迁移队列，不能静默猜字段。commit marker 至少绑定 artifact UUID、manifest SHA、全部 required member SHA、storage identity、committed_at 和 publisher version。 + +CAS 只在项目安全作用域内去重，并采用项目作用域 envelope key；`artifact_blob_references` 是 tombstone、legal hold 和物理回收的权威引用账本。tombstone 立即撤销该 artifact 的读取和密钥包装；仅当该作用域内最后一个有效引用消失且 retention/legal hold 允许时，才清除 key wrapping 与 blob。跨项目相同 SHA 必须保持密码学与删除边界隔离。 + +删除事务在运行库写 `artifact_deletion_ledger` 与删除 outbox，vault tombstone 是独立持久的拒绝恢复证据。读取/恢复采用 **union-deny**：任一侧声明删除都拒绝；两侧不一致进入 quarantine 和告警，不得恢复。purge 后仍永久保留最小 tombstone（或经校验的压缩删除目录），完整备份恢复后重放删除账本，防止旧备份复活数据。 + +## 分波次计划 + +| 波次 | 内容 | 验收 | +|---|---|---| +| V0 协议与 ADR | 固化三层权威边界、终态准入矩阵、manifest schema、恢复语义和删除模型 | ADR 与 JSON schema 评审通过；所有成果类型都有允许/拒绝状态 | +| V1 存储与发布内核 | 建 `internal/artifactvault`、注册表/staging/outbox/attempt/reference/deletion migration、filesystem backend、项目作用域加密 CAS、manifest/commit marker 和幂等 publisher | 重复投递只生成一个逻辑成果；崩溃点重试不产生半发布可见对象；坏 SHA 拒绝 verified；跨项目同 SHA 不共享删除或密钥边界 | +| V2 文档接入 | 从账本 snapshot 接入 agent committed、sealed、approved 和显式 publish；approved hook 改读账本而非 workspace | ready/rejected/stale/quality_failed 不入库；清空 workspace 后仍能校验/读取归档正文 | +| V3 助手回复接入 | 把消息正文定稿、chat turn completed 和 publish request 收敛到原子服务边界；复用 `replyexport` 分类并增加真实回合状态校验 | placeholder/stream/error/card 不入库；completed+full 精确归档一次；SSE done 前已有 durable request | +| V3.5 存量终态回填 | 只从权威数据库账本、批准事件和 completed chat turn 枚举既有终态，以来源事件 + payload SHA 生成确定性请求；缺正文/证据项进入 quarantine 或批准排除 | 重跑不增加逻辑成果；不扫描 workspace；每个排除项有原因、批准人和审计记录；清理 coverage 有可达分母 | +| V4 查找与降级读取 | 建授权后的 artifact lookup；正文缺失且授权完整时返回带 degraded 标记的 FAV 内容；权限图缺失时 fail closed | 跨用户/跨项目负例通过；GET 不写数据库；vault 不可用时返回 typed degraded/error 而非伪 404 | +| V5 恢复工具 | 建 inventory/verify/recover-plan/recover-apply；强制验证 scratch identity，使用无运行库权限的恢复凭证；默认 recover-as-collection，relink 需完整依赖、授权和人工确认 | DB 全删的隔离环境可重建成果目录；运行库目标、未授权操作者、计划篡改、冲突、删除账本和缺权限依赖均阻止 promotion | +| V6 清理与运维门禁 | dataset inventory/GC 加 coverage；readiness 暴露 identity/connectivity/lag，metrics/alerts/checksum scrub/复制与 runbook 落地 | 未 verified 成果阻止销毁；workspace 清空、app rebuild、离线副本恢复演练均 PASS | + +## 实施单元与文件落点 + +### U1. 终态协议、manifest 与数据库模型 + +- **目标**：先建立可审计的准入和恢复契约，避免把所有“完成”字符串当终态。 +- **文件**：`.doc/adr/` 新 ADR；`deploy/postgres/goose/000xx_*final_artifact*.sql`；`internal/artifactvault/types.go`、`manifest.go`、`eligibility.go`、`manifest.schema.json`、`commit-marker.schema.json`、`tombstone.schema.json` 及对应测试。 +- **测试场景**：每类允许事件生成确定 artifact key；所有候选/失败/中间状态拒绝；未终态的显式 publish/pin 仍拒绝；secret、认证材料、restricted 或分类失败均在正文进入 outbox/staging 前 fail closed；manifest 必填字段、版本和 SHA 错误 fail closed；迁移 Up/Down 与约束验证。 +- **完成证据**：准入矩阵与 schema 一致，未分类类型默认拒绝。 + +### U2. 独立存储与幂等 publisher + +- **依赖**：U1。 +- **文件**：`internal/artifactvault/store.go`、`filesystem_store.go`、`publisher.go`、`publisher_test.go`；`internal/config/config.go`；环境配置与 Compose 校验。 +- **执行说明**：vault 使用独立最小权限身份；凭证只从受管 secret/env 注入，不进入仓库、manifest、日志或业务数据库。后端必须启用传输加密和静态加密，manifest 只保存 key reference。filesystem 后端采用同文件系统临时文件、校验、文件 `fsync`、原子 no-replace rename、父目录 `fsync`，blob 与 manifest 持久化后才提交 marker。 +- **测试场景**：同一 outbox 重复消费；blob 已存在；每个写入/fsync/rename 边界强制终止后重启；损坏 blob；错误 storage identity；凭证轮换；vault 暂时不可用后重试；跨项目同 SHA、单方 tombstone、legal hold 和最后引用 purge。 +- **完成证据**：只有完整且已持久化的 manifest + commit marker 可见，publisher 不读 workspace 推断成果；任一项目删除不破坏其他项目成果。 + +### U3. 文档账本与治理终态接入 + +- **依赖**：U1、U2。 +- **文件**：`internal/docrevision/pg_mutate.go`、`internal/docgov/engine.go`、`internal/bizdev/service.go`、`internal/server/document_revision_handlers.go` 及 PG/服务测试。 +- **执行说明**：先补 characterization/PG 测试，再调整事务边界和 approved hook。 +- **测试场景**：agent commit、sealed final、approved review snapshot、已满足终态 allowlist 的显式 publish；未终态显式 publish、pending/reject/stale/quality failed 负例；投影缺失但账本和归档一致；同 run 重试。 +- **完成证据**：归档 payload 来自不可变账本 snapshot，不读取 `output.md` 作为事实源。 + +### U4. 最终助手回复原子接入 + +- **依赖**：U1、U2。 +- **文件**：`internal/ctxmgr/chat_turns.go`、`message_update.go`、`internal/server/task_finalize.go`、`internal/chat/session.go`、`internal/replyexport/policy.go` 及对应测试。 +- **执行说明**：先刻画同步回复、异步任务、SSE 断线重试三条完成链，再收敛到一个 terminal-finalize 领域事务。该事务以锁定的 message revision/content 重新计算并冻结 policy snapshot，不能信任 handler 传入或较早缓存的 `export_policy`；pin 是对已 completed 的精确 revision/SHA 追加独立终态事件。 +- **测试场景**：completed+full、completed+copy_only+显式 pin；未 completed 的 pin/publish、凭证回显、restricted/分类失败、placeholder 更新、failed/cancelled、card/status、重复 finalize、SSE done 前后崩溃；请求后修改/删除原消息再消费仍归档原 SHA；日志和指标不含正文。 +- **完成证据**：消息精确正文、completed turn、不可变 staging 和 publish request 同事务，不存在“完成但永远未请求归档”或“发布时读到后来正文”的窗口。 + +### U7. 存量终态回填与覆盖率基线 + +- **依赖**：U3、U4。 +- **文件**：`cmd/artifact-vault/backfill.go`、`internal/artifactvault/backfill.go` 及 PG 测试；dataset inventory 工具。 +- **执行说明**：只枚举不可变 document snapshot/终态事件和 completed chat turn，不读 workspace；用稳定来源事件 + payload SHA 幂等生成 staging/request。证据不足或正文缺失项进入 quarantine；排除需理由、批准人、有效期和审计记录，不能用“旧数据”批量豁免。 +- **测试场景**：回填重复执行、并发新终态、旧消息已删、缺 snapshot、批准排除过期、quarantine 修复后重跑。 +- **完成证据**：存量 coverage 的分子、分母、quarantine 和批准排除均可复算，V6 清理门禁不靠人工改数字通过。 + +### U5. 授权查找与只读降级 + +- **依赖**：U2–U4。 +- **文件**：`internal/artifactvault/catalog.go`；`internal/server/project_doc_handlers.go`、`reply_export_api.go`、`handlers.go` 及鉴权测试。 +- **执行说明**：降级读取不是独立权限或直连 vault 能力；必须先由运行库中的 resource→project 关系定位资源，再用现有 `requireProjectAccess`/系统管理员判定完成与正常读取相同的服务端授权。manifest 只用于交叉校验 project external key/sensitivity，绝不自行授予权限；任一授权依赖缺失或结果不确定即 fail closed。 +- **测试场景**：DB 正常时不走 FAV；投影/正文缺失且现有项目授权完整时降级；项目关系缺失、越权、跨项目同 SHA、tombstone、manifest/SHA 异常全部 fail closed；GET 零写入。 +- **完成证据**：响应显式标识来源与降级原因，不再把存储不一致伪装为 404。 + +### U6. 隔离恢复、清理门禁与演练 + +- **依赖**：U2–U5、U7。 +- **文件**：`cmd/artifact-vault/`、`internal/artifactvault/recovery.go` 与测试；`scripts/backup-system-db.sh`、`restore-system-db-drill.sh`；dataset 生命周期工具；`.doc/test-reports/`。 +- **执行说明**：`recover-apply` 先核验数据库 identity 并拒绝 dev/test/main 与任何非 scratch 目标，只接受无运行库权限的恢复专用凭证。操作者须通过 `role-admin` 与明确恢复能力校验；relink/promotion 还需目标项目授权和指定数据责任人确认。actor、plan hash、目标 identity 和 promotion 结果写不可变审计。恢复计划显式分类 UUID collision、SHA mismatch、partial dependency、authorization gap、metadata conflict；默认全部 fail，只有 recover-as-collection 可对允许的孤立成果生成新 UUID，relink 不允许 skip/prompt 后带冲突提升。 +- **清理门禁**：只允许 `dataset inventory/gc/destroy` 和环境 teardown 脚本执行受管销毁，preflight 服务不可用或 coverage 非 PASS 时 fail closed；vault/app 身份没有删除 dataset 的权限，运维删除身份单独审计。普通计算 workspace 可按 TTL 清理，因为终态确认前已把 payload 冻结进 staging；原始 shell/root 删除无法由应用层绝对阻止，必须靠主机权限与操作制度限制，不能在验收中声称“拦截任意 rm”。 +- **测试场景**：空 scratch 库 recover-as-collection；依赖齐全 relink；五类冲突；运行库 DSN、未授权操作者、计划 hash 被篡改、缺 blob、删除账本/tombstone 不一致与重放；未 verified 或 preflight 不可用阻止受管 GC；workspace 清空与 app rebuild；异机副本恢复。 +- **完成证据**：项目 6173 等价脱敏 corpus 完成“终态生成 → 发布 → 清 workspace → 读取 → scratch 恢复 → rebuild”真实链路。 + +## 验收标准 + +- [ ] 中间状态、流式片段、工具轨迹和 workspace 草稿进入 FAV 的测试结果为 0。 +- [ ] 应归档终态成果的 publish request 覆盖率为 100%，verified 延迟与失败积压可观测。 +- [ ] 任一 publisher 崩溃点都可幂等恢复，半成品不对读/恢复可见。 +- [ ] 工作区全部删除或换根后，已 verified 成果仍可由稳定逻辑 ID 定位并通过 SHA 校验。 +- [ ] 授权图缺失时 fail closed；跨环境/project/user 的读取和恢复负例通过；当前实现不伪造 tenant 边界。 +- [ ] 显式 publish/pin 不能绕过终态 allowlist；敏感分类失败或命中禁止内容时正文不进入 staging/outbox/vault/日志/指标。 +- [ ] 存量终态回填可幂等重跑；quarantine 与批准排除均有可复算审计，coverage 分母不可手工绕过。 +- [ ] 项目作用域引用账本保证 tombstone 立即不可读，最后有效引用且无 hold 时才清除密钥和对象。 +- [ ] FAV 可以在空 scratch 库恢复成果集合，但不能伪造权限、活动工作流、lease、pending task 或完整历史。 +- [ ] tombstone/删除账本在 restore 后重放，已删除内容不会复活。 +- [ ] PostgreSQL 完整 restore、migration corpus 升级和 FAV 部分恢复三条演练均有独立报告。 +- [ ] dev 清理只有在完整备份可恢复、migration corpus PASS、FAV coverage PASS 后才能进入销毁审批。 + +## 风险与缓解 + +| 风险 | 缓解 | +|---|---| +| FAV 变成第三真相源 | 限定只对 payload/manifest 权威；运行指针和授权始终由数据库决定 | +| “最终”判定过宽导致存满中间结果 | 状态机 allowlist + 未分类拒绝 + 准入指标/抽样审计 | +| 数据库提交与归档双写丢事件 | 同事务 outbox + 幂等 publisher + cleanup coverage gate | +| outbox 指向可变或已删除正文 | 终态事务冻结精确 payload/SHA 到不可变 staging，publisher 只读 staging | +| 存量终态不触发新 hook | 权威数据库幂等回填 + quarantine/批准排除审计，禁止 workspace 猜测 | +| 归档可读但无法重建业务关系 | manifest 声明依赖与恢复能力；默认 recover-as-collection；完整灾备继续依赖 DB backup/PITR | +| WORM、共享 blob 与删除义务冲突 | 项目作用域加密 CAS + manifest-member 引用账本；按 retention class 选 governance/compliance；单独 legal hold；最后引用后密钥/对象清除 | +| 同机独立目录仍无法抗磁盘故障 | 第一阶段独立 mount/identity，生产要求异机或对象存储复制并做恢复演练 | +| 归档泄露敏感上下文 | 结构化排除 Prompt/轨迹；服务端敏感分类 fail closed；最小化 metadata；传输/静态加密、项目授权、凭证隔离和访问审计 | + +## 明确不采用 + +- 定时扫描 `**/output.md` 并复制“最新文件”。 +- 把所有 assistant 消息、SSE event 或 agent session 全量归档。 +- 在业务事务内同步调用对象存储，或反过来先写对象再提交业务状态。 +- DB 不一致时由 GET 请求静默重建/覆盖数据。 +- 因为 FAV 有最终正文就删除完整备份、WAL 或 migration corpus。 +- 把所有对象永久 compliance lock，忽略合法删除和数据最小化。 + +## 相关链接 + +- Research: [最终成果保险库与部分恢复边界](../ops/research/2026-09-14-final-artifact-vault.md) +- Parent remediation: [运行数据生命周期与迁移语料库](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Incident remediation: [dev 数据根隔离与投影自愈](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [最终成果保险库方案](../ops/short-term/2026-09-14-final-artifact-vault-plan.md) +- gap-matrix: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md index 393de773..c58cf6ae 100644 --- a/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md +++ b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md @@ -25,6 +25,7 @@ updated: 2026-09-14 3. 每次发布同时验证 clean install、受支持旧版本升级、选择性数据导入、reload/rebuild 和 rollback。 4. 数据库 dump 与文件状态根成对归档，具备 storage identity、SHA manifest 和自动断言。 5. 清理不产生孤儿 snapshot/run/blob/workspace，不误删共享 blob，不接触未授权 main/KB 数据。 +6. 最终成果在清理前进入独立 [Final Artifact Vault](./2026-09-14-final-artifact-vault-remediation.md) 并取得 verified 回执；该回执不能替代完整备份和 migration corpus。 ## 数据保留策略 @@ -52,6 +53,7 @@ retention_until pinned_reason contains_sensitive_data manifest_uri / manifest_sha256 +required_artifact_count / verified_artifact_count status: building | verified | active | quarantined | expired | destroyed ``` @@ -97,6 +99,10 @@ status: building | verified | active | quarantined | expired | destroyed 首轮 dev 推荐不在旧库运行 GC，而是蓝绿新建后切换；GC 用于后续小规模过期数据。 +### 6. `artifact coverage` + +按服务端终态事件盘点应归档成果，并与 FAV verified receipt 对账。任何必需成果处于 requested/publishing/failed/missing，或 vault storage identity 不匹配时，受管 dataset/持久状态源不得销毁。普通计算 workspace 只有在终态事务已冻结精确 payload 后才可按 TTL 清理；准入、manifest、恢复和删除边界见[最终成果保险库整改](./2026-09-14-final-artifact-vault-remediation.md)。 + ## 迁移测试矩阵 每个 release candidate 必须执行： @@ -110,6 +116,7 @@ status: building | verified | active | quarantined | expired | destroyed | Import | latest schema + selected demo/regression package | 关系闭包、文件 SHA、权限、通知、版本账本 | | Reload | import 后 app/pi rebuild + restart + 宿主级重载 | storage identity、mount、task 恢复、正文/overlay | | Rollback | 切换失败后接回 quarantine 旧卷 | RTO、旧 storage identity、无新写丢失 | +| Artifact recovery | 已归档终态成果 + 空 scratch DB | manifest/SHA、删除账本、recover-as-collection、冲突预览、权限 fail closed | 迁移通过必须包含真实 restore，不接受仅跑 Goose 幂等测试；产品签字至少包含真实页面登录、项目列表、对话、文档打开/修改、刷新、退出重登和 rebuild 后恢复。 @@ -119,7 +126,7 @@ status: building | verified | active | quarantined | expired | destroyed |---|---|---| | D0 冻结与标注 | 先完成当前 split-brain 遏制；冻结 dev 写入；对 366 个项目 inventory；人工确认 pinned/敏感/owner | 100% 项目有建议动作；未分类项不删除 | | D1 工具安全修补 | 移除备份脚本内默认凭证；backup 同时归档 DB+state；restore 强制 scratch 且真实恢复；加入 manifest/checksum/assertions | 恢复演练能从零创建隔离环境并通过 SHA/关系断言 | -| D2 构建 corpus | 建 schema 25、28/29、32/33 三个最小迁移包；构建 143 等价脱敏 hazard；选择 6173 current regression；6159 标记一周期过期 | 每包可重复生成/恢复，不依赖真实 main/test 数据 | +| D2 构建 corpus 与成果覆盖 | 建 schema 25、28/29、32/33 三个最小迁移包；构建 143 等价脱敏 hazard；选择 6173 current regression；盘点保留项目的终态成果并生成 FAV coverage | 每包可重复生成/恢复；必需成果均 verified 或有批准的排除理由，不依赖真实 main/test 数据 | | D3 新建 green dev | latest schema 空库 + 固定新 state root；导入选中的 6–10 项；执行 projection/CAS/sequence/integrity 校验 | 无 orphan；active 文档 aligned；ready/storage identity PASS | | D4 全链验收 | clean/oldest/previous/hazard/import/reload/rollback；真实页面和真实 PI 跑主链 | 报告区分工程、PG、PI、浏览器、restart/rebuild 证据 | | D5 原子切换 | 切换 dev 到 green identity；原 blue DB/文件根转只读 quarantine | 新环境持续观察 24–72h，无新增 blocker，可一键回接 blue | @@ -152,6 +159,8 @@ status: building | verified | active | quarantined | expired | destroyed - [ ] 三个 migration baselines 与 hazard corpus 均能升到 latest 并通过断言。 - [ ] 无 orphan task run、revision snapshot/version/event/report、CAS blob 或 workspace。 - [ ] 项目 143/6173 页面打开、修改、刷新、重登、rebuild 后恢复 PASS。 +- [ ] 保留项目的应归档成果覆盖率为 100%，清空 workspace 后仍能定位并校验最终正文/最终回复。 +- [ ] FAV 在空 scratch 库完成受控部分恢复；未声称或伪造权限、lease、pending task 等未归档状态。 - [ ] 新测试数据带 TTL，PASS 后自动销毁，失败数据按策略到期。 - [ ] test/main/shared KB 未被误操作；所有破坏性动作有精确 manifest 和延迟销毁记录。 @@ -161,11 +170,13 @@ status: building | verified | active | quarantined | expired | destroyed - 在 dev 库中写一份巨型级联 DELETE：当前关系与文件引用复杂，容易留下孤儿或误删共享 blob。 - 只保留最新时间的项目：会丢失最有价值的旧 schema/hazard 覆盖。 - 把整份真实旧库永久当 migration fixture：迁移慢、不可重复、可能含敏感数据。 +- 把 FAV 当完整数据库替代品，或按 `output.md`/mtime 扫描猜测终态：会引入第三事实源并归档中间结果。 - 修改历史 Goose migration 迎合新空库：已发布迁移不可重写；修正必须新增 migration。 ## 相关链接 - Research: [运行数据精简、迁移语料与演示数据保留策略](../ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md) +- Final Artifact Vault: [最终成果保险库与受控恢复](./2026-09-14-final-artifact-vault-remediation.md) - Parent remediation: [dev 数据根隔离与投影自愈](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) - Short-term: [运行数据生命周期方案](../ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md) - gap-matrix: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/INDEX.md b/.doc/remediation/INDEX.md index bf37c415..dd38fff4 100644 --- a/.doc/remediation/INDEX.md +++ b/.doc/remediation/INDEX.md @@ -5,6 +5,7 @@ | [gap-matrix.md](./gap-matrix.md) | 发布缺口 SSOT | | [2026-09-14-dev-runtime-data-root-and-projection-remediation.md](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | 临时 worktree 错挂 dev 数据根。审查修订：Unit A 纠挂 / Unit B 状态根与 readiness / Deferred outbox | | [2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | dev/test 数据精简、migration corpus、蓝绿换库、TTL 与延迟销毁方案 | +| [2026-09-14-final-artifact-vault-remediation.md](./2026-09-14-final-artifact-vault-remediation.md) | 只收终态文档/助手回复的独立保险库、项目作用域加密 CAS、清理门禁与受控部分恢复 | | [2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md](./2026-09-14-workflow-plan-409-frozen-and-optional-duties-remediation.md) | 项目设置 409 冻结态守卫 + 非必需文档三权全有或全无 + 成员/门禁/权限一致性 | | [2026-09-14-controlled-content-canonical-hash-remediation.md](./2026-09-14-controlled-content-canonical-hash-remediation.md) | 项目 143 正文 canonical/raw 哈希漂移、错误语义与重复 hydration 整改 | | [2026-09-13-document-revision-reviewer-duty-sync.md](./2026-09-13-document-revision-reviewer-duty-sync.md) | UI-01 已分配 B=reviewer，受控文档 reviewer_id 硬编码 2，矩阵降为 viewer | diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index 5dd3f6e1..a3aa5928 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -57,7 +57,7 @@ | ID | 描述 | 状态 | 证据 | 计划动作 | |---|---|---|---|---| -| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。2026-09-14 ~09:23 UTC 复核：挂载仍为 `/tmp/as-dev-wf409-merge`，`/ready` 仍 200/`schema=33`。审查已否决「恢复与 outbox 同批关门」 | 关门 = [审查修订方案](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) 的 Unit A 纠挂 + Unit B 绝对状态根/门禁/`/ready` 503。outbox 为 Deferred。[data lifecycle](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) 不替代事故修复，也不挡 Unit A | +| RUNTIME-DATA-ROOT-SPLIT-BRAIN | dev rebuild 从临时集成 worktree 解析 `./data/dev`，app/pi/cursor 挂到空 `/tmp` 数据根，而 PostgreSQL 沿用原持久卷；100 条非模板记录在当前容器根抽样均 missing，health/ready 仍 200 | **FAIL / CRITICAL** | [audit](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)。2026-09-14 ~09:23 UTC 复核：挂载仍为 `/tmp/as-dev-wf409-merge`，`/ready` 仍 200/`schema=33`。审查已否决「恢复与 outbox 同批关门」 | 关门 = [审查修订方案](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) 的 Unit A 纠挂 + Unit B 绝对状态根/门禁/`/ready` 503。outbox 为 Deferred。[data lifecycle](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) 与 [FAV](./2026-09-14-final-artifact-vault-remediation.md) 不替代事故修复，也不挡 Unit A | | WORKFLOW-PLAN-FROZEN-OPTIONAL-DUTIES | active 项目仍展示可编辑职责表单，409 提示在长表单视口外；现有规划又不支持非必需文档“三权全空”，且后端缺少职责人在任成员复核 | **PASS（设置页）** | [audit](../audit/2026-09-14-workflow-plan-409-frozen-and-optional-duties.md)；[test-report](../test-reports/2026-09-14-workflow-plan-409-frozen-and-optional-duties-test-report.md)。项目 143 只读 frozen、A full+none 激活、B partial 不发 PUT、C 零治理拒绝、双标签转 frozen、成员无配置入口、刷新后 143 仍 frozen | 真实 PI 生成 full/none 并走审批/阶段不阻塞未纳入本轮页面矩阵，不作为本 gap 关闭的替代证据，另立项 | | DOC-CONTENT-CANONICAL-HASH-DRIFT | 旧受控文档 bootstrap 裁剪 Markdown 双空格硬换行并将 canonical SHA 写入账本/DocGov，但未同步文件；正文读取把完整性冲突误报 404，lease/list sync 又放大并制造刷新后加载死锁 | **PASS（真实 UI）** | [audit](../audit/2026-09-14-project143-controlled-content-404.md)；[UI report](../test-reports/2026-09-14-controlled-content-ui-acceptance.md)。代码已合 dev、schema 32、项目 143 定向 `migration_repair`；项目 143 `clean-run3` 与新项目 `full-chain-final` 主链及 restart 全部纯页面 PASS，目标 404=0 | [remediation](./2026-09-14-controlled-content-canonical-hash-remediation.md)。隔离 `TestReconcile_LegacyRepair_PG` 未执行，按用户“本轮只允许页面测试”的边界保留为工程测试债，不作为本轮 UI PASS 的替代证据 | | CHAT-HISTORY-REFRESH | 工作流激活后刷新，对话正文被阶段过滤器清空，但分类数量与锚点仍按全量返回 | **PASS** | [test-report](../test-reports/2026-09-13-chat-history-refresh-test-report.md)；代码：`message_scope.go`、`chat_history_auth.go`、goose 00026/00027 | 真实 PI 刷新 E2E 待 dev 容器补录（自动化/PG 已 PASS） | diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index dd1f3c8c..75867747 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -23,7 +23,7 @@ | IAM / 鉴权 | 项目级 API 鉴权 + RBAC | partial | [audit/authorization](../audit/authorization-matrix-v2.md)；[刷新恢复专项审计](../audit/2026-09-13-chat-history-refresh-disappears.md) | 历史读取端点补统一会话 owner + 阶段授权；旧 API 410 标注(BUG-11 partial) | `authz` | | 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；当前 dev 因数据根 split-brain 再次不可读。Unit A 纠挂后标 incident-mitigated；`done` 等 Unit B 负例，不等 outbox | `docgov` | | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | -| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | planned | [调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md) | 当前仅有 PostgreSQL 文档快照、消息正文和运行态 CAS；缺独立 storage identity、终态 outbox、manifest/receipt、授权降级读取和 scratch 恢复工具 | `artifact-vault` | +| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | planned | [调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md) | 方案完成并经安全/可行性/对抗复核；尚未实现独立 storage identity、不可变 staging/outbox、项目作用域加密 CAS、manifest/receipt、授权降级读取和 scratch 恢复工具 | `artifact-vault` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | | 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus 为 sibling，不挡事故关闭 | `infra` | From 20d50ef9f1b54f5df32f0a0a7a47f1c3e807e2b2 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:25 +0000 Subject: [PATCH 06/13] docs(adr): record final artifact vault authority and fail-closed rules Lock the three-layer contract, no tenant_id, union-deny delete, and scratch-only recover before storage work. Co-authored-by: Cursor --- .doc/adr/2026-09-14-final-artifact-vault.md | 25 +++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 .doc/adr/2026-09-14-final-artifact-vault.md diff --git a/.doc/adr/2026-09-14-final-artifact-vault.md b/.doc/adr/2026-09-14-final-artifact-vault.md new file mode 100644 index 00000000..7fa993c8 --- /dev/null +++ b/.doc/adr/2026-09-14-final-artifact-vault.md @@ -0,0 +1,25 @@ +# ADR：最终成果保险库（FAV） + +## 状态 + +已接受（2026-09-14，落地计划 V0–V6） + +## 背景 + +运行库 PostgreSQL 与 workspace 投影会一起丢失或错挂。需要独立保存**已确认终态**的精确字节，但不能成为第三套完整业务真相源，也不能用目录扫描猜测“最终文件”。 + +## 决策 + +1. **三层权威**：PostgreSQL 管项目/权限/工作流/版本关系；workspace 是可清理投影；FAV 只对已归档 payload、manifest、来源证据和删除状态权威。 +2. **无 tenant_id**：安全边界沿用 `environment + project_members + role-admin`。未来组织层另立 ADR。 +3. **事件准入**：只接受方案准入表中的数据库事实。显式 publish/pin 不能绕过终态 allowlist。禁止扫描 workspace。 +4. **同事务冻结**：终态事务把精确 payload/SHA 写入 staging 并写 unpublished outbox；publisher 不得重读可变消息行。 +5. **union-deny 删除**：运行库删除账本或 vault tombstone 任一侧删除即拒绝读取/恢复。 +6. **恢复**：只进 scratch 库，默认 `recover-as-collection`。禁止 GET 自动写回或覆盖现有 head。 +7. **互补**：FAV、migration corpus、完整 DB backup/PITR 互相不能替代。不作为 409/split-brain 绕行。 + +## 后果 + +- 实现从 `feature/final-artifact-vault` 推进；mutex=`artifact-vault`。 +- Goose 从 **00034** 增加 FAV 表（00033 已被 task-run 治理占用）。 +- 降级读取默认关闭（`FAV_DEGRADED_READ`），Unit A 纠挂完成前不得对现网开启。 From 028b0eb0152e3af6846155b91e0400ac7b6d4605 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:25 +0000 Subject: [PATCH 07/13] feat(db): add FAV staging, outbox, and deletion ledger tables Use goose 00034 because 00033 already belongs to task-run governance. Co-authored-by: Cursor --- .../00034_2026-09-14-final-artifact-vault.sql | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 deploy/postgres/goose/00034_2026-09-14-final-artifact-vault.sql diff --git a/deploy/postgres/goose/00034_2026-09-14-final-artifact-vault.sql b/deploy/postgres/goose/00034_2026-09-14-final-artifact-vault.sql new file mode 100644 index 00000000..4064a813 --- /dev/null +++ b/deploy/postgres/goose/00034_2026-09-14-final-artifact-vault.sql @@ -0,0 +1,93 @@ +-- +goose Up +CREATE TABLE IF NOT EXISTS final_artifacts ( + artifact_uuid TEXT PRIMARY KEY, + idempotency_key TEXT NOT NULL UNIQUE, + artifact_type TEXT NOT NULL, + logical_resource_id TEXT NOT NULL, + environment TEXT NOT NULL, + project_id BIGINT, + project_external_key TEXT NOT NULL, + source_event_id TEXT NOT NULL, + source_event_kind TEXT NOT NULL DEFAULT '', + source_snapshot_id BIGINT, + source_message_id BIGINT, + source_run_id TEXT NOT NULL DEFAULT '', + payload_sha256 CHAR(64) NOT NULL, + manifest_sha256 CHAR(64), + vault_storage_identity TEXT, + terminal_state TEXT NOT NULL DEFAULT '', + policy_version TEXT NOT NULL DEFAULT '', + publication_status TEXT NOT NULL, + retention_class TEXT NOT NULL DEFAULT 'governance', + requested_at TIMESTAMPTZ NOT NULL DEFAULT now(), + verified_at TIMESTAMPTZ, + tombstoned_at TIMESTAMPTZ, + CONSTRAINT chk_fav_type CHECK (artifact_type IN ('agent_commit','sealed_version','approved_document','freeform_publish','assistant_reply')), + CONSTRAINT chk_fav_status CHECK (publication_status IN ('requested','publishing','verified','retryable_failed','tombstoned','purged','quarantine')) +); + +CREATE TABLE IF NOT EXISTS artifact_payload_staging ( + staging_uuid TEXT PRIMARY KEY, + project_id BIGINT, + source_event_id TEXT NOT NULL, + payload_sha256 CHAR(64) NOT NULL, + encrypted_payload BYTEA NOT NULL, + classification TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + releasable_at TIMESTAMPTZ, + UNIQUE (source_event_id, payload_sha256) +); + +CREATE TABLE IF NOT EXISTS artifact_blob_references ( + artifact_uuid TEXT NOT NULL REFERENCES final_artifacts(artifact_uuid), + project_scope TEXT NOT NULL, + payload_sha256 CHAR(64) NOT NULL, + envelope_key_ref TEXT NOT NULL DEFAULT '', + reference_status TEXT NOT NULL DEFAULT 'active', + retention_class TEXT NOT NULL DEFAULT 'governance', + legal_hold BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (artifact_uuid, payload_sha256) +); + +CREATE TABLE IF NOT EXISTS artifact_publication_attempts ( + artifact_uuid TEXT NOT NULL, + idempotency_key TEXT NOT NULL, + attempt_no INT NOT NULL, + error_code TEXT NOT NULL DEFAULT '', + receipt TEXT NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (artifact_uuid, attempt_no) +); + +CREATE TABLE IF NOT EXISTS artifact_deletion_ledger ( + artifact_uuid TEXT NOT NULL, + reason TEXT NOT NULL, + authority TEXT NOT NULL DEFAULT '', + effective_at TIMESTAMPTZ NOT NULL, + applied_at TIMESTAMPTZ, + audit_ref TEXT NOT NULL DEFAULT '' +); + +CREATE TABLE IF NOT EXISTS artifact_quarantine ( + logical_resource_id TEXT NOT NULL, + reason TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE IF NOT EXISTS artifact_backfill_exclusions ( + logical_resource_id TEXT PRIMARY KEY, + reason TEXT NOT NULL, + approved_by TEXT NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- +goose Down +DROP TABLE IF EXISTS artifact_backfill_exclusions; +DROP TABLE IF EXISTS artifact_quarantine; +DROP TABLE IF EXISTS artifact_deletion_ledger; +DROP TABLE IF EXISTS artifact_publication_attempts; +DROP TABLE IF EXISTS artifact_blob_references; +DROP TABLE IF EXISTS artifact_payload_staging; +DROP TABLE IF EXISTS final_artifacts; From ffb93dc55dbd0b326f6238a9315434ae373bc40b Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:25 +0000 Subject: [PATCH 08/13] feat(artifactvault): add eligibility, encrypted CAS, and idempotent publisher Keep vault roots off ./data/dev and reject unknown types or sensitive payloads before staging. Co-authored-by: Cursor --- internal/artifactvault/backfill.go | 55 +++ internal/artifactvault/backfill_test.go | 39 ++ .../artifactvault/commit-marker.schema.json | 27 + internal/artifactvault/coverage.go | 30 ++ internal/artifactvault/crypto.go | 53 ++ internal/artifactvault/eligibility.go | 107 ++++ internal/artifactvault/eligibility_test.go | 87 ++++ internal/artifactvault/enumerate.go | 211 ++++++++ internal/artifactvault/filesystem_store.go | 123 +++++ internal/artifactvault/ingest.go | 129 +++++ internal/artifactvault/ingest_test.go | 35 ++ internal/artifactvault/manifest.go | 100 ++++ internal/artifactvault/manifest.schema.json | 67 +++ internal/artifactvault/publisher.go | 103 ++++ internal/artifactvault/publisher_test.go | 150 ++++++ internal/artifactvault/recovery.go | 67 +++ internal/artifactvault/recovery_test.go | 27 + internal/artifactvault/registry.go | 467 ++++++++++++++++++ internal/artifactvault/tombstone.schema.json | 16 + internal/artifactvault/types.go | 208 ++++++++ scripts/artifact-vault-preflight.sh | 9 + 21 files changed, 2110 insertions(+) create mode 100644 internal/artifactvault/backfill.go create mode 100644 internal/artifactvault/backfill_test.go create mode 100644 internal/artifactvault/commit-marker.schema.json create mode 100644 internal/artifactvault/coverage.go create mode 100644 internal/artifactvault/crypto.go create mode 100644 internal/artifactvault/eligibility.go create mode 100644 internal/artifactvault/eligibility_test.go create mode 100644 internal/artifactvault/enumerate.go create mode 100644 internal/artifactvault/filesystem_store.go create mode 100644 internal/artifactvault/ingest.go create mode 100644 internal/artifactvault/ingest_test.go create mode 100644 internal/artifactvault/manifest.go create mode 100644 internal/artifactvault/manifest.schema.json create mode 100644 internal/artifactvault/publisher.go create mode 100644 internal/artifactvault/publisher_test.go create mode 100644 internal/artifactvault/recovery.go create mode 100644 internal/artifactvault/recovery_test.go create mode 100644 internal/artifactvault/registry.go create mode 100644 internal/artifactvault/tombstone.schema.json create mode 100644 internal/artifactvault/types.go create mode 100755 scripts/artifact-vault-preflight.sh diff --git a/internal/artifactvault/backfill.go b/internal/artifactvault/backfill.go new file mode 100644 index 00000000..ea4959d4 --- /dev/null +++ b/internal/artifactvault/backfill.go @@ -0,0 +1,55 @@ +package artifactvault + +import ( + "context" + "fmt" +) + +// TerminalRow is an authoritative DB terminal fact for backfill. Workspace paths are forbidden. +type TerminalRow struct { + Facts Facts + Request PublishRequest + MissingEvidence bool + Reason string +} + +type BackfillReport struct { + Created int + Reused int + Quarantine int + Excluded int +} + +func Backfill(ctx context.Context, reg Registry, rows []TerminalRow, exclude map[string]string) (BackfillReport, error) { + var out BackfillReport + for _, row := range rows { + if row.MissingEvidence || len(row.Request.Payload) == 0 { + _ = reg.Quarantine(ctx, nz(row.Reason, "missing_evidence"), row.Request.LogicalResourceID) + out.Quarantine++ + continue + } + if reason, ok := exclude[row.Request.LogicalResourceID]; ok && reason != "" { + out.Excluded++ + continue + } + before := reg.Coverage(ctx).Required + if _, _, err := reg.Request(ctx, row.Facts, row.Request); err != nil { + _ = reg.Quarantine(ctx, err.Error(), row.Request.LogicalResourceID) + out.Quarantine++ + continue + } + if reg.Coverage(ctx).Required == before { + out.Reused++ + continue + } + out.Created++ + } + return out, nil +} + +func ValidateBackfillRow(row TerminalRow) error { + if row.Request.SourceEventID == "" || row.Request.LogicalResourceID == "" { + return fmt.Errorf("artifactvault: backfill row missing source identity") + } + return nil +} diff --git a/internal/artifactvault/backfill_test.go b/internal/artifactvault/backfill_test.go new file mode 100644 index 00000000..ca75ceba --- /dev/null +++ b/internal/artifactvault/backfill_test.go @@ -0,0 +1,39 @@ +package artifactvault + +import ( + "context" + "testing" + "time" +) + +func TestBackfillIdempotentNoWorkspace(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + now := time.Now().UTC() + row := TerminalRow{ + Facts: Facts{ArtifactType: TypeSealedVersion, VersionStatus: "sealed", SealedAt: &now, HeadSnapshotID: 1, Classification: ClassificationOK, Payload: []byte("snap")}, + Request: PublishRequest{ArtifactType: TypeSealedVersion, Environment: "dev", ProjectExternalKey: "p", LogicalResourceID: "doc1", SourceEventKind: "seal", SourceEventID: "e1", Payload: []byte("snap"), TerminalState: "sealed"}, + } + r1, err := Backfill(ctx, reg, []TerminalRow{row, row}, nil) + if err != nil || r1.Created != 1 || r1.Reused != 1 { + t.Fatalf("%+v %v", r1, err) + } + r2, _ := Backfill(ctx, reg, []TerminalRow{row}, nil) + if r2.Created != 0 && r2.Reused == 0 { + t.Fatalf("rerun %+v", r2) + } + missing := row + missing.MissingEvidence = true + missing.Request.LogicalResourceID = "doc-missing" + r3, _ := Backfill(ctx, reg, []TerminalRow{missing}, nil) + if r3.Quarantine != 1 { + t.Fatalf("quarantine %+v", r3) + } + ex := row + ex.Request.LogicalResourceID = "doc-ex" + ex.Request.SourceEventID = "e-ex" + r4, _ := Backfill(ctx, reg, []TerminalRow{ex}, map[string]string{"doc-ex": "approved-old"}) + if r4.Excluded != 1 { + t.Fatalf("exclude %+v", r4) + } +} diff --git a/internal/artifactvault/commit-marker.schema.json b/internal/artifactvault/commit-marker.schema.json new file mode 100644 index 00000000..958b60fa --- /dev/null +++ b/internal/artifactvault/commit-marker.schema.json @@ -0,0 +1,27 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-commit-marker-v1.json", + "title": "FAV Commit Marker", + "type": "object", + "additionalProperties": false, + "required": [ + "artifact_uuid", + "manifest_sha256", + "member_shas", + "vault_storage_identity", + "committed_at", + "publisher_version" + ], + "properties": { + "artifact_uuid": { "type": "string", "minLength": 8 }, + "manifest_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "member_shas": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "pattern": "^[a-f0-9]{64}$" } + }, + "vault_storage_identity": { "type": "string", "minLength": 8 }, + "committed_at": { "type": "string" }, + "publisher_version": { "type": "string" } + } +} diff --git a/internal/artifactvault/coverage.go b/internal/artifactvault/coverage.go new file mode 100644 index 00000000..6f7e490f --- /dev/null +++ b/internal/artifactvault/coverage.go @@ -0,0 +1,30 @@ +package artifactvault + +import "fmt" + +func CoveragePASS(c Coverage) bool { + if c.Required <= 0 { + return false + } + return c.Verified == c.Required && c.Quarantine == 0 +} + +func CanPurge(legalHold bool, activeRefs int) error { + if legalHold { + return fmt.Errorf("artifactvault: legal hold") + } + if activeRefs > 0 { + return fmt.Errorf("artifactvault: remaining references") + } + return nil +} + +func PreflightDestroy(c Coverage, backupOK, corpusOK, vaultReady bool) error { + if !vaultReady || !CoveragePASS(c) { + return fmt.Errorf("artifactvault: coverage gate failed required=%d verified=%d quarantine=%d", c.Required, c.Verified, c.Quarantine) + } + if !backupOK || !corpusOK { + return fmt.Errorf("artifactvault: complementary backup/corpus not PASS") + } + return nil +} diff --git a/internal/artifactvault/crypto.go b/internal/artifactvault/crypto.go new file mode 100644 index 00000000..9b3f8d35 --- /dev/null +++ b/internal/artifactvault/crypto.go @@ -0,0 +1,53 @@ +package artifactvault + +import ( + "crypto/aes" + "crypto/cipher" + "crypto/rand" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" +) + +// ProjectKey derives a per-project envelope key. The master secret never enters manifests. +func ProjectKey(master []byte, env, projectExternalKey string) (key []byte, ref string, err error) { + if len(master) < 16 { + return nil, "", fmt.Errorf("artifactvault: vault master key too short") + } + sum := sha256.Sum256(append(append(master, []byte(env)...), []byte(projectExternalKey)...)) + ref = "proj:" + env + ":" + projectExternalKey + ":" + hex.EncodeToString(sum[:8]) + return sum[:], ref, nil +} + +func Encrypt(key, plaintext []byte) ([]byte, error) { + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + nonce := make([]byte, gcm.NonceSize()) + if _, err := io.ReadFull(rand.Reader, nonce); err != nil { + return nil, err + } + return gcm.Seal(nonce, nonce, plaintext, nil), nil +} + +func Decrypt(key, envelope []byte) ([]byte, error) { + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + ns := gcm.NonceSize() + if len(envelope) < ns { + return nil, fmt.Errorf("artifactvault: short envelope") + } + return gcm.Open(nil, envelope[:ns], envelope[ns:], nil) +} diff --git a/internal/artifactvault/eligibility.go b/internal/artifactvault/eligibility.go new file mode 100644 index 00000000..5e7b73af --- /dev/null +++ b/internal/artifactvault/eligibility.go @@ -0,0 +1,107 @@ +package artifactvault + +import ( + "errors" + "strings" +) + +var ( + ErrNotTerminal = errors.New("artifactvault: facts are not a terminal state") + ErrExplicitBypass = errors.New("artifactvault: explicit publish/pin cannot bypass terminal allowlist") + ErrSensitiveRejected = errors.New("artifactvault: sensitive or unclassified payload rejected") + ErrUnknownArtifactType = errors.New("artifactvault: unknown artifact type rejected") + ErrSnapshotInconsistent = errors.New("artifactvault: snapshot/event inconsistent") +) + +// EvaluateEligibility is the only domain gate. Unclassified types default to reject. +func EvaluateEligibility(f Facts) error { + if err := rejectSensitive(f.Classification, f.Payload); err != nil { + return err + } + switch f.ArtifactType { + case TypeAgentCommit: + if f.CandidateStatus != "committed" || f.OutputSnapshotID <= 0 { + return ErrNotTerminal + } + if f.ChangeKind != "" && (f.ChangeKind != "agent_commit" || f.ChangeToSnapshotID != f.OutputSnapshotID) { + return ErrSnapshotInconsistent + } + return nil + case TypeSealedVersion: + if f.VersionStatus != "sealed" || f.SealedAt == nil || f.HeadSnapshotID <= 0 { + return ErrNotTerminal + } + return nil + case TypeApprovedDocument: + if f.DocState != "approved" || f.ApprovedAt == nil || f.ReviewSnapshotID <= 0 { + return ErrNotTerminal + } + return nil + case TypeFreeformPublish: + if !f.ExplicitPublish || !f.HasImmutableSnapshot { + if f.ExplicitPublish && !f.HasImmutableSnapshot { + return ErrExplicitBypass + } + return ErrNotTerminal + } + return nil + case TypeAssistantReply: + if f.ChatTurnStatus != "completed" || f.CompletedAt == nil || f.AssistantMessageID <= 0 { + if f.ExplicitPin { + return ErrExplicitBypass + } + return ErrNotTerminal + } + switch f.ExportPolicy { + case "full": + return nil + case "copy_only": + if f.ExplicitPin && f.PinAfterCompleted { + return nil + } + if f.ExplicitPin { + return ErrExplicitBypass + } + return ErrNotTerminal + default: + return ErrNotTerminal + } + default: + return ErrUnknownArtifactType + } +} + +func rejectSensitive(class string, payload []byte) error { + if class == "" { + class = ClassifyPayload(payload) + } + switch class { + case ClassificationSecret, ClassificationRestricted, ClassificationFailed: + return ErrSensitiveRejected + case ClassificationOK: + return nil + default: + return ErrSensitiveRejected + } +} + +// ClassifyPayload is a fail-closed server-side gate. It never logs the body. +func ClassifyPayload(payload []byte) string { + text := strings.ToLower(string(payload)) + if text == "" { + return ClassificationFailed + } + needles := []string{ + "-----begin ", "private_key", "api_key=", "secret=", "authorization: bearer", + "password=", "access_token=", "aws_secret", + } + for _, n := range needles { + if strings.Contains(text, n) { + return ClassificationSecret + } + } + if strings.Contains(text, "restricted:") || strings.Contains(text, "classification_failed") { + return ClassificationRestricted + } + return ClassificationOK +} diff --git a/internal/artifactvault/eligibility_test.go b/internal/artifactvault/eligibility_test.go new file mode 100644 index 00000000..04121980 --- /dev/null +++ b/internal/artifactvault/eligibility_test.go @@ -0,0 +1,87 @@ +package artifactvault + +import ( + "testing" + "time" +) + +func TestEvaluateEligibility_Allowlist(t *testing.T) { + ok := Facts{ + ArtifactType: TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 9, + ChangeKind: "agent_commit", ChangeToSnapshotID: 9, Classification: ClassificationOK, Payload: []byte("doc"), + } + if err := EvaluateEligibility(ok); err != nil { + t.Fatalf("committed should pass: %v", err) + } + for _, status := range []string{"ready", "needs_confirmation", "rejected", "stale", "quality_failed"} { + bad := ok + bad.CandidateStatus = status + if err := EvaluateEligibility(bad); err == nil { + t.Fatalf("status %s must reject", status) + } + } +} + +func TestEvaluateEligibility_ApprovedRequiresReviewSnapshot(t *testing.T) { + now := time.Now().UTC() + f := Facts{ + ArtifactType: TypeApprovedDocument, DocState: "approved", ApprovedAt: &now, + Classification: ClassificationOK, Payload: []byte("approved"), + } + if err := EvaluateEligibility(f); err == nil { + t.Fatal("approved without review_snapshot_id must reject") + } + f.ReviewSnapshotID = 3 + if err := EvaluateEligibility(f); err != nil { + t.Fatalf("approved with snapshot should pass: %v", err) + } +} + +func TestEvaluateEligibility_TaskWithoutCompletedTurn(t *testing.T) { + f := Facts{ + ArtifactType: TypeAssistantReply, ChatTurnStatus: "", AssistantMessageID: 8, + ExportPolicy: "full", Classification: ClassificationOK, Payload: []byte("hello world from assistant"), + } + if err := EvaluateEligibility(f); err == nil { + t.Fatal("missing completed turn must reject") + } + if err := EvaluateEligibility(Facts{ + ArtifactType: TypeAssistantReply, ExplicitPin: true, ChatTurnStatus: "pending", + Classification: ClassificationOK, Payload: []byte("hello world from assistant"), + }); err != ErrExplicitBypass { + t.Fatalf("pin before completed: %v", err) + } +} + +func TestEvaluateEligibility_SensitiveRejectedBeforeStaging(t *testing.T) { + now := time.Now().UTC() + f := Facts{ + ArtifactType: TypeAssistantReply, ChatTurnStatus: "completed", CompletedAt: &now, + AssistantMessageID: 1, ExportPolicy: "full", Payload: []byte("api_key=sk-secret"), + } + if err := EvaluateEligibility(f); err != ErrSensitiveRejected { + t.Fatalf("secret payload: %v", err) + } +} + +func TestEvaluateEligibility_UnknownTypeRejected(t *testing.T) { + if err := EvaluateEligibility(Facts{ArtifactType: "workspace_scan", Classification: ClassificationOK, Payload: []byte("x")}); err != ErrUnknownArtifactType { + t.Fatalf("got %v", err) + } +} + +func TestIdempotencyKeyStable(t *testing.T) { + req := PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "p1", + LogicalResourceID: "doc_a", SourceEventKind: "agent_commit", SourceEventID: "ev1", + } + a := IdempotencyKey(req, "abc") + b := IdempotencyKey(req, "abc") + if a != b || a == "" { + t.Fatalf("key unstable %q %q", a, b) + } + req.SourceEventID = "ev2" + if IdempotencyKey(req, "abc") == a { + t.Fatal("new event must new key") + } +} diff --git a/internal/artifactvault/enumerate.go b/internal/artifactvault/enumerate.go new file mode 100644 index 00000000..058867dd --- /dev/null +++ b/internal/artifactvault/enumerate.go @@ -0,0 +1,211 @@ +package artifactvault + +import ( + "context" + "fmt" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// EnumerateTerminals lists only immutable ledger / completed-turn facts. Workspace paths are forbidden. +func EnumerateTerminals(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + if pool == nil { + return nil, fmt.Errorf("artifactvault: no database") + } + var rows []TerminalRow + approved, err := enumerateApproved(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, approved...) + commits, err := enumerateAgentCommits(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, commits...) + sealed, err := enumerateSealed(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, sealed...) + replies, err := enumerateAssistant(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, replies...) + return rows, nil +} + +func LoadExclusions(ctx context.Context, pool *pgxpool.Pool) (map[string]string, error) { + out := map[string]string{} + if pool == nil { + return out, nil + } + q, err := pool.Query(ctx, `SELECT logical_resource_id, reason FROM artifact_backfill_exclusions WHERE expires_at > now()`) + if err != nil { + return nil, err + } + defer q.Close() + for q.Next() { + var id, reason string + if err := q.Scan(&id, &reason); err != nil { + return nil, err + } + out[id] = reason + } + return out, q.Err() +} + +func enumerateApproved(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT d.doc_id, d.project_id, d.review_snapshot_id, d.approved_at, COALESCE(b.content_markdown,'') + FROM document_records d + LEFT JOIN document_snapshots s ON s.id = d.review_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE d.current_state = 'approved' AND d.approved_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var projectID, snapID int64 + var approvedAt *time.Time + var md string + if err := q.Scan(&docID, &projectID, &snapID, &approvedAt, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: ApprovedFacts("approved", approvedAt, snapID), + Request: PublishRequest{ + ArtifactType: TypeApprovedDocument, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: docID, + SourceEventKind: "approved", SourceEventID: fmt.Sprintf("approved:%s:%d", docID, snapID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "approved", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_review_snapshot" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateAgentCommits(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT e.doc_id, e.id, e.to_snapshot_id, COALESCE(r.project_id,0), COALESCE(b.content_markdown,'') + FROM document_change_events e + LEFT JOIN document_records r ON r.doc_id = e.doc_id + LEFT JOIN document_snapshots s ON s.id = e.to_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE e.change_kind = 'agent_commit'`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var eventID, snapID, projectID int64 + var md string + if err := q.Scan(&docID, &eventID, &snapID, &projectID, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: AgentCommitFacts(snapID), + Request: PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: docID, + SourceEventKind: "agent_commit", SourceEventID: fmt.Sprintf("ev:%d", eventID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "committed", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_commit_snapshot" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateSealed(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT v.doc_id, v.id, v.head_snapshot_id, v.sealed_at, COALESCE(r.project_id,0), COALESCE(b.content_markdown,'') + FROM document_versions v + LEFT JOIN document_records r ON r.doc_id = v.doc_id + LEFT JOIN document_snapshots s ON s.id = v.head_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE v.status = 'sealed' AND v.sealed_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var verID, snapID, projectID int64 + var sealedAt *time.Time + var md string + if err := q.Scan(&docID, &verID, &snapID, &sealedAt, &projectID, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: SealedFacts(sealedAt, snapID), + Request: PublishRequest{ + ArtifactType: TypeSealedVersion, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: fmt.Sprintf("%s:v%d", docID, verID), + SourceEventKind: "version_sealed", SourceEventID: fmt.Sprintf("ver:%d", verID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "sealed", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_sealed_head" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateAssistant(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT t.session_id, t.client_turn_id, t.assistant_message_id, t.completed_at, t.project_id, + COALESCE(m.content,''), COALESCE(m.export_policy,'none') + FROM chat_turns t + LEFT JOIN chat_messages m ON m.id = t.assistant_message_id + WHERE t.status = 'completed' AND t.completed_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var sessionID, clientTurnID, content, policy string + var msgID, projectID int64 + var completedAt *time.Time + if err := q.Scan(&sessionID, &clientTurnID, &msgID, &completedAt, &projectID, &content, &policy); err != nil { + return nil, err + } + logical := fmt.Sprintf("msg:%d", msgID) + row := TerminalRow{ + Facts: AssistantFacts("completed", completedAt, msgID, policy), + Request: PublishRequest{ + ArtifactType: TypeAssistantReply, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: logical, + SourceEventKind: "assistant_completed", SourceEventID: sessionID + ":" + clientTurnID, + SourceMessageID: msgID, Payload: []byte(content), TerminalState: "completed", + }, + } + if msgID <= 0 || content == "" { + row.MissingEvidence = true + row.Reason = "missing_completed_body" + } + out = append(out, row) + } + return out, q.Err() +} diff --git a/internal/artifactvault/filesystem_store.go b/internal/artifactvault/filesystem_store.go new file mode 100644 index 00000000..9ba946e5 --- /dev/null +++ b/internal/artifactvault/filesystem_store.go @@ -0,0 +1,123 @@ +package artifactvault + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + + "chat-agent-system/internal/fsutil" +) + +type FilesystemStore struct { + Root string + StorageIdentity string + MasterKey []byte +} + +func NewFilesystemStore(root, identity string, master []byte) (*FilesystemStore, error) { + if strings.TrimSpace(root) == "" || strings.Contains(root, "data/dev") && !strings.Contains(root, "vault") { + if strings.HasSuffix(filepath.Clean(root), filepath.Join("data", "dev")) { + return nil, fmt.Errorf("artifactvault: refuse vault root %s (must not be runtime data/dev)", root) + } + } + if identity == "" { + return nil, fmt.Errorf("artifactvault: storage identity required") + } + if err := os.MkdirAll(root, 0o700); err != nil { + return nil, err + } + sent := filepath.Join(root, ".vault-storage-identity.json") + if b, err := os.ReadFile(sent); err == nil { + var got map[string]string + _ = json.Unmarshal(b, &got) + if got["uuid"] != "" && got["uuid"] != identity { + return nil, fmt.Errorf("artifactvault: storage identity mismatch") + } + } else { + _ = fsutil.WriteFileAtomic(sent, []byte(`{"uuid":"`+identity+`"}`), 0o444) + } + return &FilesystemStore{Root: root, StorageIdentity: identity, MasterKey: master}, nil +} + +func (s *FilesystemStore) Identity() string { return s.StorageIdentity } + +func (s *FilesystemStore) blobPath(env, projectKey, sha string) string { + return filepath.Join(s.Root, "scopes", env, projectKey, "blobs", "sha256", sha[:2], sha[2:4], sha) +} + +func (s *FilesystemStore) PutBlob(env, projectKey, sha string, plain []byte) (keyRef string, err error) { + if SHA256Hex(plain) != sha { + return "", fmt.Errorf("artifactvault: blob sha mismatch") + } + key, ref, err := ProjectKey(s.MasterKey, env, projectKey) + if err != nil { + return "", err + } + envBytes, err := Encrypt(key, plain) + if err != nil { + return "", err + } + path := s.blobPath(env, projectKey, sha) + if _, err := os.Stat(path); err == nil { + return ref, nil + } + if err := fsutil.WriteFileAtomicExclusive(path, envBytes, 0o600); err != nil { + if strings.Contains(err.Error(), "refuse overwrite") { + return ref, nil + } + return "", err + } + return ref, nil +} + +func (s *FilesystemStore) GetBlob(env, projectKey, sha string) ([]byte, error) { + key, _, err := ProjectKey(s.MasterKey, env, projectKey) + if err != nil { + return nil, err + } + raw, err := os.ReadFile(s.blobPath(env, projectKey, sha)) + if err != nil { + return nil, err + } + plain, err := Decrypt(key, raw) + if err != nil { + return nil, err + } + if SHA256Hex(plain) != sha { + return nil, fmt.Errorf("artifactvault: decrypted sha mismatch") + } + return plain, nil +} + +func (s *FilesystemStore) PutManifest(artifactUUID, manifestSHA string, raw []byte) error { + path := filepath.Join(s.Root, "manifests", artifactUUID, manifestSHA+".json") + return fsutil.WriteFileAtomicExclusive(path, raw, 0o644) +} + +func (s *FilesystemStore) PutCommit(artifactUUID, manifestSHA string, raw []byte) error { + path := filepath.Join(s.Root, "commits", artifactUUID, manifestSHA+".json") + return fsutil.WriteFileAtomicExclusive(path, raw, 0o644) +} + +func (s *FilesystemStore) HasCommit(artifactUUID, manifestSHA string) bool { + _, err := os.Stat(filepath.Join(s.Root, "commits", artifactUUID, manifestSHA+".json")) + return err == nil +} + +func (s *FilesystemStore) PutTombstone(artifactUUID, eventUUID string, raw []byte) error { + path := filepath.Join(s.Root, "tombstones", artifactUUID, eventUUID+".json") + return fsutil.WriteFileAtomicExclusive(path, raw, 0o644) +} + +func (s *FilesystemStore) HasTombstone(artifactUUID string) bool { + dir := filepath.Join(s.Root, "tombstones", artifactUUID) + ents, err := os.ReadDir(dir) + return err == nil && len(ents) > 0 +} + +func (s *FilesystemStore) Ready() (identityOK, connected, lagOK bool) { + _, err := os.Stat(s.Root) + return s.StorageIdentity != "", err == nil, err == nil +} diff --git a/internal/artifactvault/ingest.go b/internal/artifactvault/ingest.go new file mode 100644 index 00000000..e228675c --- /dev/null +++ b/internal/artifactvault/ingest.go @@ -0,0 +1,129 @@ +package artifactvault + +import ( + "context" + "fmt" + "strconv" + "time" +) + +// Ingest is the single write path from domain terminals into staging/outbox. +type Ingest struct { + Env string + Reg Registry +} + +func ProjectKeyFromID(projectID int64) string { + if projectID <= 0 { + return "unscoped" + } + return strconv.FormatInt(projectID, 10) +} + +func (i *Ingest) Request(ctx context.Context, facts Facts, req PublishRequest) error { + if i == nil || i.Reg == nil { + return nil + } + if req.Environment == "" { + req.Environment = i.Env + } + if req.ProjectExternalKey == "" { + req.ProjectExternalKey = ProjectKeyFromID(req.ProjectID) + } + if req.PolicyVersion == "" { + req.PolicyVersion = PolicyVersion + } + if req.Classification == "" { + req.Classification = ClassifyPayload(req.Payload) + } + facts.Payload = req.Payload + facts.Classification = req.Classification + _, _, err := i.Reg.Request(ctx, facts, req) + return err +} + +func ApprovedFacts(docState string, approvedAt *time.Time, snapshotID int64) Facts { + return Facts{ + ArtifactType: TypeApprovedDocument, + DocState: docState, + ApprovedAt: approvedAt, + ReviewSnapshotID: snapshotID, + } +} + +func AgentCommitFacts(outputSnapshotID int64) Facts { + return Facts{ + ArtifactType: TypeAgentCommit, + CandidateStatus: "committed", + OutputSnapshotID: outputSnapshotID, + ChangeKind: "agent_commit", + ChangeToSnapshotID: outputSnapshotID, + } +} + +func SealedFacts(sealedAt *time.Time, headSnapshotID int64) Facts { + return Facts{ + ArtifactType: TypeSealedVersion, + VersionStatus: "sealed", + SealedAt: sealedAt, + HeadSnapshotID: headSnapshotID, + } +} + +func AssistantFacts(turnStatus string, completedAt *time.Time, msgID int64, policy string) Facts { + return Facts{ + ArtifactType: TypeAssistantReply, + ChatTurnStatus: turnStatus, + CompletedAt: completedAt, + AssistantMessageID: msgID, + ExportPolicy: policy, + } +} + +func FreeformFacts(explicit bool, hasSnapshot bool) Facts { + return Facts{ + ArtifactType: TypeFreeformPublish, + ExplicitPublish: explicit, + HasImmutableSnapshot: hasSnapshot, + } +} + +func (i *Ingest) AgentCommit(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) error { + return i.Request(ctx, AgentCommitFacts(snapshotID), PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID, SourceEventKind: "agent_commit", SourceEventID: eventID, + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "committed", + }) +} + +func (i *Ingest) SealedVersion(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, sealedAt *time.Time, payload []byte) error { + return i.Request(ctx, SealedFacts(sealedAt, snapshotID), PublishRequest{ + ArtifactType: TypeSealedVersion, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID + ":sealed", SourceEventKind: "version_sealed", SourceEventID: eventID, + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "sealed", + }) +} + +func (i *Ingest) Approved(ctx context.Context, projectID int64, docID string, snapshotID int64, approvedAt *time.Time, payload []byte) error { + return i.Request(ctx, ApprovedFacts("approved", approvedAt, snapshotID), PublishRequest{ + ArtifactType: TypeApprovedDocument, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID, SourceEventKind: "approved", SourceEventID: fmt.Sprintf("approved:%s:%d", docID, snapshotID), + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "approved", + }) +} + +func (i *Ingest) FreeformPublish(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) error { + return i.Request(ctx, FreeformFacts(true, snapshotID > 0 || len(payload) > 0), PublishRequest{ + ArtifactType: TypeFreeformPublish, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID, SourceEventKind: "freeform_publish", SourceEventID: eventID, + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "published", + }) +} + +func (i *Ingest) AssistantReply(ctx context.Context, projectID int64, logical, eventID string, msgID int64, runID, policy string, completedAt *time.Time, payload []byte) error { + return i.Request(ctx, AssistantFacts("completed", completedAt, msgID, policy), PublishRequest{ + ArtifactType: TypeAssistantReply, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: logical, SourceEventKind: "assistant_completed", SourceEventID: eventID, + SourceMessageID: msgID, SourceRunID: runID, Payload: payload, TerminalState: "completed", + }) +} diff --git a/internal/artifactvault/ingest_test.go b/internal/artifactvault/ingest_test.go new file mode 100644 index 00000000..9c51e006 --- /dev/null +++ b/internal/artifactvault/ingest_test.go @@ -0,0 +1,35 @@ +package artifactvault + +import ( + "context" + "testing" + "time" +) + +func TestIngestIdempotentAndRejectsNonTerminal(t *testing.T) { + ctx := context.Background() + ing := &Ingest{Env: "dev", Reg: NewMemoryRegistry()} + now := time.Now().UTC() + payload := []byte("# committed body") + if err := ing.AgentCommit(ctx, 9, "doc_a", "ev-1", 4, payload); err != nil { + t.Fatal(err) + } + if err := ing.AgentCommit(ctx, 9, "doc_a", "ev-1", 4, payload); err != nil { + t.Fatal(err) + } + if ing.Reg.Coverage(ctx).Required != 1 { + t.Fatalf("idempotent ingest required=%d", ing.Reg.Coverage(ctx).Required) + } + if err := ing.AssistantReply(ctx, 9, "msg:1", "s:t", 1, "run", "none", &now, []byte("hello world from assistant")); err == nil { + t.Fatal("none policy must reject") + } +} + +func TestCoverageGateFailClosed(t *testing.T) { + if CoveragePASS(Coverage{}) { + t.Fatal("empty coverage must fail") + } + if err := PreflightDestroy(Coverage{Required: 2, Verified: 1}, true, true, true); err == nil { + t.Fatal("incomplete coverage must fail destroy") + } +} diff --git a/internal/artifactvault/manifest.go b/internal/artifactvault/manifest.go new file mode 100644 index 00000000..366c0d6f --- /dev/null +++ b/internal/artifactvault/manifest.go @@ -0,0 +1,100 @@ +package artifactvault + +import ( + "bytes" + "encoding/json" + "fmt" + "strings" + "time" +) + +func BuildManifest(art Artifact, payloadSHA string, size int, keyRef string) Manifest { + var snap, msg *int64 + if art.SourceSnapshotID > 0 { + v := art.SourceSnapshotID + snap = &v + } + if art.SourceMessageID > 0 { + v := art.SourceMessageID + msg = &v + } + m := Manifest{ + ArtifactSchemaVersion: SchemaMajorMinor, + ArtifactUUID: art.UUID, + ArtifactType: art.ArtifactType, + LogicalResourceID: art.LogicalResourceID, + Environment: art.Environment, + ProjectID: art.ProjectID, + ProjectExternalKey: art.ProjectExternalKey, + SourceEventID: art.SourceEventID, + SourceEventKind: art.SourceEventKind, + SourceSnapshotID: snap, + SourceMessageID: msg, + SourceRunID: art.SourceRunID, + PayloadMembers: []PayloadMember{{ + Name: "payload", MediaType: "text/markdown", ByteSize: size, SHA256: payloadSHA, Required: true, + }}, + TerminalEvidence: TerminalEvidence{ + State: art.TerminalState, DecidedAt: time.Now().UTC().Format(time.RFC3339), PolicyVersion: nz(art.PolicyVersion, PolicyVersion), + }, + RetentionClass: art.RetentionClass, + EncryptionKeyRef: keyRef, + VaultStorageIdentity: art.VaultStorageIdentity, + PublisherVersion: PublisherVersion, + PublishedAt: time.Now().UTC().Format(time.RFC3339), + } + raw, _ := json.Marshal(canonicalManifest(m)) + m.ManifestSHA256 = SHA256Hex(raw) + return m +} + +func canonicalManifest(m Manifest) Manifest { + cp := m + cp.ManifestSHA256 = "" + return cp +} + +func ValidateManifest(m Manifest) error { + if !strings.HasPrefix(m.ArtifactSchemaVersion, "1.") { + return fmt.Errorf("artifactvault: unsupported manifest major %q", m.ArtifactSchemaVersion) + } + if m.ArtifactUUID == "" || m.LogicalResourceID == "" || m.ProjectExternalKey == "" || m.VaultStorageIdentity == "" { + return fmt.Errorf("artifactvault: manifest missing required identity fields") + } + if len(m.PayloadMembers) == 0 { + return fmt.Errorf("artifactvault: manifest has no payload members") + } + for _, mem := range m.PayloadMembers { + if len(mem.SHA256) != 64 { + return fmt.Errorf("artifactvault: invalid member sha") + } + } + got := SHA256Hex(mustJSON(canonicalManifest(m))) + if m.ManifestSHA256 != "" && m.ManifestSHA256 != got { + return fmt.Errorf("artifactvault: manifest sha mismatch") + } + return nil +} + +func ValidateCommitMarker(c CommitMarker) error { + if c.ArtifactUUID == "" || len(c.ManifestSHA256) != 64 || len(c.MemberSHAs) == 0 || c.VaultStorageIdentity == "" || c.CommittedAt == "" || c.PublisherVersion == "" { + return fmt.Errorf("artifactvault: incomplete commit marker") + } + return nil +} + +func mustJSON(v any) []byte { + b, err := json.Marshal(v) + if err != nil { + return nil + } + return b +} + +func EncodeJSON(v any) []byte { + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetEscapeHTML(false) + _ = enc.Encode(v) + return bytes.TrimSpace(buf.Bytes()) +} diff --git a/internal/artifactvault/manifest.schema.json b/internal/artifactvault/manifest.schema.json new file mode 100644 index 00000000..b1e0ebd8 --- /dev/null +++ b/internal/artifactvault/manifest.schema.json @@ -0,0 +1,67 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-manifest-v1.json", + "title": "FAV Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "artifact_schema_version", + "artifact_uuid", + "artifact_type", + "logical_resource_id", + "environment", + "project_external_key", + "source_event_id", + "payload_members", + "terminal_evidence", + "vault_storage_identity", + "manifest_sha256" + ], + "properties": { + "artifact_schema_version": { "type": "string", "pattern": "^1\\.[0-9]+$" }, + "artifact_uuid": { "type": "string", "minLength": 8 }, + "artifact_type": { "type": "string", "enum": ["agent_commit", "sealed_version", "approved_document", "freeform_publish", "assistant_reply"] }, + "logical_resource_id": { "type": "string", "minLength": 1 }, + "environment": { "type": "string", "enum": ["dev", "test", "main"] }, + "project_id": { "type": "integer" }, + "project_external_key": { "type": "string", "minLength": 1 }, + "source_event_id": { "type": "string", "minLength": 1 }, + "source_event_kind": { "type": "string" }, + "source_snapshot_id": { "type": ["integer", "null"] }, + "source_message_id": { "type": ["integer", "null"] }, + "source_run_id": { "type": "string" }, + "payload_members": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": ["name", "media_type", "byte_size", "sha256", "required"], + "properties": { + "name": { "type": "string" }, + "media_type": { "type": "string" }, + "byte_size": { "type": "integer", "minimum": 0 }, + "sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "required": { "type": "boolean" } + } + } + }, + "terminal_evidence": { + "type": "object", + "required": ["state", "decided_at", "policy_version"], + "properties": { + "state": { "type": "string" }, + "decided_at": { "type": "string" }, + "actor_ref": { "type": "string" }, + "policy_version": { "type": "string" } + } + }, + "lineage": { "type": "object" }, + "retention_class": { "type": "string" }, + "sensitivity": { "type": "string" }, + "encryption_key_ref": { "type": "string" }, + "vault_storage_identity": { "type": "string", "minLength": 8 }, + "publisher_version": { "type": "string" }, + "published_at": { "type": "string" }, + "manifest_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" } + } +} diff --git a/internal/artifactvault/publisher.go b/internal/artifactvault/publisher.go new file mode 100644 index 00000000..92683cec --- /dev/null +++ b/internal/artifactvault/publisher.go @@ -0,0 +1,103 @@ +package artifactvault + +import ( + "context" + "encoding/json" + "fmt" + "time" +) + +type Publisher struct { + Reg Registry + Store *FilesystemStore +} + +func (p *Publisher) Drain(ctx context.Context) error { + arts, err := p.Reg.LoadRequested(ctx) + if err != nil { + return err + } + var first error + for _, a := range arts { + if err := p.PublishOne(ctx, a); err != nil && first == nil { + first = err + } + } + return first +} + +func (p *Publisher) PublishOne(ctx context.Context, art Artifact) error { + if p.Store == nil { + return fmt.Errorf("artifactvault: store required") + } + if p.Reg.IsDenied(ctx, art.UUID) { + return fmt.Errorf("artifactvault: artifact denied") + } + _ = p.Reg.MarkPublishing(ctx, art.UUID) + st, err := p.Reg.LoadStaging(ctx, art.SourceEventID) + if err != nil { + _ = p.Reg.MarkFailed(ctx, art.UUID, "staging_missing") + return err + } + if SHA256Hex(st.Payload) != art.PayloadSHA256 || SHA256Hex(st.Payload) != st.PayloadSHA256 { + _ = p.Reg.MarkFailed(ctx, art.UUID, "sha_mismatch") + return fmt.Errorf("artifactvault: staging sha mismatch") + } + if ClassifyPayload(st.Payload) != ClassificationOK { + _ = p.Reg.MarkFailed(ctx, art.UUID, "sensitive") + return ErrSensitiveRejected + } + art.VaultStorageIdentity = p.Store.Identity() + keyRef, err := p.Store.PutBlob(art.Environment, art.ProjectExternalKey, art.PayloadSHA256, st.Payload) + if err != nil { + _ = p.Reg.MarkFailed(ctx, art.UUID, "blob") + return err + } + man := BuildManifest(art, art.PayloadSHA256, len(st.Payload), keyRef) + if err := ValidateManifest(man); err != nil { + _ = p.Reg.MarkFailed(ctx, art.UUID, "manifest") + return err + } + manRaw := EncodeJSON(man) + if err := p.Store.PutManifest(art.UUID, man.ManifestSHA256, manRaw); err != nil { + _ = p.Reg.MarkFailed(ctx, art.UUID, "manifest_write") + return err + } + marker := CommitMarker{ + ArtifactUUID: art.UUID, ManifestSHA256: man.ManifestSHA256, + MemberSHAs: []string{art.PayloadSHA256}, VaultStorageIdentity: p.Store.Identity(), + CommittedAt: time.Now().UTC().Format(time.RFC3339), PublisherVersion: PublisherVersion, + } + if err := ValidateCommitMarker(marker); err != nil { + return err + } + if err := p.Store.PutCommit(art.UUID, man.ManifestSHA256, EncodeJSON(marker)); err != nil { + _ = p.Reg.MarkFailed(ctx, art.UUID, "commit") + return err + } + _ = p.Reg.RecordReference(ctx, art.UUID, art.ProjectExternalKey, art.PayloadSHA256, keyRef) + return p.Reg.MarkVerified(ctx, art.UUID, man.ManifestSHA256) +} + +func (p *Publisher) DegradedPayload(ctx context.Context, env, logical, projectKey string) ([]byte, string, error) { + art, err := p.Reg.GetByLogical(ctx, env, logical) + if err != nil { + return nil, "", err + } + if p.Reg.IsDenied(ctx, art.UUID) || p.Store.HasTombstone(art.UUID) { + return nil, "", fmt.Errorf("artifactvault: union-deny") + } + if !p.Store.HasCommit(art.UUID, art.ManifestSHA256) && art.ManifestSHA256 != "" { + return nil, "", fmt.Errorf("artifactvault: unpublished") + } + b, err := p.Store.GetBlob(env, projectKey, art.PayloadSHA256) + if err != nil { + return nil, "", err + } + return b, art.PayloadSHA256, nil +} + +func MustJSON(v any) string { + b, _ := json.Marshal(v) + return string(b) +} diff --git a/internal/artifactvault/publisher_test.go b/internal/artifactvault/publisher_test.go new file mode 100644 index 00000000..2e05a3da --- /dev/null +++ b/internal/artifactvault/publisher_test.go @@ -0,0 +1,150 @@ +package artifactvault + +import ( + "context" + "os" + "path/filepath" + "testing" + "time" +) + +func testStore(t *testing.T) *FilesystemStore { + t.Helper() + dir := t.TempDir() + st, err := NewFilesystemStore(dir, "vault-id-test", []byte("0123456789abcdef0123456789abcdef")) + if err != nil { + t.Fatal(err) + } + return st +} + +func TestPublisherIdempotentAndCrashSafe(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + facts := Facts{ + ArtifactType: TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 4, + ChangeKind: "agent_commit", ChangeToSnapshotID: 4, Classification: ClassificationOK, + Payload: []byte("# final\n"), + } + req := PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "proj-a", + LogicalResourceID: "doc_a", SourceEventKind: "agent_commit", SourceEventID: "ev-1", + Payload: []byte("# final\n"), TerminalState: "committed", + } + art1, _, err := reg.Request(ctx, facts, req) + if err != nil { + t.Fatal(err) + } + art2, _, err := reg.Request(ctx, facts, req) + if err != nil || art1.UUID != art2.UUID { + t.Fatalf("idempotent request: %v %s %s", err, art1.UUID, art2.UUID) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + cov := reg.Coverage(ctx) + if cov.Verified != 1 { + t.Fatalf("verified=%d", cov.Verified) + } + b, sha, err := pub.DegradedPayload(ctx, "dev", "doc_a", "proj-a") + if err != nil || string(b) != "# final\n" || sha == "" { + t.Fatalf("degraded: %v %q", err, b) + } +} + +func TestPublisherRejectsBadSHAAndCrossProjectKeys(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + facts := Facts{ + ArtifactType: TypeSealedVersion, VersionStatus: "sealed", SealedAt: ptrTime(), + HeadSnapshotID: 2, Classification: ClassificationOK, Payload: []byte("same-bytes"), + } + aReq := PublishRequest{ArtifactType: TypeSealedVersion, Environment: "dev", ProjectExternalKey: "p1", LogicalResourceID: "d1", SourceEventKind: "seal", SourceEventID: "s1", Payload: []byte("same-bytes"), TerminalState: "sealed"} + bReq := aReq + bReq.ProjectExternalKey = "p2" + bReq.LogicalResourceID = "d2" + bReq.SourceEventID = "s2" + if _, _, err := reg.Request(ctx, facts, aReq); err != nil { + t.Fatal(err) + } + if _, _, err := reg.Request(ctx, facts, bReq); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + if reg.ActiveRefs(ctx, "p1", SHA256Hex([]byte("same-bytes"))) != 1 { + t.Fatal("p1 refs") + } + if reg.ActiveRefs(ctx, "p2", SHA256Hex([]byte("same-bytes"))) != 1 { + t.Fatal("p2 refs") + } + _ = reg.Tombstone(ctx, "unused", "test", "test") +} + +func TestFilesystemRefusesRuntimeDataDev(t *testing.T) { + root := filepath.Join(t.TempDir(), "data", "dev") + _ = os.MkdirAll(root, 0o755) + if _, err := NewFilesystemStore(root, "id", []byte("0123456789abcdef0123456789abcdef")); err == nil { + t.Fatal("must refuse data/dev") + } +} + +func TestTombstoneUnionDeny(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + now := time.Now().UTC() + facts := Facts{ArtifactType: TypeApprovedDocument, DocState: "approved", ApprovedAt: &now, ReviewSnapshotID: 8, Classification: ClassificationOK, Payload: []byte("ok body")} + req := PublishRequest{ArtifactType: TypeApprovedDocument, Environment: "dev", ProjectExternalKey: "p", LogicalResourceID: "doc", SourceEventKind: "approved", SourceEventID: "ap1", Payload: []byte("ok body"), TerminalState: "approved"} + art, _, err := reg.Request(ctx, facts, req) + if err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + _ = reg.Tombstone(ctx, art.UUID, "legal", "admin") + if _, _, err := pub.DegradedPayload(ctx, "dev", "doc", "p"); err == nil { + t.Fatal("tombstone must deny read") + } +} + +func TestLastReferencePurgeAndLegalHold(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + now := time.Now().UTC() + payload := []byte("shared-bytes") + facts := Facts{ArtifactType: TypeSealedVersion, VersionStatus: "sealed", SealedAt: &now, HeadSnapshotID: 2, Classification: ClassificationOK, Payload: payload} + req := PublishRequest{ArtifactType: TypeSealedVersion, Environment: "dev", ProjectExternalKey: "p1", LogicalResourceID: "d1", SourceEventKind: "seal", SourceEventID: "s1", Payload: payload, TerminalState: "sealed"} + if _, _, err := reg.Request(ctx, facts, req); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + if reg.ActiveRefs(ctx, "p1", SHA256Hex(payload)) != 1 { + t.Fatal("expected one active ref") + } + if err := CanPurge(false, 1); err == nil { + t.Fatal("must not purge while refs remain") + } + if err := CanPurge(true, 0); err == nil { + t.Fatal("legal hold blocks purge") + } + if err := CanPurge(false, 0); err != nil { + t.Fatal(err) + } +} + +func ptrTime() *time.Time { t := time.Now().UTC(); return &t } diff --git a/internal/artifactvault/recovery.go b/internal/artifactvault/recovery.go new file mode 100644 index 00000000..bec72832 --- /dev/null +++ b/internal/artifactvault/recovery.go @@ -0,0 +1,67 @@ +package artifactvault + +import ( + "fmt" + "strings" +) + +type RecoverMode string + +const ( + RecoverAsCollection RecoverMode = "recover-as-collection" + RecoverRelink RecoverMode = "relink" +) + +type RecoverPlan struct { + Mode RecoverMode + TargetIdentity string + SourceIdentity string + PlanHash string + Conflicts []string + Actor string + HasAdmin bool + HasProjectAuth bool + DependenciesOK bool + ScratchOnly bool +} + +func ValidateRecoverTarget(dsn, identity string) error { + low := strings.ToLower(dsn + " " + identity) + for _, ban := range []string{"agent_system_dev", "agent_system_test", "agent_system_main", "env=dev", "env=test", "env=main"} { + if strings.Contains(low, ban) && !strings.Contains(low, "scratch") { + return fmt.Errorf("artifactvault: refuse recover into operational database") + } + } + if !strings.Contains(low, "scratch") { + return fmt.Errorf("artifactvault: recover target must be scratch") + } + return nil +} + +func EvaluateRecoverPlan(p RecoverPlan, denied bool) error { + if err := ValidateRecoverTarget(p.TargetIdentity, p.SourceIdentity); err != nil { + return err + } + if !p.ScratchOnly { + return fmt.Errorf("artifactvault: scratch-only required") + } + if !p.HasAdmin { + return fmt.Errorf("artifactvault: operator lacks role-admin") + } + if denied { + return fmt.Errorf("artifactvault: union-deny deletion ledger") + } + if len(p.Conflicts) > 0 && p.Mode == RecoverRelink { + return fmt.Errorf("artifactvault: relink cannot proceed with conflicts") + } + if p.Mode == RecoverRelink && (!p.HasProjectAuth || !p.DependenciesOK) { + return fmt.Errorf("artifactvault: relink authorization or dependency gap") + } + if p.Mode == "" { + p.Mode = RecoverAsCollection + } + if p.PlanHash == "" { + return fmt.Errorf("artifactvault: plan hash required") + } + return nil +} diff --git a/internal/artifactvault/recovery_test.go b/internal/artifactvault/recovery_test.go new file mode 100644 index 00000000..c4152ba3 --- /dev/null +++ b/internal/artifactvault/recovery_test.go @@ -0,0 +1,27 @@ +package artifactvault + +import "testing" + +func TestRecoverRejectsOperationalDSN(t *testing.T) { + if err := ValidateRecoverTarget("postgres://x/agent_system_dev", "dev"); err == nil { + t.Fatal("dev dsn") + } + if err := ValidateRecoverTarget("postgres://x/fav_scratch", "scratch-1"); err != nil { + t.Fatal(err) + } +} + +func TestRecoverPlanGates(t *testing.T) { + p := RecoverPlan{Mode: RecoverAsCollection, TargetIdentity: "scratch-db", SourceIdentity: "vault", PlanHash: "abc", HasAdmin: true, ScratchOnly: true} + if err := EvaluateRecoverPlan(p, false); err != nil { + t.Fatal(err) + } + if err := EvaluateRecoverPlan(p, true); err == nil { + t.Fatal("denied") + } + p.Mode = RecoverRelink + p.Conflicts = []string{"uuid_collision"} + if err := EvaluateRecoverPlan(p, false); err == nil { + t.Fatal("relink+conflict") + } +} diff --git a/internal/artifactvault/registry.go b/internal/artifactvault/registry.go new file mode 100644 index 00000000..37ee6193 --- /dev/null +++ b/internal/artifactvault/registry.go @@ -0,0 +1,467 @@ +package artifactvault + +import ( + "context" + "fmt" + "sync" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +type Registry interface { + Request(ctx context.Context, facts Facts, req PublishRequest) (*Artifact, *Staging, error) + LoadRequested(ctx context.Context) ([]Artifact, error) + LoadStaging(ctx context.Context, sourceEventID string) (*Staging, error) + MarkPublishing(ctx context.Context, uuid string) error + MarkVerified(ctx context.Context, uuid, manifestSHA string) error + MarkFailed(ctx context.Context, uuid, code string) error + GetByLogical(ctx context.Context, env, logical string) (*Artifact, error) + Tombstone(ctx context.Context, uuid, reason, authority string) error + IsDenied(ctx context.Context, uuid string) bool + RecordReference(ctx context.Context, uuid, projectKey, sha, keyRef string) error + ActiveRefs(ctx context.Context, projectKey, sha string) int + Quarantine(ctx context.Context, reason string, logical string) error + QuarantineCount(ctx context.Context) int + Coverage(ctx context.Context) Coverage +} + +type Coverage struct { + Required int + Verified int + Quarantine int + Excluded int +} + +type MemoryRegistry struct { + mu sync.Mutex + arts map[string]*Artifact + byKey map[string]string + staging map[string]*Staging + denied map[string]bool + refs map[string]int + quarantine int + required int +} + +func NewMemoryRegistry() *MemoryRegistry { + return &MemoryRegistry{ + arts: map[string]*Artifact{}, byKey: map[string]string{}, staging: map[string]*Staging{}, + denied: map[string]bool{}, refs: map[string]int{}, + } +} + +func (m *MemoryRegistry) Request(_ context.Context, facts Facts, req PublishRequest) (*Artifact, *Staging, error) { + if err := EvaluateEligibility(facts); err != nil { + return nil, nil, err + } + if err := ValidateEnv(req.Environment); err != nil { + return nil, nil, err + } + sha := SHA256Hex(req.Payload) + key := IdempotencyKey(req, sha) + m.mu.Lock() + defer m.mu.Unlock() + if id, ok := m.byKey[key]; ok { + return m.arts[id], m.staging[req.SourceEventID], nil + } + uuid := "art-" + sha[:16] + "-" + fmt.Sprintf("%d", len(m.arts)+1) + art := &Artifact{ + UUID: uuid, IdempotencyKey: key, ArtifactType: req.ArtifactType, + LogicalResourceID: req.LogicalResourceID, Environment: req.Environment, + ProjectID: req.ProjectID, ProjectExternalKey: req.ProjectExternalKey, + SourceEventID: req.SourceEventID, SourceEventKind: req.SourceEventKind, + SourceSnapshotID: req.SourceSnapshotID, SourceMessageID: req.SourceMessageID, + SourceRunID: req.SourceRunID, PayloadSHA256: sha, TerminalState: req.TerminalState, + PolicyVersion: nz(req.PolicyVersion, PolicyVersion), PublicationStatus: StatusRequested, + RetentionClass: nz(req.RetentionClass, "governance"), RequestedAt: time.Now().UTC(), + } + st := &Staging{ + UUID: "stg-" + uuid, ProjectID: req.ProjectID, SourceEventID: req.SourceEventID, + PayloadSHA256: sha, Payload: append([]byte(nil), req.Payload...), Classification: ClassificationOK, + CreatedAt: time.Now().UTC(), + } + m.arts[uuid] = art + m.byKey[key] = uuid + m.staging[req.SourceEventID] = st + m.required++ + return art, st, nil +} + +func (m *MemoryRegistry) LoadRequested(_ context.Context) ([]Artifact, error) { + m.mu.Lock() + defer m.mu.Unlock() + out := make([]Artifact, 0) + for _, a := range m.arts { + if a.PublicationStatus == StatusRequested || a.PublicationStatus == StatusRetryableFailed { + out = append(out, *a) + } + } + return out, nil +} + +func (m *MemoryRegistry) LoadStaging(_ context.Context, sourceEventID string) (*Staging, error) { + m.mu.Lock() + defer m.mu.Unlock() + st := m.staging[sourceEventID] + if st == nil { + return nil, fmt.Errorf("artifactvault: staging missing") + } + cp := *st + cp.Payload = append([]byte(nil), st.Payload...) + return &cp, nil +} + +func (m *MemoryRegistry) MarkPublishing(_ context.Context, uuid string) error { + m.mu.Lock() + defer m.mu.Unlock() + if a := m.arts[uuid]; a != nil { + a.PublicationStatus = StatusPublishing + } + return nil +} + +func (m *MemoryRegistry) MarkVerified(_ context.Context, uuid, manifestSHA string) error { + m.mu.Lock() + defer m.mu.Unlock() + if a := m.arts[uuid]; a != nil { + now := time.Now().UTC() + a.PublicationStatus = StatusVerified + a.ManifestSHA256 = manifestSHA + a.VerifiedAt = &now + } + return nil +} + +func (m *MemoryRegistry) MarkFailed(_ context.Context, uuid, _ string) error { + m.mu.Lock() + defer m.mu.Unlock() + if a := m.arts[uuid]; a != nil { + a.PublicationStatus = StatusRetryableFailed + } + return nil +} + +func (m *MemoryRegistry) GetByLogical(_ context.Context, env, logical string) (*Artifact, error) { + m.mu.Lock() + defer m.mu.Unlock() + for _, a := range m.arts { + if a.Environment == env && a.LogicalResourceID == logical && a.PublicationStatus == StatusVerified && !m.denied[a.UUID] { + cp := *a + return &cp, nil + } + } + return nil, fmt.Errorf("artifactvault: not found") +} + +func (m *MemoryRegistry) Tombstone(_ context.Context, uuid, _, _ string) error { + m.mu.Lock() + defer m.mu.Unlock() + m.denied[uuid] = true + if a := m.arts[uuid]; a != nil { + now := time.Now().UTC() + a.PublicationStatus = StatusTombstoned + a.TombstonedAt = &now + } + return nil +} + +func (m *MemoryRegistry) IsDenied(_ context.Context, uuid string) bool { + m.mu.Lock() + defer m.mu.Unlock() + return m.denied[uuid] +} + +func (m *MemoryRegistry) RecordReference(_ context.Context, _, projectKey, sha, _ string) error { + m.mu.Lock() + defer m.mu.Unlock() + m.refs[projectKey+"|"+sha]++ + return nil +} + +func (m *MemoryRegistry) ActiveRefs(_ context.Context, projectKey, sha string) int { + m.mu.Lock() + defer m.mu.Unlock() + return m.refs[projectKey+"|"+sha] +} + +func (m *MemoryRegistry) Quarantine(_ context.Context, _, _ string) error { + m.mu.Lock() + defer m.mu.Unlock() + m.quarantine++ + return nil +} + +func (m *MemoryRegistry) QuarantineCount(_ context.Context) int { + m.mu.Lock() + defer m.mu.Unlock() + return m.quarantine +} + +func (m *MemoryRegistry) Coverage(_ context.Context) Coverage { + m.mu.Lock() + defer m.mu.Unlock() + v := 0 + for _, a := range m.arts { + if a.PublicationStatus == StatusVerified { + v++ + } + } + return Coverage{Required: m.required, Verified: v, Quarantine: m.quarantine} +} + +// PGRegistry persists FAV rows. Request uses a dedicated transaction unless WithTx is used. +type PGRegistry struct { + pool *pgxpool.Pool +} + +func NewPGRegistry(pool *pgxpool.Pool) *PGRegistry { return &PGRegistry{pool: pool} } + +func (p *PGRegistry) Request(ctx context.Context, facts Facts, req PublishRequest) (*Artifact, *Staging, error) { + if p == nil || p.pool == nil { + return nil, nil, fmt.Errorf("artifactvault: no database") + } + tx, err := p.pool.BeginTx(ctx, pgx.TxOptions{}) + if err != nil { + return nil, nil, err + } + defer func() { _ = tx.Rollback(ctx) }() + art, st, err := requestTX(ctx, tx, facts, req) + if err != nil { + return nil, nil, err + } + if err := tx.Commit(ctx); err != nil { + return nil, nil, err + } + return art, st, nil +} + +func requestTX(ctx context.Context, tx pgx.Tx, facts Facts, req PublishRequest) (*Artifact, *Staging, error) { + if err := EvaluateEligibility(facts); err != nil { + return nil, nil, err + } + sha := SHA256Hex(req.Payload) + key := IdempotencyKey(req, sha) + var existing string + err := tx.QueryRow(ctx, `SELECT artifact_uuid FROM final_artifacts WHERE idempotency_key=$1`, key).Scan(&existing) + if err == nil && existing != "" { + st := &Staging{SourceEventID: req.SourceEventID, PayloadSHA256: sha, Payload: req.Payload} + return &Artifact{UUID: existing, IdempotencyKey: key, PublicationStatus: StatusRequested}, st, nil + } + uuid := "art-" + sha[:16] + _, err = tx.Exec(ctx, ` + INSERT INTO artifact_payload_staging (staging_uuid, project_id, source_event_id, payload_sha256, encrypted_payload, classification) + VALUES ($1,$2,$3,$4,$5,$6) + ON CONFLICT (source_event_id, payload_sha256) DO NOTHING + `, "stg-"+uuid, req.ProjectID, req.SourceEventID, sha, req.Payload, ClassificationOK) + if err != nil { + return nil, nil, err + } + _, err = tx.Exec(ctx, ` + INSERT INTO final_artifacts ( + artifact_uuid, idempotency_key, artifact_type, logical_resource_id, environment, project_id, + project_external_key, source_event_id, source_event_kind, source_snapshot_id, source_message_id, + source_run_id, payload_sha256, terminal_state, policy_version, publication_status, retention_class + ) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17) + ON CONFLICT (idempotency_key) DO NOTHING + `, uuid, key, req.ArtifactType, req.LogicalResourceID, req.Environment, req.ProjectID, + req.ProjectExternalKey, req.SourceEventID, req.SourceEventKind, nullInt(req.SourceSnapshotID), + nullInt(req.SourceMessageID), req.SourceRunID, sha, req.TerminalState, nz(req.PolicyVersion, PolicyVersion), + StatusRequested, nz(req.RetentionClass, "governance")) + if err != nil { + return nil, nil, err + } + return &Artifact{UUID: uuid, IdempotencyKey: key, ArtifactType: req.ArtifactType, LogicalResourceID: req.LogicalResourceID, + Environment: req.Environment, ProjectExternalKey: req.ProjectExternalKey, SourceEventID: req.SourceEventID, + PayloadSHA256: sha, PublicationStatus: StatusRequested, TerminalState: req.TerminalState}, &Staging{Payload: req.Payload, PayloadSHA256: sha, SourceEventID: req.SourceEventID}, nil +} + +func nullInt(v int64) any { + if v == 0 { + return nil + } + return v +} + +func (p *PGRegistry) LoadRequested(ctx context.Context) ([]Artifact, error) { + if p == nil || p.pool == nil { + return nil, fmt.Errorf("artifactvault: no database") + } + rows, err := p.pool.Query(ctx, ` + SELECT artifact_uuid, idempotency_key, artifact_type, logical_resource_id, environment, + COALESCE(project_id,0), project_external_key, source_event_id, source_event_kind, + COALESCE(source_snapshot_id,0), COALESCE(source_message_id,0), source_run_id, + payload_sha256, COALESCE(manifest_sha256,''), COALESCE(vault_storage_identity,''), + terminal_state, policy_version, publication_status, retention_class, requested_at + FROM final_artifacts + WHERE publication_status IN ('requested','retryable_failed') + ORDER BY requested_at ASC`) + if err != nil { + return nil, err + } + defer rows.Close() + out := make([]Artifact, 0) + for rows.Next() { + var a Artifact + if err := rows.Scan(&a.UUID, &a.IdempotencyKey, &a.ArtifactType, &a.LogicalResourceID, &a.Environment, + &a.ProjectID, &a.ProjectExternalKey, &a.SourceEventID, &a.SourceEventKind, + &a.SourceSnapshotID, &a.SourceMessageID, &a.SourceRunID, &a.PayloadSHA256, &a.ManifestSHA256, + &a.VaultStorageIdentity, &a.TerminalState, &a.PolicyVersion, &a.PublicationStatus, &a.RetentionClass, &a.RequestedAt); err != nil { + return nil, err + } + out = append(out, a) + } + return out, rows.Err() +} + +func (p *PGRegistry) LoadStaging(ctx context.Context, sourceEventID string) (*Staging, error) { + if p == nil || p.pool == nil { + return nil, fmt.Errorf("artifactvault: no database") + } + var st Staging + err := p.pool.QueryRow(ctx, ` + SELECT staging_uuid, COALESCE(project_id,0), source_event_id, payload_sha256, encrypted_payload, classification, created_at + FROM artifact_payload_staging WHERE source_event_id=$1 ORDER BY created_at DESC LIMIT 1`, sourceEventID). + Scan(&st.UUID, &st.ProjectID, &st.SourceEventID, &st.PayloadSHA256, &st.Payload, &st.Classification, &st.CreatedAt) + if err != nil { + return nil, err + } + return &st, nil +} + +func (p *PGRegistry) MarkPublishing(ctx context.Context, uuid string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, `UPDATE final_artifacts SET publication_status=$2 WHERE artifact_uuid=$1`, uuid, StatusPublishing) + if err != nil { + return err + } + _, err = p.pool.Exec(ctx, ` + INSERT INTO artifact_publication_attempts (artifact_uuid, idempotency_key, attempt_no) + SELECT artifact_uuid, idempotency_key, COALESCE((SELECT MAX(attempt_no) FROM artifact_publication_attempts a WHERE a.artifact_uuid=$1),0)+1 + FROM final_artifacts WHERE artifact_uuid=$1`, uuid) + return err +} + +func (p *PGRegistry) MarkVerified(ctx context.Context, uuid, manifestSHA string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, `UPDATE final_artifacts SET publication_status=$2, manifest_sha256=$3, verified_at=now() WHERE artifact_uuid=$1`, uuid, StatusVerified, manifestSHA) + return err +} + +func (p *PGRegistry) MarkFailed(ctx context.Context, uuid, code string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, `UPDATE final_artifacts SET publication_status=$2 WHERE artifact_uuid=$1`, uuid, StatusRetryableFailed) + if err != nil { + return err + } + _, _ = p.pool.Exec(ctx, ` + UPDATE artifact_publication_attempts SET error_code=$2 + WHERE artifact_uuid=$1 AND attempt_no=(SELECT MAX(attempt_no) FROM artifact_publication_attempts WHERE artifact_uuid=$1)`, uuid, code) + return nil +} + +func (p *PGRegistry) GetByLogical(ctx context.Context, env, logical string) (*Artifact, error) { + if p == nil || p.pool == nil { + return nil, fmt.Errorf("artifactvault: no database") + } + var a Artifact + err := p.pool.QueryRow(ctx, ` + SELECT artifact_uuid, idempotency_key, artifact_type, logical_resource_id, environment, + COALESCE(project_id,0), project_external_key, source_event_id, source_event_kind, + COALESCE(source_snapshot_id,0), COALESCE(source_message_id,0), source_run_id, + payload_sha256, COALESCE(manifest_sha256,''), COALESCE(vault_storage_identity,''), + terminal_state, policy_version, publication_status + FROM final_artifacts + WHERE environment=$1 AND logical_resource_id=$2 AND publication_status='verified' + ORDER BY verified_at DESC NULLS LAST LIMIT 1`, env, logical). + Scan(&a.UUID, &a.IdempotencyKey, &a.ArtifactType, &a.LogicalResourceID, &a.Environment, + &a.ProjectID, &a.ProjectExternalKey, &a.SourceEventID, &a.SourceEventKind, + &a.SourceSnapshotID, &a.SourceMessageID, &a.SourceRunID, &a.PayloadSHA256, &a.ManifestSHA256, + &a.VaultStorageIdentity, &a.TerminalState, &a.PolicyVersion, &a.PublicationStatus) + if err != nil { + return nil, err + } + return &a, nil +} + +func (p *PGRegistry) Tombstone(ctx context.Context, uuid, reason, authority string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, `UPDATE final_artifacts SET publication_status=$2, tombstoned_at=now() WHERE artifact_uuid=$1`, uuid, StatusTombstoned) + if err != nil { + return err + } + _, err = p.pool.Exec(ctx, `INSERT INTO artifact_deletion_ledger (artifact_uuid, reason, authority, effective_at) VALUES ($1,$2,$3,now())`, uuid, reason, authority) + return err +} + +func (p *PGRegistry) IsDenied(ctx context.Context, uuid string) bool { + if p == nil || p.pool == nil { + return false + } + var n int + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM artifact_deletion_ledger WHERE artifact_uuid=$1`, uuid).Scan(&n) + return n > 0 +} + +func (p *PGRegistry) RecordReference(ctx context.Context, uuid, projectKey, sha, keyRef string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, ` + INSERT INTO artifact_blob_references (artifact_uuid, project_scope, payload_sha256, envelope_key_ref, reference_status) + VALUES ($1,$2,$3,$4,'active') + ON CONFLICT (artifact_uuid, payload_sha256) DO NOTHING`, uuid, projectKey, sha, keyRef) + return err +} + +func (p *PGRegistry) ActiveRefs(ctx context.Context, projectKey, sha string) int { + if p == nil || p.pool == nil { + return 0 + } + var n int + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM artifact_blob_references WHERE project_scope=$1 AND payload_sha256=$2 AND reference_status='active'`, projectKey, sha).Scan(&n) + return n +} + +func (p *PGRegistry) Quarantine(ctx context.Context, reason, logical string) error { + if p == nil || p.pool == nil { + return nil + } + _, err := p.pool.Exec(ctx, `INSERT INTO artifact_quarantine (logical_resource_id, reason) VALUES ($1,$2)`, logical, reason) + return err +} + +func (p *PGRegistry) QuarantineCount(ctx context.Context) int { + if p == nil || p.pool == nil { + return 0 + } + var n int + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM artifact_quarantine`).Scan(&n) + return n +} + +func (p *PGRegistry) Coverage(ctx context.Context) Coverage { + var c Coverage + if p == nil || p.pool == nil { + return c + } + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM final_artifacts`).Scan(&c.Required) + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM final_artifacts WHERE publication_status='verified'`).Scan(&c.Verified) + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM artifact_quarantine`).Scan(&c.Quarantine) + _ = p.pool.QueryRow(ctx, `SELECT COUNT(*) FROM artifact_backfill_exclusions WHERE expires_at > now()`).Scan(&c.Excluded) + return c +} + +// RequestWithTx writes staging + unpublished outbox inside an existing domain transaction. +func RequestWithTx(ctx context.Context, tx pgx.Tx, facts Facts, req PublishRequest) (*Artifact, *Staging, error) { + return requestTX(ctx, tx, facts, req) +} diff --git a/internal/artifactvault/tombstone.schema.json b/internal/artifactvault/tombstone.schema.json new file mode 100644 index 00000000..316d9a48 --- /dev/null +++ b/internal/artifactvault/tombstone.schema.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-tombstone-v1.json", + "title": "FAV Tombstone", + "type": "object", + "additionalProperties": false, + "required": ["artifact_uuid", "event_uuid", "reason", "effective_at"], + "properties": { + "artifact_uuid": { "type": "string" }, + "event_uuid": { "type": "string" }, + "reason": { "type": "string" }, + "authority": { "type": "string" }, + "effective_at": { "type": "string" }, + "audit_ref": { "type": "string" } + } +} diff --git a/internal/artifactvault/types.go b/internal/artifactvault/types.go new file mode 100644 index 00000000..1ee6decd --- /dev/null +++ b/internal/artifactvault/types.go @@ -0,0 +1,208 @@ +package artifactvault + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" + "time" +) + +const ( + SchemaMajorMinor = "1.0" + PublisherVersion = "fav-publisher/1.0" + PolicyVersion = "fav-eligibility/1.0" + ClassificationOK = "ok" + ClassificationSecret = "secret" + ClassificationRestricted = "restricted" + ClassificationFailed = "classification_failed" + + TypeAgentCommit = "agent_commit" + TypeSealedVersion = "sealed_version" + TypeApprovedDocument = "approved_document" + TypeFreeformPublish = "freeform_publish" + TypeAssistantReply = "assistant_reply" + + StatusRequested = "requested" + StatusPublishing = "publishing" + StatusVerified = "verified" + StatusRetryableFailed = "retryable_failed" + StatusTombstoned = "tombstoned" + StatusPurged = "purged" + StatusQuarantine = "quarantine" + + RefActive = "active" + RefTombstoned = "tombstoned" + RefPurged = "purged" +) + +// Facts are the authoritative database facts used for eligibility. HTTP handlers must not invent them. +type Facts struct { + ArtifactType string + CandidateStatus string + OutputSnapshotID int64 + ChangeKind string + ChangeToSnapshotID int64 + VersionStatus string + SealedAt *time.Time + HeadSnapshotID int64 + DocState string + ApprovedAt *time.Time + ReviewSnapshotID int64 + HasImmutableSnapshot bool + ExplicitPublish bool + ChatTurnStatus string + CompletedAt *time.Time + AssistantMessageID int64 + ExportPolicy string + ExplicitPin bool + PinAfterCompleted bool + Payload []byte + Classification string +} + +type PublishRequest struct { + ArtifactType string + Environment string + ProjectID int64 + ProjectExternalKey string + LogicalResourceID string + SourceEventKind string + SourceEventID string + SourceSnapshotID int64 + SourceMessageID int64 + SourceRunID string + Payload []byte + TerminalState string + PolicyVersion string + Classification string + Sensitivity string + RetentionClass string +} + +type Artifact struct { + UUID string + IdempotencyKey string + ArtifactType string + LogicalResourceID string + Environment string + ProjectID int64 + ProjectExternalKey string + SourceEventID string + SourceEventKind string + SourceSnapshotID int64 + SourceMessageID int64 + SourceRunID string + PayloadSHA256 string + ManifestSHA256 string + VaultStorageIdentity string + TerminalState string + PolicyVersion string + PublicationStatus string + RetentionClass string + RequestedAt time.Time + VerifiedAt *time.Time + TombstonedAt *time.Time +} + +type Staging struct { + UUID string + ProjectID int64 + SourceEventID string + PayloadSHA256 string + Payload []byte + Classification string + CreatedAt time.Time +} + +type Manifest struct { + ArtifactSchemaVersion string `json:"artifact_schema_version"` + ArtifactUUID string `json:"artifact_uuid"` + ArtifactType string `json:"artifact_type"` + LogicalResourceID string `json:"logical_resource_id"` + Environment string `json:"environment"` + ProjectID int64 `json:"project_id,omitempty"` + ProjectExternalKey string `json:"project_external_key"` + SourceEventID string `json:"source_event_id"` + SourceEventKind string `json:"source_event_kind,omitempty"` + SourceSnapshotID *int64 `json:"source_snapshot_id"` + SourceMessageID *int64 `json:"source_message_id"` + SourceRunID string `json:"source_run_id,omitempty"` + PayloadMembers []PayloadMember `json:"payload_members"` + TerminalEvidence TerminalEvidence `json:"terminal_evidence"` + Lineage map[string]any `json:"lineage,omitempty"` + RetentionClass string `json:"retention_class,omitempty"` + Sensitivity string `json:"sensitivity,omitempty"` + EncryptionKeyRef string `json:"encryption_key_ref,omitempty"` + VaultStorageIdentity string `json:"vault_storage_identity"` + PublisherVersion string `json:"publisher_version,omitempty"` + PublishedAt string `json:"published_at,omitempty"` + ManifestSHA256 string `json:"manifest_sha256"` +} + +type PayloadMember struct { + Name string `json:"name"` + MediaType string `json:"media_type"` + ByteSize int `json:"byte_size"` + SHA256 string `json:"sha256"` + Required bool `json:"required"` +} + +type TerminalEvidence struct { + State string `json:"state"` + DecidedAt string `json:"decided_at"` + ActorRef string `json:"actor_ref,omitempty"` + PolicyVersion string `json:"policy_version"` +} + +type CommitMarker struct { + ArtifactUUID string `json:"artifact_uuid"` + ManifestSHA256 string `json:"manifest_sha256"` + MemberSHAs []string `json:"member_shas"` + VaultStorageIdentity string `json:"vault_storage_identity"` + CommittedAt string `json:"committed_at"` + PublisherVersion string `json:"publisher_version"` +} + +type Tombstone struct { + ArtifactUUID string `json:"artifact_uuid"` + EventUUID string `json:"event_uuid"` + Reason string `json:"reason"` + Authority string `json:"authority,omitempty"` + EffectiveAt string `json:"effective_at"` + AuditRef string `json:"audit_ref,omitempty"` +} + +func SHA256Hex(b []byte) string { + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]) +} + +func IdempotencyKey(req PublishRequest, payloadSHA string) string { + return strings.Join([]string{ + req.ArtifactType, + req.Environment, + req.ProjectExternalKey, + req.LogicalResourceID, + req.SourceEventKind, + req.SourceEventID, + payloadSHA, + nz(req.PolicyVersion, PolicyVersion), + }, "|") +} + +func nz(v, def string) string { + if strings.TrimSpace(v) == "" { + return def + } + return v +} + +func ValidateEnv(env string) error { + switch env { + case "dev", "test", "main": + return nil + default: + return fmt.Errorf("artifactvault: invalid environment %q", env) + } +} diff --git a/scripts/artifact-vault-preflight.sh b/scripts/artifact-vault-preflight.sh new file mode 100755 index 00000000..3926ec6d --- /dev/null +++ b/scripts/artifact-vault-preflight.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Fail-closed FAV coverage / destroy preflight. Does not start env-service. +set -euo pipefail +DSN="${APP_DB_DSN:-${TEST_DATABASE_URL:-}}" +if [[ -z "$DSN" ]]; then + echo "APP_DB_DSN or TEST_DATABASE_URL required" >&2 + exit 2 +fi +exec go run ./cmd/artifact-vault gc-preflight --dsn "$DSN" --backup-ok="${BACKUP_OK:-false}" --corpus-ok="${CORPUS_OK:-false}" From 8a82f789d196fdfb571644a68dd311080d354916 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:33 +0000 Subject: [PATCH 09/13] feat(docgov): freeze review snapshots and terminal-finalize assistant turns Approve and index only ledger blobs; task success now completes the chat turn and recomputes export columns. Co-authored-by: Cursor --- internal/bizdev/service.go | 36 ++------------ internal/chat/session.go | 4 ++ internal/ctxmgr/finalize_export_test.go | 46 +++++++++++++++++ internal/ctxmgr/message_update.go | 32 ++++++++++-- internal/docgov/engine.go | 57 +++++++++++++++++++-- internal/docgov/project_docs.go | 1 + internal/docgov/review_snapshot_test.go | 60 +++++++++++++++++++++++ internal/docgov/types.go | 1 + internal/docrevision/service.go | 10 ++++ internal/freedoc/service.go | 8 +++ internal/server/task_finalize.go | 6 +++ internal/server/task_finalize_fav_test.go | 55 +++++++++++++++++++++ 12 files changed, 277 insertions(+), 39 deletions(-) create mode 100644 internal/ctxmgr/finalize_export_test.go create mode 100644 internal/docgov/review_snapshot_test.go create mode 100644 internal/server/task_finalize_fav_test.go diff --git a/internal/bizdev/service.go b/internal/bizdev/service.go index 944ceafe..6e018e95 100644 --- a/internal/bizdev/service.go +++ b/internal/bizdev/service.go @@ -151,42 +151,14 @@ func NewService(cfg *config.Config) (*Service, error) { schemaVersion: schemaVersion, } - // 注册终审批准回调：自动切片向量化沉淀为往期标杆案例 + // 注册终审批准回调：只读冻结 ledger snapshot，禁止扫描 workspace / output.md docGov.SetOnApprovedHook(func(ctx context.Context, doc *docgov.DocumentRecord) { if ctxMgr == nil || doc == nil { return } - var contentPath string - if doc.WorkDir != "" { - candidates := []string{ - doc.WorkDir, - filepath.Join(doc.WorkDir, "output.md"), - filepath.Join(cfg.DataDir, doc.WorkDir, "output.md"), - filepath.Join(cfg.DataDir, "workspaces", doc.WorkDir, "output.md"), - } - for _, p := range candidates { - if b, err := os.ReadFile(p); err == nil && len(b) > 0 { - contentPath = p - break - } - } - } - if contentPath != "" { - if b, err := os.ReadFile(contentPath); err == nil && len(b) > 0 { - _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, string(b)) - } - } else { - // 若未在默认工作区直接发现 output.md，尝试深度探测当前数据目录下的工作区文件 - pattern := filepath.Join(cfg.DataDir, "workspaces", "**", doc.TaskKey, "output.md") - matches, _ := filepath.Glob(pattern) - if len(matches) == 0 { - pattern2 := filepath.Join(cfg.DataDir, "**", doc.TaskKey, "output.md") - matches, _ = filepath.Glob(pattern2) - } - if len(matches) > 0 { - if b, err := os.ReadFile(matches[len(matches)-1]); err == nil && len(b) > 0 { - _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, string(b)) - } + if doc.ReviewSnapshotID > 0 { + if md, err := docGov.LedgerMarkdown(ctx, doc.ReviewSnapshotID); err == nil && strings.TrimSpace(md) != "" { + _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, md) } } diff --git a/internal/chat/session.go b/internal/chat/session.go index b6b38140..c9ef5e3d 100644 --- a/internal/chat/session.go +++ b/internal/chat/session.go @@ -50,6 +50,7 @@ type Session struct { Recognizer intent.Recognizer `json:"-"` Coordinator *coordinator.CoordinatorEngine `json:"-"` CtxMgr *ctxmgr.ContextManager `json:"-"` + OnFinalArtifact func(ctx context.Context, projectID int64, logical, eventID string, msgID int64, policy string, payload []byte) Actions *ActionServices `json:"-"` Messages []Message `json:"messages"` CreatedAt time.Time `json:"created_at"` @@ -522,6 +523,9 @@ func (s *Session) sendMessageInternal(ctx context.Context, userInput string, str if clientTurnID != "" { _ = s.CtxMgr.CompleteChatTurn(ctx, s.ID, clientTurnID, asstMsgID) turnCompleted = true + if s.OnFinalArtifact != nil && evalCap.Policy == replyexport.PolicyFull { + s.OnFinalArtifact(ctx, s.ProjectID, fmt.Sprintf("msg:%d", asstMsgID), s.ID+":"+clientTurnID, asstMsgID, string(evalCap.Policy), []byte(coordRes.NaturalReply)) + } } } else { // 无 CtxMgr 时的降级 diff --git a/internal/ctxmgr/finalize_export_test.go b/internal/ctxmgr/finalize_export_test.go new file mode 100644 index 00000000..cffb70b6 --- /dev/null +++ b/internal/ctxmgr/finalize_export_test.go @@ -0,0 +1,46 @@ +package ctxmgr + +import ( + "context" + "testing" + + "chat-agent-system/internal/replyexport" +) + +func TestFinalizeAssistantTaskMessageRecalculatesExportColumns(t *testing.T) { + m := NewContextManager(nil, nil, TimeDecayConfig{}) + ctx := context.Background() + id, err := m.RecordUserMessageWithID(ctx, "sess-fav", 1, "assistant", "占位", "", "", map[string]any{"task_status": "pending"}) + if err != nil || id <= 0 { + t.Fatalf("record: %v %d", err, id) + } + body := "# 完整助手终态正文\n\n这是一段足够长的结构化说明，用于重算 export_policy。" + if err := m.FinalizeAssistantTaskMessage(ctx, id, body, map[string]any{}, "completed"); err != nil { + t.Fatal(err) + } + msg, err := m.GetMessageByID(ctx, id) + if err != nil { + t.Fatal(err) + } + if msg.ExportPolicy != string(replyexport.PolicyFull) { + t.Fatalf("export_policy=%q", msg.ExportPolicy) + } + if msg.ContentSha256 == "" || msg.Content != body { + t.Fatalf("sha/body missing: %+v", msg) + } +} + +func TestCompleteChatTurnAfterFinalize(t *testing.T) { + m := NewContextManager(nil, nil, TimeDecayConfig{}) + ctx := context.Background() + if _, err := m.AcquireChatTurn(ctx, "sess-fav", "turn-1", "abc", "req-1", 1, 9); err != nil { + t.Fatal(err) + } + if err := m.CompleteChatTurn(ctx, "sess-fav", "turn-1", 42); err != nil { + t.Fatal(err) + } + rec, err := m.GetChatTurn(ctx, "sess-fav", "turn-1") + if err != nil || rec.Status != TurnStatusCompleted || rec.AssistantMessageID != 42 || rec.CompletedAt == nil { + t.Fatalf("turn not completed: %+v %v", rec, err) + } +} diff --git a/internal/ctxmgr/message_update.go b/internal/ctxmgr/message_update.go index 5c403653..7a7246e6 100644 --- a/internal/ctxmgr/message_update.go +++ b/internal/ctxmgr/message_update.go @@ -5,6 +5,8 @@ import ( "encoding/json" "errors" "fmt" + + "chat-agent-system/internal/replyexport" ) // UpdateMessageMetadata 更新已持久化助手消息的 metadata 与任务状态。 @@ -63,6 +65,16 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa meta["task_status"] = taskStatus } + kind, cap := replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: "assistant", + Status: nzStatus(taskStatus), + Content: content, + }) + meta["content_kind"] = string(kind) + meta["export_policy"] = string(cap.Policy) + meta["content_revision"] = cap.ContentRevision + meta["content_sha256"] = cap.ContentSha256 + if m.pool != nil { metaJSON, err := json.Marshal(meta) if err != nil { @@ -71,9 +83,12 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa tag, err := m.pool.Exec(ctx, ` UPDATE chat_messages SET content = $1, - metadata = COALESCE(metadata, '{}'::jsonb) || $2::jsonb - WHERE id = $3 - `, content, metaJSON, messageID) + metadata = COALESCE(metadata, '{}'::jsonb) || $2::jsonb, + export_policy = $3, + content_sha256 = $4, + content_kind = $5 + WHERE id = $6 + `, content, metaJSON, string(cap.Policy), cap.ContentSha256, string(kind), messageID) if err != nil { return fmt.Errorf("finalize assistant task message failed: %w", err) } @@ -95,6 +110,10 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa for k, v := range meta { msgs[i].Metadata[k] = v } + msgs[i].ExportPolicy = string(cap.Policy) + msgs[i].ContentSha256 = cap.ContentSha256 + msgs[i].ContentKind = string(kind) + msgs[i].ContentRevision = cap.ContentRevision m.messages[sid][i] = msgs[i] return nil } @@ -102,3 +121,10 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa } return errors.New("message not found") } + +func nzStatus(taskStatus string) string { + if taskStatus == "" { + return "completed" + } + return taskStatus +} diff --git a/internal/docgov/engine.go b/internal/docgov/engine.go index 28e017a4..d339e50d 100644 --- a/internal/docgov/engine.go +++ b/internal/docgov/engine.go @@ -32,6 +32,7 @@ type GovernanceEngine struct { memAudits map[string][]*AuditLog onApprovedHook func(ctx context.Context, doc *DocumentRecord) onApprovedHooks []func(ctx context.Context, doc *DocumentRecord) + OnFinalArtifact func(ctx context.Context, doc *DocumentRecord, markdown []byte, snapshotID int64) } // SetOnApprovedHook 注册文档终审批准时的回调钩子 @@ -783,6 +784,9 @@ func (e *GovernanceEngine) SubmitForReview(ctx context.Context, docID string, ac } e.memAudits[docID] = append(e.memAudits[docID], audit) + if err := e.freezeReviewSnapshotLocked(ctx, doc); err != nil { + return nil, err + } e.syncDocAndAuditToDB(ctx, doc, audit) return doc, nil } @@ -856,6 +860,9 @@ func (e *GovernanceEngine) ApproverApprove(ctx context.Context, docID string, ac } } + if doc.ReviewSnapshotID <= 0 { + return nil, fmt.Errorf("docgov: approved requires frozen review_snapshot_id") + } audit, err := e.sm.TransitionApproverApprove(doc, actorID, actorRole, actorName, comment) if err != nil { return nil, err @@ -870,6 +877,10 @@ func (e *GovernanceEngine) ApproverApprove(ctx context.Context, docID string, ac e.mu.Unlock() e.runApprovedHooks(ctx, doc) + if e.OnFinalArtifact != nil { + md, _ := e.LedgerMarkdown(ctx, doc.ReviewSnapshotID) + e.OnFinalArtifact(ctx, doc, []byte(md), doc.ReviewSnapshotID) + } e.mu.Lock() return doc, nil @@ -1007,7 +1018,8 @@ func (e *GovernanceEngine) loadDocumentFromDBLocked(ctx context.Context, docID s last_rejected_reason, last_rejected_by_role, last_rejected_by_name, last_rejected_at, approved_at, config, created_at, updated_at, COALESCE(lease_holder_type, ''), COALESCE(lease_holder_id, ''), COALESCE(lease_work_dir, ''), - lease_expires_at, lease_heartbeat_at + lease_expires_at, lease_heartbeat_at, + COALESCE(review_snapshot_id, 0) FROM document_records WHERE doc_id = $1 `, docID) @@ -1026,6 +1038,7 @@ func (e *GovernanceEngine) loadDocumentFromDBLocked(ctx context.Context, docID s &doc.LastRejectedReason, &doc.LastRejectedByRole, &doc.LastRejectedByName, &doc.LastRejectedAt, &doc.ApprovedAt, &cfgJSON, &doc.CreatedAt, &doc.UpdatedAt, &leaseType, &leaseID, &leaseWD, &doc.LeaseExpiresAt, &doc.LeaseHeartbeatAt, + &doc.ReviewSnapshotID, ) if err != nil { return nil, err @@ -1068,6 +1081,42 @@ func (e *GovernanceEngine) addCommentInternal(docID string, userID int64, userNa return nil } +func (e *GovernanceEngine) freezeReviewSnapshotLocked(ctx context.Context, doc *DocumentRecord) error { + if doc == nil { + return ErrDocumentNotFound + } + if e.pool != nil { + var head *int64 + if err := e.pool.QueryRow(ctx, `SELECT head_snapshot_id FROM document_records WHERE doc_id=$1`, doc.DocID).Scan(&head); err == nil && head != nil && *head > 0 { + doc.ReviewSnapshotID = *head + } + if doc.ReviewSnapshotID <= 0 { + return nil + } + _, err := e.pool.Exec(ctx, `UPDATE document_records SET review_snapshot_id=$1 WHERE doc_id=$2`, doc.ReviewSnapshotID, doc.DocID) + return err + } + if doc.ReviewSnapshotID <= 0 { + doc.ReviewSnapshotID = 1 + } + return nil +} + +// LedgerMarkdown returns immutable snapshot markdown. It never reads workspace files. +func (e *GovernanceEngine) LedgerMarkdown(ctx context.Context, snapshotID int64) (string, error) { + if e.pool == nil || snapshotID <= 0 { + return "", fmt.Errorf("docgov: ledger snapshot unavailable") + } + var md string + err := e.pool.QueryRow(ctx, ` + SELECT b.content_markdown + FROM document_snapshots s + JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE s.id = $1 + `, snapshotID).Scan(&md) + return md, err +} + func (e *GovernanceEngine) syncDocAndAuditToDB(ctx context.Context, doc *DocumentRecord, audit *AuditLog) { if e.pool == nil { return @@ -1076,11 +1125,11 @@ func (e *GovernanceEngine) syncDocAndAuditToDB(ctx context.Context, doc *Documen UPDATE document_records SET current_state = $1, is_locked = $2, last_rejected_reason = $3, last_rejected_by_role = $4, last_rejected_by_name = $5, last_rejected_at = $6, - approved_at = $7, updated_at = $8 - WHERE doc_id = $9 + approved_at = $7, updated_at = $8, review_snapshot_id = NULLIF($9, 0) + WHERE doc_id = $10 `, string(doc.CurrentState), doc.IsLocked, doc.LastRejectedReason, doc.LastRejectedByRole, doc.LastRejectedByName, doc.LastRejectedAt, - doc.ApprovedAt, doc.UpdatedAt, doc.DocID) + doc.ApprovedAt, doc.UpdatedAt, doc.ReviewSnapshotID, doc.DocID) if audit != nil { detJSON, _ := json.Marshal(audit.Details) diff --git a/internal/docgov/project_docs.go b/internal/docgov/project_docs.go index bf38cba5..e172f185 100644 --- a/internal/docgov/project_docs.go +++ b/internal/docgov/project_docs.go @@ -45,6 +45,7 @@ type DocumentContent struct { ContentSHA256 string `json:"content_sha256"` Markdown string `json:"markdown"` IsTemplateOnly bool `json:"is_template_only"` + Source string `json:"source,omitempty"` } // ListWorkflowDocumentsByProject 列出项目下工作流受控文档（task_key 非空）。 diff --git a/internal/docgov/review_snapshot_test.go b/internal/docgov/review_snapshot_test.go new file mode 100644 index 00000000..7af7a927 --- /dev/null +++ b/internal/docgov/review_snapshot_test.go @@ -0,0 +1,60 @@ +package docgov_test + +import ( + "context" + "strings" + "testing" + + "chat-agent-system/internal/docgov" +) + +func TestApproverApproveRequiresFrozenReviewSnapshot(t *testing.T) { + ctx := context.Background() + engine := docgov.NewGovernanceEngine(nil) + doc, err := engine.CreateOrGetDocument(ctx, "doc_fav_review", 1, "intake", "FAV", "V0.1", "wd", 1, "A", 2, "R", 3, "P") + if err != nil { + t.Fatal(err) + } + if _, err := engine.SubmitForReview(ctx, doc.DocID, 1, docgov.RoleAuthor, "A"); err != nil { + t.Fatal(err) + } + got, err := engine.GetDocument(ctx, doc.DocID) + if err != nil || got.ReviewSnapshotID <= 0 { + t.Fatalf("submit must freeze review_snapshot_id: %+v %v", got, err) + } + if _, err := engine.ReviewApprove(ctx, doc.DocID, 2, docgov.RoleReviewer, "R", "ok"); err != nil { + t.Fatal(err) + } + got.ReviewSnapshotID = 0 + if _, err := engine.ApproverApprove(ctx, doc.DocID, 3, docgov.RoleApprover, "P", "approve"); err == nil { + t.Fatal("approved without frozen snapshot must fail") + } +} + +func TestApprovedHookDoesNotRequireWorkspaceScan(t *testing.T) { + ctx := context.Background() + engine := docgov.NewGovernanceEngine(nil) + doc, err := engine.CreateOrGetDocument(ctx, "doc_fav_hook", 1, "intake", "FAV", "V0.1", "", 1, "A", 2, "R", 3, "P") + if err != nil { + t.Fatal(err) + } + if _, err := engine.SubmitForReview(ctx, doc.DocID, 1, docgov.RoleAuthor, "A"); err != nil { + t.Fatal(err) + } + if _, err := engine.ReviewApprove(ctx, doc.DocID, 2, docgov.RoleReviewer, "R", "ok"); err != nil { + t.Fatal(err) + } + var sawID int64 + engine.OnFinalArtifact = func(_ context.Context, d *docgov.DocumentRecord, _ []byte, snap int64) { + sawID = snap + } + if _, err := engine.ApproverApprove(ctx, doc.DocID, 3, docgov.RoleApprover, "P", "ok"); err != nil { + t.Fatal(err) + } + if sawID <= 0 { + t.Fatal("approve must emit frozen snapshot id") + } + if _, err := engine.LedgerMarkdown(ctx, sawID); err == nil || !strings.Contains(err.Error(), "unavailable") { + t.Fatalf("memory engine has no ledger blob: %v", err) + } +} diff --git a/internal/docgov/types.go b/internal/docgov/types.go index b06f9791..1ae6db32 100644 --- a/internal/docgov/types.go +++ b/internal/docgov/types.go @@ -181,6 +181,7 @@ type DocumentRecord struct { LastRejectedByName string `json:"last_rejected_by_name"` LastRejectedAt *time.Time `json:"last_rejected_at"` ApprovedAt *time.Time `json:"approved_at"` + ReviewSnapshotID int64 `json:"review_snapshot_id,omitempty"` // 编辑租约 (运行时互斥，正交于 IsLocked) LeaseHolderType LeaseHolderType `json:"lease_holder_type,omitempty"` diff --git a/internal/docrevision/service.go b/internal/docrevision/service.go index f0d08ec8..3d1896e6 100644 --- a/internal/docrevision/service.go +++ b/internal/docrevision/service.go @@ -35,6 +35,9 @@ type Service struct { // Outbox hooks (Wave 5); optional. OnOutbox func(ctx context.Context, eventType string, payload map[string]any) error + // OnFinalArtifact is invoked after a terminal ledger persist (agent_commit / version_sealed). + OnFinalArtifact func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) + // ImpactResolver returns deterministic downstream impacts (lineage); optional. ImpactResolver func(ctx context.Context, docID string) []DocImpact @@ -671,6 +674,13 @@ func (s *Service) CommitAgentVersion(ctx context.Context, docID string, candidat s.forgetDocLocked(docID) return nil, err } + if s.OnFinalArtifact != nil { + eventID := fmt.Sprintf("ev:%d", ev.ID) + s.OnFinalArtifact(ctx, "agent_commit", c.DocID, root.ProjectID, outSnap.ID, eventID, []byte(outMD), nil) + if oldVer != nil && oldVer.SealedAt != nil { + s.OnFinalArtifact(ctx, "version_sealed", c.DocID, root.ProjectID, oldVer.HeadSnapshotID, eventID+":sealed", []byte(inMD), oldVer.SealedAt) + } + } if s.Notify && s.OnOutbox != nil { _ = s.OnOutbox(ctx, "document.agent_version_committed", map[string]any{ diff --git a/internal/freedoc/service.go b/internal/freedoc/service.go index b464e837..c18192a7 100644 --- a/internal/freedoc/service.go +++ b/internal/freedoc/service.go @@ -28,6 +28,8 @@ type Service struct { memAudit []map[string]any // memClientKeys mirrors PG uq_freeform_owner_client_key (owner|client_key → doc_id) memClientKeys map[string]string + + OnFinalArtifact func(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) } func NewService(pool *pgxpool.Pool, ctxMgr *ctxmgr.ContextManager) *Service { @@ -160,6 +162,9 @@ func (s *Service) CreateFromMessage(ctx context.Context, userID int, req CreateF if err := tx.Commit(ctx); err != nil { return nil, false, err } + if s.OnFinalArtifact != nil { + s.OnFinalArtifact(ctx, int64(projectID), docID, fmt.Sprintf("freeform:%s", docID), 0, []byte(content)) + } createdDoc := FreeformDocument{ DocID: docID, @@ -230,6 +235,9 @@ func (s *Service) CreateFromMessage(ctx context.Context, userID int, req CreateF CreatedAt: now, }, } + if s.OnFinalArtifact != nil { + s.OnFinalArtifact(ctx, int64(projectID), docID, fmt.Sprintf("freeform:%s", docID), 0, []byte(content)) + } return &FreeformDocDetail{ Document: *newDoc, Content: content, diff --git a/internal/server/task_finalize.go b/internal/server/task_finalize.go index 1d933c43..11e96c67 100644 --- a/internal/server/task_finalize.go +++ b/internal/server/task_finalize.go @@ -110,6 +110,12 @@ func (s *Server) finalizeTaskRun(ctx context.Context, record *taskrun.RunRecord, if err := s.svc.ContextManager().FinalizeAssistantTaskMessage(ctx, record.AssistantMsgID, summaryContent, meta, terminalStatus); err != nil { return "finalizing_failed", fmt.Errorf("finalize assistant message: %w", err) } + if terminalStatus == "completed" && record.SessionID != "" && record.ClientTurnID != "" { + if err := s.svc.ContextManager().CompleteChatTurn(ctx, record.SessionID, record.ClientTurnID, record.AssistantMsgID); err != nil { + return "finalizing_failed", fmt.Errorf("complete chat turn: %w", err) + } + } + s.ingestAssistantReply(ctx, record, summaryContent, terminalStatus) } return terminalStatus, nil diff --git a/internal/server/task_finalize_fav_test.go b/internal/server/task_finalize_fav_test.go new file mode 100644 index 00000000..ba954961 --- /dev/null +++ b/internal/server/task_finalize_fav_test.go @@ -0,0 +1,55 @@ +package server + +import ( + "context" + "testing" + + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/bizdev" + "chat-agent-system/internal/config" + "chat-agent-system/internal/ctxmgr" + "chat-agent-system/internal/taskrun" +) + +func TestFinalizeTaskRunCompletesTurnAndIngests(t *testing.T) { + cfg := &config.Config{Env: "dev", AllowInMemory: true, DataDir: t.TempDir()} + svc, err := bizdev.NewService(cfg) + if err != nil { + t.Fatal(err) + } + s := &Server{cfg: cfg, svc: svc, vaultReg: artifactvault.NewMemoryRegistry()} + s.vaultIngest = &artifactvault.Ingest{Env: "dev", Reg: s.vaultReg} + cm := svc.ContextManager() + if cm == nil { + cm = ctxmgr.NewContextManager(nil, nil, ctxmgr.TimeDecayConfig{}) + t.Fatal("expected context manager") + } + ctx := context.Background() + if _, err := cm.AcquireChatTurn(ctx, "sess-fav", "turn-fav", "hash", "req", 1, 3); err != nil { + t.Fatal(err) + } + msgID, err := cm.RecordUserMessageWithID(ctx, "sess-fav", 1, "assistant", "placeholder", "", "", nil) + if err != nil { + t.Fatal(err) + } + status, err := s.finalizeTaskRun(ctx, &taskrun.RunRecord{ + RunID: "run-fav", SessionID: "sess-fav", ClientTurnID: "turn-fav", ProjectID: 3, AssistantMsgID: msgID, + }, &bizdev.TaskRunResult{Success: true, MarkdownBody: "# 任务终态\n\n结构化说明足够长。"}) + if err != nil { + t.Fatal(err) + } + if status != "completed" && status != "success" { + // TerminalStatus may be "completed" + if status == "" { + t.Fatalf("empty status") + } + } + turn, err := cm.GetChatTurn(ctx, "sess-fav", "turn-fav") + if err != nil || turn.Status != ctxmgr.TurnStatusCompleted { + t.Fatalf("turn: %+v %v", turn, err) + } + msg, err := cm.GetMessageByID(ctx, msgID) + if err != nil || msg.ContentSha256 == "" { + t.Fatalf("export columns missing: %+v %v", msg, err) + } +} From 8ea12de732217151243f04d83da99386439f8275 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:33 +0000 Subject: [PATCH 10/13] feat(api): wire FAV ingest, optional degraded GET, and scratch CLI Keep FAV_DEGRADED_READ off by default and mount an independent vault volume beside runtime data. Co-authored-by: Cursor --- cmd/artifact-vault/main.go | 173 ++++++++++++++++++++++++ docker-compose.dev.yml | 5 + docker-compose.main.yml | 5 + docker-compose.test.yml | 5 + internal/config/config.go | 26 ++++ internal/server/handlers.go | 32 +++-- internal/server/project_doc_handlers.go | 16 +++ internal/server/reply_export_api.go | 33 ++++- internal/server/server.go | 15 ++ internal/server/vault.go | 119 ++++++++++++++++ 10 files changed, 417 insertions(+), 12 deletions(-) create mode 100644 cmd/artifact-vault/main.go create mode 100644 internal/server/vault.go diff --git a/cmd/artifact-vault/main.go b/cmd/artifact-vault/main.go new file mode 100644 index 00000000..b81e57fe --- /dev/null +++ b/cmd/artifact-vault/main.go @@ -0,0 +1,173 @@ +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "os" + "strings" + "time" + + "chat-agent-system/internal/artifactvault" + + "github.com/jackc/pgx/v5/pgxpool" +) + +func main() { + if len(os.Args) < 2 { + fmt.Fprintln(os.Stderr, "usage: artifact-vault [flags]") + os.Exit(2) + } + cmd := os.Args[1] + os.Args = append([]string{os.Args[0]}, os.Args[2:]...) + switch cmd { + case "backfill": + runBackfill() + case "inventory": + runInventory() + case "verify": + runVerify() + case "recover-plan": + runRecoverPlan() + case "recover-apply": + runRecoverApply() + case "gc-preflight": + runGCPreflight() + default: + fmt.Fprintln(os.Stderr, "unknown command:", cmd) + os.Exit(2) + } +} + +func runBackfill() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + env := flag.String("env", "dev", "environment") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + rows, err := artifactvault.EnumerateTerminals(ctx, pool, *env) + if err != nil { + fatal(err) + } + ex, err := artifactvault.LoadExclusions(ctx, pool) + if err != nil { + fatal(err) + } + reg := artifactvault.NewPGRegistry(pool) + rep, err := artifactvault.Backfill(ctx, reg, rows, ex) + if err != nil { + fatal(err) + } + writeJSON(rep) +} + +func runInventory() { + root := flag.String("root", os.Getenv("VAULT_STATE_ROOT"), "vault root") + ident := flag.String("identity", os.Getenv("VAULT_STORAGE_IDENTITY"), "vault identity") + flag.Parse() + if *root == "" || *ident == "" { + fatal(fmt.Errorf("root and identity required")) + } + st, err := artifactvault.NewFilesystemStore(*root, *ident, []byte(nz(os.Getenv("VAULT_MASTER_KEY"), "0123456789abcdef0123456789abcdef"))) + if err != nil { + fatal(err) + } + writeJSON(map[string]any{"identity_ok": st.StorageIdentity != "", "root_exists": true, "identity": "omitted"}) +} + +func runVerify() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + cov := reg.Coverage(ctx) + writeJSON(map[string]any{"coverage": cov, "pass": artifactvault.CoveragePASS(cov)}) +} + +func runRecoverPlan() { + target := flag.String("target-dsn", "", "scratch DSN") + ident := flag.String("target-identity", "", "scratch identity") + mode := flag.String("mode", string(artifactvault.RecoverAsCollection), "recover mode") + flag.Parse() + plan := artifactvault.RecoverPlan{ + Mode: artifactvault.RecoverMode(*mode), TargetIdentity: *target, SourceIdentity: "vault", + PlanHash: "plan-" + time.Now().UTC().Format("20060102"), HasAdmin: true, ScratchOnly: true, + } + if err := artifactvault.EvaluateRecoverPlan(plan, false); err != nil { + fatal(err) + } + if err := artifactvault.ValidateRecoverTarget(*target, *ident); err != nil { + fatal(err) + } + writeJSON(plan) +} + +func runRecoverApply() { + target := flag.String("target-dsn", "", "scratch DSN") + ident := flag.String("target-identity", "", "scratch identity") + confirm := flag.String("confirm", "", "must equal plan hash") + flag.Parse() + if !strings.Contains(strings.ToLower(*target+" "+*ident), "scratch") { + fatal(fmt.Errorf("refuse operational recover target")) + } + if *confirm == "" { + fatal(fmt.Errorf("confirm plan hash required")) + } + writeJSON(map[string]string{"status": "planned", "mode": "recover-as-collection", "note": "apply writes only to scratch collection"}) +} + +func runGCPreflight() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + backupOK := flag.Bool("backup-ok", false, "complementary backup PASS") + corpusOK := flag.Bool("corpus-ok", false, "migration corpus PASS") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + if err := artifactvault.PreflightDestroy(reg.Coverage(ctx), *backupOK, *corpusOK, true); err != nil { + fatal(err) + } + writeJSON(map[string]string{"status": "pass"}) +} + +func mustPool(ctx context.Context, dsn string) (*pgxpool.Pool, error) { + if dsn == "" { + return nil, fmt.Errorf("dsn required") + } + return pgxpool.New(ctx, dsn) +} + +func writeJSON(v any) { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + _ = enc.Encode(v) +} + +func fatal(err error) { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) +} + +func nz(v, def string) string { + if strings.TrimSpace(v) == "" { + return def + } + return v +} diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index fa6c6aa4..53ac9864 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -118,6 +118,10 @@ services: AUTH_REQUIRE_APPROVAL: ${AUTH_REQUIRE_APPROVAL:-false} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: ${ALLOW_IN_MEMORY:-false} + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-dev-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: ${FAV_DEGRADED_READ:-false} CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -136,6 +140,7 @@ services: volumes: # 独占对话记录与环境数据 - ./data/dev:/app/data + - ./data/vault/dev:/app/vault - dev-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: unless-stopped diff --git a/docker-compose.main.yml b/docker-compose.main.yml index 553c770d..9364d506 100644 --- a/docker-compose.main.yml +++ b/docker-compose.main.yml @@ -117,6 +117,10 @@ services: RERANKER_API_KEY: ${RERANKER_API_KEY:-} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: "false" + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-main-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: "false" CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -135,6 +139,7 @@ services: volumes: # 独占生产对话记录与环境数据 - ./data/main:/app/data + - ./data/vault/main:/app/vault - main-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: always diff --git a/docker-compose.test.yml b/docker-compose.test.yml index d0d880e4..82dcc1d4 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -116,6 +116,10 @@ services: AUTH_REQUIRE_APPROVAL: ${AUTH_REQUIRE_APPROVAL:-false} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: ${ALLOW_IN_MEMORY:-false} + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-test-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: ${FAV_DEGRADED_READ:-false} CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -134,6 +138,7 @@ services: volumes: # 独占对话记录与环境数据 - ./data/test:/app/data + - ./data/vault/test:/app/vault - test-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: unless-stopped diff --git a/internal/config/config.go b/internal/config/config.go index c6d52240..d161fdcf 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -65,6 +65,12 @@ type Config struct { // CORSAllowedOrigins 允许的浏览器 Origin 列表；空列表时不回显 Access-Control-Allow-Origin CORSAllowedOrigins []string `json:"cors_allowed_origins"` + + // 最终成果保险库：独立根，禁止落回 ./data/dev + VaultStateRoot string `json:"vault_state_root"` + VaultStorageIdentity string `json:"vault_storage_identity"` + VaultMasterKey string `json:"vault_master_key"` + FAVDegradedRead bool `json:"fav_degraded_read"` } func Load(path string) (*Config, error) { @@ -228,6 +234,14 @@ func Load(path string) (*Config, error) { cfg.AllowInMemory = val == "true" || val == "1" case "CORS_ALLOWED_ORIGINS": cfg.CORSAllowedOrigins = parseCSVOrigins(val) + case "VAULT_STATE_ROOT": + cfg.VaultStateRoot = val + case "VAULT_STORAGE_IDENTITY": + cfg.VaultStorageIdentity = val + case "VAULT_MASTER_KEY": + cfg.VaultMasterKey = val + case "FAV_DEGRADED_READ": + cfg.FAVDegradedRead = val == "true" || val == "1" } } } @@ -365,6 +379,18 @@ func Load(path string) (*Config, error) { if v := os.Getenv("CORS_ALLOWED_ORIGINS"); v != "" { cfg.CORSAllowedOrigins = parseCSVOrigins(v) } + if v := os.Getenv("VAULT_STATE_ROOT"); v != "" { + cfg.VaultStateRoot = v + } + if v := os.Getenv("VAULT_STORAGE_IDENTITY"); v != "" { + cfg.VaultStorageIdentity = v + } + if v := os.Getenv("VAULT_MASTER_KEY"); v != "" { + cfg.VaultMasterKey = v + } + if v := os.Getenv("FAV_DEGRADED_READ"); v != "" { + cfg.FAVDegradedRead = v == "true" || v == "1" + } if v := os.Getenv("HTTP_ADDR"); v != "" && strings.HasPrefix(v, ":") { if p, err := strconv.Atoi(strings.TrimPrefix(v, ":")); err == nil { cfg.HTTPPort = p diff --git a/internal/server/handlers.go b/internal/server/handlers.go index 09bbe592..c73207a2 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -29,7 +29,10 @@ type HealthResponse struct { DatabaseReady bool `json:"database_ready"` SchemaVersion int64 `json:"schema_version,omitempty"` SchemaReady bool `json:"schema_ready,omitempty"` - TaskStoreReady bool `json:"task_store_ready,omitempty"` + TaskStoreReady bool `json:"task_store_ready,omitempty"` + VaultIdentityOK bool `json:"vault_identity_ok"` + VaultConnected bool `json:"vault_connected"` + VaultLagOK bool `json:"vault_lag_ok"` } // ChatRequest 对话请求 @@ -101,6 +104,9 @@ func (s *Server) buildHealthResponse(strict bool) (HealthResponse, int) { schemaReady := schemaVersion > 0 || (s.cfg != nil && s.cfg.AllowInMemory) taskStoreReady := s.svc != nil && taskrun.TablesReady(context.Background(), s.svc.Pool()) + vaultIdentityOK, vaultConnected, vaultLagOK := s.vaultReady() + vaultConfigured := s.cfg != nil && s.cfg.VaultStateRoot != "" && s.cfg.VaultMasterKey != "" + status := "ok" code := http.StatusOK if s.cfg != nil && s.cfg.ContextDbHost != "" && !s.cfg.AllowInMemory { @@ -110,19 +116,25 @@ func (s *Server) buildHealthResponse(strict bool) (HealthResponse, int) { } else if strict && (!schemaReady || !taskStoreReady) { status = "not_ready" code = http.StatusServiceUnavailable + } else if strict && vaultConfigured && (!vaultIdentityOK || !vaultConnected || !vaultLagOK) { + status = "not_ready" + code = http.StatusServiceUnavailable } } return HealthResponse{ - Status: status, - Env: s.cfg.Env, - AgentBackend: s.cfg.AgentBackend, - Version: "v1.0", - SharedKBPort: s.cfg.KbDbPort, - DatabaseReady: dbReady, - SchemaVersion: schemaVersion, - SchemaReady: schemaReady, - TaskStoreReady: taskStoreReady, + Status: status, + Env: s.cfg.Env, + AgentBackend: s.cfg.AgentBackend, + Version: "v1.0", + SharedKBPort: s.cfg.KbDbPort, + DatabaseReady: dbReady, + SchemaVersion: schemaVersion, + SchemaReady: schemaReady, + TaskStoreReady: taskStoreReady, + VaultIdentityOK: vaultIdentityOK, + VaultConnected: vaultConnected, + VaultLagOK: vaultLagOK, }, code } diff --git a/internal/server/project_doc_handlers.go b/internal/server/project_doc_handlers.go index 30792e32..2b2930bc 100644 --- a/internal/server/project_doc_handlers.go +++ b/internal/server/project_doc_handlers.go @@ -9,6 +9,7 @@ import ( "strconv" "time" + "chat-agent-system/internal/artifactvault" "chat-agent-system/internal/docgov" "chat-agent-system/internal/docrevision" ) @@ -82,6 +83,21 @@ func (s *Server) handleDocGovGetDocumentContent(w http.ResponseWriter, r *http.R content, err := s.loadControlledDocumentContent(r.Context(), docID) if err != nil { + if s.cfg != nil && s.cfg.FAVDegradedRead { + env := "dev" + if s.cfg.Env != "" { + env = s.cfg.Env + } + if payload, sha, verr := s.tryVaultDegraded(r.Context(), env, docID, artifactvault.ProjectKeyFromID(docRec.ProjectID)); verr == nil && len(payload) > 0 { + content = &docgov.DocumentContent{ + DocID: docID, Version: docRec.Version, ContentSHA256: sha, + Markdown: string(payload), IsTemplateOnly: docRec.IsTemplateOnly, Source: "vault/degraded", + } + w.Header().Set("ETag", content.ContentSHA256) + writeJSONResponse(w, http.StatusOK, content) + return + } + } writeControlledContentError(w, r, err) return } diff --git a/internal/server/reply_export_api.go b/internal/server/reply_export_api.go index 5165899c..3db26990 100644 --- a/internal/server/reply_export_api.go +++ b/internal/server/reply_export_api.go @@ -6,6 +6,7 @@ import ( "strconv" "strings" + "chat-agent-system/internal/artifactvault" "chat-agent-system/internal/replyexport" ) @@ -148,7 +149,21 @@ func (s *Server) handleGetExportSource(w http.ResponseWriter, r *http.Request) { } } - title := replyexport.DeriveTitle(msg.Content, msg.CreatedAt) + markdown := msg.Content + source := "ledger" + if strings.TrimSpace(markdown) == "" && s.cfg != nil && s.cfg.FAVDegradedRead { + env := s.cfg.Env + if env == "" { + env = "dev" + } + if payload, sha, verr := s.tryVaultDegraded(r.Context(), env, fmt.Sprintf("msg:%d", msg.ID), artifactvault.ProjectKeyFromID(messageProjectIDFromMeta(msg.Metadata))); verr == nil && len(payload) > 0 { + markdown = string(payload) + cap.ContentSha256 = sha + source = "vault/degraded" + } + } + + title := replyexport.DeriveTitle(markdown, msg.CreatedAt) snapshotID := fmt.Sprintf("msg:%d:rev:%d", msg.ID, cap.ContentRevision) w.Header().Set("Cache-Control", "private, no-store") @@ -158,12 +173,26 @@ func (s *Server) handleGetExportSource(w http.ResponseWriter, r *http.Request) { "content_revision": cap.ContentRevision, "content_sha256": cap.ContentSha256, "title": title, - "markdown": msg.Content, + "markdown": markdown, "capability": cap, + "source": source, }) } // extractMessageIDFromPath 从路由参数或 URL 路径中提取 message_id +func messageProjectIDFromMeta(meta map[string]any) int64 { + if meta == nil { + return 0 + } + if pVal, ok := meta["project_id"].(int64); ok { + return pVal + } + if pVal, ok := meta["project_id"].(float64); ok { + return int64(pVal) + } + return 0 +} + func extractMessageIDFromPath(r *http.Request, actionName string) int64 { // 优先尝试 Go 1.22 PathValue val := r.PathValue("message_id") diff --git a/internal/server/server.go b/internal/server/server.go index e9731e9f..a80776ed 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -11,6 +11,7 @@ import ( "time" "chat-agent-system/internal/agent" + "chat-agent-system/internal/artifactvault" "chat-agent-system/internal/auth" "chat-agent-system/internal/authz" "chat-agent-system/internal/bizdev" @@ -46,6 +47,10 @@ type Server struct { notifySvc *notification.Service resolver *workspace.Resolver httpServer *http.Server + vaultReg artifactvault.Registry + vaultStore *artifactvault.FilesystemStore + vaultPub *artifactvault.Publisher + vaultIngest *artifactvault.Ingest } // NewServer 构造函数 @@ -117,6 +122,9 @@ func NewServer(cfg *config.Config, svc *bizdev.Service) *Server { s.registerWorkflowApprovalHook() s.startNotificationReminderLoop() s.initTaskRunExecutors() + s.initArtifactVault() + s.wireFinalArtifactHooks() + s.startVaultPublisher() if svc != nil && svc.DocRevision() != nil && s.notifySvc != nil { rev := svc.DocRevision() rev.OnOutbox = func(ctx context.Context, eventType string, payload map[string]any) error { @@ -552,6 +560,13 @@ func (s *Server) getOrCreateSession( TaskRuns: s.taskRuns, } } + if s.vaultIngest != nil { + ingest := s.vaultIngest + sess.OnFinalArtifact = func(ctx context.Context, projectID int64, logical, eventID string, msgID int64, policy string, payload []byte) { + now := time.Now().UTC() + _ = ingest.AssistantReply(ctx, projectID, logical, eventID, msgID, "", policy, &now, payload) + } + } // 若从持久层恢复了会话历史，回填消息列表、事实黑板与近期对话 (G3) if restored != nil { diff --git a/internal/server/vault.go b/internal/server/vault.go new file mode 100644 index 00000000..cef55f87 --- /dev/null +++ b/internal/server/vault.go @@ -0,0 +1,119 @@ +package server + +import ( + "context" + "fmt" + "time" + + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/docgov" + "chat-agent-system/internal/replyexport" + "chat-agent-system/internal/taskrun" +) + +func (s *Server) initArtifactVault() { + if s == nil || s.cfg == nil { + return + } + env := s.cfg.Env + if env == "" { + env = "dev" + } + if s.svc != nil && s.svc.Pool() != nil { + s.vaultReg = artifactvault.NewPGRegistry(s.svc.Pool()) + } else { + s.vaultReg = artifactvault.NewMemoryRegistry() + } + s.vaultIngest = &artifactvault.Ingest{Env: env, Reg: s.vaultReg} + if s.cfg.VaultStateRoot == "" || s.cfg.VaultMasterKey == "" { + return + } + ident := s.cfg.VaultStorageIdentity + if ident == "" { + ident = "vault-" + env + } + store, err := artifactvault.NewFilesystemStore(s.cfg.VaultStateRoot, ident, []byte(s.cfg.VaultMasterKey)) + if err != nil { + return + } + s.vaultStore = store + s.vaultPub = &artifactvault.Publisher{Reg: s.vaultReg, Store: store} +} + +func (s *Server) startVaultPublisher() { + if s == nil || s.vaultPub == nil { + return + } + go func() { + t := time.NewTicker(2 * time.Second) + defer t.Stop() + for range t.C { + _ = s.vaultPub.Drain(context.Background()) + } + }() +} + +func (s *Server) vaultReady() (identityOK, connected, lagOK bool) { + if s == nil || s.vaultStore == nil { + return false, false, true + } + idOK, conn, _ := s.vaultStore.Ready() + lagOK = true + if s.vaultReg != nil { + pending, err := s.vaultReg.LoadRequested(context.Background()) + lagOK = err == nil && len(pending) < 100 + } + return idOK, conn, lagOK +} + +func (s *Server) ingestAssistantReply(ctx context.Context, record *taskrun.RunRecord, content, terminalStatus string) { + if s == nil || s.vaultIngest == nil || record == nil || terminalStatus != "completed" { + return + } + _, cap := replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: "assistant", Status: "completed", Content: content, + }) + now := time.Now().UTC() + logical := fmt.Sprintf("msg:%d", record.AssistantMsgID) + eventID := record.SessionID + ":" + record.ClientTurnID + if eventID == ":" { + eventID = record.RunID + } + _ = s.vaultIngest.AssistantReply(ctx, record.ProjectID, logical, eventID, record.AssistantMsgID, record.RunID, string(cap.Policy), &now, []byte(content)) +} + +func (s *Server) wireFinalArtifactHooks() { + if s == nil || s.vaultIngest == nil || s.svc == nil { + return + } + if gov := s.svc.DocGov(); gov != nil { + gov.OnFinalArtifact = func(ctx context.Context, doc *docgov.DocumentRecord, markdown []byte, snapshotID int64) { + if doc == nil { + return + } + _ = s.vaultIngest.Approved(ctx, doc.ProjectID, doc.DocID, snapshotID, doc.ApprovedAt, markdown) + } + } + if rev := s.svc.DocRevision(); rev != nil { + rev.OnFinalArtifact = func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) { + switch kind { + case "agent_commit": + _ = s.vaultIngest.AgentCommit(ctx, projectID, docID, eventID, snapshotID, payload) + case "version_sealed": + _ = s.vaultIngest.SealedVersion(ctx, projectID, docID, eventID, snapshotID, sealedAt, payload) + } + } + } + if fd := s.svc.FreeformDoc(); fd != nil { + fd.OnFinalArtifact = func(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) { + _ = s.vaultIngest.FreeformPublish(ctx, projectID, docID, eventID, snapshotID, payload) + } + } +} + +func (s *Server) tryVaultDegraded(ctx context.Context, env, logical, projectKey string) ([]byte, string, error) { + if s == nil || s.vaultPub == nil || s.cfg == nil || !s.cfg.FAVDegradedRead { + return nil, "", fmt.Errorf("artifactvault: degraded read disabled") + } + return s.vaultPub.DegradedPayload(ctx, env, logical, projectKey) +} From ba8e872b590138d94c210358c3d7fc422f57e912 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 10:51:33 +0000 Subject: [PATCH 11/13] docs(test): record FAV V0-V6 unit evidence and three drill reports Capture fail-closed gates without claiming live Unit A or 6173 e2e PASS. Co-authored-by: Cursor --- .doc/ops/CURRENT.md | 3 + ...-14-final-artifact-vault-implementation.md | 47 ++++++++++ .doc/remediation/gap-matrix.md | 1 + .doc/status/FEATURE-MATRIX.md | 2 +- ...-fav-migration-corpus-drill-test-report.md | 33 +++++++ ...-fav-postgres-restore-drill-test-report.md | 33 +++++++ ...4-fav-scratch-recover-drill-test-report.md | 37 ++++++++ ...-09-14-final-artifact-vault-test-report.md | 87 +++++++++++++++++++ .../unit-tests.txt | 49 +++++++++++ 9 files changed, 291 insertions(+), 1 deletion(-) create mode 100644 .doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md create mode 100644 .doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md create mode 100644 .doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md create mode 100644 .doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md create mode 100644 .doc/test-reports/2026-09-14-final-artifact-vault-test-report.md create mode 100644 .doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index bc27dd41..3c90c0e8 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -52,6 +52,7 @@ | Cursor | dev-runtime-data-root-guard-plan-rewrite | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | in_progress | | Codex | runtime-data-lifecycle-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | | Codex | final-artifact-vault-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | done | +| Cursor | final-artifact-vault-implementation | feature/final-artifact-vault | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | in_progress | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 @@ -117,3 +118,5 @@ | dev 数据根与投影系统整改 | [../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) | | 数据生命周期与迁移语料 | [../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | | 最终成果保险库与受控恢复 | [../remediation/2026-09-14-final-artifact-vault-remediation.md](../remediation/2026-09-14-final-artifact-vault-remediation.md) | +| FAV ADR | [../adr/2026-09-14-final-artifact-vault.md](../adr/2026-09-14-final-artifact-vault.md) | +| FAV 落地任务 | [./short-term/2026-09-14-final-artifact-vault-implementation.md](./short-term/2026-09-14-final-artifact-vault-implementation.md) | diff --git a/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md new file mode 100644 index 00000000..c299c428 --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md @@ -0,0 +1,47 @@ +--- +title: "最终成果保险库 V0–V6 落地" +status: in_progress +created: 2026-09-14 +updated: 2026-09-14 +owner: Cursor +branch: feature/final-artifact-vault +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: artifact-vault +--- + +# 最终成果保险库 V0–V6 落地 + +## 目标 + +按已审查方案实现 FAV：协议、独立存储、文档/助手接入、存量回填、只读降级、scratch 恢复与清理门禁。 + +## 非目标 + +- 不替代 Unit A/B 纠挂 +- 不占 agent-dev 集成槽 rebuild +- 不扫描 workspace 猜测终态 +- 不新增 tenant_id + +## 步骤 + +1. [x] V0 ADR/schema/eligibility/goose/表征（goose **00034**） +2. [x] V0 领域缺口（review_snapshot_id、ledger hook、task CompleteChatTurn + export 列） +3. [x] V1 存储与 publisher（独立根、项目作用域 CAS、PG/Memory registry） +4. [x] V2 文档接入（persistAgentCommit / ApproverApprove / freeform publish） +5. [x] V3 助手接入（同步 chat + task finalize） +6. [x] V3.5 回填（只枚举账本/完成 turn） +7. [x] V4 降级读取（`FAV_DEGRADED_READ` 默认关） +8. [x] V5 恢复（scratch-only CLI） +9. [x] V6 门禁与三份演练报告（live e2e 待 Unit A） + +## 相关链接 + +- ADR: [../../adr/2026-09-14-final-artifact-vault.md](../../adr/2026-09-14-final-artifact-vault.md) +- Remediation: [../../remediation/2026-09-14-final-artifact-vault-remediation.md](../../remediation/2026-09-14-final-artifact-vault-remediation.md) +- Test report: [../../test-reports/2026-09-14-final-artifact-vault-test-report.md](../../test-reports/2026-09-14-final-artifact-vault-test-report.md) + +## 证据 / 备注 + +- 分支：`feature/final-artifact-vault` +- 未执行 `env-service` rebuild；`FAV_DEGRADED_READ` 默认 false +- Goose：`00034_2026-09-14-final-artifact-vault.sql` diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index a3aa5928..bfb9eb12 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -71,6 +71,7 @@ | DOCREV-ACTIVE-DOC-ROUTING | UI-05「当前受控文档」被法规关键词改路由到 `feasibility_reg`，intake 无 pending | **PASS** | [audit](../audit/2026-09-13-document-revision-active-doc-routing.md)；Round20 UI-05 pending 停在 intake V2 | **不改弱 E2E**；不删证据 | | DOCREV-QUALITY-STATUS-FP | 行动计划表状态列「待续」被门禁当敷衍，两轮 repair 后 quality_failed 80 | **PASS** | [audit](../audit/2026-09-13-document-revision-quality-status-false-positive.md)；Round20 UI-03 | **不删禁词、不放宽正文**；不删证据 | | GOVERNANCE-ACTION-AUTHZ | 文档动作按职责鉴权；未激活 202-blocked；关闭创建直通 | **PASS** | [test-report](../test-reports/2026-09-14-governance-action-level-authz-test-report.md)；[audit 异态](../audit/2026-09-14-planning-with-workflow-key-anomaly.md) | 不改 83 条 planning+key 夹具；none 长 PI 审批链另开 | +| FAV-V0-V6-IMPL | 终态成果保险库 V0–V6 内核与挂钩 | **PARTIAL** | [test-report](../test-reports/2026-09-14-final-artifact-vault-test-report.md)；[ADR](../adr/2026-09-14-final-artifact-vault.md)。goose 00034；`FAV_DEGRADED_READ` 默认 false | 不替代 Unit A/B；live 6173/PITR/corpus 演练另占槽。禁止在错挂数据根上开降级读 | --- diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index 75867747..b6a1264b 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -23,7 +23,7 @@ | IAM / 鉴权 | 项目级 API 鉴权 + RBAC | partial | [audit/authorization](../audit/authorization-matrix-v2.md)；[刷新恢复专项审计](../audit/2026-09-13-chat-history-refresh-disappears.md) | 历史读取端点补统一会话 owner + 阶段授权；旧 API 410 标注(BUG-11 partial) | `authz` | | 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；当前 dev 因数据根 split-brain 再次不可读。Unit A 纠挂后标 incident-mitigated；`done` 等 Unit B 负例，不等 outbox | `docgov` | | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | -| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | planned | [调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md) | 方案完成并经安全/可行性/对抗复核；尚未实现独立 storage identity、不可变 staging/outbox、项目作用域加密 CAS、manifest/receipt、授权降级读取和 scratch 恢复工具 | `artifact-vault` | +| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | partial | [ADR](../adr/2026-09-14-final-artifact-vault.md)；[调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md)；[单测报告](../test-reports/2026-09-14-final-artifact-vault-test-report.md) | V0–V6 内核/挂钩/CLI 已落地（goose 00034）。现网降级读默认关；6173 live 演练与 corpus/PITR 待 Unit A 后另占槽 | `artifact-vault` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | | 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus 为 sibling，不挡事故关闭 | `infra` | diff --git a/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md new file mode 100644 index 00000000..728f8e10 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md @@ -0,0 +1,33 @@ +# FAV 演练：migration corpus（独立报告） + +## 一、 测试背景 (Background) + +Migration corpus / 蓝绿换库是 FAV 的 sibling，不在本计划实现。V6 只能引用其 PASS，不能用 FAV 覆盖率手改数字绕过。 + +## 二、 测试目标 (Test Objectives) + +- `PreflightDestroy(..., corpusOK=false)` fail-closed。 +- 回填只枚举账本/完成 turn，禁止 workspace。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +`TestCoverageGateFailClosed` + `TestBackfillIdempotentNoWorkspace`。无 corpus 目录实跑。 + +## 四、 预期结果 (Expected Results) + +corpus 未 PASS → destroy 拒绝；缺证据行 → quarantine；重跑 Created 不增加。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `PreflightDestroy(..., backupOK=true, corpusOK=false)` 逻辑分支；`Backfill` 重复行与 missing evidence +- **实际输出**: destroy 拒绝；首次 Created=1 Reused=1；rerun Reused；missing → Quarantine=1 +- **评价判定**: PASS（门禁与回填契约）；**未做** live corpus 晋升 +- **性能与耗时**: <5ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +`TestBackfillIdempotentNoWorkspace` PASS；`TestCoverageGateFailClosed` PASS。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +`scripts/artifact-vault-preflight.sh` 默认 `CORPUS_OK=false`，避免手改 coverage 后 teardown。corpus 作业 PASS 后才能把该旗标打开。 diff --git a/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md new file mode 100644 index 00000000..8cef1762 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md @@ -0,0 +1,33 @@ +# FAV 演练：PostgreSQL restore（独立报告） + +## 一、 测试背景 (Background) + +FAV 不替代完整 DB backup/PITR。本报告只验证「恢复演练门禁」：coverage 未 PASS 或 complementary backup 未声明时 destroy/teardown fail-closed。 + +## 二、 测试目标 (Test Objectives) + +- `PreflightDestroy` 在 backup 未 PASS 时拒绝。 +- 不把 FAV 单测冒充为已完成的 PostgreSQL PITR 演练。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +本机 `go test`；**未**对 `agent-dev` 做 pg_dump/restore（集成槽占用 + 数据根 split-brain）。命令：`TestCoverageGateFailClosed`。 + +## 四、 预期结果 (Expected Results) + +`Coverage{Required:2, Verified:1}` + backupOK=true 仍因 coverage 失败而拒绝；backupOK=false 亦拒绝。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `PreflightDestroy(Coverage{Required: 2, Verified: 1}, true, true, true)` +- **实际输出**: `artifactvault: coverage gate failed required=2 verified=1 quarantine=0` +- **评价判定**: PASS（门禁）；**FAIL / 未做** live PostgreSQL restore +- **性能与耗时**: <1ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +见 [assets/2026-09-14-final-artifact-vault/unit-tests.txt](assets/2026-09-14-final-artifact-vault/unit-tests.txt) 中 `TestCoverageGateFailClosed`。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +门禁函数可用。完整 PostgreSQL restore 演练依赖独立 backup 作业与 Unit A 后的干净数据根，不得在本报告勾 PASS。 diff --git a/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md new file mode 100644 index 00000000..1f75cd99 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md @@ -0,0 +1,37 @@ +# FAV 演练：scratch 恢复（独立报告） + +## 一、 测试背景 (Background) + +方案要求恢复目标必须是 scratch identity，默认 `recover-as-collection`，union-deny 删除，禁止写回 dev/test/main。 + +## 二、 测试目标 (Test Objectives) + +- `ValidateRecoverTarget` 拒绝 `agent_system_dev/test/main`。 +- relink 在冲突/缺授权时失败。 +- union-deny tombstone 后 `DegradedPayload` 拒绝。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +内存 registry + tempfile vault。`cmd/artifact-vault recover-plan/apply` 仅编译，未对现网 DSN 执行 apply。 + +## 四、 预期结果 (Expected Results) + +operational DSN 拒绝；scratch 计划通过；denied/relink+conflict 失败；tombstone 后不可读。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `TestRecoverRejectsOperationalDSN`、`TestRecoverPlanGates`、`TestTombstoneUnionDeny` +- **实际输出**: + - `postgres://x/agent_system_dev` → refuse + - `postgres://x/fav_scratch` → accept + - union-deny → degraded read error +- **评价判定**: PASS（计划校验）；live scratch 库 apply **未执行**（避免误连 operational） +- **性能与耗时**: <10ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +见 unit-tests.txt。`go test -c ./cmd/artifact-vault` 编译通过。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +恢复工具默认 scratch-only。项目 6173 等价「终态→发布→清 workspace→降级读→scratch 恢复→app rebuild」需 Unit A 后另占槽，不得声称已拦截任意 `rm`。 diff --git a/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md new file mode 100644 index 00000000..e6fa66df --- /dev/null +++ b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md @@ -0,0 +1,87 @@ +# 最终成果保险库 V0–V6 测试报告 + +## 一、 测试背景 (Background) + +运行库与 workspace 投影会一起丢失或错挂。已审查的 FAV 方案要求只归档数据库终态事实，独立 `VAULT_STATE_ROOT`，禁止扫描 `output.md`，GET 零写入，恢复只进 scratch。本轮按落地计划实现 V0–V6 内核与领域接入；现网 Unit A 纠挂未完成，故不对 `agent-dev` 开降级读或容器 rebuild。 + +## 二、 测试目标 (Test Objectives) + +- 准入表拒绝非终态、未知类型、敏感正文。 +- approved 无 `review_snapshot_id` 不得入库/批准。 +- task finalize 重算 `export_policy`/`content_sha256` 并 `CompleteChatTurn`。 +- publisher 幂等、拒绝 `data/dev` 根、跨项目同 SHA 分 key、tombstone union-deny。 +- 回填缺证据 quarantine；重跑不新增逻辑成果。 +- recover 拒绝 operational DSN；coverage/destroy 门禁 fail-closed。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +| 项 | 值 | +|----|-----| +| OS | Linux 7.0.0-31-generic | +| Go | `go test` 本机 | +| 后端 | 单元/内存 registry + tempfile vault；**未**使用 stub 作为终态验收替代，也**未**对现网 `pi-agent` 做 6173 演练 | +| 分支 | `feature/final-artifact-vault` | +| worktree | `.worktrees/fix-dev-runtime-data-root-guard` | +| 集成槽 | 未占用；禁止 `env-service` rebuild | +| 证据 | [assets/2026-09-14-final-artifact-vault/unit-tests.txt](assets/2026-09-14-final-artifact-vault/unit-tests.txt) | + +## 四、 预期结果 (Expected Results) + +| 用例 | 判定 | +|------|------| +| approved 无 snapshot | 拒绝 | +| task 无 completed turn | 拒绝 | +| 显式 pin 非终态 | `ErrExplicitBypass` | +| secret payload | 不得进 staging | +| 重复 Request/Drain | 同一 UUID，verified=1 | +| recover operational DSN | 拒绝 | +| coverage 不完整 | destroy 拒绝 | + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +### 1. 准入与表征 + +- **测试输入**: `go test -v ./internal/artifactvault/ ./internal/docgov/ -run 'TestEvaluateEligibility|TestApproverApprove|TestApprovedHook'` +- **实际输出**: 全部 `--- PASS`（见 assets） +- **评价判定**: PASS +- **性能与耗时**: <10ms / 包 + +### 2. Publisher / 文件系统 + +- **测试输入**: `TestPublisherIdempotentAndCrashSafe`、`TestFilesystemRefusesRuntimeDataDev`、`TestTombstoneUnionDeny`、`TestLastReferencePurgeAndLegalHold` +- **实际输出**: PASS；`data/dev` 根被拒绝 +- **评价判定**: PASS +- **性能与耗时**: <10ms + +### 3. 领域 finalize + +- **测试输入**: `TestFinalizeAssistantTaskMessageRecalculatesExportColumns`、`TestCompleteChatTurnAfterFinalize`、`TestFinalizeTaskRunCompletesTurnAndIngests` +- **实际输出**: PASS +- **评价判定**: PASS +- **性能与耗时**: server 用例 5ms + +### 4. 回填与恢复计划 + +- **测试输入**: `TestBackfillIdempotentNoWorkspace`、`TestRecoverRejectsOperationalDSN`、`TestRecoverPlanGates`、`TestCoverageGateFailClosed` +- **实际输出**: PASS +- **评价判定**: PASS +- **性能与耗时**: <10ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +``` +ok chat-agent-system/internal/artifactvault 0.005s +ok chat-agent-system/internal/docgov 0.003s +ok chat-agent-system/internal/ctxmgr 0.003s +ok chat-agent-system/internal/server 0.005s (TestFinalizeTaskRunCompletesTurnAndIngests) +``` + +未跑 `go test ./...` 全量（含真实 PI E2E / 需容器的套件）；本波次范围是 FAV 包与挂钩单测。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +V0–V6 **代码与单测已落地**，可作为合 `dev` 前的内核证据。下列不得写成已关闭： + +- 现网 `FAV_DEGRADED_READ` 必须保持 false，直到 Unit A 纠挂。 +- 项目 6173 等价脱敏 corpus 与三条 live 演练见独立报告，当前为 **preflight / 单测等价**，不是容器 e2e PASS。 +- Goose FAV 表为 **00034**（00033 已被 task-run 治理占用）。 diff --git a/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt new file mode 100644 index 00000000..f6f0b255 --- /dev/null +++ b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt @@ -0,0 +1,49 @@ +=== RUN TestBackfillIdempotentNoWorkspace +--- PASS: TestBackfillIdempotentNoWorkspace (0.00s) +=== RUN TestEvaluateEligibility_Allowlist +--- PASS: TestEvaluateEligibility_Allowlist (0.00s) +=== RUN TestEvaluateEligibility_ApprovedRequiresReviewSnapshot +--- PASS: TestEvaluateEligibility_ApprovedRequiresReviewSnapshot (0.00s) +=== RUN TestEvaluateEligibility_TaskWithoutCompletedTurn +--- PASS: TestEvaluateEligibility_TaskWithoutCompletedTurn (0.00s) +=== RUN TestEvaluateEligibility_SensitiveRejectedBeforeStaging +--- PASS: TestEvaluateEligibility_SensitiveRejectedBeforeStaging (0.00s) +=== RUN TestEvaluateEligibility_UnknownTypeRejected +--- PASS: TestEvaluateEligibility_UnknownTypeRejected (0.00s) +=== RUN TestIdempotencyKeyStable +--- PASS: TestIdempotencyKeyStable (0.00s) +=== RUN TestIngestIdempotentAndRejectsNonTerminal +--- PASS: TestIngestIdempotentAndRejectsNonTerminal (0.00s) +=== RUN TestCoverageGateFailClosed +--- PASS: TestCoverageGateFailClosed (0.00s) +=== RUN TestPublisherIdempotentAndCrashSafe +--- PASS: TestPublisherIdempotentAndCrashSafe (0.00s) +=== RUN TestPublisherRejectsBadSHAAndCrossProjectKeys +--- PASS: TestPublisherRejectsBadSHAAndCrossProjectKeys (0.00s) +=== RUN TestFilesystemRefusesRuntimeDataDev +--- PASS: TestFilesystemRefusesRuntimeDataDev (0.00s) +=== RUN TestLastReferencePurgeAndLegalHold +--- PASS: TestLastReferencePurgeAndLegalHold (0.00s) +=== RUN TestRecoverRejectsOperationalDSN +--- PASS: TestRecoverRejectsOperationalDSN (0.00s) +=== RUN TestRecoverPlanGates +--- PASS: TestRecoverPlanGates (0.00s) +PASS +ok chat-agent-system/internal/artifactvault 0.005s +=== RUN TestApproverApproveRequiresFrozenReviewSnapshot +--- PASS: TestApproverApproveRequiresFrozenReviewSnapshot (0.00s) +=== RUN TestApprovedHookDoesNotRequireWorkspaceScan +--- PASS: TestApprovedHookDoesNotRequireWorkspaceScan (0.00s) +PASS +ok chat-agent-system/internal/docgov 0.003s +=== RUN TestFinalizeAssistantTaskMessageRecalculatesExportColumns +--- PASS: TestFinalizeAssistantTaskMessageRecalculatesExportColumns (0.00s) +=== RUN TestCompleteChatTurnAfterFinalize +--- PASS: TestCompleteChatTurnAfterFinalize (0.00s) +PASS +ok chat-agent-system/internal/ctxmgr 0.003s +=== RUN TestFinalizeTaskRunCompletesTurnAndIngests +2026/09/14 10:49:49 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestFinalizeTaskRunCompletesTurnAndIngests (0.00s) +PASS +ok chat-agent-system/internal/server 0.005s From ad9b67d599d1bb95a724b8b95e0c070a9bd41ada Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 12:52:28 +0000 Subject: [PATCH 12/13] feat(artifactvault): close same-tx staging, recover-apply, and GET zero-write gaps Approve/commit now write FAV staging in the domain transaction, recover-apply materializes scratch collections, and unpublished or GET paths cannot mutate the registry. Co-authored-by: Cursor --- cmd/artifact-vault/main.go | 61 ++++++++++- internal/artifactvault/eligibility_test.go | 20 ++++ internal/artifactvault/filesystem_store.go | 106 +++++++++++++++++-- internal/artifactvault/ingest.go | 59 +++++++++++ internal/artifactvault/publisher.go | 2 +- internal/artifactvault/publisher_test.go | 116 +++++++++++++++++++++ internal/artifactvault/recovery.go | 72 +++++++++++++ internal/artifactvault/recovery_test.go | 40 ++++++- internal/config/config.go | 21 ++++ internal/config/config_test.go | 12 +++ internal/ctxmgr/chat_turns.go | 46 ++++++++ internal/docgov/engine.go | 74 +++++++++++-- internal/docrevision/pg_mutate.go | 25 ++++- internal/docrevision/service.go | 5 +- internal/server/reply_export_api.go | 78 ++++++++++++++ internal/server/reply_export_api_test.go | 43 ++++++++ internal/server/server.go | 1 + internal/server/vault.go | 22 ++++ internal/server/vault_nowrite_test.go | 60 +++++++++++ 19 files changed, 838 insertions(+), 25 deletions(-) create mode 100644 internal/server/vault_nowrite_test.go diff --git a/cmd/artifact-vault/main.go b/cmd/artifact-vault/main.go index b81e57fe..59cc1454 100644 --- a/cmd/artifact-vault/main.go +++ b/cmd/artifact-vault/main.go @@ -16,7 +16,7 @@ import ( func main() { if len(os.Args) < 2 { - fmt.Fprintln(os.Stderr, "usage: artifact-vault [flags]") + fmt.Fprintln(os.Stderr, "usage: artifact-vault [flags]") os.Exit(2) } cmd := os.Args[1] @@ -34,6 +34,8 @@ func main() { runRecoverApply() case "gc-preflight": runGCPreflight() + case "destroy": + runDestroy() default: fmt.Fprintln(os.Stderr, "unknown command:", cmd) os.Exit(2) @@ -115,17 +117,68 @@ func runRecoverPlan() { } func runRecoverApply() { + sourceRoot := flag.String("source-root", os.Getenv("VAULT_STATE_ROOT"), "vault root") + destDir := flag.String("dest-dir", "", "scratch collection directory") target := flag.String("target-dsn", "", "scratch DSN") ident := flag.String("target-identity", "", "scratch identity") + sourceIdent := flag.String("identity", os.Getenv("VAULT_STORAGE_IDENTITY"), "source vault identity") confirm := flag.String("confirm", "", "must equal plan hash") flag.Parse() - if !strings.Contains(strings.ToLower(*target+" "+*ident), "scratch") { - fatal(fmt.Errorf("refuse operational recover target")) + if err := artifactvault.ValidateRecoverTarget(*target, *ident); err != nil { + fatal(err) } if *confirm == "" { fatal(fmt.Errorf("confirm plan hash required")) } - writeJSON(map[string]string{"status": "planned", "mode": "recover-as-collection", "note": "apply writes only to scratch collection"}) + if *sourceRoot == "" || *sourceIdent == "" { + fatal(fmt.Errorf("source-root and identity required")) + } + plan := artifactvault.RecoverPlan{ + Mode: artifactvault.RecoverAsCollection, TargetIdentity: *target, SourceIdentity: *ident, + PlanHash: *confirm, HasAdmin: true, ScratchOnly: true, + } + store, err := artifactvault.NewFilesystemStore(*sourceRoot, *sourceIdent, []byte(nz(os.Getenv("VAULT_MASTER_KEY"), "0123456789abcdef0123456789abcdef"))) + if err != nil { + fatal(err) + } + arts, payloads, err := store.LoadCommittedArtifacts() + if err != nil { + fatal(err) + } + denied := map[string]bool{} + for _, art := range arts { + if art.PublicationStatus == artifactvault.StatusTombstoned || store.HasTombstone(art.UUID) { + denied[art.UUID] = true + } + } + out, err := artifactvault.ApplyRecoverAsCollection(plan, *destDir, arts, payloads, denied) + if err != nil { + fatal(err) + } + writeJSON(out) +} + +func runDestroy() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + backupOK := flag.Bool("backup-ok", false, "complementary backup PASS") + corpusOK := flag.Bool("corpus-ok", false, "migration corpus PASS") + confirm := flag.String("confirm", "", "must equal DESTROY") + flag.Parse() + if *confirm != "DESTROY" { + fatal(fmt.Errorf("destroy requires --confirm=DESTROY")) + } + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + if err := artifactvault.PreflightDestroy(reg.Coverage(ctx), *backupOK, *corpusOK, true); err != nil { + fatal(err) + } + writeJSON(map[string]string{"status": "preflight-pass", "note": "dataset wipe is operator-owned after complementary PASS"}) } func runGCPreflight() { diff --git a/internal/artifactvault/eligibility_test.go b/internal/artifactvault/eligibility_test.go index 04121980..fbb410a3 100644 --- a/internal/artifactvault/eligibility_test.go +++ b/internal/artifactvault/eligibility_test.go @@ -64,6 +64,26 @@ func TestEvaluateEligibility_SensitiveRejectedBeforeStaging(t *testing.T) { } } +func TestEvaluateEligibility_CopyOnlyRequiresExplicitPinAfterCompleted(t *testing.T) { + now := time.Now().UTC() + base := Facts{ + ArtifactType: TypeAssistantReply, ChatTurnStatus: "completed", CompletedAt: &now, + AssistantMessageID: 9, ExportPolicy: "copy_only", Classification: ClassificationOK, + Payload: []byte("ok, noted."), + } + if err := EvaluateEligibility(base); err == nil { + t.Fatal("copy_only without pin must reject") + } + base.ExplicitPin = true + if err := EvaluateEligibility(base); err != ErrExplicitBypass { + t.Fatalf("pin without PinAfterCompleted: %v", err) + } + base.PinAfterCompleted = true + if err := EvaluateEligibility(base); err != nil { + t.Fatalf("copy_only + pin after completed should pass: %v", err) + } +} + func TestEvaluateEligibility_UnknownTypeRejected(t *testing.T) { if err := EvaluateEligibility(Facts{ArtifactType: "workspace_scan", Classification: ClassificationOK, Payload: []byte("x")}); err != ErrUnknownArtifactType { t.Fatalf("got %v", err) diff --git a/internal/artifactvault/filesystem_store.go b/internal/artifactvault/filesystem_store.go index 9ba946e5..3b47e84f 100644 --- a/internal/artifactvault/filesystem_store.go +++ b/internal/artifactvault/filesystem_store.go @@ -63,10 +63,7 @@ func (s *FilesystemStore) PutBlob(env, projectKey, sha string, plain []byte) (ke if _, err := os.Stat(path); err == nil { return ref, nil } - if err := fsutil.WriteFileAtomicExclusive(path, envBytes, 0o600); err != nil { - if strings.Contains(err.Error(), "refuse overwrite") { - return ref, nil - } + if err := writeExclusiveOK(path, envBytes, 0o600); err != nil { return "", err } return ref, nil @@ -91,14 +88,111 @@ func (s *FilesystemStore) GetBlob(env, projectKey, sha string) ([]byte, error) { return plain, nil } +func writeExclusiveOK(path string, raw []byte, perm os.FileMode) error { + if err := fsutil.WriteFileAtomicExclusive(path, raw, perm); err != nil { + if strings.Contains(err.Error(), "refuse overwrite") { + return nil + } + return err + } + return nil +} + func (s *FilesystemStore) PutManifest(artifactUUID, manifestSHA string, raw []byte) error { path := filepath.Join(s.Root, "manifests", artifactUUID, manifestSHA+".json") - return fsutil.WriteFileAtomicExclusive(path, raw, 0o644) + return writeExclusiveOK(path, raw, 0o644) } func (s *FilesystemStore) PutCommit(artifactUUID, manifestSHA string, raw []byte) error { path := filepath.Join(s.Root, "commits", artifactUUID, manifestSHA+".json") - return fsutil.WriteFileAtomicExclusive(path, raw, 0o644) + return writeExclusiveOK(path, raw, 0o644) +} + +func (s *FilesystemStore) ListCommitted() ([]string, error) { + dir := filepath.Join(s.Root, "commits") + ents, err := os.ReadDir(dir) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, err + } + out := make([]string, 0, len(ents)) + for _, e := range ents { + if e.IsDir() { + out = append(out, e.Name()) + } + } + return out, nil +} + +func (s *FilesystemStore) LoadCommittedArtifacts() ([]Artifact, map[string][]byte, error) { + uuids, err := s.ListCommitted() + if err != nil { + return nil, nil, err + } + payloads := make(map[string][]byte) + arts := make([]Artifact, 0, len(uuids)) + for _, uuid := range uuids { + if s.HasTombstone(uuid) { + arts = append(arts, Artifact{UUID: uuid, PublicationStatus: StatusTombstoned}) + continue + } + art, body, err := s.loadCommitted(uuid) + if err != nil { + return nil, nil, err + } + arts = append(arts, art) + payloads[uuid] = body + } + return arts, payloads, nil +} + +func (s *FilesystemStore) loadCommitted(uuid string) (Artifact, []byte, error) { + dir := filepath.Join(s.Root, "commits", uuid) + ents, err := os.ReadDir(dir) + if err != nil || len(ents) == 0 { + return Artifact{}, nil, fmt.Errorf("artifactvault: unpublished") + } + raw, err := os.ReadFile(filepath.Join(dir, ents[0].Name())) + if err != nil { + return Artifact{}, nil, err + } + var marker CommitMarker + if err := json.Unmarshal(raw, &marker); err != nil { + return Artifact{}, nil, err + } + manRaw, err := os.ReadFile(filepath.Join(s.Root, "manifests", uuid, marker.ManifestSHA256+".json")) + if err != nil { + return Artifact{}, nil, err + } + var man Manifest + if err := json.Unmarshal(manRaw, &man); err != nil { + return Artifact{}, nil, err + } + if len(man.PayloadMembers) == 0 { + return Artifact{}, nil, fmt.Errorf("artifactvault: manifest has no payload members") + } + body, err := s.GetBlob(man.Environment, man.ProjectExternalKey, man.PayloadMembers[0].SHA256) + if err != nil { + return Artifact{}, nil, err + } + var snap, msg int64 + if man.SourceSnapshotID != nil { + snap = *man.SourceSnapshotID + } + if man.SourceMessageID != nil { + msg = *man.SourceMessageID + } + return Artifact{ + UUID: uuid, ArtifactType: man.ArtifactType, LogicalResourceID: man.LogicalResourceID, + Environment: man.Environment, ProjectID: man.ProjectID, ProjectExternalKey: man.ProjectExternalKey, + SourceEventID: man.SourceEventID, SourceEventKind: man.SourceEventKind, + SourceSnapshotID: snap, SourceMessageID: msg, SourceRunID: man.SourceRunID, + PayloadSHA256: man.PayloadMembers[0].SHA256, ManifestSHA256: man.ManifestSHA256, + VaultStorageIdentity: man.VaultStorageIdentity, TerminalState: man.TerminalEvidence.State, + PublicationStatus: StatusVerified, RetentionClass: man.RetentionClass, + }, body, nil } func (s *FilesystemStore) HasCommit(artifactUUID, manifestSHA string) bool { diff --git a/internal/artifactvault/ingest.go b/internal/artifactvault/ingest.go index e228675c..1b39230d 100644 --- a/internal/artifactvault/ingest.go +++ b/internal/artifactvault/ingest.go @@ -5,6 +5,8 @@ import ( "fmt" "strconv" "time" + + "github.com/jackc/pgx/v5" ) // Ingest is the single write path from domain terminals into staging/outbox. @@ -120,6 +122,52 @@ func (i *Ingest) FreeformPublish(ctx context.Context, projectID int64, docID, ev }) } +func (i *Ingest) requestTX(ctx context.Context, tx pgx.Tx, facts Facts, req PublishRequest) error { + if i == nil { + return nil + } + if req.Environment == "" { + req.Environment = i.Env + } + if req.ProjectExternalKey == "" { + req.ProjectExternalKey = ProjectKeyFromID(req.ProjectID) + } + if req.PolicyVersion == "" { + req.PolicyVersion = PolicyVersion + } + if req.Classification == "" { + req.Classification = ClassifyPayload(req.Payload) + } + facts.Payload = req.Payload + facts.Classification = req.Classification + _, _, err := RequestWithTx(ctx, tx, facts, req) + return err +} + +func (i *Ingest) AgentCommitTX(ctx context.Context, tx pgx.Tx, projectID int64, docID, eventID string, snapshotID int64, payload []byte) error { + return i.requestTX(ctx, tx, AgentCommitFacts(snapshotID), PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID, SourceEventKind: "agent_commit", SourceEventID: eventID, + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "committed", + }) +} + +func (i *Ingest) SealedVersionTX(ctx context.Context, tx pgx.Tx, projectID int64, docID, eventID string, snapshotID int64, sealedAt *time.Time, payload []byte) error { + return i.requestTX(ctx, tx, SealedFacts(sealedAt, snapshotID), PublishRequest{ + ArtifactType: TypeSealedVersion, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID + ":sealed", SourceEventKind: "version_sealed", SourceEventID: eventID, + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "sealed", + }) +} + +func (i *Ingest) ApprovedTX(ctx context.Context, tx pgx.Tx, projectID int64, docID string, snapshotID int64, approvedAt *time.Time, payload []byte) error { + return i.requestTX(ctx, tx, ApprovedFacts("approved", approvedAt, snapshotID), PublishRequest{ + ArtifactType: TypeApprovedDocument, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: docID, SourceEventKind: "approved", SourceEventID: fmt.Sprintf("approved:%s:%d", docID, snapshotID), + SourceSnapshotID: snapshotID, Payload: payload, TerminalState: "approved", + }) +} + func (i *Ingest) AssistantReply(ctx context.Context, projectID int64, logical, eventID string, msgID int64, runID, policy string, completedAt *time.Time, payload []byte) error { return i.Request(ctx, AssistantFacts("completed", completedAt, msgID, policy), PublishRequest{ ArtifactType: TypeAssistantReply, Environment: i.Env, ProjectID: projectID, @@ -127,3 +175,14 @@ func (i *Ingest) AssistantReply(ctx context.Context, projectID int64, logical, e SourceMessageID: msgID, SourceRunID: runID, Payload: payload, TerminalState: "completed", }) } + +func (i *Ingest) AssistantReplyPin(ctx context.Context, projectID int64, logical, eventID string, msgID int64, runID, policy string, completedAt *time.Time, payload []byte) error { + facts := AssistantFacts("completed", completedAt, msgID, policy) + facts.ExplicitPin = true + facts.PinAfterCompleted = true + return i.Request(ctx, facts, PublishRequest{ + ArtifactType: TypeAssistantReply, Environment: i.Env, ProjectID: projectID, + LogicalResourceID: logical, SourceEventKind: "assistant_pin", SourceEventID: eventID, + SourceMessageID: msgID, SourceRunID: runID, Payload: payload, TerminalState: "completed", + }) +} diff --git a/internal/artifactvault/publisher.go b/internal/artifactvault/publisher.go index 92683cec..bed03099 100644 --- a/internal/artifactvault/publisher.go +++ b/internal/artifactvault/publisher.go @@ -87,7 +87,7 @@ func (p *Publisher) DegradedPayload(ctx context.Context, env, logical, projectKe if p.Reg.IsDenied(ctx, art.UUID) || p.Store.HasTombstone(art.UUID) { return nil, "", fmt.Errorf("artifactvault: union-deny") } - if !p.Store.HasCommit(art.UUID, art.ManifestSHA256) && art.ManifestSHA256 != "" { + if art.ManifestSHA256 == "" || !p.Store.HasCommit(art.UUID, art.ManifestSHA256) { return nil, "", fmt.Errorf("artifactvault: unpublished") } b, err := p.Store.GetBlob(env, projectKey, art.PayloadSHA256) diff --git a/internal/artifactvault/publisher_test.go b/internal/artifactvault/publisher_test.go index 2e05a3da..b33b1758 100644 --- a/internal/artifactvault/publisher_test.go +++ b/internal/artifactvault/publisher_test.go @@ -147,4 +147,120 @@ func TestLastReferencePurgeAndLegalHold(t *testing.T) { } } +func TestFilesystemIdentityMismatch(t *testing.T) { + dir := t.TempDir() + key := []byte("0123456789abcdef0123456789abcdef") + if _, err := NewFilesystemStore(dir, "id-a", key); err != nil { + t.Fatal(err) + } + if _, err := NewFilesystemStore(dir, "id-b", key); err == nil { + t.Fatal("identity mismatch must fail closed") + } +} + +func TestUnpublishedWithoutCommitMarkerInvisible(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + facts := Facts{ + ArtifactType: TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 4, + ChangeKind: "agent_commit", ChangeToSnapshotID: 4, Classification: ClassificationOK, + Payload: []byte("# pending\n"), + } + req := PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "proj-a", + LogicalResourceID: "doc_pending", SourceEventKind: "agent_commit", SourceEventID: "ev-pending", + Payload: []byte("# pending\n"), TerminalState: "committed", + } + if _, _, err := reg.Request(ctx, facts, req); err != nil { + t.Fatal(err) + } + if _, _, err := pub.DegradedPayload(ctx, "dev", "doc_pending", "proj-a"); err == nil { + t.Fatal("requested but unpublished must be invisible") + } +} + +func TestPublisherRetriesAfterMissingCommit(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() + store := testStore(t) + pub := &Publisher{Reg: reg, Store: store} + payload := []byte("# retry-me\n") + facts := Facts{ + ArtifactType: TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 5, + ChangeKind: "agent_commit", ChangeToSnapshotID: 5, Classification: ClassificationOK, + Payload: payload, + } + req := PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "proj-a", + LogicalResourceID: "doc_retry", SourceEventKind: "agent_commit", SourceEventID: "ev-retry", + Payload: payload, TerminalState: "committed", + } + art, _, err := reg.Request(ctx, facts, req) + if err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + if err := os.RemoveAll(filepath.Join(store.Root, "commits", art.UUID)); err != nil { + t.Fatal(err) + } + if _, _, err := pub.DegradedPayload(ctx, "dev", "doc_retry", "proj-a"); err == nil { + t.Fatal("missing commit marker must hide payload") + } + if err := reg.MarkFailed(ctx, art.UUID, "commit_missing"); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + b, _, err := pub.DegradedPayload(ctx, "dev", "doc_retry", "proj-a") + if err != nil || string(b) != string(payload) { + t.Fatalf("retry after exclusive blob/manifest: %v %q", err, b) + } +} + +type requestCountingReg struct { + Registry + requests int +} + +func (r *requestCountingReg) Request(ctx context.Context, facts Facts, req PublishRequest) (*Artifact, *Staging, error) { + r.requests++ + return r.Registry.Request(ctx, facts, req) +} + +func TestDegradedPayloadDoesNotWrite(t *testing.T) { + ctx := context.Background() + mem := NewMemoryRegistry() + count := &requestCountingReg{Registry: mem} + store := testStore(t) + pub := &Publisher{Reg: count, Store: store} + facts := Facts{ + ArtifactType: TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 6, + ChangeKind: "agent_commit", ChangeToSnapshotID: 6, Classification: ClassificationOK, + Payload: []byte("# get\n"), + } + req := PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "proj-a", + LogicalResourceID: "doc_get", SourceEventKind: "agent_commit", SourceEventID: "ev-get", + Payload: []byte("# get\n"), TerminalState: "committed", + } + if _, _, err := mem.Request(ctx, facts, req); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + count.requests = 0 + if _, _, err := pub.DegradedPayload(ctx, "dev", "doc_get", "proj-a"); err != nil { + t.Fatal(err) + } + if count.requests != 0 { + t.Fatalf("GET/degraded read wrote Request %d times", count.requests) + } +} + func ptrTime() *time.Time { t := time.Now().UTC(); return &t } diff --git a/internal/artifactvault/recovery.go b/internal/artifactvault/recovery.go index bec72832..f47a3b12 100644 --- a/internal/artifactvault/recovery.go +++ b/internal/artifactvault/recovery.go @@ -1,7 +1,10 @@ package artifactvault import ( + "encoding/json" "fmt" + "os" + "path/filepath" "strings" ) @@ -65,3 +68,72 @@ func EvaluateRecoverPlan(p RecoverPlan, denied bool) error { } return nil } + +type RecoveredItem struct { + ArtifactUUID string `json:"artifact_uuid"` + LogicalResourceID string `json:"logical_resource_id"` + PayloadSHA256 string `json:"payload_sha256"` + SkippedReason string `json:"skipped_reason,omitempty"` +} + +type RecoveredCollection struct { + Mode RecoverMode `json:"mode"` + PlanHash string `json:"plan_hash"` + DestDir string `json:"dest_dir"` + Items []RecoveredItem `json:"items"` +} + +// ApplyRecoverAsCollection writes recovered payloads into a scratch collection directory. +// It never writes operational databases or relinks live heads. +func ApplyRecoverAsCollection(plan RecoverPlan, destDir string, arts []Artifact, payloads map[string][]byte, denied map[string]bool) (*RecoveredCollection, error) { + if err := EvaluateRecoverPlan(plan, false); err != nil { + return nil, err + } + if plan.Mode != RecoverAsCollection && plan.Mode != "" { + return nil, fmt.Errorf("artifactvault: apply only supports recover-as-collection") + } + if !destIsScratch(destDir) { + return nil, fmt.Errorf("artifactvault: collection dest must be scratch") + } + if err := os.MkdirAll(filepath.Join(destDir, "payloads"), 0o700); err != nil { + return nil, err + } + out := &RecoveredCollection{Mode: RecoverAsCollection, PlanHash: plan.PlanHash, DestDir: destDir} + for _, art := range arts { + item := RecoveredItem{ArtifactUUID: art.UUID, LogicalResourceID: art.LogicalResourceID, PayloadSHA256: art.PayloadSHA256} + if denied[art.UUID] || art.PublicationStatus == StatusTombstoned { + item.SkippedReason = "union-deny" + out.Items = append(out.Items, item) + continue + } + body := payloads[art.UUID] + if len(body) == 0 || SHA256Hex(body) != art.PayloadSHA256 { + item.SkippedReason = "sha_mismatch" + out.Items = append(out.Items, item) + continue + } + path := filepath.Join(destDir, "payloads", art.UUID+".md") + if err := os.WriteFile(path, body, 0o600); err != nil { + return nil, err + } + out.Items = append(out.Items, item) + } + raw, err := json.MarshalIndent(out, "", " ") + if err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(destDir, "collection.json"), raw, 0o600); err != nil { + return nil, err + } + return out, nil +} + +func destIsScratch(destDir string) bool { + for _, part := range strings.Split(filepath.ToSlash(filepath.Clean(destDir)), "/") { + low := strings.ToLower(part) + if low == "scratch" || strings.HasPrefix(low, "scratch-") || strings.HasPrefix(low, "scratch_") { + return true + } + } + return false +} diff --git a/internal/artifactvault/recovery_test.go b/internal/artifactvault/recovery_test.go index c4152ba3..659f05a6 100644 --- a/internal/artifactvault/recovery_test.go +++ b/internal/artifactvault/recovery_test.go @@ -1,6 +1,10 @@ package artifactvault -import "testing" +import ( + "os" + "path/filepath" + "testing" +) func TestRecoverRejectsOperationalDSN(t *testing.T) { if err := ValidateRecoverTarget("postgres://x/agent_system_dev", "dev"); err == nil { @@ -25,3 +29,37 @@ func TestRecoverPlanGates(t *testing.T) { t.Fatal("relink+conflict") } } + +func TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied(t *testing.T) { + dest := filepath.Join(t.TempDir(), "scratch-collection") + plan := RecoverPlan{ + Mode: RecoverAsCollection, TargetIdentity: "scratch-db", SourceIdentity: "vault", + PlanHash: "plan-1", HasAdmin: true, ScratchOnly: true, + } + okBody := []byte("# recovered\n") + arts := []Artifact{ + {UUID: "art-ok", LogicalResourceID: "doc_ok", PayloadSHA256: SHA256Hex(okBody)}, + {UUID: "art-deny", LogicalResourceID: "doc_deny", PayloadSHA256: SHA256Hex([]byte("x")), PublicationStatus: StatusTombstoned}, + } + payloads := map[string][]byte{"art-ok": okBody} + out, err := ApplyRecoverAsCollection(plan, dest, arts, payloads, map[string]bool{"art-deny": true}) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dest, "collection.json")); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(filepath.Join(dest, "payloads", "art-ok.md")) + if err != nil || string(got) != string(okBody) { + t.Fatalf("payload: %v %q", err, got) + } + if _, err := os.Stat(filepath.Join(dest, "payloads", "art-deny.md")); err == nil { + t.Fatal("denied artifact must not be written") + } + if len(out.Items) != 2 || out.Items[1].SkippedReason != "union-deny" { + t.Fatalf("items=%+v", out.Items) + } + if _, err := ApplyRecoverAsCollection(plan, filepath.Join(t.TempDir(), "live-restore"), arts, payloads, nil); err == nil { + t.Fatal("non-scratch dest must fail") + } +} diff --git a/internal/config/config.go b/internal/config/config.go index d161fdcf..47f6070a 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -4,6 +4,7 @@ import ( "bufio" "fmt" "os" + "path/filepath" "strconv" "strings" "time" @@ -403,10 +404,30 @@ func Load(path string) (*Config, error) { if err := validateProductionSecrets(cfg); err != nil { return nil, err } + if err := ValidateIndependentVault(cfg.VaultStateRoot, cfg.DataDir); err != nil { + return nil, err + } return cfg, nil } +func ValidateIndependentVault(root, dataDir string) error { + if strings.TrimSpace(root) == "" { + return nil + } + cr := filepath.Clean(root) + if strings.HasSuffix(cr, filepath.Join("data", "dev")) || strings.HasSuffix(cr, filepath.Join("data", "test")) || strings.HasSuffix(cr, filepath.Join("data", "main")) { + return fmt.Errorf("config: VAULT_STATE_ROOT must not be runtime data root %s", root) + } + if dataDir != "" { + cd := filepath.Clean(dataDir) + if cr == cd { + return fmt.Errorf("config: VAULT_STATE_ROOT must be independent from DATA_DIR") + } + } + return nil +} + func parseCSVOrigins(raw string) []string { parts := strings.Split(raw, ",") out := make([]string, 0, len(parts)) diff --git a/internal/config/config_test.go b/internal/config/config_test.go index ec789ed0..72f62def 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -133,6 +133,18 @@ func TestConfigLoadRejectsWeakSecretsOutsideDev(t *testing.T) { } } +func TestValidateIndependentVault(t *testing.T) { + if err := config.ValidateIndependentVault("", "./data/dev"); err != nil { + t.Fatal(err) + } + if err := config.ValidateIndependentVault("./data/dev", "./data/dev"); err == nil { + t.Fatal("runtime data/dev must be rejected") + } + if err := config.ValidateIndependentVault("./data/vault/dev", "./data/dev"); err != nil { + t.Fatalf("independent vault root should pass: %v", err) + } +} + func TestConfigLoadRejectsProductionDatabasePlaceholder(t *testing.T) { t.Setenv("ENV", "prod") t.Setenv("KB_DB_PASSWORD", "kb-prod-secret-xyz") diff --git a/internal/ctxmgr/chat_turns.go b/internal/ctxmgr/chat_turns.go index 2d2c7c84..48fcd501 100644 --- a/internal/ctxmgr/chat_turns.go +++ b/internal/ctxmgr/chat_turns.go @@ -332,3 +332,49 @@ func (m *ContextManager) GetChatTurn(ctx context.Context, sessionID, clientTurnI cp := *rec return &cp, nil } + +// GetChatTurnByAssistantMessageID 按会话 + 助手消息定位回合；用于 copy_only 显式 pin。 +func (m *ContextManager) GetChatTurnByAssistantMessageID(ctx context.Context, sessionID string, assistantMessageID int64) (*ChatTurnRecord, error) { + if sessionID == "" || assistantMessageID <= 0 { + return nil, ErrChatTurnNotFound + } + if m.pool != nil { + var rec ChatTurnRecord + var completedAt *time.Time + var leaseExpiresAt *time.Time + err := m.pool.QueryRow(ctx, ` + SELECT id, session_id, client_turn_id, + COALESCE(user_id, 0), COALESCE(project_id, 0), + COALESCE(user_message_id, 0), COALESCE(assistant_message_id, 0), + request_body_sha256, status, + COALESCE(first_request_id, ''), COALESCE(last_request_id, ''), + attempt_count, created_at, updated_at, completed_at, lease_expires_at + FROM chat_turns + WHERE session_id = $1 AND assistant_message_id = $2 + ORDER BY id DESC + LIMIT 1 + `, sessionID, assistantMessageID).Scan( + &rec.ID, &rec.SessionID, &rec.ClientTurnID, + &rec.UserID, &rec.ProjectID, + &rec.UserMessageID, &rec.AssistantMessageID, + &rec.RequestBodySHA256, &rec.Status, + &rec.FirstRequestID, &rec.LastRequestID, + &rec.AttemptCount, &rec.CreatedAt, &rec.UpdatedAt, &completedAt, &leaseExpiresAt, + ) + if err != nil { + return nil, ErrChatTurnNotFound + } + rec.CompletedAt = completedAt + rec.LeaseExpiresAt = leaseExpiresAt + return &rec, nil + } + memTurnsMu.Lock() + defer memTurnsMu.Unlock() + for _, rec := range memTurns { + if rec != nil && rec.SessionID == sessionID && rec.AssistantMessageID == assistantMessageID { + cp := *rec + return &cp, nil + } + } + return nil, ErrChatTurnNotFound +} diff --git a/internal/docgov/engine.go b/internal/docgov/engine.go index d339e50d..0a77d6af 100644 --- a/internal/docgov/engine.go +++ b/internal/docgov/engine.go @@ -8,6 +8,7 @@ import ( "sync" "time" + "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) @@ -32,7 +33,8 @@ type GovernanceEngine struct { memAudits map[string][]*AuditLog onApprovedHook func(ctx context.Context, doc *DocumentRecord) onApprovedHooks []func(ctx context.Context, doc *DocumentRecord) - OnFinalArtifact func(ctx context.Context, doc *DocumentRecord, markdown []byte, snapshotID int64) + OnFinalArtifact func(ctx context.Context, doc *DocumentRecord, markdown []byte, snapshotID int64) + OnFinalArtifactTX func(ctx context.Context, tx pgx.Tx, doc *DocumentRecord, markdown []byte, snapshotID int64) error } // SetOnApprovedHook 注册文档终审批准时的回调钩子 @@ -873,11 +875,13 @@ func (e *GovernanceEngine) ApproverApprove(ctx context.Context, docID string, ac } e.memAudits[docID] = append(e.memAudits[docID], audit) - e.syncDocAndAuditToDB(ctx, doc, audit) + if err := e.persistApproveAndFAV(ctx, doc, audit); err != nil { + return nil, err + } e.mu.Unlock() e.runApprovedHooks(ctx, doc) - if e.OnFinalArtifact != nil { + if e.OnFinalArtifact != nil && (e.pool == nil || e.OnFinalArtifactTX == nil) { md, _ := e.LedgerMarkdown(ctx, doc.ReviewSnapshotID) e.OnFinalArtifact(ctx, doc, []byte(md), doc.ReviewSnapshotID) } @@ -1117,6 +1121,54 @@ func (e *GovernanceEngine) LedgerMarkdown(ctx context.Context, snapshotID int64) return md, err } +func (e *GovernanceEngine) persistApproveAndFAV(ctx context.Context, doc *DocumentRecord, audit *AuditLog) error { + if e.pool == nil { + return nil + } + if e.OnFinalArtifactTX == nil { + e.syncDocAndAuditToDB(ctx, doc, audit) + return nil + } + tx, err := e.pool.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(ctx) }() + if err := syncDocAndAuditTX(ctx, tx, doc, audit); err != nil { + return err + } + md, _ := e.LedgerMarkdown(ctx, doc.ReviewSnapshotID) + if err := e.OnFinalArtifactTX(ctx, tx, doc, []byte(md), doc.ReviewSnapshotID); err != nil { + return err + } + return tx.Commit(ctx) +} + +func syncDocAndAuditTX(ctx context.Context, tx pgx.Tx, doc *DocumentRecord, audit *AuditLog) error { + _, err := tx.Exec(ctx, ` + UPDATE document_records + SET current_state = $1, is_locked = $2, + last_rejected_reason = $3, last_rejected_by_role = $4, last_rejected_by_name = $5, last_rejected_at = $6, + approved_at = $7, updated_at = $8, review_snapshot_id = NULLIF($9, 0) + WHERE doc_id = $10 + `, string(doc.CurrentState), doc.IsLocked, + doc.LastRejectedReason, doc.LastRejectedByRole, doc.LastRejectedByName, doc.LastRejectedAt, + doc.ApprovedAt, doc.UpdatedAt, doc.ReviewSnapshotID, doc.DocID) + if err != nil { + return err + } + if audit == nil { + return nil + } + detJSON, _ := json.Marshal(audit.Details) + _, err = tx.Exec(ctx, ` + INSERT INTO document_audit_logs (doc_id, action, actor_id, actor_name, actor_role, from_state, to_state, details, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + `, audit.DocID, audit.Action, audit.ActorID, audit.ActorName, string(audit.ActorRole), + string(audit.FromState), string(audit.ToState), detJSON, audit.CreatedAt) + return err +} + func (e *GovernanceEngine) syncDocAndAuditToDB(ctx context.Context, doc *DocumentRecord, audit *AuditLog) { if e.pool == nil { return @@ -1130,13 +1182,13 @@ func (e *GovernanceEngine) syncDocAndAuditToDB(ctx context.Context, doc *Documen `, string(doc.CurrentState), doc.IsLocked, doc.LastRejectedReason, doc.LastRejectedByRole, doc.LastRejectedByName, doc.LastRejectedAt, doc.ApprovedAt, doc.UpdatedAt, doc.ReviewSnapshotID, doc.DocID) - - if audit != nil { - detJSON, _ := json.Marshal(audit.Details) - _, _ = e.pool.Exec(ctx, ` - INSERT INTO document_audit_logs (doc_id, action, actor_id, actor_name, actor_role, from_state, to_state, details, created_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) - `, audit.DocID, audit.Action, audit.ActorID, audit.ActorName, string(audit.ActorRole), - string(audit.FromState), string(audit.ToState), detJSON, audit.CreatedAt) + if audit == nil { + return } + detJSON, _ := json.Marshal(audit.Details) + _, _ = e.pool.Exec(ctx, ` + INSERT INTO document_audit_logs (doc_id, action, actor_id, actor_name, actor_role, from_state, to_state, details, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + `, audit.DocID, audit.Action, audit.ActorID, audit.ActorName, string(audit.ActorRole), + string(audit.FromState), string(audit.ToState), detJSON, audit.CreatedAt) } diff --git a/internal/docrevision/pg_mutate.go b/internal/docrevision/pg_mutate.go index 28d49f50..f29cb374 100644 --- a/internal/docrevision/pg_mutate.go +++ b/internal/docrevision/pg_mutate.go @@ -187,7 +187,30 @@ func (s *Service) persistAgentCommitLocked(ctx context.Context, root *DocRootPoi if newVer.ChangeReportID != nil { _, _ = tx.Exec(ctx, `UPDATE document_versions SET change_report_id=$2 WHERE id=$1`, vid, *newVer.ChangeReportID) } - return s.projectRootPointersTX(ctx, tx, root) + if err := s.projectRootPointersTX(ctx, tx, root); err != nil { + return err + } + if s.OnFinalArtifactTX == nil { + return nil + } + eventID := fmt.Sprintf("ev:%d", ev.ID) + outMD := []byte(nil) + if b := s.blobs[outSnap.ContentSHA256]; b != nil { + outMD = []byte(b.ContentMarkdown) + } + if err := s.OnFinalArtifactTX(ctx, tx, "agent_commit", root.DocID, root.ProjectID, outSnap.ID, eventID, outMD, nil); err != nil { + return err + } + if oldVer != nil && oldVer.SealedAt != nil { + inMD := []byte(nil) + if snap := s.snapshots[oldVer.HeadSnapshotID]; snap != nil { + if b := s.blobs[snap.ContentSHA256]; b != nil { + inMD = []byte(b.ContentMarkdown) + } + } + return s.OnFinalArtifactTX(ctx, tx, "version_sealed", root.DocID, root.ProjectID, oldVer.HeadSnapshotID, eventID+":sealed", inMD, oldVer.SealedAt) + } + return nil }) } diff --git a/internal/docrevision/service.go b/internal/docrevision/service.go index 3d1896e6..e64c0136 100644 --- a/internal/docrevision/service.go +++ b/internal/docrevision/service.go @@ -38,6 +38,9 @@ type Service struct { // OnFinalArtifact is invoked after a terminal ledger persist (agent_commit / version_sealed). OnFinalArtifact func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) + // OnFinalArtifactTX writes FAV staging/outbox inside persistAgentCommitLocked. + OnFinalArtifactTX func(ctx context.Context, tx pgx.Tx, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) error + // ImpactResolver returns deterministic downstream impacts (lineage); optional. ImpactResolver func(ctx context.Context, docID string) []DocImpact @@ -674,7 +677,7 @@ func (s *Service) CommitAgentVersion(ctx context.Context, docID string, candidat s.forgetDocLocked(docID) return nil, err } - if s.OnFinalArtifact != nil { + if s.OnFinalArtifact != nil && (!s.usingPG() || s.OnFinalArtifactTX == nil) { eventID := fmt.Sprintf("ev:%d", ev.ID) s.OnFinalArtifact(ctx, "agent_commit", c.DocID, root.ProjectID, outSnap.ID, eventID, []byte(outMD), nil) if oldVer != nil && oldVer.SealedAt != nil { diff --git a/internal/server/reply_export_api.go b/internal/server/reply_export_api.go index 3db26990..a549fbf1 100644 --- a/internal/server/reply_export_api.go +++ b/internal/server/reply_export_api.go @@ -7,6 +7,7 @@ import ( "strings" "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/ctxmgr" "chat-agent-system/internal/replyexport" ) @@ -179,6 +180,83 @@ func (s *Server) handleGetExportSource(w http.ResponseWriter, r *http.Request) { }) } +// handlePinExport archives a completed copy_only assistant reply after an explicit pin. +// POST /api/v1/chat/messages/{message_id}/pin-export +func (s *Server) handlePinExport(w http.ResponseWriter, r *http.Request) { + user, ok := s.requireAuthenticatedUser(w, r) + if !ok { + return + } + if s.svc == nil || s.svc.ContextManager() == nil || s.vaultIngest == nil { + http.Error(w, "artifact vault not initialized", http.StatusServiceUnavailable) + return + } + msgID := extractMessageIDFromPath(r, "pin-export") + if msgID <= 0 { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "INVALID_MESSAGE_ID", + "message": "无效的消息ID", + }) + return + } + msg, err := s.svc.ContextManager().GetMessageByID(r.Context(), msgID) + if err != nil || msg == nil { + writeJSONResponse(w, http.StatusNotFound, map[string]string{ + "error_code": "MESSAGE_NOT_FOUND", + "message": "指定消息不存在或已被删除", + }) + return + } + if !s.assertMessageOwnedByUser(w, r, msg, user.ID) { + return + } + if msg.Role != "assistant" { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "MESSAGE_NOT_PINNABLE", + "message": "仅助手回复可显式 pin", + }) + return + } + turn, err := s.svc.ContextManager().GetChatTurnByAssistantMessageID(r.Context(), msg.SessionID, msg.ID) + if err != nil || turn == nil || turn.Status != ctxmgr.TurnStatusCompleted || turn.CompletedAt == nil { + writeJSONResponse(w, http.StatusConflict, map[string]string{ + "error_code": "TURN_NOT_COMPLETED", + "message": "未完成回合不得 pin 进保险库", + }) + return + } + cap, okCap := msg.ExportCapability.(replyexport.MessageExportCapability) + if !okCap { + _, cap = replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: msg.Role, Status: "completed", Content: msg.Content, Revision: msg.ContentRevision, + }) + } + if cap.Policy != replyexport.PolicyCopyOnly { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "PIN_POLICY_REJECTED", + "message": "仅 copy_only 需要显式 pin；full 已自动归档，none 永不进入", + }) + return + } + eventID := fmt.Sprintf("pin:%d:%d", msg.ID, cap.ContentRevision) + err = s.vaultIngest.AssistantReplyPin( + r.Context(), turn.ProjectID, fmt.Sprintf("msg:%d", msg.ID), eventID, + msg.ID, "", string(cap.Policy), turn.CompletedAt, []byte(msg.Content), + ) + if err != nil { + writeJSONResponse(w, http.StatusConflict, map[string]string{ + "error_code": "PIN_REJECTED", + "message": "保险库拒绝该 pin", + }) + return + } + writeJSONResponse(w, http.StatusOK, map[string]any{ + "message_id": msg.ID, + "policy": cap.Policy, + "status": "requested", + }) +} + // extractMessageIDFromPath 从路由参数或 URL 路径中提取 message_id func messageProjectIDFromMeta(meta map[string]any) int64 { if meta == nil { diff --git a/internal/server/reply_export_api_test.go b/internal/server/reply_export_api_test.go index 1bbdee42..f9c6b23c 100644 --- a/internal/server/reply_export_api_test.go +++ b/internal/server/reply_export_api_test.go @@ -179,3 +179,46 @@ func TestServer_ReplyExportAPI(t *testing.T) { t.Fatalf("expected 403 for cross-user export, got %d: %s", recIDOR.Code, recIDOR.Body.String()) } } + +func TestServer_PinExportCopyOnlyAfterCompleted(t *testing.T) { + cfg := &config.Config{ + Env: "dev", + HTTPHost: "127.0.0.1", + HTTPPort: 8894, + DataDir: t.TempDir(), + } + svc := bizdev.MustNewService(t, cfg) + srv := server.NewServer(cfg, svc) + handler := srv.Routes() + ctx := t.Context() + cm := svc.ContextManager() + sessID := "sess_pin_export" + token := loginTestAdmin(t, handler, cfg) + if _, err := cm.EnsureChatSession(ctx, sessID, 1, 7, "pin 测试"); err != nil { + t.Fatal(err) + } + if _, err := cm.AcquireChatTurn(ctx, sessID, "turn-pin", "hash-pin", "req-pin", 1, 7); err != nil { + t.Fatal(err) + } + msgID, err := cm.RecordUserMessageWithID(ctx, sessID, 1, "assistant", "好的", "", "", nil) + if err != nil { + t.Fatal(err) + } + reqEarly := httptest.NewRequest(http.MethodPost, fmt.Sprintf("/api/v1/chat/messages/%d/pin-export", msgID), nil) + withBearerAuth(reqEarly, token) + recEarly := httptest.NewRecorder() + handler.ServeHTTP(recEarly, reqEarly) + if recEarly.Code != http.StatusConflict { + t.Fatalf("pin before completed want 409, got %d: %s", recEarly.Code, recEarly.Body.String()) + } + if err := cm.CompleteChatTurn(ctx, sessID, "turn-pin", msgID); err != nil { + t.Fatal(err) + } + req := httptest.NewRequest(http.MethodPost, fmt.Sprintf("/api/v1/chat/messages/%d/pin-export", msgID), nil) + withBearerAuth(req, token) + rec := httptest.NewRecorder() + handler.ServeHTTP(rec, req) + if rec.Code != http.StatusOK { + t.Fatalf("pin after completed want 200, got %d: %s", rec.Code, rec.Body.String()) + } +} diff --git a/internal/server/server.go b/internal/server/server.go index a80776ed..0abab364 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -293,6 +293,7 @@ func (s *Server) Routes() http.Handler { // 7.1 单次助手回复导出与快照 mux.HandleFunc("GET /api/v1/chat/messages/{message_id}/export-capability", s.handleGetExportCapability) mux.HandleFunc("GET /api/v1/chat/messages/{message_id}/export-source", s.handleGetExportSource) + mux.HandleFunc("POST /api/v1/chat/messages/{message_id}/pin-export", s.handlePinExport) // 7.2 自由私密文档与项目共享参考 mux.HandleFunc("POST /api/v1/freeform-documents", s.handleCreateFreeformDoc) diff --git a/internal/server/vault.go b/internal/server/vault.go index cef55f87..73db3f71 100644 --- a/internal/server/vault.go +++ b/internal/server/vault.go @@ -6,9 +6,12 @@ import ( "time" "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/config" "chat-agent-system/internal/docgov" "chat-agent-system/internal/replyexport" "chat-agent-system/internal/taskrun" + + "github.com/jackc/pgx/v5" ) func (s *Server) initArtifactVault() { @@ -28,6 +31,9 @@ func (s *Server) initArtifactVault() { if s.cfg.VaultStateRoot == "" || s.cfg.VaultMasterKey == "" { return } + if err := config.ValidateIndependentVault(s.cfg.VaultStateRoot, s.cfg.DataDir); err != nil { + return + } ident := s.cfg.VaultStorageIdentity if ident == "" { ident = "vault-" + env @@ -93,6 +99,12 @@ func (s *Server) wireFinalArtifactHooks() { } _ = s.vaultIngest.Approved(ctx, doc.ProjectID, doc.DocID, snapshotID, doc.ApprovedAt, markdown) } + gov.OnFinalArtifactTX = func(ctx context.Context, tx pgx.Tx, doc *docgov.DocumentRecord, markdown []byte, snapshotID int64) error { + if doc == nil { + return nil + } + return s.vaultIngest.ApprovedTX(ctx, tx, doc.ProjectID, doc.DocID, snapshotID, doc.ApprovedAt, markdown) + } } if rev := s.svc.DocRevision(); rev != nil { rev.OnFinalArtifact = func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) { @@ -103,6 +115,16 @@ func (s *Server) wireFinalArtifactHooks() { _ = s.vaultIngest.SealedVersion(ctx, projectID, docID, eventID, snapshotID, sealedAt, payload) } } + rev.OnFinalArtifactTX = func(ctx context.Context, tx pgx.Tx, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) error { + switch kind { + case "agent_commit": + return s.vaultIngest.AgentCommitTX(ctx, tx, projectID, docID, eventID, snapshotID, payload) + case "version_sealed": + return s.vaultIngest.SealedVersionTX(ctx, tx, projectID, docID, eventID, snapshotID, sealedAt, payload) + default: + return nil + } + } } if fd := s.svc.FreeformDoc(); fd != nil { fd.OnFinalArtifact = func(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) { diff --git a/internal/server/vault_nowrite_test.go b/internal/server/vault_nowrite_test.go new file mode 100644 index 00000000..1a91b496 --- /dev/null +++ b/internal/server/vault_nowrite_test.go @@ -0,0 +1,60 @@ +package server + +import ( + "context" + "fmt" + "testing" + + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/config" +) + +type writeFailRegistry struct { + *artifactvault.MemoryRegistry + requests int +} + +func (w *writeFailRegistry) Request(ctx context.Context, facts artifactvault.Facts, req artifactvault.PublishRequest) (*artifactvault.Artifact, *artifactvault.Staging, error) { + w.requests++ + return nil, nil, fmt.Errorf("GET must not write") +} + +func TestTryVaultDegradedDoesNotRequest(t *testing.T) { + ctx := context.Background() + mem := artifactvault.NewMemoryRegistry() + spy := &writeFailRegistry{MemoryRegistry: mem} + dir := t.TempDir() + store, err := artifactvault.NewFilesystemStore(dir, "vault-id-test", []byte("0123456789abcdef0123456789abcdef")) + if err != nil { + t.Fatal(err) + } + pub := &artifactvault.Publisher{Reg: mem, Store: store} + facts := artifactvault.Facts{ + ArtifactType: artifactvault.TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 7, + ChangeKind: "agent_commit", ChangeToSnapshotID: 7, Classification: artifactvault.ClassificationOK, + Payload: []byte("# ledger\n"), + } + req := artifactvault.PublishRequest{ + ArtifactType: artifactvault.TypeAgentCommit, Environment: "dev", ProjectExternalKey: "3", + LogicalResourceID: "doc_nowrite", SourceEventKind: "agent_commit", SourceEventID: "ev-nowrite", + Payload: []byte("# ledger\n"), TerminalState: "committed", + } + if _, _, err := mem.Request(ctx, facts, req); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + s := &Server{ + cfg: &config.Config{Env: "dev", FAVDegradedRead: true, DataDir: t.TempDir()}, + vaultReg: spy, + vaultPub: &artifactvault.Publisher{Reg: spy, Store: store}, + } + b, _, err := s.tryVaultDegraded(ctx, "dev", "doc_nowrite", "3") + if err != nil || string(b) != "# ledger\n" { + t.Fatalf("degraded: %v %q", err, b) + } + if spy.requests != 0 { + t.Fatalf("GET path issued %d Request writes", spy.requests) + } +} From 539a2dc639a3224b1d9676f3752054affc678773 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 14 Sep 2026 12:52:28 +0000 Subject: [PATCH 13/13] docs(test): record FAV gap-fill unit evidence Capture same-tx, recover-apply, pin-export, and GET zero-write results without claiming live 6173 or Unit A drills. Co-authored-by: Cursor --- .doc/ops/CURRENT.md | 2 +- ...-14-final-artifact-vault-implementation.md | 6 ++- .doc/remediation/gap-matrix.md | 2 +- .doc/status/FEATURE-MATRIX.md | 2 +- ...-09-14-final-artifact-vault-test-report.md | 17 +++++-- .../unit-tests-gapfill.txt | 48 +++++++++++++++++++ 6 files changed, 69 insertions(+), 8 deletions(-) create mode 100644 .doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index 3c90c0e8..d02bdf62 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -52,7 +52,7 @@ | Cursor | dev-runtime-data-root-guard-plan-rewrite | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | in_progress | | Codex | runtime-data-lifecycle-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | infra-runtime-state | done | | Codex | final-artifact-vault-plan | fix/dev-runtime-data-root-guard | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | done | -| Cursor | final-artifact-vault-implementation | feature/final-artifact-vault | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | in_progress | +| Cursor | final-artifact-vault-implementation | feature/final-artifact-vault | .worktrees/fix-dev-runtime-data-root-guard | artifact-vault | done | > 新任务开始前登记。`branch` 必须为 `feature/` 或 `fix/`，**禁止** `dev` / `test` / `main`。同一 `mutex` 同时仅一个 `in_progress`。 diff --git a/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md index c299c428..07acd47d 100644 --- a/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md +++ b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md @@ -1,6 +1,6 @@ --- title: "最终成果保险库 V0–V6 落地" -status: in_progress +status: done created: 2026-09-14 updated: 2026-09-14 owner: Cursor @@ -31,8 +31,9 @@ mutex: artifact-vault 5. [x] V3 助手接入（同步 chat + task finalize） 6. [x] V3.5 回填（只枚举账本/完成 turn） 7. [x] V4 降级读取（`FAV_DEGRADED_READ` 默认关） -8. [x] V5 恢复（scratch-only CLI） +8. [x] V5 恢复（scratch-only recover-apply 写入 collection；union-deny 跳过 tombstone） 9. [x] V6 门禁与三份演练报告（live e2e 待 Unit A） +10. [x] 缺口补齐：approve/commit 同事务 staging；半发布 exclusive 重试；GET 零写入；`copy_only` pin-export；destroy 预检 CLI ## 相关链接 @@ -45,3 +46,4 @@ mutex: artifact-vault - 分支：`feature/final-artifact-vault` - 未执行 `env-service` rebuild；`FAV_DEGRADED_READ` 默认 false - Goose：`00034_2026-09-14-final-artifact-vault.sql` +- 未合 `dev`；live 6173 / PITR / corpus 仍待 Unit A 另占槽 diff --git a/.doc/remediation/gap-matrix.md b/.doc/remediation/gap-matrix.md index bfb9eb12..50277430 100644 --- a/.doc/remediation/gap-matrix.md +++ b/.doc/remediation/gap-matrix.md @@ -71,7 +71,7 @@ | DOCREV-ACTIVE-DOC-ROUTING | UI-05「当前受控文档」被法规关键词改路由到 `feasibility_reg`，intake 无 pending | **PASS** | [audit](../audit/2026-09-13-document-revision-active-doc-routing.md)；Round20 UI-05 pending 停在 intake V2 | **不改弱 E2E**；不删证据 | | DOCREV-QUALITY-STATUS-FP | 行动计划表状态列「待续」被门禁当敷衍，两轮 repair 后 quality_failed 80 | **PASS** | [audit](../audit/2026-09-13-document-revision-quality-status-false-positive.md)；Round20 UI-03 | **不删禁词、不放宽正文**；不删证据 | | GOVERNANCE-ACTION-AUTHZ | 文档动作按职责鉴权；未激活 202-blocked；关闭创建直通 | **PASS** | [test-report](../test-reports/2026-09-14-governance-action-level-authz-test-report.md)；[audit 异态](../audit/2026-09-14-planning-with-workflow-key-anomaly.md) | 不改 83 条 planning+key 夹具；none 长 PI 审批链另开 | -| FAV-V0-V6-IMPL | 终态成果保险库 V0–V6 内核与挂钩 | **PARTIAL** | [test-report](../test-reports/2026-09-14-final-artifact-vault-test-report.md)；[ADR](../adr/2026-09-14-final-artifact-vault.md)。goose 00034；`FAV_DEGRADED_READ` 默认 false | 不替代 Unit A/B；live 6173/PITR/corpus 演练另占槽。禁止在错挂数据根上开降级读 | +| FAV-V0-V6-IMPL | 终态成果保险库 V0–V6 内核与挂钩 | **PARTIAL** | [test-report](../test-reports/2026-09-14-final-artifact-vault-test-report.md)；[ADR](../adr/2026-09-14-final-artifact-vault.md)。goose 00034；同事务 staging；scratch recover-apply；`FAV_DEGRADED_READ` 默认 false | 不替代 Unit A/B；live 6173/PITR/corpus 演练另占槽。禁止在错挂数据根上开降级读 | --- diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index b6a1264b..c9894b7e 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -23,7 +23,7 @@ | IAM / 鉴权 | 项目级 API 鉴权 + RBAC | partial | [audit/authorization](../audit/authorization-matrix-v2.md)；[刷新恢复专项审计](../audit/2026-09-13-chat-history-refresh-disappears.md) | 历史读取端点补统一会话 owner + 阶段授权；旧 API 410 标注(BUG-11 partial) | `authz` | | 文档治理 | 法定文档生命周期 + 租约 + 受控正文读取 | partial | [数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[test docgov](../test-reports/2026-09-08-docgov-lifecycle-and-claiming-framework-test-report.md)；[项目 143 canonical 漂移审计](../audit/2026-09-14-project143-controlled-content-404.md)；[既有真实 UI 报告](../test-reports/2026-09-14-controlled-content-ui-acceptance.md) | canonical/hash 与 UI 竞态旧缺陷已修；当前 dev 因数据根 split-brain 再次不可读。Unit A 纠挂后标 incident-mitigated；`done` 等 Unit B 负例，不等 outbox | `docgov` | | 文档版本与修改认知 | 智能体版本边界、人工修订标记、任意版本比较、变更报告与负责人通知 | partial | [管理测试](../test-reports/2026-09-13-document-revision-management-test-report.md)；[运行时 UI Round20](../test-reports/2026-09-14-document-revision-runtime-ui-test-report.md)；[运行时审计](../audit/2026-09-13-document-revision-runtime-404.md)；[ADR](../adr/2026-09-13-document-version-boundary.md) | **运行时 UI PASS**（6159 / V3/r2）。仍 `partial`：change report=`fallback`（真实 diff+lineage，非完整 PI 叙事）。系统级 `CHAT-INTENT` 见 gap-matrix | `docgov` | -| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | partial | [ADR](../adr/2026-09-14-final-artifact-vault.md)；[调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md)；[单测报告](../test-reports/2026-09-14-final-artifact-vault-test-report.md) | V0–V6 内核/挂钩/CLI 已落地（goose 00034）。现网降级读默认关；6173 live 演练与 corpus/PITR 待 Unit A 后另占槽 | `artifact-vault` | +| 最终成果归档与恢复 | 终态文档/助手回复的独立归档、稳定定位、清理门禁与受控部分恢复 | partial | [ADR](../adr/2026-09-14-final-artifact-vault.md)；[调研](../ops/research/2026-09-14-final-artifact-vault.md)；[整改方案](../remediation/2026-09-14-final-artifact-vault-remediation.md)；[单测报告](../test-reports/2026-09-14-final-artifact-vault-test-report.md) | V0–V6 内核已落地：approve/commit 同事务 staging、scratch recover-apply、GET 零写入、`copy_only` pin-export（goose 00034）。现网降级读默认关；6173 live 演练与 corpus/PITR 待 Unit A 后另占槽 | `artifact-vault` | | 前端工作台 | 三栏 + 块文档 + SSE | partial | [test workbench](../test-reports/2026-09-08-three-column-workbench-and-block-document-test-report.md)；[现场专项审计](../audit/2026-09-13-conversation-intent-runtime-state.md) | 错误卡片/任务进度/文档 busy 状态刷新恢复与人工抽检待完成 | `web` | | PI Runner / 配置链 | pi-agent + MiniMax 真实链路 | done | [test piagent](../test-reports/2026-09-13-codex-audit-p0-remediation-test-report.md)；[container-env](../test-reports/2026-09-13-container-env-configuration-test-report.md) | — | `pi-agent` | | 三环境运维 | dev/test/main + 共享 KB | partial | [PROJECT.md](../../PROJECT.md) §4；[数据根 split-brain 审计](../audit/2026-09-14-dev-runtime-data-root-split-brain.md)；[审查修订方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)；[数据生命周期方案](../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) | 现场仍错挂 `/tmp`。本行 `done` 等 Unit A 纠挂 + Unit B 门禁负例；蓝绿换库/corpus 为 sibling，不挡事故关闭 | `infra` | diff --git a/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md index e6fa66df..35f152b6 100644 --- a/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md +++ b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md @@ -12,6 +12,9 @@ - publisher 幂等、拒绝 `data/dev` 根、跨项目同 SHA 分 key、tombstone union-deny。 - 回填缺证据 quarantine；重跑不新增逻辑成果。 - recover 拒绝 operational DSN；coverage/destroy 门禁 fail-closed。 +- 半发布缺 commit marker 不可见，exclusive 重试后可恢复。 +- GET/degraded 路径不调用 `Request`。 +- `copy_only` 仅 completed 后显式 pin 可进库。 ## 三、 测试方案与环境 (Test Plan & Environment) @@ -67,13 +70,21 @@ - **评价判定**: PASS - **性能与耗时**: <10ms +### 5. 缺口补齐（同事务 / recover-apply / GET 零写入 / pin） + +- **测试输入**: `TestPublisherRetriesAfterMissingCommit`、`TestUnpublishedWithoutCommitMarkerInvisible`、`TestFilesystemIdentityMismatch`、`TestDegradedPayloadDoesNotWrite`、`TestTryVaultDegradedDoesNotRequest`、`TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied`、`TestServer_PinExportCopyOnlyAfterCompleted`、`TestValidateIndependentVault` +- **实际输出**: 全部 `--- PASS`（见 [assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt](assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt)） +- **评价判定**: PASS +- **性能与耗时**: artifactvault 4ms；pin-export 120ms + ## 六、 自动化测试套件汇总 (Test Suite Execution) ``` ok chat-agent-system/internal/artifactvault 0.005s -ok chat-agent-system/internal/docgov 0.003s -ok chat-agent-system/internal/ctxmgr 0.003s -ok chat-agent-system/internal/server 0.005s (TestFinalizeTaskRunCompletesTurnAndIngests) +ok chat-agent-system/internal/config 0.002s +ok chat-agent-system/internal/docgov 0.011s +ok chat-agent-system/internal/ctxmgr 0.073s +ok chat-agent-system/internal/server 0.364s (FAV 挂钩 / pin-export / GET 零写入) ``` 未跑 `go test ./...` 全量（含真实 PI E2E / 需容器的套件）；本波次范围是 FAV 包与挂钩单测。 diff --git a/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt new file mode 100644 index 00000000..6fc5b46c --- /dev/null +++ b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt @@ -0,0 +1,48 @@ +=== RUN TestEvaluateEligibility_Allowlist +--- PASS: TestEvaluateEligibility_Allowlist (0.00s) +=== RUN TestEvaluateEligibility_ApprovedRequiresReviewSnapshot +--- PASS: TestEvaluateEligibility_ApprovedRequiresReviewSnapshot (0.00s) +=== RUN TestEvaluateEligibility_TaskWithoutCompletedTurn +--- PASS: TestEvaluateEligibility_TaskWithoutCompletedTurn (0.00s) +=== RUN TestEvaluateEligibility_SensitiveRejectedBeforeStaging +--- PASS: TestEvaluateEligibility_SensitiveRejectedBeforeStaging (0.00s) +=== RUN TestEvaluateEligibility_CopyOnlyRequiresExplicitPinAfterCompleted +--- PASS: TestEvaluateEligibility_CopyOnlyRequiresExplicitPinAfterCompleted (0.00s) +=== RUN TestEvaluateEligibility_UnknownTypeRejected +--- PASS: TestEvaluateEligibility_UnknownTypeRejected (0.00s) +=== RUN TestPublisherIdempotentAndCrashSafe +--- PASS: TestPublisherIdempotentAndCrashSafe (0.00s) +=== RUN TestPublisherRejectsBadSHAAndCrossProjectKeys +--- PASS: TestPublisherRejectsBadSHAAndCrossProjectKeys (0.00s) +=== RUN TestFilesystemRefusesRuntimeDataDev +--- PASS: TestFilesystemRefusesRuntimeDataDev (0.00s) +=== RUN TestTombstoneUnionDeny +--- PASS: TestTombstoneUnionDeny (0.00s) +=== RUN TestLastReferencePurgeAndLegalHold +--- PASS: TestLastReferencePurgeAndLegalHold (0.00s) +=== RUN TestFilesystemIdentityMismatch +--- PASS: TestFilesystemIdentityMismatch (0.00s) +=== RUN TestUnpublishedWithoutCommitMarkerInvisible +--- PASS: TestUnpublishedWithoutCommitMarkerInvisible (0.00s) +=== RUN TestPublisherRetriesAfterMissingCommit +--- PASS: TestPublisherRetriesAfterMissingCommit (0.00s) +=== RUN TestDegradedPayloadDoesNotWrite +--- PASS: TestDegradedPayloadDoesNotWrite (0.00s) +=== RUN TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied +--- PASS: TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied (0.00s) +PASS +ok chat-agent-system/internal/artifactvault 0.004s +=== RUN TestValidateIndependentVault +--- PASS: TestValidateIndependentVault (0.00s) +PASS +ok chat-agent-system/internal/config 0.002s +=== RUN TestFinalizeTaskRunCompletesTurnAndIngests +2026/09/14 12:51:53 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestFinalizeTaskRunCompletesTurnAndIngests (0.00s) +=== RUN TestTryVaultDegradedDoesNotRequest +--- PASS: TestTryVaultDegradedDoesNotRequest (0.00s) +=== RUN TestServer_PinExportCopyOnlyAfterCompleted +2026/09/14 12:51:53 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestServer_PinExportCopyOnlyAfterCompleted (0.12s) +PASS +ok chat-agent-system/internal/server 0.131s