diff --git a/.doc/adr/2026-09-14-final-artifact-vault.md b/.doc/adr/2026-09-14-final-artifact-vault.md new file mode 100644 index 00000000..7fa993c8 --- /dev/null +++ b/.doc/adr/2026-09-14-final-artifact-vault.md @@ -0,0 +1,25 @@ +# ADR：最终成果保险库（FAV） + +## 状态 + +已接受（2026-09-14，落地计划 V0–V6） + +## 背景 + +运行库 PostgreSQL 与 workspace 投影会一起丢失或错挂。需要独立保存**已确认终态**的精确字节，但不能成为第三套完整业务真相源，也不能用目录扫描猜测“最终文件”。 + +## 决策 + +1. **三层权威**：PostgreSQL 管项目/权限/工作流/版本关系；workspace 是可清理投影；FAV 只对已归档 payload、manifest、来源证据和删除状态权威。 +2. **无 tenant_id**：安全边界沿用 `environment + project_members + role-admin`。未来组织层另立 ADR。 +3. **事件准入**：只接受方案准入表中的数据库事实。显式 publish/pin 不能绕过终态 allowlist。禁止扫描 workspace。 +4. **同事务冻结**：终态事务把精确 payload/SHA 写入 staging 并写 unpublished outbox；publisher 不得重读可变消息行。 +5. **union-deny 删除**：运行库删除账本或 vault tombstone 任一侧删除即拒绝读取/恢复。 +6. **恢复**：只进 scratch 库，默认 `recover-as-collection`。禁止 GET 自动写回或覆盖现有 head。 +7. **互补**：FAV、migration corpus、完整 DB backup/PITR 互相不能替代。不作为 409/split-brain 绕行。 + +## 后果 + +- 实现从 `feature/final-artifact-vault` 推进；mutex=`artifact-vault`。 +- Goose 从 **00034** 增加 FAV 表（00033 已被 task-run 治理占用）。 +- 降级读取默认关闭（`FAV_DEGRADED_READ`），Unit A 纠挂完成前不得对现网开启。 diff --git a/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md new file mode 100644 index 00000000..b10b43d9 --- /dev/null +++ b/.doc/audit/2026-09-14-dev-runtime-data-root-split-brain.md @@ -0,0 +1,167 @@ +--- +title: "dev rebuild 从临时 worktree 挂载空数据根导致正文投影 409" +status: done +created: 2026-09-14 +updated: 2026-09-14 +severity: critical +--- + +# dev rebuild 从临时 worktree 挂载空数据根导致正文投影 409 + +## 现象 + +用户刷新项目 143 后，前端请求： + +```text +GET /api/v1/docgov/documents/doc_proj143_intake/content 409 +DOCUMENT_CONTENT_PROJECTION_MISSING +docgov: document content projection missing +``` + +错误发生在 `projectDocuments.ts` → `controlledDocHydrate.ts` 的受控正文水合链。它不是并发编辑的 `HEAD_CONFLICT`，也不是此前已修过的 canonical hash drift。 + +本轮只做只读现场核对与整改设计：没有操作用户会话、没有修改数据库/正文、没有重建共享容器。 + +## 结论 + +直接根因是 **dev 应用、PI Runner、Cursor Runner 被重建成挂载临时集成 worktree 的空 `data/dev`，而 PostgreSQL 仍沿用原 dev 持久卷**，形成数据库与文件事实源的 split-brain： + +```text +PostgreSQL persistent volume + document_records.work_dir=/app/data/workspaces/bizdev/intake_1789270446856290128 + │ + ├─ 主仓 data/dev：output.md 存在，SHA 与 DB/head 一致 + │ + └─ 当前容器 /app/data：绑定 /tmp/as-dev-wf409-merge/data/dev，文件不存在 +``` + +`loadControlledDocumentContent` 先取得账本 head，再调用 `InspectProjection`；容器内看不到 `output.md`，因此正确分类为 `MISSING` 并返回 409。前端只是如实暴露后端错误，不是本次根因。 + +## 现场证据 + +### E1 — 容器挂载源是临时 worktree + +2026-09-14 08:54 UTC 的 `docker inspect shuimu-agent-app-dev`： + +| 项 | 现场值 | +|---|---| +| Compose working dir | `/tmp/as-dev-wf409-merge` | +| Compose config | `/tmp/as-dev-wf409-merge/docker-compose.dev.yml` | +| app `/app/data` source | `/tmp/as-dev-wf409-merge/data/dev` | +| pi-agent `/app/data/workspaces` source | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| cursor-agent `/app/data/workspaces` source | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| app created | `2026-09-14T08:40:44Z` | + +数据库容器的数据仍来自命名卷 `agent-dev-context-pgdata`，不会随 Compose 工作目录切换。由此产生“旧数据库 + 新空文件根”的不一致组合。 + +### E2 — 项目 143 数据本身当前完整 + +`document_records`： + +| 字段 | 值 | +|---|---| +| doc_id | `doc_proj143_intake` | +| work_dir | `/app/data/workspaces/bizdev/intake_1789270446856290128` | +| content_sha256 | `f283ba94c9c037897f021960f51abacee7242c3db8d10f9f2a5e32cde6acf039` | +| updated_at | `2026-09-14 05:14:10.302146+00` | + +账本 open version 1 的 head snapshot 302 是人工修订，SHA 同为 `f283...f039`；主仓文件： + +```text +/home/weichen/projects/Agent-System/data/dev/workspaces/bizdev/intake_1789270446856290128/output.md +size=40075 +sha256=f283ba94c9c037897f021960f51abacee7242c3db8d10f9f2a5e32cde6acf039 +``` + +同一路径在当前 app 容器内 `stat` 返回 `No such file or directory`。所以无需再次对项目 143 做内容修复；需要恢复正确数据根挂载。 + +### E3 — 影响不是单文档 + +对 dev 数据库 100 条“非模板且 work_dir 非空”的记录做只读比较： + +| 根目录 | aligned | missing | +|---|---:|---:| +| 主仓 `/home/weichen/projects/Agent-System/data/dev` | 32 | 68 | +| 当前容器挂载 `/tmp/as-dev-wf409-merge/data/dev` | 0 | 100 | + +主仓下的 68 条 missing 包含历史测试/临时路径，不能直接等同于 68 个真实用户缺陷；但当前容器对本次抽样 **100/100 都不可见**，足以确认环境级事故。恢复前任何真实文档读取或 PI 写入都不可信。 + +错误根在容器启动后已出现 27 个文件，其中 9 个 CAS blob 的 mtime 为 `08:40:45 UTC`；虽然 08:54 UTC 尚未找到新的 `output.md`，错误根已经不是可直接丢弃的“绝对空目录”，恢复前仍须做完整差异保全。 + +只读核对显示 test/main 当前 app 与 PI Runner 仍挂主仓 `data/test`、`data/main`，本轮没有观察到同样的运行事故；但其 Compose 仍使用相对 bind，仍受同一设计缺陷影响。 + +### E4 — 健康检查假阳性 + +同一时刻 `/health` 与 `/ready` 都返回 200，只检查数据库、schema 和 task store；未验证： + +- 容器绑定的数据根身份； +- app 与 runner 是否挂载同一根； +- 活跃项目的正文投影是否可达； +- 数据库记录与文件根是否属于同一环境实例。 + +因此“healthy + schema 33”不能证明文档运行态可用。 + +## 根因分析 + +### R1 — Compose 相对 bind mount 与调用 worktree 耦合（直接根因） + +`docker-compose.dev.yml` 使用： + +```yaml +- ./data/dev:/app/data +- ./data/dev/workspaces:/app/data/workspaces +``` + +Compose 按 project working directory 解析相对路径。`scripts/env-service.sh` 又把 `ROOT_DIR` 定义为脚本所在 worktree 并 `cd` 进去，所以从任何复制/linked worktree 执行 `dev rebuild`，都会把该 worktree 自己的 `data/dev` 当成运行数据根。 + +### R2 — 纪律是文档约束，不是可执行门禁 + +`scripts/agent-worktree.sh` 只打印“禁止在此目录运行 env-service”，`AGENTS.md` 也规定 worktree 不得操作共享 Docker；但 `env-service.sh` 没有检测 linked worktree、主 worktree、分支、数据根或集成槽位。一次误调用即可静默替换全套容器挂载。 + +### R3 — 数据库与文件没有共享存储身份 + +数据库使用稳定 named volume；正文使用随调用目录变化的 bind mount。两者没有共同的 environment/storage UUID，也没有启动握手，因此任意一个被指向另一份状态时，进程仍可启动并连接成功。 + +### R4 — 账本权威仍被兼容投影阻断 + +当前设计声明 ledger head 是权威正文，但 GET 在账本存在时仍要求 `output.md` 投影必须存在且 aligned。于是兼容投影丢失会阻断 UI 读取权威账本。该 fail-closed 对真实 fork 有价值，却把“可安全重建的 missing”与“内容分叉”按同样的可用性后果处理。 + +### R5 — 正文更新仍是非原子双写 + +人工 PATCH 先提交 PostgreSQL 账本，再单独写 `output.md`；候选 commit/reject 也采用 DB 状态与文件投影分步更新。失败后只依赖同幂等键重试，没有持久 projection job/status 和后台补偿。即使修复本次挂载问题，进程崩溃、权限错误或磁盘故障仍可能再次留下 head 已更新、投影缺失的状态。 + +### R6 — 发布验收没有“从不同源码目录 rebuild 后仍接回同一状态”的负例 + +既有真实 UI 验收验证过 restart 后恢复，但 restart 不会重新解析 Compose bind mount；本次事故由 rebuild/recreate 触发。现有门禁没有覆盖 linked worktree、临时集成目录或错误数据根的启动拒绝。 + +## 为什么会“每次修改后又出现” + +修改本身并不会删除项目 143 正文。本次时间线表明：最新人工修订和主仓 `output.md` 在 05:14 UTC 已一致落盘；08:40 UTC 后续 dev rebuild 把运行容器切到另一份空数据根，刷新才出现 409。 + +因此用户感知的“改完再刷新就坏”来自两个事件经常相邻：开发修改 → Agent 在临时 worktree 合并/rebuild → Compose 改绑数据目录。只修某个文档、某个 hydrate effect 或某次 hash 漂移，都无法阻止它复发。 + +## 严重度与影响 + +`critical`： + +- dev 环境所有受控正文、工作流产物和 runner 工作区均可能不可见或写入错误根； +- 错误根上的新输出可能与主仓状态分叉； +- health/ready 仍为绿色，集成报告可能产生假 PASS； +- test/main 使用同类相对 bind 写法，设计缺陷可横向复制，虽本轮未观察其被错挂载。 + +## 关联整改 + +- Remediation: [dev 运行数据根隔离与文档投影自愈整改](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [dev runtime data root guard 诊断与设计](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) +- 前序问题: [项目 143 canonical hash 漂移](./2026-09-14-project143-controlled-content-404.md) + +## 开放问题 + +- 恢复前需再次冻结并备份错误根，确认 08:40 UTC 后是否已有新会话、附件、runner 输出或 agent session 写入；08:54 UTC 抽样未发现 `output.md`，不能替代完整差异清单。 +- test/main 当前挂载源需在实施 Unit A 只读核对；本轮未扩大操作范围。 + +## 审查附记（2026-09-14 ~09:23 UTC） + +- 现场复核：`shuimu-agent-app-dev` 仍 created `08:40:44Z`，bind source 仍为 `/tmp/as-dev-wf409-merge/data/dev`；`/health` `/ready` 仍 200，`schema_version=33`。 +- 主仓 CURRENT 集成槽仍为 `implementer_cursor` / `e2e`。worktree 副本不得写成 idle。 +- 方案已审查修订：见 [整改方案](../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md)。用户选择事故抢占 + 先改方案再批实施。08:54 的 `f283...f039` 不得再当 Wave 0 硬门槛。 diff --git a/.doc/ops/research/2026-09-14-final-artifact-vault.md b/.doc/ops/research/2026-09-14-final-artifact-vault.md new file mode 100644 index 00000000..67c52cf7 --- /dev/null +++ b/.doc/ops/research/2026-09-14-final-artifact-vault.md @@ -0,0 +1,149 @@ +--- +title: "最终成果保险库与部分恢复边界" +created: 2026-09-14 +status: done +confidence: high +--- + +# 调研：最终成果保险库与部分恢复边界 + +## 结论 + +提案可行，建议采用，但不应实现成“把最终文件复制到一个固定目录”。应建设独立的**最终成果保险库（Final Artifact Vault，FAV）**：只接收服务端已经确认终态、具有稳定标识和内容哈希的成果，以不可变 blob、语义 manifest 和提交回执保存，并与工作区清理完全解耦。 + +FAV 是 PostgreSQL 完整备份/PITR 的互补层，不是第三套完整业务事实源。它只对“已归档成果的精确内容与出处”负责，可以在数据库损坏或选择性重建时恢复文档、最终回复和有限的来源关系；成员权限、工作流活动态、租约、待办任务、幂等记录等仍须由数据库备份或其他权威记录恢复。 + +## 当前能力与缺口 + +| 能力 | 当前事实 | 与提案的差距 | +|---|---|---| +| 受控文档版本 | `document_content_blobs` 与 `document_snapshots` 在 PostgreSQL 内保存不可变 Markdown；工作区 `output.md` 是投影 | 数据库与账本一起丢失时，没有独立副本；当前 GET 仍会被缺失工作区投影阻断 | +| 助手最终回复 | `chat_messages.content` 保存正文，`chat_turns.assistant_message_id` 可在 completed 回合绑定最终消息 | 占位正文会原地更新，没有独立不可变回复快照；删除 session 会级联删除消息 | +| 回复导出 | `replyexport` 已区分 completed、status/error/card，提供 `content_sha256` 与 `export_policy` | `export-source` 仍从当前消息行临时构造 snapshot，不是持久归档；部分调用在缺少真实回合状态时默认按 completed 评估 | +| 附件 CAS | `storage_blobs`/`file_references` + `data//storage/blobs` 按 SHA 去重 | blob 物理文件仍在运行状态根，引用元数据仍在数据库；不独立于工作区/数据库事故 | +| 数据备份 | 现有脚本主要导出 PostgreSQL；数据生命周期方案已要求 DB + state 成对打包 | 尚无跨文档/回复的终态准入、持续发布、独立索引或成果级恢复 API | + +当前事故已经证明“数据库记录还在”不代表“运行容器能找到正文”。FAV 可以解决成果副本对工作区根的依赖，但不能替代固定 `RUNTIME_STATE_ROOT`、storage identity、projection outbox 和数据库备份。 + +## 提案中应保留的部分 + +1. **独立持久位置**：成果存储不位于 Git worktree、临时集成目录、任务 workspace 或普通 GC 扫描范围。 +2. **只收终态成果**：拒绝流式片段、候选稿、工具轨迹、修复过程、临时报告、租约和未完成任务。 +3. **支持系统定位**：使用稳定 artifact UUID、逻辑资源 ID、来源事件和内容 SHA，而不是依赖数据库自增 ID 或 workspace 路径。 +4. **辅助部分恢复**：允许在数据库关系不匹配时校验并导入已归档内容，同时保留来源和冲突证据。 +5. **与清理联动**：工作区或旧数据集只有在必需成果全部发布并验证后才可进入销毁阶段。 + +## 必须修正的部分 + +### 1. “最终”必须由状态机判定 + +不能按文件名、目录、时间或“任务进程退出 0”判断最终。建议的准入规则： + +| 成果类型 | 可归档条件 | 明确排除 | +|---|---|---| +| 智能体生成文档 | agent candidate 已 `committed`，且 output snapshot 成为账本 head；归档该次生成的不可变 snapshot | ready、needs_confirmation、rejected、stale、quality_failed、workspace 草稿 | +| 受控文档正式稿 | `approved` 时归档被冻结的 review snapshot；版本被 sealed 时可归档该版 final head | drafting/reviewing 中尚未冻结的当前文件、lease 中内容 | +| 自由文档 | 已形成不可变 version，且用户显式“发布/归档”或被正式成果引用；显式操作不能绕过终态检查 | 自动保存中的连续修订、临时私密摘录 | +| 助手回复 | `chat_turn.status=completed`、绑定唯一 `assistant_message_id`、正文 SHA 匹配，且 `export_policy=full`；`copy_only` 只在 completed 后显式 pin 时进入 | status/error/cancelled/system、纯操作卡、流式 chunk、placeholder、中间 agent 消息；未完成回复即使 pin 也拒绝 | +| 最终报告/附件 | 被上述成果 manifest 引用并通过合规/病毒/类型校验 | 未引用附件、日志、Prompt、工具输入输出、agent session | + +“每次成功保存的人工 revision”仍进入 PostgreSQL 不可变账本，但不自动进入 FAV。这样既保留文档历史，又不会把编辑过程伪装成最终成果。 + +### 2. 保险库不是第三套完整真相源 + +建议采用三层责任边界： + +| 层 | 权威范围 | 可否清理 | +|---|---|---| +| PostgreSQL 运行库 | 项目/用户/权限、工作流、任务、当前指针、完整版本账本和归档发布状态 | 按环境生命周期与完整备份策略处理 | +| Runtime workspace/state | Agent 计算现场、`output.md` 投影、临时转换文件和附件运行副本 | 可重建/可清理，但须受发布门禁保护 | +| Final Artifact Vault | 已准入成果的精确字节、manifest、来源证据、保留/删除状态和发布回执 | 独立保留；只能按成果保留策略和删除账本处理 | + +正常读写仍以数据库授权和业务指针为准。数据库仍能完成授权判定但正文缺失时，可从 FAV 只读降级返回并标记 `source=vault/degraded`；若权限关系也缺失，FAV 不得绕过授权直接对用户开放，只能进入管理员隔离恢复流程。 + +### 3. 发布必须避免双写裂缝 + +数据库事务内同时提交业务终态、精确 payload/SHA 的不可变 staging 和 `artifact.publish.requested` outbox；事务外 publisher 只从 staging 幂等写 blob、manifest，校验后最后写 commit marker，再回填 receipt。对象存储 I/O 不进入数据库事务。不能让异步 publisher 重读仍可更新或随 session 级联删除的消息行。 + +清理流程只认 `verified` receipt，不认“消息发过”“目录存在”或“有一个 SHA”。重复投递必须由 `artifact_uuid + source_event_id + payload_sha256` 去重；相同逻辑资源的新终态生成新 manifest，不覆盖旧 manifest。 + +### 4. Manifest 必须表达恢复语义 + +每个归档单元至少包含： + +```text +artifact_schema_version +artifact_uuid / artifact_type / logical_resource_id +project_external_key / environment +source_event_id / source_run_id / source_message_id / source_snapshot_id +payload members: name / media_type / byte_size / sha256 / required +terminal_evidence: state / decided_at / actor_ref / policy_version +lineage: parent artifact / document version / chat turn +retention_class / sensitivity / encryption_key_ref +source_database_identity / source_schema_version / source_commit +published_at / publisher_version / manifest_sha256 +``` + +manifest 使用稳定外部标识，同时保留旧数据库 ID 作为来源提示。它必须能在没有原数据库的情况下建立离线目录，但不得声称自己包含未列出的业务依赖。 + +### 5. 恢复必须经过隔离与对账 + +FAV 支持两种恢复结果： + +- `recover-as-collection`：把验证通过的成果导入新的恢复集合/文档，保留原来源与 hash，不伪造原工作流状态。这是默认模式。 +- `relink`：只有项目、用户、权限、会话和版本依赖均存在且无冲突时，才把成果重新连接到原逻辑资源。 + +恢复顺序为：离线 inventory → manifest/schema/SHA 校验 → 删除账本/保留策略检查 → scratch DB 导入 → 依赖与 ID 映射 → 冲突预览 → 人工确认 → 单事务 promotion → 审计报告。禁止 GET 接口因发现不一致而自动写回数据库，也禁止直接覆盖既有 head 或消息。 + +## 可恢复与不可恢复边界 + +| 可从 FAV 恢复 | 不能只靠 FAV 恢复 | +|---|---| +| 最终 Markdown/附件字节与 MIME/SHA | 用户认证凭证、session、token | +| 已归档助手最终回复及其稳定 turn/message 来源 | 项目成员关系、文档 duty、当时的完整授权图 | +| 文档版本/快照的有限来源、父子和生成 run 线索 | 活跃 lease、pending candidate、未完成 task/run | +| 最终报告、批准/提交终态证据 | 通知投递状态、outbox 消费位置、worker 状态 | +| 恢复所需 schema/producer/storage identity 元数据 | 未进入准入清单的草稿、中间过程和完整数据库约束 | + +因此灾难恢复仍须保留 PostgreSQL base backup/WAL 或经验证的完整逻辑备份；FAV 解决的是“最终成果可独立找回”和“有限业务内容可重建”。 + +## 存储、保留与安全 + +- 第一阶段可用独立 filesystem backend，但必须有独立 `vault_storage_identity`、最小权限凭证和 mount，不允许退回 `./data/dev`；凭证只从受管 secret/env 注入。生产目标应支持 TLS、静态加密及异机/对象存储复制。 +- 当前仓库没有 tenant 实体，安全边界应使用现有 `environment + project_members/role-admin` 授权模型，不得伪造 `tenant_id`。blob 采用**项目作用域加密 CAS**：同项目按 SHA-256 去重，跨项目相同 SHA 不共享密钥和删除边界；未来引入组织/租户层须另立 ADR 和迁移。 +- manifest-member 到 blob 的引用账本决定删除：tombstone 先撤销单个成果读取与 key wrapping；仅当项目作用域内最后有效引用消失且 retention/legal hold 允许时物理清除。manifest 和 commit marker 只追加，定期执行 checksum scrub 和离线目录重建。 +- filesystem backend 必须以同文件系统临时文件、校验、文件 `fsync`、原子 no-replace rename、父目录 `fsync` 实现持久化，确认 blob/manifest 落盘后才提交 marker；不能把“最后写 marker”误当成掉电安全。 +- WORM/对象锁按数据类别启用，不对所有内容一律永久锁定。普通保留使用可治理模式；法律保全单独使用 legal hold。 +- project/user 授权仍由服务端复用正常读取的既有判定；授权关系缺失或不确定时 fail closed。归档正文不得进入日志、metrics 或通知 payload。 +- 自动归档前执行服务端敏感内容准入：结构化排除 Prompt/工具轨迹，secret、认证材料、restricted 或分类失败均 fail closed，且正文不进入 outbox/staging。 +- 建立删除/法务保留账本。满足删除条件时先禁止读取和恢复，再删除密钥/对象；任何历史 restore 都必须重放 tombstone，避免已删除内容复活。 +- manifest 不保存 secret、Prompt、内部推理、工具轨迹或认证材料；敏感元数据最小化，必要时使用不可逆外部键映射。 + +## 与数据精简方案的关系 + +FAV 与 migration corpus、完整备份是三个不同产品： + +| 机制 | 目的 | 内容 | +|---|---|---| +| Final Artifact Vault | 持续保存终态成果，供定位、只读降级和有限恢复 | 小而长期、按终态事件持续追加 | +| Migration corpus | 验证旧 schema 到新 schema 的迁移、异常和 reload | 小而可重复、脱敏、带 assertions | +| DB + state backup/PITR | 完整灾难恢复与回滚 | 大而完整、按 RPO/RTO 生成 | + +清理不能因为“FAV 有最终正文”就跳过完整备份，也不能用完整备份替代成果级准入。正确门禁是三者分别验证：最终成果覆盖率、迁移 corpus 可升级、完整备份可恢复。 + +首次启用 FAV 时还必须做一次**存量终态回填**：只从权威数据库的不可变 snapshot、批准/提交事件和 completed chat turn 枚举，以来源事件 + payload SHA 幂等产生请求。缺正文或终态证据的记录进入 quarantine；批准排除必须记录原因、批准人和有效期。禁止为了达到 coverage 扫描 workspace 或批量把“旧数据”标为已覆盖。 + +## 外部依据 + +- [RFC 8493 BagIt](https://www.rfc-editor.org/rfc/rfc8493.html) 要求 manifest 覆盖 payload 并验证完整性，支持本方案的 manifest/checksum 设计。 +- [OCI Descriptor](https://github.com/opencontainers/image-spec/blob/main/descriptor.md) 以 digest 标识并校验内容，支持内容寻址 blob。 +- [AWS Transactional Outbox guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html) 说明业务写入与 outbox 同事务、消费者幂等的必要性。 +- [PostgreSQL PITR](https://www.postgresql.org/docs/17/continuous-archiving.html)、[pg_restore](https://www.postgresql.org/docs/17/app-pgrestore.html) 与 [pg_verifybackup](https://www.postgresql.org/docs/16/app-pgverifybackup.html) 说明完整恢复、选择性恢复依赖和真实 restore 演练的边界。 +- [AWS S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) 说明 governance/compliance/legal hold 的差异，支持按数据类别选择保留模式。 + +## 建议动作 + +1. 采用[最终成果保险库整改方案](../../remediation/2026-09-14-final-artifact-vault-remediation.md)，先定义终态协议和 manifest，再选择 filesystem 或对象存储实现。 +2. 将 FAV 发布完成率加入 dataset inventory/GC 的硬门禁，但不把 FAV 设为当前 409 的绕行修复。 +3. 首次实现优先覆盖受控文档 approved/agent committed snapshot 与 `export_policy=full` 的 completed assistant reply。 +4. 先在 dev 用项目 6173 的等价脱敏 corpus 做“生成 → 发布 → 清 workspace → 读取/恢复 → rebuild”演练，再考虑 main 保留策略。 diff --git a/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md new file mode 100644 index 00000000..daa1ca58 --- /dev/null +++ b/.doc/ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md @@ -0,0 +1,150 @@ +--- +title: "运行数据精简、迁移语料与演示数据保留策略" +created: 2026-09-14 +status: done +confidence: high +--- + +# 调研：运行数据精简、迁移语料与演示数据保留策略 + +## 结论 + +用户提出的“及时清理过往数据，只保留最新、有用且必要的一部分，并用保留数据验证迁移和重载”方向合理，尤其适合当前快速迭代期的 dev/test 环境。 + +但必须补充两个边界： + +1. 清理历史运行数据只能降低数据转换量和现场噪声，不能替代对受支持旧 schema 的迁移兼容；应把有价值的旧状态压缩成小型、可重复、脱敏的 migration corpus。 +2. 不应在当前库上逐表删除后继续使用；当前跨项目数据存在非完整级联和 DB/文件双事实源，优先采用“新库 + 新状态根 + 选择性导入 + 旧卷隔离”的蓝绿轮换。 +3. 对最终成果再增加独立 [Final Artifact Vault](./2026-09-14-final-artifact-vault.md)，但它只辅助成果定位与部分恢复，不能替代完整 DB+state 备份或旧 schema migration corpus。 + +## 当前数据画像 + +2026-09-14 只读现场： + +| 环境 | schema | 项目数/特征 | 文件状态 | 判断 | +|---|---:|---|---|---| +| dev | 33 | 366 个项目；149 active、217 planning；全部产生于 9 月 11–14 日 | 主仓 `data/dev` 约 100MB | 大量 PG/E2E 夹具，适合立即精简，但须先解决当前错误挂载事故 | +| test | 29 | 约 37 个项目 | 主仓 `data/test` 约 20KB | schema 落后且数据价值低，应改为可重建/短 TTL | +| main | 25 | 10 个历史 seed 项目，当前无受控文档 | 主仓 `data/main` 约 20KB | 不应用作日常迁移试验；清理需另行确认业务/展示用途 | +| shared KB | 独立 | 跨环境共享知识库 | `data/shared` | 不属于本方案的项目数据清理范围 | + +dev 主要表约有：`projects=366`、`chat_sessions=421`、`chat_messages=1923`、`task_runs=307`、`document_records=153`、`document_snapshots=283`、`document_change_events=257`。数据库总体不大，主要问题不是容量，而是数据来源混杂、迁移噪声和无法判定哪些记录应保留。 + +## 现有清理为何不能直接做 + +### 1. `DELETE projects` 不是完整项目删除 + +当前外键有 101 条，部分项目子表 CASCADE，但存在关键例外： + +- `task_runs.project_id` 是 `ON DELETE SET NULL`，项目删除后 run/event 仍可残留； +- `document_versions`、`document_snapshots`、`document_change_events`、`document_change_reports`、`document_agent_candidates` 依赖 `doc_id`/snapshot/version，但不完整依附 `projects`； +- `document_content_blobs` 与 `storage_blobs` 需要引用计数/mark-and-sweep，不能看到文件就删； +- 工作区、agent sessions、CAS 文件位于状态根，数据库 cascade 不会删除文件； +- notification/outbox/audit/lineage 的保留语义不同，不能一律级联。 + +所以简单删除项目会制造孤儿数据，反而加重后续迁移与排障成本。 + +### 2. 现有备份/恢复不足以支撑安全轮换 + +- `backup-system-db.sh` 只导出 PostgreSQL，不包含与之匹配的 `data/` 文件状态； +- `restore-system-db-drill.sh` 校验 dump 列表并运行迁移测试，但当前没有真正把 dump restore 到新隔离数据库； +- restore 未强制隔离目标，默认路径存在误触共享 dev 库的风险； +- 没有 dataset manifest、项目选择、文件 SHA inventory、脱敏、预期断言与导入校验； +- 备份脚本的凭证来源需要改成统一安全配置，不应保留脚本内默认值。 + +在这些缺口补齐前，不能用“有 dump”作为可删除证明。 + +## 建议的数据分类 + +| 类别 | 用途 | 典型内容 | 是否进入日常运行库 | 保留策略 | +|---|---|---|---|---| +| `operational-pinned` | 当前人工使用/正在排障 | 项目 143 等明确认领项目 | dev 可保留；main 按业务规则 | 显式 pin，禁止自动清理 | +| `demo-current` | 产品展示 | 每个正式工作流 1 个完整、可解释项目 | dev/demo 环境 | 只保留当前版本；新版验证后替换旧版 | +| `regression-golden` | 功能回归 | 文档版本、人工修订、并发冲突、reload 等完整链 | 隔离验收库，可选只读展示副本 | 小型、确定性、脱敏；按能力而非数量保留 | +| `migration-baseline` | 旧 schema → 当前 schema | 旧 release 的最小 DB + 匹配文件包 | 仅隔离迁移测试 | 最多保留 3 个 release 基线，滚动替换 | +| `ephemeral-test` | PG/E2E 单次运行 | 自动测试项目、会话、run、workspace | 每次独立数据库/状态根 | PASS 后立即删；FAIL 保留 3–7 天 | +| `quarantine` | 清理回滚与取证 | 被替换的完整卷/状态根 | 不挂载到运行容器 | 只读 7–14 天，恢复演练通过后销毁 | + +## 建议的最小保留集合 + +当前 dev 可先采用 6–10 个项目的上限，而不是按日期盲留： + +1. 项目 143：暂作旧数据/canonical repair/active workflow 回归样本；若含真实内容，不进入 Git，后续用脱敏等价 fixture 替换。 +2. 项目 6173：目前最新且证据完整的受控正文 UI 全链，含两份受控文档、V3/r2、并发、权限与 restart 验证，可作为当前回归/展示候选。 +3. 项目 6159：保留一个发布周期作为上一轮文档账本对照；6173 corpus 稳定后删除，避免永久重复。 +4. 三个正式工作流各生成 1 个 `demo-current`：`main`、`product_manager`、`test_manager`；必须由固定 seed/真实页面脚本可重建，不能依赖手工挑出的偶然状态。 +5. 1 个失败/异常样本不进入展示库，只进入隔离 `regression-golden`：覆盖 projection missing、hard-break、旧消息 scope、高 ID sequence 等历史风险。 + +最新项目不等于最有价值。选择评分应优先：证据完整度、覆盖的能力、是否可脱敏重建、DB/文件一致性、是否有预期断言；时间只作为同类样本的淘汰条件。 + +本轮已只读确认 143、6159、6173 共 5 份受控文档在主仓状态根的 `output.md` 均存在且 SHA 与 `document_records` 一致；这说明它们具备候选资格，但仍需通过敏感性、owner 和可恢复性审查后才能进入最终 retain allowlist。 + +## 迁移语料库策略 + +不保存每个 Goose 版本的整库。按产品 release 边界保留： + +- `oldest-supported`：当前承诺支持的最老 release； +- `previous-release`：上一发布版； +- `current-pre-upgrade`：本次上线前快照，下一次轮换为 previous。 + +当前建议先建立三个最小基线： + +- schema 25：文档账本前/早期治理状态，覆盖 main 当前落后路径； +- schema 28/29：文档 revision 首次引入后的状态，覆盖 test 当前路径； +- schema 32/33：canonical repair、task 状态和当前数据根契约。 + +这些基线应由迁移 fixture builder 在隔离库中构造或从已批准快照脱敏得到，不能直接把 main/test 的运行数据当测试资产。 + +每个 corpus package 必须包含： + +```text +manifest.json +database.dump +state.tar.zst +checksums.txt +assertions.yaml +``` + +manifest 至少记录 dataset 版本、source schema、目标 schema、生成 commit、storage identity、项目/会话/文档/run 数量、文件总数和 SHA、脱敏状态、过期时间；`assertions.yaml` 记录迁移后必须成立的版本、状态、正文 hash、序列与权限断言。 + +二进制 dump/状态包不进 Git；仓库只保留 manifest schema、fixture builder、脱敏规则、选择规则和非敏感断言。实际包放受控备份目录或对象存储。 + +## 每次发布的数据轮换模型 + +```text +旧运行库/状态根（只读） + │ export + manifest + checksums + ▼ +迁移语料构建区（脱敏/裁剪/恢复验证） + │ + ├── oldest-supported corpus + ├── previous-release corpus + └── current demo/regression corpus + │ + ▼ +新数据库 + 新状态根（latest schema） + │ import selected datasets + ▼ +工程门禁 + 页面验收 + rebuild/reload + │ PASS + ▼ +切换运行 identity；旧卷 quarantine 7–14 天后销毁 +``` + +这比在原 dev 库上大规模 DELETE 更适合当前阶段：新库天然消除未知孤儿，导入过程本身又验证了备份、迁移、文件配对、序列和重载。 + +## 风险边界 + +- main 真实业务数据不能套用 dev 的 7–14 天策略；须先定义业务、合规、审计和用户删除规则。 +- 项目 143 等真实内容不能直接固化为仓库 fixture；应保留 hash/结构特征并脱敏重建。 +- migration corpus 不能只有 happy path，否则清场后会失去发现 legacy bug 的能力。 +- 任何清理必须先完成隔离 restore，并验证数据库与文件包属于同一 storage identity。 +- 任何清理还必须验证应归档终态成果均已取得 FAV verified receipt；“有归档正文”不能替代完整关系闭包恢复。 +- 错误 `/tmp` 数据根属于当前事故证据；在 split-brain 恢复报告签字前不纳入普通清理。 + +## 建议动作 + +- 采用 [运行数据生命周期与迁移语料整改方案](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)。 +- 首轮只处理 dev；test 改为短 TTL 隔离环境；main 仅盘点，不删除；shared KB 排除。 +- 先实现 dataset inventory/export/restore/verify，再执行蓝绿精简，不先写清理 SQL。 +- 并行实施[最终成果保险库整改](../../remediation/2026-09-14-final-artifact-vault-remediation.md)，把成果覆盖率加入 GC/销毁门禁。 diff --git a/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md new file mode 100644 index 00000000..d3e1f40c --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-dev-runtime-data-root-guard.md @@ -0,0 +1,61 @@ +--- +title: "dev runtime data root split-brain 诊断、方案审查与修订" +status: in_progress +created: 2026-09-14 +updated: 2026-09-14 +owner: Cursor +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: infra-runtime-state +--- + +# dev runtime data root split-brain 诊断、方案审查与修订 + +## 目标 + +定位项目 143 刷新 409 的当前根因，经审查后把整改方案改成可执行、可分批批准的文本；**不实施**纠挂或业务代码，直到用户再批 Unit A / Unit B。 + +## 范围 + +- 诊断：用户报错链、dev PostgreSQL、主仓文件、容器 mount、health/ready、Compose / `env-service.sh` +- 审查：原文六波次方案的内部矛盾、可行性、抢占与身份首启风险 +- 修订：Unit A / Unit B / Deferred 拆分与可执行 Wave 0 Runbook + +## 非目标 + +- 不操作真实用户会话 +- 不写数据库、不覆盖正文、不删除错误根 +- 本轮不 rebuild、不占集成槽执行纠挂（抢占授权已写入方案，实施另批） +- 不变更运行中的 test/main + +## 步骤 + +1. [x] 完成仓库启动链、既有 canonical hash/refresh 事故文档复核。 +2. [x] 比对项目 143 DB record、ledger head、主仓 `output.md` 与容器内路径。 +3. [x] 审计 app/pi/cursor mounts、Compose working dir、健康检查与影响面。 +4. [x] 完成初版 audit、remediation、gap/feature/current SSOT。 +5. [x] 多视角审查原文方案；现场复核挂载与集成槽仍为事故态。 +6. [x] 按审查结论重写整改方案（Unit A/B/Deferred + 可执行 Runbook）。 +7. [ ] 用户再批后实施 Unit A（事故抢占纠挂）。 +8. [ ] 用户再批后实施 Unit B（绝对状态根 + 门禁 + `/ready`）。 + +## 验收标准 + +- [x] 直接根因有数据库、文件和容器三方证据。 +- [x] 明确区分旧 hash drift、真正 ETag 409 与本次 projection missing。 +- [x] 修订方案不再把 Wave 0 依赖尚未存在的 identity / outbox；不再把 Deferred 当事故关门条件。 +- [x] 未污染真实会话或修改共享运行态。 +- [ ] Unit A/B 实施与验收（另批）。 + +## 相关链接 + +- Audit: [../../audit/2026-09-14-dev-runtime-data-root-split-brain.md](../../audit/2026-09-14-dev-runtime-data-root-split-brain.md) +- Remediation: [../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md](../../remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Test report: 待 Unit A 实施后创建 + +## 证据 / 备注 + +- 诊断分支：`fix/dev-runtime-data-root-guard`，基线 `dev@a8f9451`，诊断 commit `f1edb70`。 +- 2026-09-14 08:54 UTC 诊断：app/pi/cursor 均挂 `/tmp/as-dev-wf409-merge/data/dev*`；DB 仍为 `agent-dev-context-pgdata`。 +- 2026-09-14 ~09:23 UTC 审查复核：同一挂载仍在；`/health` `/ready` 200，`schema_version=33`；主仓 CURRENT 槽位仍为 `implementer_cursor` e2e。 +- 用户选择：Wave 0 = 事故抢占；先改方案再批实施。 diff --git a/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md new file mode 100644 index 00000000..07acd47d --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-final-artifact-vault-implementation.md @@ -0,0 +1,49 @@ +--- +title: "最终成果保险库 V0–V6 落地" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Cursor +branch: feature/final-artifact-vault +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: artifact-vault +--- + +# 最终成果保险库 V0–V6 落地 + +## 目标 + +按已审查方案实现 FAV：协议、独立存储、文档/助手接入、存量回填、只读降级、scratch 恢复与清理门禁。 + +## 非目标 + +- 不替代 Unit A/B 纠挂 +- 不占 agent-dev 集成槽 rebuild +- 不扫描 workspace 猜测终态 +- 不新增 tenant_id + +## 步骤 + +1. [x] V0 ADR/schema/eligibility/goose/表征（goose **00034**） +2. [x] V0 领域缺口（review_snapshot_id、ledger hook、task CompleteChatTurn + export 列） +3. [x] V1 存储与 publisher（独立根、项目作用域 CAS、PG/Memory registry） +4. [x] V2 文档接入（persistAgentCommit / ApproverApprove / freeform publish） +5. [x] V3 助手接入（同步 chat + task finalize） +6. [x] V3.5 回填（只枚举账本/完成 turn） +7. [x] V4 降级读取（`FAV_DEGRADED_READ` 默认关） +8. [x] V5 恢复（scratch-only recover-apply 写入 collection；union-deny 跳过 tombstone） +9. [x] V6 门禁与三份演练报告（live e2e 待 Unit A） +10. [x] 缺口补齐：approve/commit 同事务 staging；半发布 exclusive 重试；GET 零写入；`copy_only` pin-export；destroy 预检 CLI + +## 相关链接 + +- ADR: [../../adr/2026-09-14-final-artifact-vault.md](../../adr/2026-09-14-final-artifact-vault.md) +- Remediation: [../../remediation/2026-09-14-final-artifact-vault-remediation.md](../../remediation/2026-09-14-final-artifact-vault-remediation.md) +- Test report: [../../test-reports/2026-09-14-final-artifact-vault-test-report.md](../../test-reports/2026-09-14-final-artifact-vault-test-report.md) + +## 证据 / 备注 + +- 分支：`feature/final-artifact-vault` +- 未执行 `env-service` rebuild；`FAV_DEGRADED_READ` 默认 false +- Goose：`00034_2026-09-14-final-artifact-vault.sql` +- 未合 `dev`；live 6173 / PITR / corpus 仍待 Unit A 另占槽 diff --git a/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md b/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md new file mode 100644 index 00000000..e983a986 --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-final-artifact-vault-plan.md @@ -0,0 +1,61 @@ +--- +title: "最终成果保险库调研与整改方案" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Codex +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: artifact-vault +--- + +# 最终成果保险库调研与整改方案 + +## 目标 + +评估并完善“只归档最终生成物和最终智能体回复”的提案，形成不会与数据库、文档账本和工作区争夺事实源，且能支持受控部分恢复的系统方案。 + +## 范围 + +- 最终文档版本、最终助手回复和必要恢复元数据的归档边界； +- 终态判定、原子发布、不可变清单、校验、授权、保留与删除； +- 工作区清理门禁、降级读取、隔离恢复和数据库重建边界； +- 与现有数据生命周期、迁移语料、文档账本和对话持久化方案衔接。 + +## 非目标 + +- 本轮不实现业务代码、数据库迁移或对象存储； +- 不归档候选稿、流式片段、工具轨迹、租约、临时工作区或其他中间结果； +- 不把保险库定义为运行数据库的完整镜像或自动覆盖线上数据的入口； +- 不执行数据删除、卷切换或恢复写入。 + +## 步骤 + +1. [x] 登记任务和独立 mutex。 +2. [x] 核对现有文档快照、CAS、最终助手消息、任务终态和备份恢复边界。 +3. [x] 评审提案，明确可采用部分、必须修正部分和反模式。 +4. [x] 编写专项调研与整改方案，并并入运行数据生命周期方案。 +5. [x] 更新 CURRENT、FEATURE-MATRIX、gap-matrix 与整改索引。 +6. [x] 完成文档一致性、安全、可行性和对抗性复核。 + +## 验收标准 + +- [x] 明确“最终”的服务端判定规则和禁止进入保险库的内容。 +- [x] 明确数据库、工作区、保险库三者权威边界，不产生静默双写。 +- [x] 方案含 manifest/checksum、幂等发布、清理前门禁、降级读取和隔离恢复流程。 +- [x] 明确可恢复与不可恢复的数据范围、安全/删除要求及恢复演练。 +- [x] 本轮未实施业务代码或破坏性操作。 + +## 相关链接 + +- Research: [最终成果保险库与部分恢复边界](../research/2026-09-14-final-artifact-vault.md) +- Remediation: [最终成果保险库与受控恢复](../../remediation/2026-09-14-final-artifact-vault-remediation.md) +- Test report: 待用户批准实施后创建 + +## 证据 / 备注 + +- 当前受控文档不可变正文位于 PostgreSQL `document_content_blobs`，工作区 `output.md` 是投影。 +- 当前附件 CAS 物理 blob 位于运行状态根，引用元数据位于 PostgreSQL。 +- 当前任务完成摘要通过 `FinalizeAssistantTaskMessage` 写回 `chat_messages`，尚无独立终态成果发布协议。 +- 文档复核补齐：显式发布不可绕过终态、不可变 payload staging、项目作用域加密 CAS/引用删除、存量回填、掉电持久化、敏感内容准入、既有项目授权降级和 scratch 恢复权限。 +- 独立 Claude 安全/对抗复核结果已收敛；whole-document peer 因结构化输出重试耗尽被丢弃，未作为结论依据。 diff --git a/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md new file mode 100644 index 00000000..32eb9432 --- /dev/null +++ b/.doc/ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md @@ -0,0 +1,60 @@ +--- +title: "运行数据精简与迁移语料库方案" +status: done +created: 2026-09-14 +updated: 2026-09-14 +owner: Codex +branch: fix/dev-runtime-data-root-guard +worktree: .worktrees/fix-dev-runtime-data-root-guard +mutex: infra-runtime-state +--- + +# 运行数据精简与迁移语料库方案 + +## 目标 + +评估及时清理旧数据建议的合理性，并设计可恢复、可迁移验证、可展示且不污染长期环境的数据生命周期方案。 + +## 范围 + +- dev/test/main 数据量、schema、项目/文档/run 关系与文件状态； +- 备份/恢复、选择性导出、迁移 corpus、蓝绿轮换； +- retention 分类、TTL、quarantine、GC 和验收门禁。 +- 最终成果保险库 coverage 与清理门禁，但不以其替代完整备份。 + +## 非目标 + +- 不执行 DELETE/TRUNCATE/drop volume； +- 不修改数据库、文件正文或容器； +- 不清理 main/shared KB； +- 不把现有项目 ID 清单视作删除授权。 + +## 步骤 + +1. [x] 只读统计三环境 schema、项目/关键表规模和 dev 文件规模。 +2. [x] 审计项目外键闭包、revision/CAS/workspace 孤儿风险。 +3. [x] 审计现有 backup/restore drill 能力与隔离缺口。 +4. [x] 制定数据分类、最小保留集、migration corpus 和蓝绿轮换方案。 +5. [x] 更新父整改、CURRENT、gap 与 remediation 索引。 +6. [x] 并入最终成果保险库、终态 coverage 和受控部分恢复边界。 + +## 验收标准 + +- [x] 明确说明建议合理的条件和不能替代 migration 的边界。 +- [x] 首轮 retain 建议基于能力覆盖，不按时间盲留。 +- [x] 方案包含备份配对、真实 restore、选择性导入、reload/rebuild、rollback 和延迟销毁。 +- [x] 本轮未实施破坏性操作。 +- [x] 明确最终成果保险库、migration corpus 和完整 DB+state backup 三者互补且分别验收。 + +## 相关链接 + +- Research: [../research/2026-09-14-runtime-data-retention-and-migration-corpus.md](../research/2026-09-14-runtime-data-retention-and-migration-corpus.md) +- Remediation: [../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md](../../remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Final Artifact Vault: [../../remediation/2026-09-14-final-artifact-vault-remediation.md](../../remediation/2026-09-14-final-artifact-vault-remediation.md) +- Test report: 待用户批准实施后创建 + +## 证据 / 备注 + +- dev schema 33 / 366 projects / 主仓状态约 100MB；test schema 29；main schema 25。 +- `DELETE projects` 无法完整清理 task run、revision ledger、blob 与文件状态。 +- 首选新 green DB/state root 导入 retain allowlist，旧 blue 只读隔离后延迟销毁。 diff --git a/.doc/ops/short-term/2026-09-23-integrate-fav-final.md b/.doc/ops/short-term/2026-09-23-integrate-fav-final.md new file mode 100644 index 00000000..beb7397a --- /dev/null +++ b/.doc/ops/short-term/2026-09-23-integrate-fav-final.md @@ -0,0 +1,35 @@ +# integrate-fav-final-2026-09-23 + +| field | value | +|---|---| +| status | in_progress | +| owner | Cursor | +| type | integrate | +| base_dev_sha | 6032dbaa16ea8389ebe871c7e1e532c016d673e7 | +| write_scope | . | +| mutex | git:dev | +| review_state | pending | +| integration_state | pending | +| deployment | not_required | +| cleanup | pending | +| remote_sync | not_requested | + +## Exception registry (FAV protection lift) + +| field | value | +|---|---| +| owner | Cursor | +| reason | 用户 2026-09-23 明确授权「都推进至完成」并选择解除保护 | +| overridden_clause | CURRENT「禁止整支 merge FAV」 | +| scope | feature/final-artifact-vault 整支合入 local_dev | +| exit_condition | 合入后 flag 默认关；MATRIX events-fav 保持 partial；live 不放量 | +| restore_action | 合入后 CURRENT 修订为「FAV live 仍不放量；FAV_DEGRADED_READ 默认 false」 | + +## Migration note + +FAV branch ships ; local_dev already has +. Renumber FAV migration to . + +## Goose renumber + +resolved: FAV branch 00034 dropped; local_dev already has equivalent 00036_final-artifact-vault.sql diff --git a/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md new file mode 100644 index 00000000..84d9d29a --- /dev/null +++ b/.doc/remediation/2026-09-14-dev-runtime-data-root-and-projection-remediation.md @@ -0,0 +1,216 @@ +--- +title: "dev 运行数据根隔离与文档投影整改方案（审查修订）" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +--- + +# dev 运行数据根隔离与文档投影整改方案（审查修订） + +## 审查结论 + +2026-09-14 多视角审查确认：**诊断成立，原文不可整包实施。** + +现场在审查当时（约 09:23 UTC）仍为 split-brain： + +| 项 | 值 | +|---|---| +| `shuimu-agent-app-dev` created | `2026-09-14T08:40:44Z` | +| Compose working dir | `/tmp/as-dev-wf409-merge` | +| app `/app/data` | `/tmp/as-dev-wf409-merge/data/dev` | +| pi/cursor workspaces | `/tmp/as-dev-wf409-merge/data/dev/workspaces` | +| `/health` `/ready` | 200，`schema_version=33` | +| 主仓 CURRENT 集成槽 | `implementer_cursor` / `e2e` / workflow-plan | + +用户已拍板： + +1. **Wave 0 授权方式 = 事故抢占**：不排队等 e2e 结束；获批实施后立即停 runner/app，只读保全两端数据根，再从主仓目录纠挂。 +2. **本轮只改方案**：消解波次矛盾后再单独批准实施。本文未授权 rebuild、未授权写库、未授权删除 `/tmp`。 + +原文把「恢复可读」和「投影 outbox / 数据治理 / 完整 SLO」绑成同一批准与同一关门条件。修订后拆成三个互不阻塞的批准单元。 + +## 批准单元 + +| 单元 | 目的 | 关闭事故？ | 状态 | +|---|---|---|---| +| **Unit A** | 事故遏制与纠挂：停写、保全、从主仓 remount | 恢复当前可读/可写 | 方案已修订，**实施待再批** | +| **Unit B** | 防复发：绝对状态根 + 脚本硬门禁 + readiness 身份门禁 | 阻止同类错挂再次显示 healthy | 方案已修订，**实施待再批** | +| **Deferred** | ledger 只读降级、projection outbox、历史 work_dir 治理、监控产品、迁仓外路径 | 否 | 记 long-term，不挡 Unit A/B 关门 | + +`RUNTIME-DATA-ROOT-SPLIT-BRAIN` 与 FEATURE-MATRIX `infra` 在 **Unit A 验收 + Unit B 负例** 后关闭。不得等 Deferred。同 worktree 上的数据生命周期方案是 sibling：可以并行设计，**不得**成为 Unit A 纠挂的前置或本 gap 的关门条件。 + +## 目标 + +1. 任意 build source / linked worktree / `/tmp` checkout 都不能改变已部署环境的持久数据根。 +2. 错挂或身份不一致时 `/ready` 必须 503；`/health` 只表示进程存活，不得作为部署就绪依据。 +3. Unit A 完成后，项目 143 等活跃受控正文在正确数据根上可打开；不以 08:54 历史 SHA 作为事后硬门槛。 +4. 真实内容分叉仍 409 并保全双方；安全 missing 的自动重建与 outbox **不在本方案实施范围**。 + +## 术语（强制） + +| 术语 | 定义 | +|---|---| +| `SOURCE_ROOT` | 本次构建源码目录，可来自受控集成 worktree | +| `RUNTIME_STATE_ROOT` | 各环境状态目录的**绝对父路径**。短期默认 = 主仓 `…/Agent-System/data` | +| `ENV_STATE_DIR` | `${RUNTIME_STATE_ROOT}/`，例如 `…/data/dev` | +| `SHARED_STATE_ROOT` | 共享知识库状态根，**不**拼接 `/`，不随 `SOURCE_ROOT` 变化 | +| bind mount source | `docker inspect` 的宿主机 `Mounts.Source` | +| build context | 镜像构建用的源码目录，允许与状态根不同 | + +禁止把 `RUNTIME_STATE_ROOT` 理解成已经带 env 的 `data/dev`。Compose 公式一律是 `${RUNTIME_STATE_ROOT}/${ENV}/...`。`.storage-identity.json`（Unit B）落在 `ENV_STATE_DIR/`。 + +禁止把 Wave 0「容器 source」读成 `SOURCE_ROOT`。Wave 0 验收的是 **bind mount source**。 + +## 核心决策 + +### D1 — 代码根与状态根分离（Unit B；Wave 0 不依赖新变量） + +- Compose 可变 bind 改为 `${RUNTIME_STATE_ROOT}/${ENV}/...`，禁止 `./data/`。 +- 短期兼容：用主仓 `data/` 作为 `RUNTIME_STATE_ROOT`。**本方案禁止把状态根迁到 `/var/lib/agent-system`。** +- 禁止硬编码个人 HOME。解析顺序：显式环境文件 → Git common-dir 推出的 **primary** 根下的 `data/`。 +- 显式 `RUNTIME_STATE_ROOT` **不能**把任意绝对路径变成合法根。解析结果必须：绝对、`realpath`、等于宿主机 allowlist、环境后缀与命令一致、不落在任何 linked worktree toplevel、不是 `/tmp` 集成目录。 +- shared compose 使用 `SHARED_STATE_ROOT`，不套用 `/${ENV}`。 +- `.regulations` 若仍从 `SOURCE_ROOT` 相对挂载，必须在 Unit B 清单里单独点名；不得假装已被 data identity 覆盖。 +- **否决记录**：Postgres 逃过本次事故是因为 named volume。本轮仍保持宿主机 bind，以便 Unit A 做两端差异保全。不用 named volume 收口 `/app/data`，也不把「固定 `--project-directory`」当成 Unit B 的全部答案——脚本必须同时导出绝对 bind，否则合法集成 rebuild 仍会解析到调用目录。 + +### D2 — 存储身份是配对仪式，不是可复制哨兵（Unit B） + +- 每个 `ENV_STATE_DIR` 放受保护的 `.storage-identity.json`：`environment`、UUID、schema version、**canonical host path**；不含 secret。 +- PostgreSQL 用**新的单行表**，不复用 `iam_system_config`。 +- **禁止自动初始化。** app 启动只校验，不 mint、不把文件 UUID 写进 DB，也不把 DB UUID 写到空根。 +- 真首启（允许 pair 命令写入同一 UUID）当且仅当：DB 无业务行（至少 `document_records` / `document_snapshots` 为空）**且**文件根无业务布局。 +- 已有业务库 + 文件根无合法 sentinel → fail closed / `/ready` 503。现场错误根已有 27 个文件，**「空目录」不是可用谓词**。 +- DB 有 UUID、文件缺失 → 503，不回写。 +- 文件有 UUID、DB 无 → 503，防止新卷认领旧根。 +- UUID 一致但 `realpath` ≠ DB 固化 canonical path → 503（克隆/备份根）。 +- runner 一致性**不**靠进程内 HTTP：`cursor-agent` 现为 `alpine sleep`，pi/cursor 只挂 `workspaces`，读不到状态根哨兵。Unit B 由部署脚本 `docker inspect` 三个容器的 `Mounts.Source` + 文件 sentinel + DB 行。进程内 `/ready` 只比 app 可见 sentinel 与 DB。 +- 未认证 `/ready` 只返回聚合布尔和稳定错误码，禁止 UUID、路径、mount source、doc_id、计数明细。 + +### D3 — 读取真源与自动重建拆开（Deferred / 后续） + +- Unit A **不**改 GET 语义；纠挂后投影文件回到视野，现有 fail-closed 即可恢复 200。 +- 以后若做「ledger head 可读」，必须：identity 已通过、GET **只读**、真实 fork 仍 409 `DOCUMENT_CONTENT_PROJECTION_DRIFT`、不得在 GET 路径写盘。 +- 不得把「此刻 stat 不到 `output.md`」单独当成可自动重建。曾出现 fork/mismatch 的文档禁止降级为 MISSING 自愈。 +- 本方案**删除**原文 D3「MISSING 后台从 head 原子重建」。任何写回只允许在独立 Deferred 产品里设计，且必须有路径沙箱（`Lstat` + `EvalSymlinks`，realpath 严格位于 `ENV_STATE_DIR`）。 + +### D4 — transactional outbox 降为 Deferred + +- 当前 409 的根因是错挂空根，不是缺 outbox。 +- 仓库已有 `InspectProjection`、`ProjectOutputMarkdown`、`fsutil.WriteFileAtomic`、`document-content-reconcile`。新 job 表 / projector 进程是第二产品。 +- PATCH 若以后暴露 `projection_status`：ledger commit 成功则 **HTTP 一律 200**，状态放 body/头；禁止用非 2xx 表示「账本未保存」。 +- fork：服务继续，该文档 409，保全双方，立即人工处置。**不是**停 app/runner。 + +## 分波次计划 + +| 波次 | 单元 | 内容 | 验收 | +|---|---|---|---| +| Wave 0 | A | 事故抢占：停 pi/cursor/app（保留 Postgres 卷）；只读保全主仓根与 `/tmp` 错误根；三方 disposition；**仅从主仓 checkout** 用现有相对 bind 纠挂三容器 | 见下方 Runbook；**不**验 storage identity，**不**改 143 正文 | +| Wave 1 | B | `env-service.sh` + compose 同时改：导出绝对 `RUNTIME_STATE_ROOT`，可变 bind 全部改公式；worktree / `/tmp` 默认 mutating 失败且不 recreate 现网 | 两个不同 `SOURCE_ROOT` 渲染后，app/pi/cursor 的 host mount source 字节级相同，且等于 allowlist 中的 `ENV_STATE_DIR` | +| Wave 2 | B | 与 Wave 1 同一批落地。CI 禁相对 `./data/`；**允许**绝对路径落在 primary checkout 的 `data/`。shared 只用 `SHARED_STATE_ROOT` | 合法配置不会被「工作树内 data/」误杀；dev 命令不能挂 test/main 根 | +| Wave 3 | B | 文件/DB identity + pair 命令；`/ready` 增加 `storage_ready` 布尔；dev compose healthcheck 改为 `/api/v1/ready` | 复现本次 split-brain 时 `/ready` 503、容器 unhealthy；`/health` 仍可 200 | +| — | Deferred | ledger GET 与 MISSING 解耦；outbox/projector；历史 work_dir 治理；五条时序指标与 60s SLO；迁仓外路径 | 另开 long-term，不进本 gap 关门 | + +Wave 1 若只改脚本、不改 compose，从 `/tmp` 做「合法 rebuild」仍会把 `./data/dev` 解析到空根。因此 **Wave 1 与 Wave 2 必须同批**。 + +## Unit A — 即时恢复 Runbook（实施待再批） + +授权：事故抢占。主仓 CURRENT 即使仍显示 `implementer_cursor` / `e2e`，实施时也要写入抢占记录，停止该 e2e 对 `agent-dev` 的使用，并通知 queue 中的 `governance-action-level-authz`。以**主仓** CURRENT 与 `docker inspect` 为准，禁止信任任何 worktree 副本里的 `idle`。 + +**冻结 = 停进程，不是文档 mutex，也不是「请大家别保存」。** 仓库没有 maintenance 开关。 + +1. 在主仓 CURRENT 登记抢占：`holder=incident-preempt`，`scope=RUNTIME-DATA-ROOT Unit A`，`mutex=infra-runtime-state`。记录三容器 ID / created / `Mounts.Source` / compose working dir。 +2. **先停写再盘点**（保留 `shuimu-context-pg-dev`）： + + ```bash + docker stop shuimu-pi-agent-dev shuimu-cursor-agent-dev shuimu-agent-app-dev + ``` + + 未停止不得开始差异扫描。test/main 容器禁止触碰。 +3. 对主仓 `…/Agent-System/data/dev` 与 `/tmp/as-dev-wf409-merge/data/dev` 做只读 inventory：相对路径、size、mtime、SHA。使用 `find -P` / 不跟随 symlink；先 `Lstat` 跳过非常规文件。特别列出 08:40 UTC 之后的新增/修改，以及 `agent_sessions`、attachments、CAS、非 `output.md`。 +4. 备份目的地必须在仓库外、权限 `0700`、与运行数据根分离。`agent_sessions` / token / `.env` 只记存在性与大小，不拷贝明文。禁止把 `/tmp` 备份写进 `data/dev` 或 git 工作区。禁止 `cp -r` 覆盖主根。 +5. 按 **DB/ledger vs 主根 vs 错误根** 出 disposition 清单（保留主根 / 回收错误根独有项 / 标 orphan）。项目 143 只是抽样锚点，不能代表 100 条记录或 68 条主仓历史 missing。冻结时刻重新计算三方 hash；**删除「必须仍为 `f283...f039`」硬门槛**。仅当冻结快照上 143 仍三方一致时，才允许「只纠挂、不改该文件」。 +6. 纠挂**必须**在主仓执行，禁止 `env-service.sh` 从 worktree 或 `/tmp/as-dev-wf409-merge` 调用： + + ```bash + PRIMARY=/home/weichen/projects/Agent-System + docker compose --project-directory "$PRIMARY" \ + -f "$PRIMARY/docker-compose.dev.yml" \ + up -d --build \ + app pi-agent cursor-agent + ``` + + 数据库 named volume 不变。纠挂后立刻 `docker inspect`：三个 bind source 必须是 `$PRIMARY/data/dev` 及其 `workspaces`，不得再指向 `/tmp`。 +7. 验证：`GET /api/v1/docgov/documents/doc_proj143_intake/content` 在冻结快照仍一致时应 200；抽查若干活跃非模板文档的投影可见性。`/ready` 仍按现状理解（此时还没有 storage identity）。 +8. 错误根只读保留到报告签字；本方案**不授权删除** `/tmp/as-dev-wf409-merge`。 +9. **禁止**在 Wave 0 对项目 143 保存 marker、禁止真实 PI 改文。那些属于 Unit B 之后的页面验收，另批。 + +### Wave 0 验收（仅这些） + +- [ ] 停写发生在 inventory 之前；Postgres 未删卷、未重建。 +- [ ] 两端 inventory + 仓库外备份 + disposition 清单落盘到 test-reports assets。 +- [ ] app/pi/cursor 的 bind mount source 回到主仓 `data/dev`（或 `…/workspaces`），compose working dir 不再是 `/tmp/as-dev-wf409-merge`。 +- [ ] 项目 143 在「冻结快照仍一致」时 content 200，文件内容与冻结 hash 一致。 +- [ ] 错误根仍在且未被覆盖/删除。 +- [ ] 主仓 CURRENT 记录抢占与恢复结果；不以 `/health` 200 作为通过。 + +## Unit B — 防复发验收矩阵 + +### 脚本与 Compose + +- [ ] primary repo、linked worktree、`/tmp` integration checkout 三种 `SOURCE_ROOT` 下，合法 config 的 state bind source 相同，且等于 allowlist 中的绝对 `ENV_STATE_DIR`。 +- [ ] 未配置 / 相对 / 不存在 / 环境名不符 / 指向 worktree 或 `/tmp` 的 state root 均 fail fast，已有容器不被 recreate。 +- [ ] 显式 `RUNTIME_STATE_ROOT` 不在 allowlist 时失败（不能靠「绝对路径」绕过）。 +- [ ] dev 命令不能挂 test/main 根。 +- [ ] `docker inspect`：app、pi-agent、cursor-agent 三者 mount source 同一 realpath 前缀。 +- [ ] CI 断言的是「禁止相对 `./data/` + 两次渲染 source 相同」，**不是**「不得出现 primary 工作树内的绝对 `data/dev`」。 + +### Readiness + +- [ ] 错误空根、业务库无 sentinel、canonical path 不符：`/ready` 503，dev compose healthcheck 显示 unhealthy。 +- [ ] `/health` 保持 liveness，部署脚本不得用它当就绪。 +- [ ] `/ready` 不泄露 UUID/路径/doc 内容。 + +## 回滚策略 + +- Unit B 代码可回滚，但 **不得回滚/重建已配对的 storage UUID**；旧相对 bind **禁止**作为回滚方案。 +- Unit A 失败：保持停写，不要把容器重新绑回 `/tmp`。 +- 任何自动修复（若未来另批）只覆盖 `MISSING + identity matched + 无历史 fork`；fork 保留证据，服务继续。 + +## 方案边界 + +| 阶段 | 允许 | 禁止 | +|---|---|---| +| 审查修订（本轮） | 改 `.doc` 方案与 SSOT | 改业务代码、rebuild、写库、覆盖/删除正文或 `/tmp` | +| Unit A（再批后） | 停 dev app/pi/cursor；只读保全；从主仓纠挂三容器 | 改 test/main 运行态；删错误根；Wave 0 写 143；从 worktree 调 `env-service` | +| Unit B（再批后） | 改仓库内 compose/脚本/`/ready`；test/main **文件**可改绝对变量 | 对运行中的 test/main 做 mutating env-service；自动 mint identity；迁仓外 | +| Deferred | 另开 long-term | 当作本事故关门条件 | + +## 监控（Unit B 最小集 / 其余 Deferred） + +Unit B 只要求：部署日志记录 source commit、compose working dir、state root、实际 mount source hash；禁止记录正文与凭证。`storage_identity_mismatch` 作为 `/ready` 布尔即可。 + +五条 Prometheus 指标、60 秒 pending SLO、`/ready` 投影计数阈值整节移出本方案。 + +## 相关链接 + +- Audit: [dev rebuild 从临时 worktree 挂载空数据根](../audit/2026-09-14-dev-runtime-data-root-split-brain.md) +- Short-term: [dev runtime data root guard](../ops/short-term/2026-09-14-dev-runtime-data-root-guard.md) +- gap-matrix 行: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` + +## 审查修订记录 + +原文矛盾已在本文消除，不再保留互斥表述： + +- `RUNTIME_STATE_ROOT` 同时被写成「每环境根」和「父目录+/」 +- 「本轮不重建」与 Wave 0 重建互相否决（现按阶段表） +- Wave 0 冻结 143 SHA 与 Runbook 要求保存 marker +- Wave 0 验收 storage identity（Wave 3 才有） +- Wave 1 宣称 bind 固定但 compose 仍相对 +- Wave 2 CI 禁止工作树内 `data/` 与短期默认根互斥 +- fork「停机式处理」与 409 继续服务互斥 +- PATCH 投影失败的 HTTP 状态分叉 +- GET `projection_status` 与 job state 混用 +- `CONTENT_FORK` 与 `DOCUMENT_CONTENT_PROJECTION_DRIFT` 对外码漂移 +- shared compose 套用每环境公式 +- 「暂停售写」未定义 +- 恢复与 outbox/治理/完整 SLO 同一关门条件 diff --git a/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md b/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md new file mode 100644 index 00000000..ed11b8bd --- /dev/null +++ b/.doc/remediation/2026-09-14-final-artifact-vault-remediation.md @@ -0,0 +1,247 @@ +--- +title: "最终成果保险库与受控恢复整改方案" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +deepened: 2026-09-14 +--- + +# 最终成果保险库与受控恢复整改方案 + +## 背景 + +提案评审见[最终成果保险库调研](../ops/research/2026-09-14-final-artifact-vault.md)。当前受控文档账本、助手消息和附件 CAS 分别覆盖局部持久化，但都仍与运行 PostgreSQL 或 `RUNTIME_STATE_ROOT` 绑定；工作区 split-brain 已证明成果定位不能依赖 `output.md` 路径。 + +本方案把 FAV 作为[运行数据生命周期方案](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md)的长期成果层。它不替代当前事故的固定数据根整改，也不替代完整数据库备份。 + +## 目标 + +1. 只有服务端确认的终态文档、终态助手回复及其必要附件进入 FAV，中间结果准入率为 0。 +2. 每个成果都有稳定 UUID、语义 manifest、payload SHA、来源终态证据、保留策略和可验证发布回执。 +3. FAV 使用独立存储身份，工作区清理、worktree 删除和 app rebuild 不影响已验证成果。 +4. 数据库正文缺失但授权图完整时可只读降级；数据库关系损坏时只能经 scratch 恢复、冲突预览和人工 promotion。 +5. 完整数据库恢复、migration corpus 与成果级恢复分别演练、分别签字。 + +## 核心决策 + +1. **互补而非替代**：PostgreSQL 是运行关系与授权权威；FAV 只对已归档 payload 与 manifest 权威；workspace 是可清理投影。 +2. **事件准入而非目录扫描**：由 `agent_commit`、`approved`、`sealed`、显式 publish 和 completed chat turn 发起准入判断，禁止扫描 workspace 猜测“final”；显式 publish/pin 只是请求，不能绕过终态 allowlist。 +3. **事务 outbox**：业务终态与 publish request 同事务；异步 publisher 幂等写对象并最后写 commit marker。 +4. **默认只归档高价值回复**：completed + `export_policy=full` 自动进入；`copy_only` 仅显式 pin；`none` 永不进入。 +5. **恢复默认生成新受控对象**：`recover-as-collection` 保留来源；只有依赖、授权和 ID 无冲突时才允许 `relink`。 +6. **清理受覆盖率门禁**：应归档而未 `verified` 的成果会阻止受管 dataset/持久状态源销毁；普通计算 workspace 在终态 payload 已冻结后可按 TTL 清理，不能再成为成果事实源。 + +### 终态准入协议（U1 必须固化） + +| 类型 | 允许的权威数据库事实 | payload 来源 | 拒绝条件 | +|---|---|---|---| +| Agent 文档提交 | `document_agent_candidates.status='committed'`，存在 `output_snapshot_id`，且对应 `document_change_events.change_kind='agent_commit'` 的 `to_snapshot_id` 与该 snapshot 一致 | `document_content_blobs` 对应不可变 snapshot | ready/needs_confirmation/rejected/stale/quality_failed、snapshot/event 不一致 | +| sealed 文档版本 | `document_versions.status='sealed'` 且 `sealed_at` 非空 | 该版本 `head_snapshot_id` | open、head 缺失 | +| approved 受控文档 | `document_records.current_state='approved'`、`approved_at` 非空且 `review_snapshot_id` 非空 | 冻结的 `review_snapshot_id`，不得取 workspace 当前文件 | reviewing/approving/rejected、review snapshot 缺失 | +| 自由文档发布 | 显式 publish 事件引用已存在的不可变 ledger snapshot | 事件指向的 snapshot | 只有 UI 动作但无不可变 version，或事件后才读取“最新”正文 | +| 助手最终回复 | `chat_turns.status='completed'`、`completed_at` 和唯一 `assistant_message_id` 均存在；同事务对精确正文计算的 policy snapshot 为 `full`，或为 `copy_only` 且存在 completed 后的显式 pin 事件 | 同事务冻结到 staging 的精确消息 revision/SHA | pending/streaming/failed/cancelled、placeholder/card/status、policy none/restricted/分类失败、pin 时未 completed | + +HTTP/SSE handler 只能调用这一套领域 eligibility 服务，不能自己根据字符串或响应结束推断终态。U1 先以当前数据库列和约束写 characterization test；若发现任一完成链缺少上述事实，先补领域状态机/事务边界，再允许接入 FAV。 + +## 高层架构 + +```mermaid +flowchart TB + A[Document or chat terminal transaction] --> B[Operational PostgreSQL] + A --> C[artifact.publish.requested outbox] + C --> D[Idempotent publisher] + D --> E[Content-addressed blobs] + D --> F[Semantic manifest] + F --> G[Commit marker and receipt] + G --> H[Cleanup coverage gate] + G --> I[Authorized degraded read] + G --> J[Scratch recovery and reconciliation] +``` + +### 发布状态机 + +```mermaid +stateDiagram-v2 + [*] --> requested + requested --> publishing + publishing --> verified + publishing --> retryable_failed + retryable_failed --> publishing + verified --> tombstoned + tombstoned --> purged +``` + +只有 `verified` 成果可用于清理门禁和恢复。manifest 写完但没有 commit marker 的对象视为未发布；`tombstoned` 内容不能被读取或恢复。 + +## 数据与存储契约 + +### 运行库注册表 + +新增等价模型： + +```text +final_artifacts + artifact_uuid, artifact_type, logical_resource_id + environment, project_id, project_external_key, source_event_id + source_snapshot_id / source_message_id / source_run_id + payload_sha256, manifest_sha256, vault_storage_identity + terminal_state, policy_version, retention_class + publication_status, requested_at, verified_at, tombstoned_at + +artifact_payload_staging + staging_uuid, project_id, source_event_id, payload_sha256 + encrypted_payload / content_blob_ref, classification, created_at, releasable_at + +artifact_blob_references + artifact_uuid, project_scope, payload_sha256, envelope_key_ref + reference_status, retention_class, legal_hold, timestamps + +artifact_publication_attempts + artifact_uuid, idempotency_key, attempt_no, error_code, receipt, timestamps + +artifact_deletion_ledger + artifact_uuid, reason, authority, effective_at, applied_at, audit_ref +``` + +实现时优先复用现有 `outbox_events` 投递框架；归档注册与文档/消息终态的原子边界需由所属服务事务负责，不能由 HTTP handler 事后拼接。 + +幂等 key 由业务终态事务确定性生成：`artifact_type + environment + project_external_key + logical_resource_id + source_event_kind + source_event_id + payload_sha256 + policy_version`。数据库对该 key 建唯一约束；同一终态重试/重新发布复用 key，新终态事件或新 payload 生成新 artifact，不用 outbox event ID 或 publisher 本地状态充当业务幂等键。 + +当前仓库没有 tenant 实体；本期安全边界沿用现有 `environment + project_members/role-admin` 模型，不新增无法校验的 `tenant_id`。若未来引入组织/租户层，须另立 ADR 和数据迁移，不能仅改路径字段。 + +终态事务必须把**精确 payload 字节或不可变 content blob 引用、SHA、来源版本**一并冻结到 staging，outbox 只引用 staging；publisher 不得在事务后重读可修改或会级联删除的消息行。staging 在成果 `verified` 或依法 `tombstoned` 前不得回收，且正文不能进入普通 outbox payload、日志或指标。 + +### Vault 对象布局 + +```text +scopes///blobs/sha256/<2>/<2>/ +manifests//.json +commits//.json +tombstones//.json +catalog/checkpoints/.json +``` + +路径不包含用户文件名、Prompt 或数据库自增 ID。manifest 记录逻辑标识、来源、成员、hash、终态证据、schema/producer/storage identity、敏感级别和保留策略；commit marker 最后生成。 + +U1 必须交付版本化 `manifest.schema.json`、`commit-marker.schema.json` 和 `tombstone.schema.json`，并由同一 Go 类型/validator 验证。schema 采用 major/minor：读端必须兼容同 major 的旧 minor；未知 major fail closed 并进入迁移队列，不能静默猜字段。commit marker 至少绑定 artifact UUID、manifest SHA、全部 required member SHA、storage identity、committed_at 和 publisher version。 + +CAS 只在项目安全作用域内去重，并采用项目作用域 envelope key；`artifact_blob_references` 是 tombstone、legal hold 和物理回收的权威引用账本。tombstone 立即撤销该 artifact 的读取和密钥包装；仅当该作用域内最后一个有效引用消失且 retention/legal hold 允许时，才清除 key wrapping 与 blob。跨项目相同 SHA 必须保持密码学与删除边界隔离。 + +删除事务在运行库写 `artifact_deletion_ledger` 与删除 outbox，vault tombstone 是独立持久的拒绝恢复证据。读取/恢复采用 **union-deny**：任一侧声明删除都拒绝；两侧不一致进入 quarantine 和告警，不得恢复。purge 后仍永久保留最小 tombstone（或经校验的压缩删除目录），完整备份恢复后重放删除账本，防止旧备份复活数据。 + +## 分波次计划 + +| 波次 | 内容 | 验收 | +|---|---|---| +| V0 协议与 ADR | 固化三层权威边界、终态准入矩阵、manifest schema、恢复语义和删除模型 | ADR 与 JSON schema 评审通过；所有成果类型都有允许/拒绝状态 | +| V1 存储与发布内核 | 建 `internal/artifactvault`、注册表/staging/outbox/attempt/reference/deletion migration、filesystem backend、项目作用域加密 CAS、manifest/commit marker 和幂等 publisher | 重复投递只生成一个逻辑成果；崩溃点重试不产生半发布可见对象；坏 SHA 拒绝 verified；跨项目同 SHA 不共享删除或密钥边界 | +| V2 文档接入 | 从账本 snapshot 接入 agent committed、sealed、approved 和显式 publish；approved hook 改读账本而非 workspace | ready/rejected/stale/quality_failed 不入库；清空 workspace 后仍能校验/读取归档正文 | +| V3 助手回复接入 | 把消息正文定稿、chat turn completed 和 publish request 收敛到原子服务边界；复用 `replyexport` 分类并增加真实回合状态校验 | placeholder/stream/error/card 不入库；completed+full 精确归档一次；SSE done 前已有 durable request | +| V3.5 存量终态回填 | 只从权威数据库账本、批准事件和 completed chat turn 枚举既有终态，以来源事件 + payload SHA 生成确定性请求；缺正文/证据项进入 quarantine 或批准排除 | 重跑不增加逻辑成果；不扫描 workspace；每个排除项有原因、批准人和审计记录；清理 coverage 有可达分母 | +| V4 查找与降级读取 | 建授权后的 artifact lookup；正文缺失且授权完整时返回带 degraded 标记的 FAV 内容；权限图缺失时 fail closed | 跨用户/跨项目负例通过；GET 不写数据库；vault 不可用时返回 typed degraded/error 而非伪 404 | +| V5 恢复工具 | 建 inventory/verify/recover-plan/recover-apply；强制验证 scratch identity，使用无运行库权限的恢复凭证；默认 recover-as-collection，relink 需完整依赖、授权和人工确认 | DB 全删的隔离环境可重建成果目录；运行库目标、未授权操作者、计划篡改、冲突、删除账本和缺权限依赖均阻止 promotion | +| V6 清理与运维门禁 | dataset inventory/GC 加 coverage；readiness 暴露 identity/connectivity/lag，metrics/alerts/checksum scrub/复制与 runbook 落地 | 未 verified 成果阻止销毁；workspace 清空、app rebuild、离线副本恢复演练均 PASS | + +## 实施单元与文件落点 + +### U1. 终态协议、manifest 与数据库模型 + +- **目标**：先建立可审计的准入和恢复契约，避免把所有“完成”字符串当终态。 +- **文件**：`.doc/adr/` 新 ADR；`deploy/postgres/goose/000xx_*final_artifact*.sql`；`internal/artifactvault/types.go`、`manifest.go`、`eligibility.go`、`manifest.schema.json`、`commit-marker.schema.json`、`tombstone.schema.json` 及对应测试。 +- **测试场景**：每类允许事件生成确定 artifact key；所有候选/失败/中间状态拒绝；未终态的显式 publish/pin 仍拒绝；secret、认证材料、restricted 或分类失败均在正文进入 outbox/staging 前 fail closed；manifest 必填字段、版本和 SHA 错误 fail closed；迁移 Up/Down 与约束验证。 +- **完成证据**：准入矩阵与 schema 一致，未分类类型默认拒绝。 + +### U2. 独立存储与幂等 publisher + +- **依赖**：U1。 +- **文件**：`internal/artifactvault/store.go`、`filesystem_store.go`、`publisher.go`、`publisher_test.go`；`internal/config/config.go`；环境配置与 Compose 校验。 +- **执行说明**：vault 使用独立最小权限身份；凭证只从受管 secret/env 注入，不进入仓库、manifest、日志或业务数据库。后端必须启用传输加密和静态加密，manifest 只保存 key reference。filesystem 后端采用同文件系统临时文件、校验、文件 `fsync`、原子 no-replace rename、父目录 `fsync`，blob 与 manifest 持久化后才提交 marker。 +- **测试场景**：同一 outbox 重复消费；blob 已存在；每个写入/fsync/rename 边界强制终止后重启；损坏 blob；错误 storage identity；凭证轮换；vault 暂时不可用后重试；跨项目同 SHA、单方 tombstone、legal hold 和最后引用 purge。 +- **完成证据**：只有完整且已持久化的 manifest + commit marker 可见，publisher 不读 workspace 推断成果；任一项目删除不破坏其他项目成果。 + +### U3. 文档账本与治理终态接入 + +- **依赖**：U1、U2。 +- **文件**：`internal/docrevision/pg_mutate.go`、`internal/docgov/engine.go`、`internal/bizdev/service.go`、`internal/server/document_revision_handlers.go` 及 PG/服务测试。 +- **执行说明**：先补 characterization/PG 测试，再调整事务边界和 approved hook。 +- **测试场景**：agent commit、sealed final、approved review snapshot、已满足终态 allowlist 的显式 publish；未终态显式 publish、pending/reject/stale/quality failed 负例；投影缺失但账本和归档一致；同 run 重试。 +- **完成证据**：归档 payload 来自不可变账本 snapshot，不读取 `output.md` 作为事实源。 + +### U4. 最终助手回复原子接入 + +- **依赖**：U1、U2。 +- **文件**：`internal/ctxmgr/chat_turns.go`、`message_update.go`、`internal/server/task_finalize.go`、`internal/chat/session.go`、`internal/replyexport/policy.go` 及对应测试。 +- **执行说明**：先刻画同步回复、异步任务、SSE 断线重试三条完成链，再收敛到一个 terminal-finalize 领域事务。该事务以锁定的 message revision/content 重新计算并冻结 policy snapshot，不能信任 handler 传入或较早缓存的 `export_policy`；pin 是对已 completed 的精确 revision/SHA 追加独立终态事件。 +- **测试场景**：completed+full、completed+copy_only+显式 pin；未 completed 的 pin/publish、凭证回显、restricted/分类失败、placeholder 更新、failed/cancelled、card/status、重复 finalize、SSE done 前后崩溃；请求后修改/删除原消息再消费仍归档原 SHA；日志和指标不含正文。 +- **完成证据**：消息精确正文、completed turn、不可变 staging 和 publish request 同事务，不存在“完成但永远未请求归档”或“发布时读到后来正文”的窗口。 + +### U7. 存量终态回填与覆盖率基线 + +- **依赖**：U3、U4。 +- **文件**：`cmd/artifact-vault/backfill.go`、`internal/artifactvault/backfill.go` 及 PG 测试；dataset inventory 工具。 +- **执行说明**：只枚举不可变 document snapshot/终态事件和 completed chat turn，不读 workspace；用稳定来源事件 + payload SHA 幂等生成 staging/request。证据不足或正文缺失项进入 quarantine；排除需理由、批准人、有效期和审计记录，不能用“旧数据”批量豁免。 +- **测试场景**：回填重复执行、并发新终态、旧消息已删、缺 snapshot、批准排除过期、quarantine 修复后重跑。 +- **完成证据**：存量 coverage 的分子、分母、quarantine 和批准排除均可复算，V6 清理门禁不靠人工改数字通过。 + +### U5. 授权查找与只读降级 + +- **依赖**：U2–U4。 +- **文件**：`internal/artifactvault/catalog.go`；`internal/server/project_doc_handlers.go`、`reply_export_api.go`、`handlers.go` 及鉴权测试。 +- **执行说明**：降级读取不是独立权限或直连 vault 能力；必须先由运行库中的 resource→project 关系定位资源，再用现有 `requireProjectAccess`/系统管理员判定完成与正常读取相同的服务端授权。manifest 只用于交叉校验 project external key/sensitivity，绝不自行授予权限；任一授权依赖缺失或结果不确定即 fail closed。 +- **测试场景**：DB 正常时不走 FAV；投影/正文缺失且现有项目授权完整时降级；项目关系缺失、越权、跨项目同 SHA、tombstone、manifest/SHA 异常全部 fail closed；GET 零写入。 +- **完成证据**：响应显式标识来源与降级原因，不再把存储不一致伪装为 404。 + +### U6. 隔离恢复、清理门禁与演练 + +- **依赖**：U2–U5、U7。 +- **文件**：`cmd/artifact-vault/`、`internal/artifactvault/recovery.go` 与测试；`scripts/backup-system-db.sh`、`restore-system-db-drill.sh`；dataset 生命周期工具；`.doc/test-reports/`。 +- **执行说明**：`recover-apply` 先核验数据库 identity 并拒绝 dev/test/main 与任何非 scratch 目标，只接受无运行库权限的恢复专用凭证。操作者须通过 `role-admin` 与明确恢复能力校验；relink/promotion 还需目标项目授权和指定数据责任人确认。actor、plan hash、目标 identity 和 promotion 结果写不可变审计。恢复计划显式分类 UUID collision、SHA mismatch、partial dependency、authorization gap、metadata conflict；默认全部 fail，只有 recover-as-collection 可对允许的孤立成果生成新 UUID，relink 不允许 skip/prompt 后带冲突提升。 +- **清理门禁**：只允许 `dataset inventory/gc/destroy` 和环境 teardown 脚本执行受管销毁，preflight 服务不可用或 coverage 非 PASS 时 fail closed；vault/app 身份没有删除 dataset 的权限，运维删除身份单独审计。普通计算 workspace 可按 TTL 清理，因为终态确认前已把 payload 冻结进 staging；原始 shell/root 删除无法由应用层绝对阻止，必须靠主机权限与操作制度限制，不能在验收中声称“拦截任意 rm”。 +- **测试场景**：空 scratch 库 recover-as-collection；依赖齐全 relink；五类冲突；运行库 DSN、未授权操作者、计划 hash 被篡改、缺 blob、删除账本/tombstone 不一致与重放；未 verified 或 preflight 不可用阻止受管 GC；workspace 清空与 app rebuild；异机副本恢复。 +- **完成证据**：项目 6173 等价脱敏 corpus 完成“终态生成 → 发布 → 清 workspace → 读取 → scratch 恢复 → rebuild”真实链路。 + +## 验收标准 + +- [ ] 中间状态、流式片段、工具轨迹和 workspace 草稿进入 FAV 的测试结果为 0。 +- [ ] 应归档终态成果的 publish request 覆盖率为 100%，verified 延迟与失败积压可观测。 +- [ ] 任一 publisher 崩溃点都可幂等恢复，半成品不对读/恢复可见。 +- [ ] 工作区全部删除或换根后，已 verified 成果仍可由稳定逻辑 ID 定位并通过 SHA 校验。 +- [ ] 授权图缺失时 fail closed；跨环境/project/user 的读取和恢复负例通过；当前实现不伪造 tenant 边界。 +- [ ] 显式 publish/pin 不能绕过终态 allowlist；敏感分类失败或命中禁止内容时正文不进入 staging/outbox/vault/日志/指标。 +- [ ] 存量终态回填可幂等重跑；quarantine 与批准排除均有可复算审计，coverage 分母不可手工绕过。 +- [ ] 项目作用域引用账本保证 tombstone 立即不可读，最后有效引用且无 hold 时才清除密钥和对象。 +- [ ] FAV 可以在空 scratch 库恢复成果集合，但不能伪造权限、活动工作流、lease、pending task 或完整历史。 +- [ ] tombstone/删除账本在 restore 后重放，已删除内容不会复活。 +- [ ] PostgreSQL 完整 restore、migration corpus 升级和 FAV 部分恢复三条演练均有独立报告。 +- [ ] dev 清理只有在完整备份可恢复、migration corpus PASS、FAV coverage PASS 后才能进入销毁审批。 + +## 风险与缓解 + +| 风险 | 缓解 | +|---|---| +| FAV 变成第三真相源 | 限定只对 payload/manifest 权威；运行指针和授权始终由数据库决定 | +| “最终”判定过宽导致存满中间结果 | 状态机 allowlist + 未分类拒绝 + 准入指标/抽样审计 | +| 数据库提交与归档双写丢事件 | 同事务 outbox + 幂等 publisher + cleanup coverage gate | +| outbox 指向可变或已删除正文 | 终态事务冻结精确 payload/SHA 到不可变 staging，publisher 只读 staging | +| 存量终态不触发新 hook | 权威数据库幂等回填 + quarantine/批准排除审计，禁止 workspace 猜测 | +| 归档可读但无法重建业务关系 | manifest 声明依赖与恢复能力；默认 recover-as-collection；完整灾备继续依赖 DB backup/PITR | +| WORM、共享 blob 与删除义务冲突 | 项目作用域加密 CAS + manifest-member 引用账本；按 retention class 选 governance/compliance；单独 legal hold；最后引用后密钥/对象清除 | +| 同机独立目录仍无法抗磁盘故障 | 第一阶段独立 mount/identity，生产要求异机或对象存储复制并做恢复演练 | +| 归档泄露敏感上下文 | 结构化排除 Prompt/轨迹；服务端敏感分类 fail closed；最小化 metadata；传输/静态加密、项目授权、凭证隔离和访问审计 | + +## 明确不采用 + +- 定时扫描 `**/output.md` 并复制“最新文件”。 +- 把所有 assistant 消息、SSE event 或 agent session 全量归档。 +- 在业务事务内同步调用对象存储，或反过来先写对象再提交业务状态。 +- DB 不一致时由 GET 请求静默重建/覆盖数据。 +- 因为 FAV 有最终正文就删除完整备份、WAL 或 migration corpus。 +- 把所有对象永久 compliance lock，忽略合法删除和数据最小化。 + +## 相关链接 + +- Research: [最终成果保险库与部分恢复边界](../ops/research/2026-09-14-final-artifact-vault.md) +- Parent remediation: [运行数据生命周期与迁移语料库](./2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md) +- Incident remediation: [dev 数据根隔离与投影自愈](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [最终成果保险库方案](../ops/short-term/2026-09-14-final-artifact-vault-plan.md) +- gap-matrix: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md new file mode 100644 index 00000000..c58cf6ae --- /dev/null +++ b/.doc/remediation/2026-09-14-runtime-data-lifecycle-and-migration-corpus-remediation.md @@ -0,0 +1,182 @@ +--- +title: "运行数据生命周期与迁移语料库整改方案" +status: planned +created: 2026-09-14 +updated: 2026-09-14 +--- + +# 运行数据生命周期与迁移语料库整改方案 + +## 背景 + +合理性和当前数据画像见[调研](../ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md)。本方案是 [dev 数据根与投影整改](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) Wave 5 的详细子方案。 + +目标不是“为了省事删数据”，而是把不可控的历史运行库转化为： + +- 少量可展示、可解释的当前数据； +- 少量可重复、可断言的迁移/回归语料； +- 大量自动过期的隔离测试数据； +- 可恢复后再销毁的旧卷。 + +## 目标 + +1. dev 常驻项目控制在 6–10 个，均有 owner、用途、数据类别和过期/固定理由。 +2. test 不再积累跨轮测试数据；每轮使用独立 DB + state root，PASS 即删，FAIL 短期保留。 +3. 每次发布同时验证 clean install、受支持旧版本升级、选择性数据导入、reload/rebuild 和 rollback。 +4. 数据库 dump 与文件状态根成对归档，具备 storage identity、SHA manifest 和自动断言。 +5. 清理不产生孤儿 snapshot/run/blob/workspace，不误删共享 blob，不接触未授权 main/KB 数据。 +6. 最终成果在清理前进入独立 [Final Artifact Vault](./2026-09-14-final-artifact-vault-remediation.md) 并取得 verified 回执；该回执不能替代完整备份和 migration corpus。 + +## 数据保留策略 + +| 环境 | 常驻内容 | 自动过期 | 明确保留 | 禁止事项 | +|---|---|---|---|---| +| dev | 3 个 workflow demo + 2–4 个当前 regression + 明确人工项目 | 普通 E2E 24h；失败运行 7 天；上一版 demo 14 天 | pinned 项目、当前/上一 release 对照 | 不按 ID/目录 mtime 直接删除；不把真实内容提交为 fixture | +| test | 无长期业务项目 | PASS 立即销毁；FAIL 3–7 天 | 最近 1 次失败环境用于排障 | 不连接 dev/test 共享库跑 destructive test | +| main | 真实业务/正式展示 | 暂不自动过期 | 由产品、审计、合规规则决定 | 本方案不授权清理或测试写入 | +| shared KB | 法规知识库 | 独立策略 | 现有 KB 备份体系 | 不随项目数据轮换 | + +## 数据标识模型 + +新增数据集注册表或等价 SSOT： + +```text +dataset_id +class: operational_pinned | demo_current | regression_golden | + migration_baseline | ephemeral_test | quarantine +environment +source_schema / target_schema +source_commit +storage_identity +owner +retention_until +pinned_reason +contains_sensitive_data +manifest_uri / manifest_sha256 +required_artifact_count / verified_artifact_count +status: building | verified | active | quarantined | expired | destroyed +``` + +项目至少增加 `dataset_id`、`data_origin`、`retention_until`、`pinned_at/pinned_reason`；自动测试创建项目时必须带 `ephemeral_test` origin 和 run namespace。未分类数据默认不可自动删除，而不是默认过期。 + +## 工具设计 + +### 1. `dataset inventory` + +只读生成候选清单：项目关系行数、doc/run/session IDs、workspace、CAS 引用、最后活动时间、生命周期、是否有 ledger/fork/missing、敏感级别、推荐动作。输出 JSON + Markdown；不接受模糊环境名。 + +### 2. `dataset export` + +按 dataset/project allowlist 导出关系闭包，不能只导 `projects` 表。必须同时打包： + +- project/chat/task/workflow/governance/document/revision/notification/outbox/audit 关系； +- 账本 blob 与受控/自由文档文件； +- file reference 指向的 CAS blob； +- workspace/agent session 中经 allowlist 认领的文件； +- schema/version/storage identity/count/SHA manifest。 + +导出后在隔离区执行脱敏；真实账号 token、auth sessions、credential、原始敏感附件默认剔除或替换。 + +### 3. `dataset restore --isolated-only` + +强制目标数据库名/端口/state root 满足隔离规则；拒绝 5433/5434/5435 及 `agent_system_dev|test|main`。实际执行 `pg_restore`，再导入文件包，不能只做 `pg_restore --list`。 + +恢复顺序：创建空库 → 恢复 source schema/data → 校验 source assertions → 执行 Goose up → 校验 target assertions → 重建 projection/index → restart/reload → 再次校验。 + +### 4. `dataset rotate` + +生成计划而不直接删除：`retain | replace | quarantine | destroy`。执行必须使用 manifest ID、期望 storage UUID 和确认短语；禁止 `--all`、模糊 glob、未解析环境变量。 + +### 5. `dataset gc` + +仅在数据库引用闭包和文件 manifest 都验证后执行： + +- 删除明确 dataset/project 的关系闭包； +- 清理无引用 document content blob/CAS blob； +- 清理已归属 workspace/session； +- `setval`/identity sequence 修复； +- 输出 before/after counts、孤儿检查与可恢复位置。 + +首轮 dev 推荐不在旧库运行 GC，而是蓝绿新建后切换；GC 用于后续小规模过期数据。 + +### 6. `artifact coverage` + +按服务端终态事件盘点应归档成果，并与 FAV verified receipt 对账。任何必需成果处于 requested/publishing/failed/missing，或 vault storage identity 不匹配时，受管 dataset/持久状态源不得销毁。普通计算 workspace 只有在终态事务已冻结精确 payload 后才可按 TTL 清理；准入、manifest、恢复和删除边界见[最终成果保险库整改](./2026-09-14-final-artifact-vault-remediation.md)。 + +## 迁移测试矩阵 + +每个 release candidate 必须执行： + +| 测试轨 | 输入 | 验证重点 | +|---|---|---| +| Clean | 空 PostgreSQL + 空 state root | migrations 1→latest、官方 workflow seed、storage identity、ready | +| Oldest | `oldest-supported` corpus | 跨版本 backfill、约束、ID sequence、文档 legacy repair | +| Previous | `previous-release` corpus | 常规升级、API/UI 兼容、性能 | +| Hazard | 最小异常 corpus | hard-break、projection missing/fork、旧消息 scope、高 ID、pending run | +| Import | latest schema + selected demo/regression package | 关系闭包、文件 SHA、权限、通知、版本账本 | +| Reload | import 后 app/pi rebuild + restart + 宿主级重载 | storage identity、mount、task 恢复、正文/overlay | +| Rollback | 切换失败后接回 quarantine 旧卷 | RTO、旧 storage identity、无新写丢失 | +| Artifact recovery | 已归档终态成果 + 空 scratch DB | manifest/SHA、删除账本、recover-as-collection、冲突预览、权限 fail closed | + +迁移通过必须包含真实 restore，不接受仅跑 Goose 幂等测试；产品签字至少包含真实页面登录、项目列表、对话、文档打开/修改、刷新、退出重登和 rebuild 后恢复。 + +## 初次精简执行波次 + +| 波次 | 动作 | 完成判据 | +|---|---|---| +| D0 冻结与标注 | 先完成当前 split-brain 遏制；冻结 dev 写入；对 366 个项目 inventory；人工确认 pinned/敏感/owner | 100% 项目有建议动作；未分类项不删除 | +| D1 工具安全修补 | 移除备份脚本内默认凭证；backup 同时归档 DB+state；restore 强制 scratch 且真实恢复；加入 manifest/checksum/assertions | 恢复演练能从零创建隔离环境并通过 SHA/关系断言 | +| D2 构建 corpus 与成果覆盖 | 建 schema 25、28/29、32/33 三个最小迁移包；构建 143 等价脱敏 hazard；选择 6173 current regression；盘点保留项目的终态成果并生成 FAV coverage | 每包可重复生成/恢复；必需成果均 verified 或有批准的排除理由，不依赖真实 main/test 数据 | +| D3 新建 green dev | latest schema 空库 + 固定新 state root；导入选中的 6–10 项；执行 projection/CAS/sequence/integrity 校验 | 无 orphan；active 文档 aligned；ready/storage identity PASS | +| D4 全链验收 | clean/oldest/previous/hazard/import/reload/rollback；真实页面和真实 PI 跑主链 | 报告区分工程、PG、PI、浏览器、restart/rebuild 证据 | +| D5 原子切换 | 切换 dev 到 green identity；原 blue DB/文件根转只读 quarantine | 新环境持续观察 24–72h，无新增 blocker，可一键回接 blue | +| D6 延迟销毁 | quarantine 满 7–14 天，再次 restore drill、差异确认、用户批准后销毁 | 记录销毁 manifest、范围、时间和不可恢复性；不得顺带清 main/KB | + +## 首轮建议保留/淘汰 + +### 保留 + +- 项目 143：临时 pinned，直到脱敏 legacy/hazard corpus 可替代。 +- 项目 6173：作为最新完整受控正文/版本/UI/restart 回归样本。 +- 项目 6159：作为 previous 对照保留一个发布周期，然后由 6173 替换。 +- 三正式工作流各一个新生成的确定性 demo；生成成功后不保留同类失败尝试。 +- 最近一次失败 E2E 的完整环境，最多 7 天，且与常驻 dev 隔离。 + +### 不直接保留 + +- 数百个无文档、无 run 或仅测试前置步骤的 planning/active 项目； +- 重复的 UI Round 中间项目和已由更新 PASS 覆盖的失败项目； +- 能由 unit/PG fixture 稳定重建的测试污染 workflow key、临时用户、session 和 run； +- 无引用 CAS/workspace，需在 mark-and-sweep 验证后删除。 + +该清单是初始建议，不是删除授权；最终 retain allowlist 应由 inventory 报告和用户确认生成。 + +## 验收标准 + +- [ ] dev 常驻项目 6–10 个，所有项目都有 dataset class/owner/retention。 +- [ ] 旧 366 项目对应的 blue DB+state 已成对备份并实际恢复成功。 +- [ ] green dev 从空库建成，选择性导入后 DB/file/storage UUID 一致。 +- [ ] 三个 migration baselines 与 hazard corpus 均能升到 latest 并通过断言。 +- [ ] 无 orphan task run、revision snapshot/version/event/report、CAS blob 或 workspace。 +- [ ] 项目 143/6173 页面打开、修改、刷新、重登、rebuild 后恢复 PASS。 +- [ ] 保留项目的应归档成果覆盖率为 100%，清空 workspace 后仍能定位并校验最终正文/最终回复。 +- [ ] FAV 在空 scratch 库完成受控部分恢复；未声称或伪造权限、lease、pending task 等未归档状态。 +- [ ] 新测试数据带 TTL，PASS 后自动销毁，失败数据按策略到期。 +- [ ] test/main/shared KB 未被误操作；所有破坏性动作有精确 manifest 和延迟销毁记录。 + +## 不建议的方案 + +- 每次直接 `docker compose down -v` 再凭记忆重建：无法证明保留数据和文件配对完整。 +- 在 dev 库中写一份巨型级联 DELETE：当前关系与文件引用复杂，容易留下孤儿或误删共享 blob。 +- 只保留最新时间的项目：会丢失最有价值的旧 schema/hazard 覆盖。 +- 把整份真实旧库永久当 migration fixture：迁移慢、不可重复、可能含敏感数据。 +- 把 FAV 当完整数据库替代品，或按 `output.md`/mtime 扫描猜测终态：会引入第三事实源并归档中间结果。 +- 修改历史 Goose migration 迎合新空库：已发布迁移不可重写；修正必须新增 migration。 + +## 相关链接 + +- Research: [运行数据精简、迁移语料与演示数据保留策略](../ops/research/2026-09-14-runtime-data-retention-and-migration-corpus.md) +- Final Artifact Vault: [最终成果保险库与受控恢复](./2026-09-14-final-artifact-vault-remediation.md) +- Parent remediation: [dev 数据根隔离与投影自愈](./2026-09-14-dev-runtime-data-root-and-projection-remediation.md) +- Short-term: [运行数据生命周期方案](../ops/short-term/2026-09-14-runtime-data-lifecycle-plan.md) +- gap-matrix: `RUNTIME-DATA-ROOT-SPLIT-BRAIN` diff --git a/.doc/system-design/module-boundaries.yaml b/.doc/system-design/module-boundaries.yaml index 8c5ca996..7992d183 100644 --- a/.doc/system-design/module-boundaries.yaml +++ b/.doc/system-design/module-boundaries.yaml @@ -468,6 +468,7 @@ event_consumers: outbox.notify: - app/docnotify - notification + - server document.agent_candidate_ready: - notify document.agent_version_committed: diff --git a/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md new file mode 100644 index 00000000..728f8e10 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-migration-corpus-drill-test-report.md @@ -0,0 +1,33 @@ +# FAV 演练：migration corpus（独立报告） + +## 一、 测试背景 (Background) + +Migration corpus / 蓝绿换库是 FAV 的 sibling，不在本计划实现。V6 只能引用其 PASS，不能用 FAV 覆盖率手改数字绕过。 + +## 二、 测试目标 (Test Objectives) + +- `PreflightDestroy(..., corpusOK=false)` fail-closed。 +- 回填只枚举账本/完成 turn，禁止 workspace。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +`TestCoverageGateFailClosed` + `TestBackfillIdempotentNoWorkspace`。无 corpus 目录实跑。 + +## 四、 预期结果 (Expected Results) + +corpus 未 PASS → destroy 拒绝；缺证据行 → quarantine；重跑 Created 不增加。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `PreflightDestroy(..., backupOK=true, corpusOK=false)` 逻辑分支；`Backfill` 重复行与 missing evidence +- **实际输出**: destroy 拒绝；首次 Created=1 Reused=1；rerun Reused；missing → Quarantine=1 +- **评价判定**: PASS（门禁与回填契约）；**未做** live corpus 晋升 +- **性能与耗时**: <5ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +`TestBackfillIdempotentNoWorkspace` PASS；`TestCoverageGateFailClosed` PASS。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +`scripts/artifact-vault-preflight.sh` 默认 `CORPUS_OK=false`，避免手改 coverage 后 teardown。corpus 作业 PASS 后才能把该旗标打开。 diff --git a/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md new file mode 100644 index 00000000..8cef1762 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-postgres-restore-drill-test-report.md @@ -0,0 +1,33 @@ +# FAV 演练：PostgreSQL restore（独立报告） + +## 一、 测试背景 (Background) + +FAV 不替代完整 DB backup/PITR。本报告只验证「恢复演练门禁」：coverage 未 PASS 或 complementary backup 未声明时 destroy/teardown fail-closed。 + +## 二、 测试目标 (Test Objectives) + +- `PreflightDestroy` 在 backup 未 PASS 时拒绝。 +- 不把 FAV 单测冒充为已完成的 PostgreSQL PITR 演练。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +本机 `go test`；**未**对 `agent-dev` 做 pg_dump/restore（集成槽占用 + 数据根 split-brain）。命令：`TestCoverageGateFailClosed`。 + +## 四、 预期结果 (Expected Results) + +`Coverage{Required:2, Verified:1}` + backupOK=true 仍因 coverage 失败而拒绝；backupOK=false 亦拒绝。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `PreflightDestroy(Coverage{Required: 2, Verified: 1}, true, true, true)` +- **实际输出**: `artifactvault: coverage gate failed required=2 verified=1 quarantine=0` +- **评价判定**: PASS（门禁）；**FAIL / 未做** live PostgreSQL restore +- **性能与耗时**: <1ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +见 [assets/2026-09-14-final-artifact-vault/unit-tests.txt](assets/2026-09-14-final-artifact-vault/unit-tests.txt) 中 `TestCoverageGateFailClosed`。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +门禁函数可用。完整 PostgreSQL restore 演练依赖独立 backup 作业与 Unit A 后的干净数据根，不得在本报告勾 PASS。 diff --git a/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md b/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md new file mode 100644 index 00000000..1f75cd99 --- /dev/null +++ b/.doc/test-reports/2026-09-14-fav-scratch-recover-drill-test-report.md @@ -0,0 +1,37 @@ +# FAV 演练：scratch 恢复（独立报告） + +## 一、 测试背景 (Background) + +方案要求恢复目标必须是 scratch identity，默认 `recover-as-collection`，union-deny 删除，禁止写回 dev/test/main。 + +## 二、 测试目标 (Test Objectives) + +- `ValidateRecoverTarget` 拒绝 `agent_system_dev/test/main`。 +- relink 在冲突/缺授权时失败。 +- union-deny tombstone 后 `DegradedPayload` 拒绝。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +内存 registry + tempfile vault。`cmd/artifact-vault recover-plan/apply` 仅编译，未对现网 DSN 执行 apply。 + +## 四、 预期结果 (Expected Results) + +operational DSN 拒绝；scratch 计划通过；denied/relink+conflict 失败；tombstone 后不可读。 + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +- **测试输入**: `TestRecoverRejectsOperationalDSN`、`TestRecoverPlanGates`、`TestTombstoneUnionDeny` +- **实际输出**: + - `postgres://x/agent_system_dev` → refuse + - `postgres://x/fav_scratch` → accept + - union-deny → degraded read error +- **评价判定**: PASS（计划校验）；live scratch 库 apply **未执行**（避免误连 operational） +- **性能与耗时**: <10ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +见 unit-tests.txt。`go test -c ./cmd/artifact-vault` 编译通过。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +恢复工具默认 scratch-only。项目 6173 等价「终态→发布→清 workspace→降级读→scratch 恢复→app rebuild」需 Unit A 后另占槽，不得声称已拦截任意 `rm`。 diff --git a/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md new file mode 100644 index 00000000..35f152b6 --- /dev/null +++ b/.doc/test-reports/2026-09-14-final-artifact-vault-test-report.md @@ -0,0 +1,98 @@ +# 最终成果保险库 V0–V6 测试报告 + +## 一、 测试背景 (Background) + +运行库与 workspace 投影会一起丢失或错挂。已审查的 FAV 方案要求只归档数据库终态事实，独立 `VAULT_STATE_ROOT`，禁止扫描 `output.md`，GET 零写入，恢复只进 scratch。本轮按落地计划实现 V0–V6 内核与领域接入；现网 Unit A 纠挂未完成，故不对 `agent-dev` 开降级读或容器 rebuild。 + +## 二、 测试目标 (Test Objectives) + +- 准入表拒绝非终态、未知类型、敏感正文。 +- approved 无 `review_snapshot_id` 不得入库/批准。 +- task finalize 重算 `export_policy`/`content_sha256` 并 `CompleteChatTurn`。 +- publisher 幂等、拒绝 `data/dev` 根、跨项目同 SHA 分 key、tombstone union-deny。 +- 回填缺证据 quarantine；重跑不新增逻辑成果。 +- recover 拒绝 operational DSN；coverage/destroy 门禁 fail-closed。 +- 半发布缺 commit marker 不可见，exclusive 重试后可恢复。 +- GET/degraded 路径不调用 `Request`。 +- `copy_only` 仅 completed 后显式 pin 可进库。 + +## 三、 测试方案与环境 (Test Plan & Environment) + +| 项 | 值 | +|----|-----| +| OS | Linux 7.0.0-31-generic | +| Go | `go test` 本机 | +| 后端 | 单元/内存 registry + tempfile vault；**未**使用 stub 作为终态验收替代，也**未**对现网 `pi-agent` 做 6173 演练 | +| 分支 | `feature/final-artifact-vault` | +| worktree | `.worktrees/fix-dev-runtime-data-root-guard` | +| 集成槽 | 未占用；禁止 `env-service` rebuild | +| 证据 | [assets/2026-09-14-final-artifact-vault/unit-tests.txt](assets/2026-09-14-final-artifact-vault/unit-tests.txt) | + +## 四、 预期结果 (Expected Results) + +| 用例 | 判定 | +|------|------| +| approved 无 snapshot | 拒绝 | +| task 无 completed turn | 拒绝 | +| 显式 pin 非终态 | `ErrExplicitBypass` | +| secret payload | 不得进 staging | +| 重复 Request/Drain | 同一 UUID，verified=1 | +| recover operational DSN | 拒绝 | +| coverage 不完整 | destroy 拒绝 | + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +### 1. 准入与表征 + +- **测试输入**: `go test -v ./internal/artifactvault/ ./internal/docgov/ -run 'TestEvaluateEligibility|TestApproverApprove|TestApprovedHook'` +- **实际输出**: 全部 `--- PASS`（见 assets） +- **评价判定**: PASS +- **性能与耗时**: <10ms / 包 + +### 2. Publisher / 文件系统 + +- **测试输入**: `TestPublisherIdempotentAndCrashSafe`、`TestFilesystemRefusesRuntimeDataDev`、`TestTombstoneUnionDeny`、`TestLastReferencePurgeAndLegalHold` +- **实际输出**: PASS；`data/dev` 根被拒绝 +- **评价判定**: PASS +- **性能与耗时**: <10ms + +### 3. 领域 finalize + +- **测试输入**: `TestFinalizeAssistantTaskMessageRecalculatesExportColumns`、`TestCompleteChatTurnAfterFinalize`、`TestFinalizeTaskRunCompletesTurnAndIngests` +- **实际输出**: PASS +- **评价判定**: PASS +- **性能与耗时**: server 用例 5ms + +### 4. 回填与恢复计划 + +- **测试输入**: `TestBackfillIdempotentNoWorkspace`、`TestRecoverRejectsOperationalDSN`、`TestRecoverPlanGates`、`TestCoverageGateFailClosed` +- **实际输出**: PASS +- **评价判定**: PASS +- **性能与耗时**: <10ms + +### 5. 缺口补齐（同事务 / recover-apply / GET 零写入 / pin） + +- **测试输入**: `TestPublisherRetriesAfterMissingCommit`、`TestUnpublishedWithoutCommitMarkerInvisible`、`TestFilesystemIdentityMismatch`、`TestDegradedPayloadDoesNotWrite`、`TestTryVaultDegradedDoesNotRequest`、`TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied`、`TestServer_PinExportCopyOnlyAfterCompleted`、`TestValidateIndependentVault` +- **实际输出**: 全部 `--- PASS`（见 [assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt](assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt)） +- **评价判定**: PASS +- **性能与耗时**: artifactvault 4ms；pin-export 120ms + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +``` +ok chat-agent-system/internal/artifactvault 0.005s +ok chat-agent-system/internal/config 0.002s +ok chat-agent-system/internal/docgov 0.011s +ok chat-agent-system/internal/ctxmgr 0.073s +ok chat-agent-system/internal/server 0.364s (FAV 挂钩 / pin-export / GET 零写入) +``` + +未跑 `go test ./...` 全量（含真实 PI E2E / 需容器的套件）；本波次范围是 FAV 包与挂钩单测。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +V0–V6 **代码与单测已落地**，可作为合 `dev` 前的内核证据。下列不得写成已关闭： + +- 现网 `FAV_DEGRADED_READ` 必须保持 false，直到 Unit A 纠挂。 +- 项目 6173 等价脱敏 corpus 与三条 live 演练见独立报告，当前为 **preflight / 单测等价**，不是容器 e2e PASS。 +- Goose FAV 表为 **00034**（00033 已被 task-run 治理占用）。 diff --git a/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt new file mode 100644 index 00000000..6fc5b46c --- /dev/null +++ b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests-gapfill.txt @@ -0,0 +1,48 @@ +=== RUN TestEvaluateEligibility_Allowlist +--- PASS: TestEvaluateEligibility_Allowlist (0.00s) +=== RUN TestEvaluateEligibility_ApprovedRequiresReviewSnapshot +--- PASS: TestEvaluateEligibility_ApprovedRequiresReviewSnapshot (0.00s) +=== RUN TestEvaluateEligibility_TaskWithoutCompletedTurn +--- PASS: TestEvaluateEligibility_TaskWithoutCompletedTurn (0.00s) +=== RUN TestEvaluateEligibility_SensitiveRejectedBeforeStaging +--- PASS: TestEvaluateEligibility_SensitiveRejectedBeforeStaging (0.00s) +=== RUN TestEvaluateEligibility_CopyOnlyRequiresExplicitPinAfterCompleted +--- PASS: TestEvaluateEligibility_CopyOnlyRequiresExplicitPinAfterCompleted (0.00s) +=== RUN TestEvaluateEligibility_UnknownTypeRejected +--- PASS: TestEvaluateEligibility_UnknownTypeRejected (0.00s) +=== RUN TestPublisherIdempotentAndCrashSafe +--- PASS: TestPublisherIdempotentAndCrashSafe (0.00s) +=== RUN TestPublisherRejectsBadSHAAndCrossProjectKeys +--- PASS: TestPublisherRejectsBadSHAAndCrossProjectKeys (0.00s) +=== RUN TestFilesystemRefusesRuntimeDataDev +--- PASS: TestFilesystemRefusesRuntimeDataDev (0.00s) +=== RUN TestTombstoneUnionDeny +--- PASS: TestTombstoneUnionDeny (0.00s) +=== RUN TestLastReferencePurgeAndLegalHold +--- PASS: TestLastReferencePurgeAndLegalHold (0.00s) +=== RUN TestFilesystemIdentityMismatch +--- PASS: TestFilesystemIdentityMismatch (0.00s) +=== RUN TestUnpublishedWithoutCommitMarkerInvisible +--- PASS: TestUnpublishedWithoutCommitMarkerInvisible (0.00s) +=== RUN TestPublisherRetriesAfterMissingCommit +--- PASS: TestPublisherRetriesAfterMissingCommit (0.00s) +=== RUN TestDegradedPayloadDoesNotWrite +--- PASS: TestDegradedPayloadDoesNotWrite (0.00s) +=== RUN TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied +--- PASS: TestApplyRecoverAsCollectionWritesScratchAndSkipsDenied (0.00s) +PASS +ok chat-agent-system/internal/artifactvault 0.004s +=== RUN TestValidateIndependentVault +--- PASS: TestValidateIndependentVault (0.00s) +PASS +ok chat-agent-system/internal/config 0.002s +=== RUN TestFinalizeTaskRunCompletesTurnAndIngests +2026/09/14 12:51:53 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestFinalizeTaskRunCompletesTurnAndIngests (0.00s) +=== RUN TestTryVaultDegradedDoesNotRequest +--- PASS: TestTryVaultDegradedDoesNotRequest (0.00s) +=== RUN TestServer_PinExportCopyOnlyAfterCompleted +2026/09/14 12:51:53 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestServer_PinExportCopyOnlyAfterCompleted (0.12s) +PASS +ok chat-agent-system/internal/server 0.131s diff --git a/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt new file mode 100644 index 00000000..f6f0b255 --- /dev/null +++ b/.doc/test-reports/assets/2026-09-14-final-artifact-vault/unit-tests.txt @@ -0,0 +1,49 @@ +=== RUN TestBackfillIdempotentNoWorkspace +--- PASS: TestBackfillIdempotentNoWorkspace (0.00s) +=== RUN TestEvaluateEligibility_Allowlist +--- PASS: TestEvaluateEligibility_Allowlist (0.00s) +=== RUN TestEvaluateEligibility_ApprovedRequiresReviewSnapshot +--- PASS: TestEvaluateEligibility_ApprovedRequiresReviewSnapshot (0.00s) +=== RUN TestEvaluateEligibility_TaskWithoutCompletedTurn +--- PASS: TestEvaluateEligibility_TaskWithoutCompletedTurn (0.00s) +=== RUN TestEvaluateEligibility_SensitiveRejectedBeforeStaging +--- PASS: TestEvaluateEligibility_SensitiveRejectedBeforeStaging (0.00s) +=== RUN TestEvaluateEligibility_UnknownTypeRejected +--- PASS: TestEvaluateEligibility_UnknownTypeRejected (0.00s) +=== RUN TestIdempotencyKeyStable +--- PASS: TestIdempotencyKeyStable (0.00s) +=== RUN TestIngestIdempotentAndRejectsNonTerminal +--- PASS: TestIngestIdempotentAndRejectsNonTerminal (0.00s) +=== RUN TestCoverageGateFailClosed +--- PASS: TestCoverageGateFailClosed (0.00s) +=== RUN TestPublisherIdempotentAndCrashSafe +--- PASS: TestPublisherIdempotentAndCrashSafe (0.00s) +=== RUN TestPublisherRejectsBadSHAAndCrossProjectKeys +--- PASS: TestPublisherRejectsBadSHAAndCrossProjectKeys (0.00s) +=== RUN TestFilesystemRefusesRuntimeDataDev +--- PASS: TestFilesystemRefusesRuntimeDataDev (0.00s) +=== RUN TestLastReferencePurgeAndLegalHold +--- PASS: TestLastReferencePurgeAndLegalHold (0.00s) +=== RUN TestRecoverRejectsOperationalDSN +--- PASS: TestRecoverRejectsOperationalDSN (0.00s) +=== RUN TestRecoverPlanGates +--- PASS: TestRecoverPlanGates (0.00s) +PASS +ok chat-agent-system/internal/artifactvault 0.005s +=== RUN TestApproverApproveRequiresFrozenReviewSnapshot +--- PASS: TestApproverApproveRequiresFrozenReviewSnapshot (0.00s) +=== RUN TestApprovedHookDoesNotRequireWorkspaceScan +--- PASS: TestApprovedHookDoesNotRequireWorkspaceScan (0.00s) +PASS +ok chat-agent-system/internal/docgov 0.003s +=== RUN TestFinalizeAssistantTaskMessageRecalculatesExportColumns +--- PASS: TestFinalizeAssistantTaskMessageRecalculatesExportColumns (0.00s) +=== RUN TestCompleteChatTurnAfterFinalize +--- PASS: TestCompleteChatTurnAfterFinalize (0.00s) +PASS +ok chat-agent-system/internal/ctxmgr 0.003s +=== RUN TestFinalizeTaskRunCompletesTurnAndIngests +2026/09/14 10:49:49 [kb] 无法连接到 PostgreSQL 知识库 (postgres://:@:0/?sslmode=disable): cannot parse `postgres://:xxxxx@:0/?sslmode=disable`: invalid port (outside range), 将降级运行在离线模式 +--- PASS: TestFinalizeTaskRunCompletesTurnAndIngests (0.00s) +PASS +ok chat-agent-system/internal/server 0.005s diff --git a/cmd/artifact-vault/main.go b/cmd/artifact-vault/main.go index 63ce2467..59cc1454 100644 --- a/cmd/artifact-vault/main.go +++ b/cmd/artifact-vault/main.go @@ -1,57 +1,210 @@ -// Command artifact-vault is the FAV operator CLI stub for U7 incremental. -// -// Full backfill / inventory / recover-apply / destroy live on -// feature/final-artifact-vault and were NOT whole-branch merged. This binary -// compiles on current dev and stays non-destructive: it only prints status / -// remaining-port inventory. package main import ( + "context" "encoding/json" + "flag" "fmt" "os" + "strings" + "time" + + "chat-agent-system/internal/artifactvault" + + "github.com/jackc/pgx/v5/pgxpool" ) func main() { if len(os.Args) < 2 { - usage() + fmt.Fprintln(os.Stderr, "usage: artifact-vault [flags]") os.Exit(2) } - switch os.Args[1] { - case "status", "version": - writeJSON(map[string]any{ - "binary": "artifact-vault", - "slice": "u7-incremental-stub", - "publisher_default": false, - "degraded_read": false, - "consumer": "fav", - "consumer_mode": "off", - "remaining_to_port": []string{ - "enumerate.go (EnumerateTerminals / LoadExclusions)", - "TerminalRow + Backfill(rows) API (diverged from dataset Candidate path)", - "full recover-plan / recover-apply / gc-preflight / destroy", - "server ingest hooks (OnFinalArtifact*)", - "durable fav event Register after separate approval", - "live staging → artifact → publication (each ≥7d observation)", - }, - "shadow_observe": true, - "would_stage": false, - "would_publish": false, - "source_branch": "feature/final-artifact-vault", - "note": "do not enable FAV_DEGRADED_READ or fav consumer without approval; subscriber shadow is read-only", - }) - case "help", "-h", "--help": - usage() + cmd := os.Args[1] + os.Args = append([]string{os.Args[0]}, os.Args[2:]...) + switch cmd { + case "backfill": + runBackfill() + case "inventory": + runInventory() + case "verify": + runVerify() + case "recover-plan": + runRecoverPlan() + case "recover-apply": + runRecoverApply() + case "gc-preflight": + runGCPreflight() + case "destroy": + runDestroy() default: - fmt.Fprintf(os.Stderr, "artifact-vault: unknown command %q (full CLI remains on feature/final-artifact-vault)\n", os.Args[1]) - usage() + fmt.Fprintln(os.Stderr, "unknown command:", cmd) os.Exit(2) } } -func usage() { - fmt.Fprintln(os.Stderr, "usage: artifact-vault ") - fmt.Fprintln(os.Stderr, "U7 stub only — backfill/recover/destroy not ported; fav stays off") +func runBackfill() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + env := flag.String("env", "dev", "environment") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + rows, err := artifactvault.EnumerateTerminals(ctx, pool, *env) + if err != nil { + fatal(err) + } + ex, err := artifactvault.LoadExclusions(ctx, pool) + if err != nil { + fatal(err) + } + reg := artifactvault.NewPGRegistry(pool) + rep, err := artifactvault.Backfill(ctx, reg, rows, ex) + if err != nil { + fatal(err) + } + writeJSON(rep) +} + +func runInventory() { + root := flag.String("root", os.Getenv("VAULT_STATE_ROOT"), "vault root") + ident := flag.String("identity", os.Getenv("VAULT_STORAGE_IDENTITY"), "vault identity") + flag.Parse() + if *root == "" || *ident == "" { + fatal(fmt.Errorf("root and identity required")) + } + st, err := artifactvault.NewFilesystemStore(*root, *ident, []byte(nz(os.Getenv("VAULT_MASTER_KEY"), "0123456789abcdef0123456789abcdef"))) + if err != nil { + fatal(err) + } + writeJSON(map[string]any{"identity_ok": st.StorageIdentity != "", "root_exists": true, "identity": "omitted"}) +} + +func runVerify() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + cov := reg.Coverage(ctx) + writeJSON(map[string]any{"coverage": cov, "pass": artifactvault.CoveragePASS(cov)}) +} + +func runRecoverPlan() { + target := flag.String("target-dsn", "", "scratch DSN") + ident := flag.String("target-identity", "", "scratch identity") + mode := flag.String("mode", string(artifactvault.RecoverAsCollection), "recover mode") + flag.Parse() + plan := artifactvault.RecoverPlan{ + Mode: artifactvault.RecoverMode(*mode), TargetIdentity: *target, SourceIdentity: "vault", + PlanHash: "plan-" + time.Now().UTC().Format("20060102"), HasAdmin: true, ScratchOnly: true, + } + if err := artifactvault.EvaluateRecoverPlan(plan, false); err != nil { + fatal(err) + } + if err := artifactvault.ValidateRecoverTarget(*target, *ident); err != nil { + fatal(err) + } + writeJSON(plan) +} + +func runRecoverApply() { + sourceRoot := flag.String("source-root", os.Getenv("VAULT_STATE_ROOT"), "vault root") + destDir := flag.String("dest-dir", "", "scratch collection directory") + target := flag.String("target-dsn", "", "scratch DSN") + ident := flag.String("target-identity", "", "scratch identity") + sourceIdent := flag.String("identity", os.Getenv("VAULT_STORAGE_IDENTITY"), "source vault identity") + confirm := flag.String("confirm", "", "must equal plan hash") + flag.Parse() + if err := artifactvault.ValidateRecoverTarget(*target, *ident); err != nil { + fatal(err) + } + if *confirm == "" { + fatal(fmt.Errorf("confirm plan hash required")) + } + if *sourceRoot == "" || *sourceIdent == "" { + fatal(fmt.Errorf("source-root and identity required")) + } + plan := artifactvault.RecoverPlan{ + Mode: artifactvault.RecoverAsCollection, TargetIdentity: *target, SourceIdentity: *ident, + PlanHash: *confirm, HasAdmin: true, ScratchOnly: true, + } + store, err := artifactvault.NewFilesystemStore(*sourceRoot, *sourceIdent, []byte(nz(os.Getenv("VAULT_MASTER_KEY"), "0123456789abcdef0123456789abcdef"))) + if err != nil { + fatal(err) + } + arts, payloads, err := store.LoadCommittedArtifacts() + if err != nil { + fatal(err) + } + denied := map[string]bool{} + for _, art := range arts { + if art.PublicationStatus == artifactvault.StatusTombstoned || store.HasTombstone(art.UUID) { + denied[art.UUID] = true + } + } + out, err := artifactvault.ApplyRecoverAsCollection(plan, *destDir, arts, payloads, denied) + if err != nil { + fatal(err) + } + writeJSON(out) +} + +func runDestroy() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + backupOK := flag.Bool("backup-ok", false, "complementary backup PASS") + corpusOK := flag.Bool("corpus-ok", false, "migration corpus PASS") + confirm := flag.String("confirm", "", "must equal DESTROY") + flag.Parse() + if *confirm != "DESTROY" { + fatal(fmt.Errorf("destroy requires --confirm=DESTROY")) + } + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + if err := artifactvault.PreflightDestroy(reg.Coverage(ctx), *backupOK, *corpusOK, true); err != nil { + fatal(err) + } + writeJSON(map[string]string{"status": "preflight-pass", "note": "dataset wipe is operator-owned after complementary PASS"}) +} + +func runGCPreflight() { + dsn := flag.String("dsn", os.Getenv("APP_DB_DSN"), "postgres DSN") + backupOK := flag.Bool("backup-ok", false, "complementary backup PASS") + corpusOK := flag.Bool("corpus-ok", false, "migration corpus PASS") + flag.Parse() + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + pool, err := mustPool(ctx, *dsn) + if err != nil { + fatal(err) + } + defer pool.Close() + reg := artifactvault.NewPGRegistry(pool) + if err := artifactvault.PreflightDestroy(reg.Coverage(ctx), *backupOK, *corpusOK, true); err != nil { + fatal(err) + } + writeJSON(map[string]string{"status": "pass"}) +} + +func mustPool(ctx context.Context, dsn string) (*pgxpool.Pool, error) { + if dsn == "" { + return nil, fmt.Errorf("dsn required") + } + return pgxpool.New(ctx, dsn) } func writeJSON(v any) { @@ -59,3 +212,15 @@ func writeJSON(v any) { enc.SetIndent("", " ") _ = enc.Encode(v) } + +func fatal(err error) { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) +} + +func nz(v, def string) string { + if strings.TrimSpace(v) == "" { + return def + } + return v +} diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 8709a4b9..99743b57 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -117,6 +117,10 @@ services: AUTH_REQUIRE_APPROVAL: ${AUTH_REQUIRE_APPROVAL:-false} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: ${ALLOW_IN_MEMORY:-false} + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-dev-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: ${FAV_DEGRADED_READ:-false} CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -146,6 +150,7 @@ services: volumes: # 独占对话记录与环境数据 - ./data/dev:/app/data + - ./data/vault/dev:/app/vault - dev-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: unless-stopped diff --git a/docker-compose.main.yml b/docker-compose.main.yml index 8cdc7cb2..4962f69f 100644 --- a/docker-compose.main.yml +++ b/docker-compose.main.yml @@ -115,6 +115,10 @@ services: MATERIAL_RAG_TOP_K: ${MATERIAL_RAG_TOP_K:-8} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: "false" + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-main-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: "false" CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -142,6 +146,7 @@ services: volumes: # 独占生产对话记录与环境数据 - ./data/main:/app/data + - ./data/vault/main:/app/vault - main-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: always diff --git a/docker-compose.test.yml b/docker-compose.test.yml index 111b527c..23a3cfa3 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -112,6 +112,10 @@ services: AUTH_REQUIRE_APPROVAL: ${AUTH_REQUIRE_APPROVAL:-false} CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-} ALLOW_IN_MEMORY: ${ALLOW_IN_MEMORY:-false} + VAULT_STATE_ROOT: /app/vault + VAULT_STORAGE_IDENTITY: ${VAULT_STORAGE_IDENTITY:-test-vault} + VAULT_MASTER_KEY: ${VAULT_MASTER_KEY:-} + FAV_DEGRADED_READ: ${FAV_DEGRADED_READ:-false} CACHE_ENABLED: ${CACHE_ENABLED:-true} INTENT_V2_SHADOW: ${INTENT_V2_SHADOW:-false} INTENT_V2_DECISION: ${INTENT_V2_DECISION:-false} @@ -139,6 +143,7 @@ services: volumes: # 独占对话记录与环境数据 - ./data/test:/app/data + - ./data/vault/test:/app/vault - test-aux-sqlite-data:/var/lib/feishu - ./.regulations:/app/.regulations:ro restart: unless-stopped diff --git a/internal/artifactvault/backfill.go b/internal/artifactvault/backfill.go index befe2f4b..9a7470ab 100644 --- a/internal/artifactvault/backfill.go +++ b/internal/artifactvault/backfill.go @@ -3,282 +3,53 @@ package artifactvault import ( "context" "fmt" - "strings" - "time" - - "github.com/jackc/pgx/v5" ) -const ( - KindSnapshot = "snapshot" - KindAgentCommit = "agent_commit" - KindApproved = "approved" - KindChatTurn = "chat_turn" - - HazardNone = "" - HazardMissingSnapshot = "missing_snapshot" - HazardProjectionFork = "projection_fork" - HazardLegacyV0 = "legacy_v0" - HazardHighID = "high_id" - HazardOldSchema = "old_schema" - HazardWorkspacePath = "workspace_path" - HazardSensitivePayload = "sensitive_payload" - - HighIDThreshold int64 = 1_000_000_000 - OldestSupportedSchema = 28 - BackfillBatchSize = 200 -) - -// Candidate is a durable ledger row eligible for FAV backfill. It never carries a workspace path. -type Candidate struct { - Kind string - SourcePK int64 - LogicalID string - ProjectID int64 - SnapshotID int64 - HeadSnapshotID int64 - MessageID int64 - EventID string - RunID string - Payload []byte - PayloadSHA string - CompletedAt *time.Time - ApprovedAt *time.Time - SourceSchema int - SchemaVer int - ProjectionSHA string - LedgerSHA string - ExportPolicy string - WorkDir string +// TerminalRow is an authoritative DB terminal fact for backfill. Workspace paths are forbidden. +type TerminalRow struct { + Facts Facts + Request PublishRequest + MissingEvidence bool + Reason string } -type ProcessResult struct { - Status string // done | quarantine | duplicate - Hazard string - ArtifactID string +type BackfillReport struct { + Created int + Reused int + Quarantine int + Excluded int } -const ( - SQLSnapshots = ` -SELECT s.id, s.doc_id, COALESCE(s.content_sha256,''), COALESCE(b.content_markdown,''), s.source_kind -FROM document_snapshots s -LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 -WHERE s.id > $1 -ORDER BY s.id -LIMIT $2` - - SQLAgentCommits = ` -SELECT e.id, e.doc_id, e.to_snapshot_id, e.idempotency_key, COALESCE(b.content_markdown,''), COALESCE(s.content_sha256,'') -FROM document_change_events e -LEFT JOIN document_snapshots s ON s.id = e.to_snapshot_id -LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 -WHERE e.change_kind = 'agent_commit' AND e.id > $1 -ORDER BY e.id -LIMIT $2` - - SQLApproved = ` -SELECT r.id, r.doc_id, r.project_id, r.approved_at, COALESCE(r.content_sha256,''), - COALESCE(v.head_snapshot_id, 0), COALESCE(hs.content_sha256,''), COALESCE(b.content_markdown,'') -FROM document_records r -LEFT JOIN document_versions v ON v.id = r.ledger_version_id -LEFT JOIN document_snapshots hs ON hs.id = v.head_snapshot_id -LEFT JOIN document_content_blobs b ON b.content_sha256 = r.content_sha256 -WHERE r.current_state = 'approved' AND r.approved_at IS NOT NULL AND r.id > $1 -ORDER BY r.id -LIMIT $2` - - SQLChatTurns = ` -SELECT t.id, COALESCE(t.project_id, 0), t.assistant_message_id, t.completed_at, t.session_id, COALESCE(m.content,'') -FROM chat_turns t -JOIN chat_messages m ON m.id = t.assistant_message_id -WHERE t.status = 'completed' AND t.completed_at IS NOT NULL - AND t.assistant_message_id IS NOT NULL AND t.id > $1 -ORDER BY t.id -LIMIT $2` -) - -func EnumeratorSQL(kind string) (string, error) { - var sql string - switch kind { - case KindSnapshot: - sql = SQLSnapshots - case KindAgentCommit: - sql = SQLAgentCommits - case KindApproved: - sql = SQLApproved - case KindChatTurn: - sql = SQLChatTurns - default: - return "", fmt.Errorf("artifactvault: unknown backfill kind %q", kind) - } - if err := ForbidWorkspaceScan(sql); err != nil { - return "", err - } - return sql, nil -} - -func ForbidWorkspaceScan(sql string) error { - low := strings.ToLower(sql) - parts := [][]string{ - {"work", "spaces/"}, - {"output", ".md"}, - {"work", "_dir"}, - {"data/", "dev"}, - {"data/", "test"}, - {"data/", "main"}, - } - for _, p := range parts { - needle := strings.Join(p, "") - if strings.Contains(low, needle) { - return fmt.Errorf("artifactvault: enumerator must not scan workspace or env data roots") +func Backfill(ctx context.Context, reg Registry, rows []TerminalRow, exclude map[string]string) (BackfillReport, error) { + var out BackfillReport + for _, row := range rows { + if row.MissingEvidence || len(row.Request.Payload) == 0 { + _ = reg.Quarantine(ctx, nz(row.Reason, "missing_evidence"), row.Request.LogicalResourceID) + out.Quarantine++ + continue } + if reason, ok := exclude[row.Request.LogicalResourceID]; ok && reason != "" { + out.Excluded++ + continue + } + before := reg.Coverage(ctx).Required + if _, _, err := reg.Request(ctx, row.Facts, row.Request); err != nil { + _ = reg.Quarantine(ctx, err.Error(), row.Request.LogicalResourceID) + out.Quarantine++ + continue + } + if reg.Coverage(ctx).Required == before { + out.Reused++ + continue + } + out.Created++ + } + return out, nil +} + +func ValidateBackfillRow(row TerminalRow) error { + if row.Request.SourceEventID == "" || row.Request.LogicalResourceID == "" { + return fmt.Errorf("artifactvault: backfill row missing source identity") } return nil } - -func ClassifyHazard(c Candidate) string { - if strings.TrimSpace(c.WorkDir) != "" || strings.Contains(strings.ToLower(c.LogicalID), strings.Join([]string{"work", "spaces/"}, "")) { - return HazardWorkspacePath - } - if c.SourcePK > HighIDThreshold { - return HazardHighID - } - if c.SourceSchema > 0 && c.SourceSchema < OldestSupportedSchema { - return HazardOldSchema - } - if c.Kind == KindSnapshot { - if len(c.Payload) == 0 || c.PayloadSHA == "" { - return HazardMissingSnapshot - } - if c.SchemaVer == 0 { - return HazardLegacyV0 - } - return HazardNone - } - if c.Kind == KindAgentCommit || c.Kind == KindApproved { - if c.SnapshotID <= 0 || len(c.Payload) == 0 { - return HazardMissingSnapshot - } - if c.Kind == KindApproved && c.HeadSnapshotID > 0 && c.LedgerSHA != "" && c.ProjectionSHA != "" && c.LedgerSHA != c.ProjectionSHA { - return HazardProjectionFork - } - } - if c.Kind == KindChatTurn && (c.MessageID <= 0 || c.CompletedAt == nil || len(c.Payload) == 0) { - return HazardMissingSnapshot - } - if ClassifyPayload(c.Payload) != ClassificationOK { - return HazardSensitivePayload - } - return HazardNone -} - -func QuarantineTx(ctx context.Context, tx pgx.Tx, reason, logical string) error { - _, err := tx.Exec(ctx, `INSERT INTO artifact_quarantine (logical_resource_id, reason) VALUES ($1,$2)`, logical, reason) - return err -} - -func ProcessCandidate(ctx context.Context, tx pgx.Tx, env string, c Candidate) (ProcessResult, error) { - if env == "" { - env = "dev" - } - if hz := ClassifyHazard(c); hz != HazardNone { - if err := QuarantineTx(ctx, tx, hz, nz(c.LogicalID, fmt.Sprintf("%s:%d", c.Kind, c.SourcePK))); err != nil { - return ProcessResult{}, err - } - return ProcessResult{Status: "quarantine", Hazard: hz}, nil - } - if c.Kind == KindSnapshot { - return ProcessResult{Status: "done"}, nil - } - ing := &Ingest{Env: env} - var err error - switch c.Kind { - case KindAgentCommit: - err = ing.AgentCommitTX(ctx, tx, c.ProjectID, c.LogicalID, nz(c.EventID, fmt.Sprintf("commit:%d", c.SourcePK)), c.SnapshotID, c.Payload) - case KindApproved: - err = ing.ApprovedTX(ctx, tx, c.ProjectID, c.LogicalID, c.SnapshotID, c.ApprovedAt, c.Payload) - case KindChatTurn: - policy := nz(c.ExportPolicy, "full") - err = ing.requestTX(ctx, tx, AssistantFacts("completed", c.CompletedAt, c.MessageID, policy), PublishRequest{ - ArtifactType: TypeAssistantReply, Environment: env, ProjectID: c.ProjectID, - LogicalResourceID: nz(c.LogicalID, fmt.Sprintf("turn:%d", c.SourcePK)), - SourceEventKind: "assistant_completed", SourceEventID: nz(c.EventID, fmt.Sprintf("turn:%d", c.SourcePK)), - SourceMessageID: c.MessageID, SourceRunID: c.RunID, Payload: c.Payload, TerminalState: "completed", - }) - default: - return ProcessResult{}, fmt.Errorf("artifactvault: unsupported backfill kind %q", c.Kind) - } - if err != nil { - return ProcessResult{}, err - } - return ProcessResult{Status: "done"}, nil -} - -type rowQuerier interface { - Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) -} - -func Enumerate(ctx context.Context, q rowQuerier, kind string, lastPK int64, limit int) ([]Candidate, error) { - sql, err := EnumeratorSQL(kind) - if err != nil { - return nil, err - } - if limit <= 0 || limit > BackfillBatchSize { - limit = BackfillBatchSize - } - rows, err := q.Query(ctx, sql, lastPK, limit) - if err != nil { - return nil, err - } - defer rows.Close() - out := make([]Candidate, 0, limit) - for rows.Next() { - c, err := scanCandidate(kind, rows) - if err != nil { - return nil, err - } - out = append(out, c) - } - return out, rows.Err() -} - -type scannable interface { - Scan(dest ...any) error -} - -func scanCandidate(kind string, row scannable) (Candidate, error) { - var c Candidate - c.Kind = kind - switch kind { - case KindSnapshot: - var srcKind string - if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.PayloadSHA, &c.Payload, &srcKind); err != nil { - return c, err - } - c.SchemaVer = 1 - if len(c.Payload) == 0 { - c.SchemaVer = 0 - } - _ = srcKind - case KindAgentCommit: - if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.SnapshotID, &c.EventID, &c.Payload, &c.PayloadSHA); err != nil { - return c, err - } - case KindApproved: - if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.ProjectID, &c.ApprovedAt, &c.LedgerSHA, &c.HeadSnapshotID, &c.ProjectionSHA, &c.Payload); err != nil { - return c, err - } - c.SnapshotID = c.HeadSnapshotID - case KindChatTurn: - var session string - if err := row.Scan(&c.SourcePK, &c.ProjectID, &c.MessageID, &c.CompletedAt, &session, &c.Payload); err != nil { - return c, err - } - c.LogicalID = session - c.EventID = fmt.Sprintf("turn:%d", c.SourcePK) - c.ExportPolicy = "full" - default: - return c, fmt.Errorf("artifactvault: scan %s", kind) - } - return c, nil -} diff --git a/internal/artifactvault/backfill_test.go b/internal/artifactvault/backfill_test.go index 2424c081..bb0fdc08 100644 --- a/internal/artifactvault/backfill_test.go +++ b/internal/artifactvault/backfill_test.go @@ -1,68 +1,39 @@ package artifactvault import ( - "strings" + "context" "testing" "time" ) -func TestForbidWorkspaceScan(t *testing.T) { - if err := ForbidWorkspaceScan("SELECT work_dir FROM document_records"); err == nil { - t.Fatal("work_dir must be rejected") - } - if err := ForbidWorkspaceScan("SELECT path FROM workspaces/foo"); err == nil { - t.Fatal("workspaces/ must be rejected") - } - if err := ForbidWorkspaceScan(SQLSnapshots + SQLAgentCommits + SQLApproved + SQLChatTurns); err != nil { - t.Fatal(err) - } -} - -func TestEnumeratorSQLCoversFourKindsOnly(t *testing.T) { - for _, kind := range []string{KindSnapshot, KindAgentCommit, KindApproved, KindChatTurn} { - sql, err := EnumeratorSQL(kind) - if err != nil { - t.Fatal(err) - } - if strings.Contains(strings.ToLower(sql), "workspace") { - t.Fatalf("%s leaked workspace: %s", kind, sql) - } - } - if _, err := EnumeratorSQL("workspace_dir"); err == nil { - t.Fatal("unknown kind") - } -} - -func TestClassifyHazardMatrix(t *testing.T) { +func TestBackfillIdempotentNoWorkspace(t *testing.T) { + ctx := context.Background() + reg := NewMemoryRegistry() now := time.Now().UTC() - okBody := []byte("# approved body for vault") - cases := []struct { - name string - c Candidate - want string - }{ - {"commit ok", Candidate{Kind: KindAgentCommit, SnapshotID: 3, Payload: okBody}, HazardNone}, - {"missing snapshot", Candidate{Kind: KindAgentCommit, SnapshotID: 0, Payload: okBody}, HazardMissingSnapshot}, - {"legacy v0", Candidate{Kind: KindSnapshot, PayloadSHA: "", Payload: nil}, HazardMissingSnapshot}, - {"legacy empty schema", Candidate{Kind: KindSnapshot, PayloadSHA: "abc", Payload: okBody, SchemaVer: 0}, HazardLegacyV0}, - {"high id", Candidate{Kind: KindApproved, SourcePK: HighIDThreshold + 1, SnapshotID: 1, Payload: okBody, ApprovedAt: &now}, HazardHighID}, - {"old schema", Candidate{Kind: KindApproved, SourceSchema: 12, SnapshotID: 1, Payload: okBody, ApprovedAt: &now}, HazardOldSchema}, - {"fork", Candidate{Kind: KindApproved, SnapshotID: 4, HeadSnapshotID: 9, LedgerSHA: "aa", ProjectionSHA: "bb", Payload: okBody, ApprovedAt: &now}, HazardProjectionFork}, - {"workspace path", Candidate{Kind: KindApproved, WorkDir: "data/dev/workspaces/x", SnapshotID: 1, Payload: okBody}, HazardWorkspacePath}, - {"secret", Candidate{Kind: KindChatTurn, MessageID: 1, CompletedAt: &now, Payload: []byte("api_key=secret")}, HazardSensitivePayload}, + row := TerminalRow{ + Facts: Facts{ArtifactType: TypeSealedVersion, VersionStatus: "sealed", SealedAt: &now, HeadSnapshotID: 1, Classification: ClassificationOK, Payload: []byte("snap")}, + Request: PublishRequest{ArtifactType: TypeSealedVersion, Environment: "dev", ProjectExternalKey: "p", LogicalResourceID: "doc1", SourceEventKind: "seal", SourceEventID: "e1", Payload: []byte("snap"), TerminalState: "sealed"}, } - for _, tc := range cases { - if got := ClassifyHazard(tc.c); got != tc.want { - t.Fatalf("%s: got %q want %q", tc.name, got, tc.want) - } + r1, err := Backfill(ctx, reg, []TerminalRow{row, row}, nil) + if err != nil || r1.Created != 1 || r1.Reused != 1 { + t.Fatalf("%+v %v", r1, err) } -} - -func TestProcessCandidateQuarantinesWorkspaceAndSecrets(t *testing.T) { - // ProcessCandidate without a real tx is exercised via hazard classification and - // memory ingest in dataset tests. Guard the refuse path here. - c := Candidate{Kind: KindAgentCommit, WorkDir: "workspaces/p/output.md", SnapshotID: 1, Payload: []byte("ok body here")} - if ClassifyHazard(c) != HazardWorkspacePath { - t.Fatal("workspace candidate must be classified before ingest") + r2, _ := Backfill(ctx, reg, []TerminalRow{row}, nil) + if r2.Created != 0 && r2.Reused == 0 { + t.Fatalf("rerun %+v", r2) + } + missing := row + missing.MissingEvidence = true + missing.Request.LogicalResourceID = "doc-missing" + r3, _ := Backfill(ctx, reg, []TerminalRow{missing}, nil) + if r3.Quarantine != 1 { + t.Fatalf("quarantine %+v", r3) + } + ex := row + ex.Request.LogicalResourceID = "doc-ex" + ex.Request.SourceEventID = "e-ex" + r4, _ := Backfill(ctx, reg, []TerminalRow{ex}, map[string]string{"doc-ex": "approved-old"}) + if r4.Excluded != 1 { + t.Fatalf("exclude %+v", r4) } } diff --git a/internal/artifactvault/commit-marker.schema.json b/internal/artifactvault/commit-marker.schema.json new file mode 100644 index 00000000..958b60fa --- /dev/null +++ b/internal/artifactvault/commit-marker.schema.json @@ -0,0 +1,27 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-commit-marker-v1.json", + "title": "FAV Commit Marker", + "type": "object", + "additionalProperties": false, + "required": [ + "artifact_uuid", + "manifest_sha256", + "member_shas", + "vault_storage_identity", + "committed_at", + "publisher_version" + ], + "properties": { + "artifact_uuid": { "type": "string", "minLength": 8 }, + "manifest_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "member_shas": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "pattern": "^[a-f0-9]{64}$" } + }, + "vault_storage_identity": { "type": "string", "minLength": 8 }, + "committed_at": { "type": "string" }, + "publisher_version": { "type": "string" } + } +} diff --git a/internal/artifactvault/eligibility.go b/internal/artifactvault/eligibility.go index bf72b431..5e7b73af 100644 --- a/internal/artifactvault/eligibility.go +++ b/internal/artifactvault/eligibility.go @@ -11,7 +11,6 @@ var ( ErrSensitiveRejected = errors.New("artifactvault: sensitive or unclassified payload rejected") ErrUnknownArtifactType = errors.New("artifactvault: unknown artifact type rejected") ErrSnapshotInconsistent = errors.New("artifactvault: snapshot/event inconsistent") - ErrUnionDeny = errors.New("artifactvault: union-deny") ) // EvaluateEligibility is the only domain gate. Unclassified types default to reject. diff --git a/internal/artifactvault/eligibility_test.go b/internal/artifactvault/eligibility_test.go index 4003062b..fbb410a3 100644 --- a/internal/artifactvault/eligibility_test.go +++ b/internal/artifactvault/eligibility_test.go @@ -90,14 +90,6 @@ func TestEvaluateEligibility_UnknownTypeRejected(t *testing.T) { } } -func TestEvaluateEligibility_IntermediateAdmissionZero(t *testing.T) { - for _, typ := range []string{"prompt", "tool_trace", "scratch", "workspace_scan"} { - if err := EvaluateEligibility(Facts{ArtifactType: typ, Classification: ClassificationOK, Payload: []byte("x")}); err != ErrUnknownArtifactType { - t.Fatalf("%s: %v", typ, err) - } - } -} - func TestIdempotencyKeyStable(t *testing.T) { req := PublishRequest{ ArtifactType: TypeAgentCommit, Environment: "dev", ProjectExternalKey: "p1", diff --git a/internal/artifactvault/enumerate.go b/internal/artifactvault/enumerate.go new file mode 100644 index 00000000..058867dd --- /dev/null +++ b/internal/artifactvault/enumerate.go @@ -0,0 +1,211 @@ +package artifactvault + +import ( + "context" + "fmt" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// EnumerateTerminals lists only immutable ledger / completed-turn facts. Workspace paths are forbidden. +func EnumerateTerminals(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + if pool == nil { + return nil, fmt.Errorf("artifactvault: no database") + } + var rows []TerminalRow + approved, err := enumerateApproved(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, approved...) + commits, err := enumerateAgentCommits(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, commits...) + sealed, err := enumerateSealed(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, sealed...) + replies, err := enumerateAssistant(ctx, pool, env) + if err != nil { + return nil, err + } + rows = append(rows, replies...) + return rows, nil +} + +func LoadExclusions(ctx context.Context, pool *pgxpool.Pool) (map[string]string, error) { + out := map[string]string{} + if pool == nil { + return out, nil + } + q, err := pool.Query(ctx, `SELECT logical_resource_id, reason FROM artifact_backfill_exclusions WHERE expires_at > now()`) + if err != nil { + return nil, err + } + defer q.Close() + for q.Next() { + var id, reason string + if err := q.Scan(&id, &reason); err != nil { + return nil, err + } + out[id] = reason + } + return out, q.Err() +} + +func enumerateApproved(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT d.doc_id, d.project_id, d.review_snapshot_id, d.approved_at, COALESCE(b.content_markdown,'') + FROM document_records d + LEFT JOIN document_snapshots s ON s.id = d.review_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE d.current_state = 'approved' AND d.approved_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var projectID, snapID int64 + var approvedAt *time.Time + var md string + if err := q.Scan(&docID, &projectID, &snapID, &approvedAt, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: ApprovedFacts("approved", approvedAt, snapID), + Request: PublishRequest{ + ArtifactType: TypeApprovedDocument, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: docID, + SourceEventKind: "approved", SourceEventID: fmt.Sprintf("approved:%s:%d", docID, snapID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "approved", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_review_snapshot" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateAgentCommits(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT e.doc_id, e.id, e.to_snapshot_id, COALESCE(r.project_id,0), COALESCE(b.content_markdown,'') + FROM document_change_events e + LEFT JOIN document_records r ON r.doc_id = e.doc_id + LEFT JOIN document_snapshots s ON s.id = e.to_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE e.change_kind = 'agent_commit'`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var eventID, snapID, projectID int64 + var md string + if err := q.Scan(&docID, &eventID, &snapID, &projectID, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: AgentCommitFacts(snapID), + Request: PublishRequest{ + ArtifactType: TypeAgentCommit, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: docID, + SourceEventKind: "agent_commit", SourceEventID: fmt.Sprintf("ev:%d", eventID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "committed", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_commit_snapshot" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateSealed(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT v.doc_id, v.id, v.head_snapshot_id, v.sealed_at, COALESCE(r.project_id,0), COALESCE(b.content_markdown,'') + FROM document_versions v + LEFT JOIN document_records r ON r.doc_id = v.doc_id + LEFT JOIN document_snapshots s ON s.id = v.head_snapshot_id + LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE v.status = 'sealed' AND v.sealed_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var docID string + var verID, snapID, projectID int64 + var sealedAt *time.Time + var md string + if err := q.Scan(&docID, &verID, &snapID, &sealedAt, &projectID, &md); err != nil { + return nil, err + } + row := TerminalRow{ + Facts: SealedFacts(sealedAt, snapID), + Request: PublishRequest{ + ArtifactType: TypeSealedVersion, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: fmt.Sprintf("%s:v%d", docID, verID), + SourceEventKind: "version_sealed", SourceEventID: fmt.Sprintf("ver:%d", verID), + SourceSnapshotID: snapID, Payload: []byte(md), TerminalState: "sealed", + }, + } + if snapID <= 0 || md == "" { + row.MissingEvidence = true + row.Reason = "missing_sealed_head" + } + out = append(out, row) + } + return out, q.Err() +} + +func enumerateAssistant(ctx context.Context, pool *pgxpool.Pool, env string) ([]TerminalRow, error) { + q, err := pool.Query(ctx, ` + SELECT t.session_id, t.client_turn_id, t.assistant_message_id, t.completed_at, t.project_id, + COALESCE(m.content,''), COALESCE(m.export_policy,'none') + FROM chat_turns t + LEFT JOIN chat_messages m ON m.id = t.assistant_message_id + WHERE t.status = 'completed' AND t.completed_at IS NOT NULL`) + if err != nil { + return nil, err + } + defer q.Close() + var out []TerminalRow + for q.Next() { + var sessionID, clientTurnID, content, policy string + var msgID, projectID int64 + var completedAt *time.Time + if err := q.Scan(&sessionID, &clientTurnID, &msgID, &completedAt, &projectID, &content, &policy); err != nil { + return nil, err + } + logical := fmt.Sprintf("msg:%d", msgID) + row := TerminalRow{ + Facts: AssistantFacts("completed", completedAt, msgID, policy), + Request: PublishRequest{ + ArtifactType: TypeAssistantReply, Environment: env, ProjectID: projectID, + ProjectExternalKey: ProjectKeyFromID(projectID), LogicalResourceID: logical, + SourceEventKind: "assistant_completed", SourceEventID: sessionID + ":" + clientTurnID, + SourceMessageID: msgID, Payload: []byte(content), TerminalState: "completed", + }, + } + if msgID <= 0 || content == "" { + row.MissingEvidence = true + row.Reason = "missing_completed_body" + } + out = append(out, row) + } + return out, q.Err() +} diff --git a/internal/artifactvault/legacy_backfill.go b/internal/artifactvault/legacy_backfill.go new file mode 100644 index 00000000..d2feb5a8 --- /dev/null +++ b/internal/artifactvault/legacy_backfill.go @@ -0,0 +1,284 @@ +package artifactvault + +import ( + "context" + "fmt" + "strings" + "time" + + "github.com/jackc/pgx/v5" +) + +const ( + KindSnapshot = "snapshot" + KindAgentCommit = "agent_commit" + KindApproved = "approved" + KindChatTurn = "chat_turn" + + HazardNone = "" + HazardMissingSnapshot = "missing_snapshot" + HazardProjectionFork = "projection_fork" + HazardLegacyV0 = "legacy_v0" + HazardHighID = "high_id" + HazardOldSchema = "old_schema" + HazardWorkspacePath = "workspace_path" + HazardSensitivePayload = "sensitive_payload" + + HighIDThreshold int64 = 1_000_000_000 + OldestSupportedSchema = 28 + BackfillBatchSize = 200 +) + +// Candidate is a durable ledger row eligible for dataset/FAV backfill enumeration. +type Candidate struct { + Kind string + SourcePK int64 + LogicalID string + ProjectID int64 + SnapshotID int64 + HeadSnapshotID int64 + MessageID int64 + EventID string + RunID string + Payload []byte + PayloadSHA string + CompletedAt *time.Time + ApprovedAt *time.Time + SourceSchema int + SchemaVer int + ProjectionSHA string + LedgerSHA string + ExportPolicy string + WorkDir string +} + +type ProcessResult struct { + Status string + Hazard string + ArtifactID string +} + +const ( + SQLSnapshots = ` +SELECT s.id, s.doc_id, COALESCE(s.content_sha256,''), COALESCE(b.content_markdown,''), s.source_kind +FROM document_snapshots s +LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 +WHERE s.id > $1 +ORDER BY s.id +LIMIT $2` + + SQLAgentCommits = ` +SELECT e.id, e.doc_id, e.to_snapshot_id, e.idempotency_key, COALESCE(b.content_markdown,''), COALESCE(s.content_sha256,'') +FROM document_change_events e +LEFT JOIN document_snapshots s ON s.id = e.to_snapshot_id +LEFT JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 +WHERE e.change_kind = 'agent_commit' AND e.id > $1 +ORDER BY e.id +LIMIT $2` + + SQLApproved = ` +SELECT r.id, r.doc_id, r.project_id, r.approved_at, COALESCE(r.content_sha256,''), + COALESCE(v.head_snapshot_id, 0), COALESCE(hs.content_sha256,''), COALESCE(b.content_markdown,'') +FROM document_records r +LEFT JOIN document_versions v ON v.id = r.ledger_version_id +LEFT JOIN document_snapshots hs ON hs.id = v.head_snapshot_id +LEFT JOIN document_content_blobs b ON b.content_sha256 = r.content_sha256 +WHERE r.current_state = 'approved' AND r.approved_at IS NOT NULL AND r.id > $1 +ORDER BY r.id +LIMIT $2` + + SQLChatTurns = ` +SELECT t.id, COALESCE(t.project_id, 0), t.assistant_message_id, t.completed_at, t.session_id, COALESCE(m.content,'') +FROM chat_turns t +JOIN chat_messages m ON m.id = t.assistant_message_id +WHERE t.status = 'completed' AND t.completed_at IS NOT NULL + AND t.assistant_message_id IS NOT NULL AND t.id > $1 +ORDER BY t.id +LIMIT $2` +) + +func EnumeratorSQL(kind string) (string, error) { + var sql string + switch kind { + case KindSnapshot: + sql = SQLSnapshots + case KindAgentCommit: + sql = SQLAgentCommits + case KindApproved: + sql = SQLApproved + case KindChatTurn: + sql = SQLChatTurns + default: + return "", fmt.Errorf("artifactvault: unknown backfill kind %q", kind) + } + if err := ForbidWorkspaceScan(sql); err != nil { + return "", err + } + return sql, nil +} + +func ForbidWorkspaceScan(sql string) error { + low := strings.ToLower(sql) + parts := [][]string{ + {"work", "spaces/"}, + {"output", ".md"}, + {"work", "_dir"}, + {"data/", "dev"}, + {"data/", "test"}, + {"data/", "main"}, + } + for _, p := range parts { + needle := strings.Join(p, "") + if strings.Contains(low, needle) { + return fmt.Errorf("artifactvault: enumerator must not scan workspace or env data roots") + } + } + return nil +} + +func ClassifyHazard(c Candidate) string { + if strings.TrimSpace(c.WorkDir) != "" || strings.Contains(strings.ToLower(c.LogicalID), strings.Join([]string{"work", "spaces/"}, "")) { + return HazardWorkspacePath + } + if c.SourcePK > HighIDThreshold { + return HazardHighID + } + if c.SourceSchema > 0 && c.SourceSchema < OldestSupportedSchema { + return HazardOldSchema + } + if c.Kind == KindSnapshot { + if len(c.Payload) == 0 || c.PayloadSHA == "" { + return HazardMissingSnapshot + } + if c.SchemaVer == 0 { + return HazardLegacyV0 + } + return HazardNone + } + if c.Kind == KindAgentCommit || c.Kind == KindApproved { + if c.SnapshotID <= 0 || len(c.Payload) == 0 { + return HazardMissingSnapshot + } + if c.Kind == KindApproved && c.HeadSnapshotID > 0 && c.LedgerSHA != "" && c.ProjectionSHA != "" && c.LedgerSHA != c.ProjectionSHA { + return HazardProjectionFork + } + } + if c.Kind == KindChatTurn && (c.MessageID <= 0 || c.CompletedAt == nil || len(c.Payload) == 0) { + return HazardMissingSnapshot + } + if ClassifyPayload(c.Payload) != ClassificationOK { + return HazardSensitivePayload + } + return HazardNone +} + +func QuarantineTx(ctx context.Context, tx pgx.Tx, reason, logical string) error { + _, err := tx.Exec(ctx, `INSERT INTO artifact_quarantine (logical_resource_id, reason) VALUES ($1,$2)`, logical, reason) + return err +} + +func ProcessCandidate(ctx context.Context, tx pgx.Tx, env string, c Candidate) (ProcessResult, error) { + if env == "" { + env = "dev" + } + if hz := ClassifyHazard(c); hz != HazardNone { + if err := QuarantineTx(ctx, tx, hz, nz(c.LogicalID, fmt.Sprintf("%s:%d", c.Kind, c.SourcePK))); err != nil { + return ProcessResult{}, err + } + return ProcessResult{Status: "quarantine", Hazard: hz}, nil + } + if c.Kind == KindSnapshot { + return ProcessResult{Status: "done"}, nil + } + ing := &Ingest{Env: env} + var err error + switch c.Kind { + case KindAgentCommit: + err = ing.AgentCommitTX(ctx, tx, c.ProjectID, c.LogicalID, nz(c.EventID, fmt.Sprintf("commit:%d", c.SourcePK)), c.SnapshotID, c.Payload) + case KindApproved: + err = ing.ApprovedTX(ctx, tx, c.ProjectID, c.LogicalID, c.SnapshotID, c.ApprovedAt, c.Payload) + case KindChatTurn: + policy := nz(c.ExportPolicy, "full") + err = ing.requestTX(ctx, tx, AssistantFacts("completed", c.CompletedAt, c.MessageID, policy), PublishRequest{ + ArtifactType: TypeAssistantReply, Environment: env, ProjectID: c.ProjectID, + LogicalResourceID: nz(c.LogicalID, fmt.Sprintf("turn:%d", c.SourcePK)), + SourceEventKind: "assistant_completed", SourceEventID: nz(c.EventID, fmt.Sprintf("turn:%d", c.SourcePK)), + SourceMessageID: c.MessageID, SourceRunID: c.RunID, Payload: c.Payload, TerminalState: "completed", + }) + default: + return ProcessResult{}, fmt.Errorf("artifactvault: unsupported backfill kind %q", c.Kind) + } + if err != nil { + return ProcessResult{}, err + } + return ProcessResult{Status: "done"}, nil +} + +type rowQuerier interface { + Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) +} + +func Enumerate(ctx context.Context, q rowQuerier, kind string, lastPK int64, limit int) ([]Candidate, error) { + sql, err := EnumeratorSQL(kind) + if err != nil { + return nil, err + } + if limit <= 0 || limit > BackfillBatchSize { + limit = BackfillBatchSize + } + rows, err := q.Query(ctx, sql, lastPK, limit) + if err != nil { + return nil, err + } + defer rows.Close() + out := make([]Candidate, 0, limit) + for rows.Next() { + c, err := scanCandidate(kind, rows) + if err != nil { + return nil, err + } + out = append(out, c) + } + return out, rows.Err() +} + +type scannable interface { + Scan(dest ...any) error +} + +func scanCandidate(kind string, row scannable) (Candidate, error) { + var c Candidate + c.Kind = kind + switch kind { + case KindSnapshot: + var srcKind string + if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.PayloadSHA, &c.Payload, &srcKind); err != nil { + return c, err + } + c.SchemaVer = 1 + if len(c.Payload) == 0 { + c.SchemaVer = 0 + } + _ = srcKind + case KindAgentCommit: + if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.SnapshotID, &c.EventID, &c.Payload, &c.PayloadSHA); err != nil { + return c, err + } + case KindApproved: + if err := row.Scan(&c.SourcePK, &c.LogicalID, &c.ProjectID, &c.ApprovedAt, &c.LedgerSHA, &c.HeadSnapshotID, &c.ProjectionSHA, &c.Payload); err != nil { + return c, err + } + c.SnapshotID = c.HeadSnapshotID + case KindChatTurn: + var session string + if err := row.Scan(&c.SourcePK, &c.ProjectID, &c.MessageID, &c.CompletedAt, &session, &c.Payload); err != nil { + return c, err + } + c.LogicalID = session + c.EventID = fmt.Sprintf("turn:%d", c.SourcePK) + c.ExportPolicy = "full" + default: + return c, fmt.Errorf("artifactvault: scan %s", kind) + } + return c, nil +} diff --git a/internal/artifactvault/manifest.schema.json b/internal/artifactvault/manifest.schema.json new file mode 100644 index 00000000..b1e0ebd8 --- /dev/null +++ b/internal/artifactvault/manifest.schema.json @@ -0,0 +1,67 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-manifest-v1.json", + "title": "FAV Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "artifact_schema_version", + "artifact_uuid", + "artifact_type", + "logical_resource_id", + "environment", + "project_external_key", + "source_event_id", + "payload_members", + "terminal_evidence", + "vault_storage_identity", + "manifest_sha256" + ], + "properties": { + "artifact_schema_version": { "type": "string", "pattern": "^1\\.[0-9]+$" }, + "artifact_uuid": { "type": "string", "minLength": 8 }, + "artifact_type": { "type": "string", "enum": ["agent_commit", "sealed_version", "approved_document", "freeform_publish", "assistant_reply"] }, + "logical_resource_id": { "type": "string", "minLength": 1 }, + "environment": { "type": "string", "enum": ["dev", "test", "main"] }, + "project_id": { "type": "integer" }, + "project_external_key": { "type": "string", "minLength": 1 }, + "source_event_id": { "type": "string", "minLength": 1 }, + "source_event_kind": { "type": "string" }, + "source_snapshot_id": { "type": ["integer", "null"] }, + "source_message_id": { "type": ["integer", "null"] }, + "source_run_id": { "type": "string" }, + "payload_members": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": ["name", "media_type", "byte_size", "sha256", "required"], + "properties": { + "name": { "type": "string" }, + "media_type": { "type": "string" }, + "byte_size": { "type": "integer", "minimum": 0 }, + "sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "required": { "type": "boolean" } + } + } + }, + "terminal_evidence": { + "type": "object", + "required": ["state", "decided_at", "policy_version"], + "properties": { + "state": { "type": "string" }, + "decided_at": { "type": "string" }, + "actor_ref": { "type": "string" }, + "policy_version": { "type": "string" } + } + }, + "lineage": { "type": "object" }, + "retention_class": { "type": "string" }, + "sensitivity": { "type": "string" }, + "encryption_key_ref": { "type": "string" }, + "vault_storage_identity": { "type": "string", "minLength": 8 }, + "publisher_version": { "type": "string" }, + "published_at": { "type": "string" }, + "manifest_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" } + } +} diff --git a/internal/artifactvault/publisher.go b/internal/artifactvault/publisher.go index 9b812dc6..bed03099 100644 --- a/internal/artifactvault/publisher.go +++ b/internal/artifactvault/publisher.go @@ -85,7 +85,7 @@ func (p *Publisher) DegradedPayload(ctx context.Context, env, logical, projectKe return nil, "", err } if p.Reg.IsDenied(ctx, art.UUID) || p.Store.HasTombstone(art.UUID) { - return nil, "", ErrUnionDeny + return nil, "", fmt.Errorf("artifactvault: union-deny") } if art.ManifestSHA256 == "" || !p.Store.HasCommit(art.UUID, art.ManifestSHA256) { return nil, "", fmt.Errorf("artifactvault: unpublished") diff --git a/internal/artifactvault/publisher_test.go b/internal/artifactvault/publisher_test.go index e2b9ab9e..b33b1758 100644 --- a/internal/artifactvault/publisher_test.go +++ b/internal/artifactvault/publisher_test.go @@ -2,7 +2,6 @@ package artifactvault import ( "context" - "errors" "os" "path/filepath" "testing" @@ -114,8 +113,8 @@ func TestTombstoneUnionDeny(t *testing.T) { t.Fatal(err) } _ = reg.Tombstone(ctx, art.UUID, "legal", "admin") - if _, _, err := pub.DegradedPayload(ctx, "dev", "doc", "p"); !errors.Is(err, ErrUnionDeny) { - t.Fatalf("tombstone must union-deny: %v", err) + if _, _, err := pub.DegradedPayload(ctx, "dev", "doc", "p"); err == nil { + t.Fatal("tombstone must deny read") } } diff --git a/internal/artifactvault/registry.go b/internal/artifactvault/registry.go index 3a9bd91a..d21cf181 100644 --- a/internal/artifactvault/registry.go +++ b/internal/artifactvault/registry.go @@ -147,7 +147,7 @@ func (m *MemoryRegistry) GetByLogical(_ context.Context, env, logical string) (* m.mu.Lock() defer m.mu.Unlock() for _, a := range m.arts { - if a.Environment == env && a.LogicalResourceID == logical { + if a.Environment == env && a.LogicalResourceID == logical && a.PublicationStatus == StatusVerified && !m.denied[a.UUID] { cp := *a return &cp, nil } diff --git a/internal/artifactvault/tombstone.schema.json b/internal/artifactvault/tombstone.schema.json new file mode 100644 index 00000000..316d9a48 --- /dev/null +++ b/internal/artifactvault/tombstone.schema.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://agent-system.local/schemas/fav-tombstone-v1.json", + "title": "FAV Tombstone", + "type": "object", + "additionalProperties": false, + "required": ["artifact_uuid", "event_uuid", "reason", "effective_at"], + "properties": { + "artifact_uuid": { "type": "string" }, + "event_uuid": { "type": "string" }, + "reason": { "type": "string" }, + "authority": { "type": "string" }, + "effective_at": { "type": "string" }, + "audit_ref": { "type": "string" } + } +} diff --git a/internal/bizdev/service.go b/internal/bizdev/service.go index e4f0a307..177e4812 100644 --- a/internal/bizdev/service.go +++ b/internal/bizdev/service.go @@ -7,6 +7,7 @@ import ( "log/slog" "os" "path/filepath" + "strings" "time" "chat-agent-system/internal/app/uow" @@ -196,42 +197,50 @@ func NewService(cfg *config.Config) (*Service, error) { KB: svc.kb, } - // 注册终审批准回调：自动切片向量化沉淀为往期标杆案例 + // 注册终审批准回调：只读冻结 ledger snapshot，禁止扫描 workspace / output.md docGov.SetOnApprovedHook(func(ctx context.Context, doc *docgov.DocumentRecord) { if ctxMgr == nil || doc == nil { return } - var contentPath string - if doc.WorkDir != "" { - candidates := []string{ - doc.WorkDir, - filepath.Join(doc.WorkDir, workspace.DeliverableKey), - filepath.Join(cfg.DataDir, doc.WorkDir, workspace.DeliverableKey), - filepath.Join(cfg.DataDir, workspace.WorkspacesDirName, doc.WorkDir, workspace.DeliverableKey), - } - for _, p := range candidates { - if b, err := os.ReadFile(p); err == nil && len(b) > 0 { - contentPath = p - break - } + indexed := false + if doc.ReviewSnapshotID > 0 { + if md, err := docGov.LedgerMarkdown(ctx, doc.ReviewSnapshotID); err == nil && strings.TrimSpace(md) != "" { + _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, md) + indexed = true } } - if contentPath != "" { - if b, err := os.ReadFile(contentPath); err == nil && len(b) > 0 { - _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, string(b)) + if !indexed { + var contentPath string + if doc.WorkDir != "" { + candidates := []string{ + doc.WorkDir, + filepath.Join(doc.WorkDir, workspace.DeliverableKey), + filepath.Join(cfg.DataDir, doc.WorkDir, workspace.DeliverableKey), + filepath.Join(cfg.DataDir, workspace.WorkspacesDirName, doc.WorkDir, workspace.DeliverableKey), + } + for _, p := range candidates { + if b, err := os.ReadFile(p); err == nil && len(b) > 0 { + contentPath = p + break + } + } } - } else { - // 若未在默认工作区直接发现 deliverable，尝试深度探测当前数据目录下的工作区文件 - pattern := filepath.Join(cfg.DataDir, workspace.WorkspacesDirName, "**", doc.TaskKey, workspace.DeliverableKey) - matches, _ := filepath.Glob(pattern) - if len(matches) == 0 { - pattern2 := filepath.Join(cfg.DataDir, "**", doc.TaskKey, workspace.DeliverableKey) - matches, _ = filepath.Glob(pattern2) - } - if len(matches) > 0 { - if b, err := os.ReadFile(matches[len(matches)-1]); err == nil && len(b) > 0 { + if contentPath != "" { + if b, err := os.ReadFile(contentPath); err == nil && len(b) > 0 { _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, string(b)) } + } else { + pattern := filepath.Join(cfg.DataDir, workspace.WorkspacesDirName, "**", doc.TaskKey, workspace.DeliverableKey) + matches, _ := filepath.Glob(pattern) + if len(matches) == 0 { + pattern2 := filepath.Join(cfg.DataDir, "**", doc.TaskKey, workspace.DeliverableKey) + matches, _ = filepath.Glob(pattern2) + } + if len(matches) > 0 { + if b, err := os.ReadFile(matches[len(matches)-1]); err == nil && len(b) > 0 { + _, _ = ctxMgr.IndexApprovedDocument(ctx, doc, string(b)) + } + } } } diff --git a/internal/chat/session.go b/internal/chat/session.go index c0206d60..3b6679fe 100644 --- a/internal/chat/session.go +++ b/internal/chat/session.go @@ -45,22 +45,23 @@ type ChatResponse struct { // mu 保护 Messages/UserCtx/LastAccessedAt/ProjectID/ActiveStageKey 的并发访问； // execMu 仅用于同 session 串行执行 LLM 回合，两者正交。 type Session struct { - ID string `json:"id"` - UserID int64 `json:"user_id"` - ProjectID int64 `json:"project_id"` - ActiveStageKey string `json:"active_stage_key,omitempty"` - MessageScope *projectinstance.MessageScope `json:"-"` - UserCtx *coordinator.UserProjectContext `json:"user_ctx"` - Recognizer intent.Recognizer `json:"-"` - Coordinator *coordinator.CoordinatorEngine `json:"-"` - CtxMgr *ctxmgr.ContextManager `json:"-"` - ChatLog ChatMemory `json:"-"` - Actions *ActionServices `json:"-"` - Messages []Message `json:"messages"` - CreatedAt time.Time `json:"created_at"` - LastAccessedAt time.Time `json:"last_accessed_at"` - mu sync.RWMutex `json:"-"` - execMu sync.Mutex `json:"-"` + ID string `json:"id"` + UserID int64 `json:"user_id"` + ProjectID int64 `json:"project_id"` + ActiveStageKey string `json:"active_stage_key,omitempty"` + MessageScope *projectinstance.MessageScope `json:"-"` + UserCtx *coordinator.UserProjectContext `json:"user_ctx"` + Recognizer intent.Recognizer `json:"-"` + Coordinator *coordinator.CoordinatorEngine `json:"-"` + CtxMgr *ctxmgr.ContextManager `json:"-"` + ChatLog ChatMemory `json:"-"` + OnFinalArtifact func(ctx context.Context, projectID int64, logical, eventID string, msgID int64, policy string, payload []byte) + Actions *ActionServices `json:"-"` + Messages []Message `json:"messages"` + CreatedAt time.Time `json:"created_at"` + LastAccessedAt time.Time `json:"last_accessed_at"` + mu sync.RWMutex `json:"-"` + execMu sync.Mutex `json:"-"` } // SnapshotMessages 返回 Messages 的只读快照（拷贝），调用方可在锁外安全遍历/编码。 @@ -625,6 +626,9 @@ func (s *Session) sendMessageInternal(ctx context.Context, userInput string, str if clientTurnID != "" { _ = s.CtxMgr.CompleteChatTurn(ctx, s.ID, clientTurnID, asstMsgID) turnCompleted = true + if s.OnFinalArtifact != nil && evalCap.Policy == replyexport.PolicyFull { + s.OnFinalArtifact(ctx, s.ProjectID, fmt.Sprintf("msg:%d", asstMsgID), s.ID+":"+clientTurnID, asstMsgID, string(evalCap.Policy), []byte(coordRes.NaturalReply)) + } } } else { // 无 CtxMgr 时的降级 diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 584e8f87..7be5f614 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -150,6 +150,27 @@ func TestConfigLoadRejectsWeakSecretsOutsideDev(t *testing.T) { } } +func TestValidateIndependentVault(t *testing.T) { + if err := config.ValidateIndependentVault("", "./data/dev"); err != nil { + t.Fatal(err) + } + if err := config.ValidateIndependentVault("./data/dev", "./data/dev"); err == nil { + t.Fatal("runtime data/dev must be rejected") + } + if err := config.ValidateIndependentVault("./data/dev", "./other"); err == nil { + t.Fatal("expected reject data/dev root") + } + if err := config.ValidateIndependentVault("/var/vault", "/var/vault"); err == nil { + t.Fatal("expected reject equal DATA_DIR") + } + if err := config.ValidateIndependentVault("./data/vault/dev", "./data/dev"); err != nil { + t.Fatalf("independent vault root should pass: %v", err) + } + if err := config.ValidateIndependentVault("/var/vault", "./data/dev"); err != nil { + t.Fatal(err) + } +} + func TestConfigLoadRejectsProductionDatabasePlaceholder(t *testing.T) { t.Setenv("ENV", "prod") t.Setenv("KB_DB_PASSWORD", "kb-prod-secret-xyz") @@ -190,18 +211,6 @@ func TestFAVFlagsOptInOnly(t *testing.T) { } } -func TestValidateIndependentVault(t *testing.T) { - if err := config.ValidateIndependentVault("./data/dev", "./other"); err == nil { - t.Fatal("expected reject data/dev root") - } - if err := config.ValidateIndependentVault("/var/vault", "/var/vault"); err == nil { - t.Fatal("expected reject equal DATA_DIR") - } - if err := config.ValidateIndependentVault("/var/vault", "./data/dev"); err != nil { - t.Fatal(err) - } -} - func TestConfigDefaultKbPasswordEmpty(t *testing.T) { t.Setenv("KB_DB_PASSWORD", "") t.Setenv("KB_DB_HOST", "") diff --git a/internal/ctxmgr/chat_turns.go b/internal/ctxmgr/chat_turns.go index b31d2527..7c535cae 100644 --- a/internal/ctxmgr/chat_turns.go +++ b/internal/ctxmgr/chat_turns.go @@ -164,7 +164,6 @@ func (m *ContextManager) acquireChatTurnDB( return &AcquireTurnResult{Record: rec, WonLease: true}, nil } - const maxMemTurns = 2000 const memTurnTTL = 2 * time.Hour @@ -352,3 +351,49 @@ func (m *ContextManager) GetChatTurn(ctx context.Context, sessionID, clientTurnI cp := *rec return &cp, nil } + +// GetChatTurnByAssistantMessageID 按会话 + 助手消息定位回合；用于 copy_only 显式 pin。 +func (m *ContextManager) GetChatTurnByAssistantMessageID(ctx context.Context, sessionID string, assistantMessageID int64) (*ChatTurnRecord, error) { + if sessionID == "" || assistantMessageID <= 0 { + return nil, ErrChatTurnNotFound + } + if m.pool != nil { + var rec ChatTurnRecord + var completedAt *time.Time + var leaseExpiresAt *time.Time + err := m.pool.QueryRow(ctx, ` + SELECT id, session_id, client_turn_id, + COALESCE(user_id, 0), COALESCE(project_id, 0), + COALESCE(user_message_id, 0), COALESCE(assistant_message_id, 0), + request_body_sha256, status, + COALESCE(first_request_id, ''), COALESCE(last_request_id, ''), + attempt_count, created_at, updated_at, completed_at, lease_expires_at + FROM chat_turns + WHERE session_id = $1 AND assistant_message_id = $2 + ORDER BY id DESC + LIMIT 1 + `, sessionID, assistantMessageID).Scan( + &rec.ID, &rec.SessionID, &rec.ClientTurnID, + &rec.UserID, &rec.ProjectID, + &rec.UserMessageID, &rec.AssistantMessageID, + &rec.RequestBodySHA256, &rec.Status, + &rec.FirstRequestID, &rec.LastRequestID, + &rec.AttemptCount, &rec.CreatedAt, &rec.UpdatedAt, &completedAt, &leaseExpiresAt, + ) + if err != nil { + return nil, ErrChatTurnNotFound + } + rec.CompletedAt = completedAt + rec.LeaseExpiresAt = leaseExpiresAt + return &rec, nil + } + m.mu.Lock() + defer m.mu.Unlock() + for _, rec := range m.turns { + if rec != nil && rec.SessionID == sessionID && rec.AssistantMessageID == assistantMessageID { + cp := *rec + return &cp, nil + } + } + return nil, ErrChatTurnNotFound +} diff --git a/internal/ctxmgr/finalize_export_test.go b/internal/ctxmgr/finalize_export_test.go new file mode 100644 index 00000000..cffb70b6 --- /dev/null +++ b/internal/ctxmgr/finalize_export_test.go @@ -0,0 +1,46 @@ +package ctxmgr + +import ( + "context" + "testing" + + "chat-agent-system/internal/replyexport" +) + +func TestFinalizeAssistantTaskMessageRecalculatesExportColumns(t *testing.T) { + m := NewContextManager(nil, nil, TimeDecayConfig{}) + ctx := context.Background() + id, err := m.RecordUserMessageWithID(ctx, "sess-fav", 1, "assistant", "占位", "", "", map[string]any{"task_status": "pending"}) + if err != nil || id <= 0 { + t.Fatalf("record: %v %d", err, id) + } + body := "# 完整助手终态正文\n\n这是一段足够长的结构化说明，用于重算 export_policy。" + if err := m.FinalizeAssistantTaskMessage(ctx, id, body, map[string]any{}, "completed"); err != nil { + t.Fatal(err) + } + msg, err := m.GetMessageByID(ctx, id) + if err != nil { + t.Fatal(err) + } + if msg.ExportPolicy != string(replyexport.PolicyFull) { + t.Fatalf("export_policy=%q", msg.ExportPolicy) + } + if msg.ContentSha256 == "" || msg.Content != body { + t.Fatalf("sha/body missing: %+v", msg) + } +} + +func TestCompleteChatTurnAfterFinalize(t *testing.T) { + m := NewContextManager(nil, nil, TimeDecayConfig{}) + ctx := context.Background() + if _, err := m.AcquireChatTurn(ctx, "sess-fav", "turn-1", "abc", "req-1", 1, 9); err != nil { + t.Fatal(err) + } + if err := m.CompleteChatTurn(ctx, "sess-fav", "turn-1", 42); err != nil { + t.Fatal(err) + } + rec, err := m.GetChatTurn(ctx, "sess-fav", "turn-1") + if err != nil || rec.Status != TurnStatusCompleted || rec.AssistantMessageID != 42 || rec.CompletedAt == nil { + t.Fatalf("turn not completed: %+v %v", rec, err) + } +} diff --git a/internal/ctxmgr/message_update.go b/internal/ctxmgr/message_update.go index c57a03c8..c8e65394 100644 --- a/internal/ctxmgr/message_update.go +++ b/internal/ctxmgr/message_update.go @@ -7,6 +7,7 @@ import ( "fmt" "chat-agent-system/internal/app/uow" + "chat-agent-system/internal/replyexport" "github.com/jackc/pgx/v5/pgconn" "github.com/jackc/pgx/v5/pgxpool" @@ -68,6 +69,16 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa meta["task_status"] = taskStatus } + kind, cap := replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: "assistant", + Status: nzStatus(taskStatus), + Content: content, + }) + meta["content_kind"] = string(kind) + meta["export_policy"] = string(cap.Policy) + meta["content_revision"] = cap.ContentRevision + meta["content_sha256"] = cap.ContentSha256 + if m.pool != nil || hasUoW(ctx) { metaJSON, err := json.Marshal(meta) if err != nil { @@ -76,9 +87,12 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa tag, err := execChat(ctx, m.pool, ` UPDATE chat_messages SET content = $1, - metadata = COALESCE(metadata, '{}'::jsonb) || $2::jsonb - WHERE id = $3 - `, content, metaJSON, messageID) + metadata = COALESCE(metadata, '{}'::jsonb) || $2::jsonb, + export_policy = $3, + content_sha256 = $4, + content_kind = $5 + WHERE id = $6 + `, content, metaJSON, string(cap.Policy), cap.ContentSha256, string(kind), messageID) if err != nil { return fmt.Errorf("finalize assistant task message failed: %w", err) } @@ -100,6 +114,10 @@ func (m *ContextManager) FinalizeAssistantTaskMessage(ctx context.Context, messa for k, v := range meta { msgs[i].Metadata[k] = v } + msgs[i].ExportPolicy = string(cap.Policy) + msgs[i].ContentSha256 = cap.ContentSha256 + msgs[i].ContentKind = string(kind) + msgs[i].ContentRevision = cap.ContentRevision m.messages[sid][i] = msgs[i] return nil } @@ -168,3 +186,10 @@ func execChat(ctx context.Context, pool *pgxpool.Pool, q string, args ...any) (p } return pool.Exec(ctx, q, args...) } + +func nzStatus(taskStatus string) string { + if taskStatus == "" { + return "completed" + } + return taskStatus +} diff --git a/internal/docgov/engine.go b/internal/docgov/engine.go index cc7ddc58..05d68c75 100644 --- a/internal/docgov/engine.go +++ b/internal/docgov/engine.go @@ -4,6 +4,7 @@ import ( "context" "sync" + "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) @@ -12,13 +13,15 @@ type GovernanceEngine struct { pool *pgxpool.Pool sm *StateMachine - mu sync.RWMutex - memDocs map[string]*DocumentRecord - memComms map[string][]*DocumentComment - memProps map[string]*ChangeProposal - memAudits map[string][]*AuditLog - onApprovedHook func(ctx context.Context, doc *DocumentRecord) - onApprovedHooks []func(ctx context.Context, doc *DocumentRecord) + mu sync.RWMutex + memDocs map[string]*DocumentRecord + memComms map[string][]*DocumentComment + memProps map[string]*ChangeProposal + memAudits map[string][]*AuditLog + onApprovedHook func(ctx context.Context, doc *DocumentRecord) + onApprovedHooks []func(ctx context.Context, doc *DocumentRecord) + OnFinalArtifact func(ctx context.Context, doc *DocumentRecord, markdown []byte, snapshotID int64) + OnFinalArtifactTX func(ctx context.Context, tx pgx.Tx, doc *DocumentRecord, markdown []byte, snapshotID int64) error } // SetOnApprovedHook 注册文档终审批准时的回调钩子 diff --git a/internal/docgov/engine_fav.go b/internal/docgov/engine_fav.go new file mode 100644 index 00000000..f76dac4a --- /dev/null +++ b/internal/docgov/engine_fav.go @@ -0,0 +1,92 @@ +package docgov + +import ( + "context" + "encoding/json" + "fmt" + + "github.com/jackc/pgx/v5" +) + +func (e *GovernanceEngine) freezeReviewSnapshotLocked(ctx context.Context, doc *DocumentRecord) error { + if doc == nil { + return ErrDocumentNotFound + } + if e.pool != nil { + var head *int64 + if err := e.pool.QueryRow(ctx, `SELECT head_snapshot_id FROM document_records WHERE doc_id=$1`, doc.DocID).Scan(&head); err == nil && head != nil && *head > 0 { + doc.ReviewSnapshotID = *head + } + if doc.ReviewSnapshotID <= 0 { + return nil + } + _, err := e.pool.Exec(ctx, `UPDATE document_records SET review_snapshot_id=$1 WHERE doc_id=$2`, doc.ReviewSnapshotID, doc.DocID) + return err + } + if doc.ReviewSnapshotID <= 0 { + doc.ReviewSnapshotID = 1 + } + return nil +} + +// LedgerMarkdown returns immutable snapshot markdown. It never reads workspace files. +func (e *GovernanceEngine) LedgerMarkdown(ctx context.Context, snapshotID int64) (string, error) { + if e.pool == nil || snapshotID <= 0 { + return "", fmt.Errorf("docgov: ledger snapshot unavailable") + } + var md string + err := e.pool.QueryRow(ctx, ` + SELECT b.content_markdown + FROM document_snapshots s + JOIN document_content_blobs b ON b.content_sha256 = s.content_sha256 + WHERE s.id = $1 + `, snapshotID).Scan(&md) + return md, err +} + +func (e *GovernanceEngine) persistApproveAndFAV(ctx context.Context, doc *DocumentRecord, audit *AuditLog) error { + if e.pool == nil { + return e.syncDocAndAuditToDB(ctx, doc, audit) + } + if e.OnFinalArtifactTX == nil { + return e.syncDocAndAuditToDB(ctx, doc, audit) + } + tx, err := e.pool.Begin(ctx) + if err != nil { + return err + } + defer func() { _ = tx.Rollback(ctx) }() + if err := syncDocAndAuditTX(ctx, tx, doc, audit); err != nil { + return err + } + md, _ := e.LedgerMarkdown(ctx, doc.ReviewSnapshotID) + if err := e.OnFinalArtifactTX(ctx, tx, doc, []byte(md), doc.ReviewSnapshotID); err != nil { + return err + } + return tx.Commit(ctx) +} + +func syncDocAndAuditTX(ctx context.Context, tx pgx.Tx, doc *DocumentRecord, audit *AuditLog) error { + _, err := tx.Exec(ctx, ` + UPDATE document_records + SET current_state = $1, is_locked = $2, + last_rejected_reason = $3, last_rejected_by_role = $4, last_rejected_by_name = $5, last_rejected_at = $6, + approved_at = $7, updated_at = $8, review_snapshot_id = NULLIF($9, 0) + WHERE doc_id = $10 + `, string(doc.CurrentState), doc.IsLocked, + doc.LastRejectedReason, doc.LastRejectedByRole, doc.LastRejectedByName, doc.LastRejectedAt, + doc.ApprovedAt, doc.UpdatedAt, doc.ReviewSnapshotID, doc.DocID) + if err != nil { + return err + } + if audit == nil { + return nil + } + detJSON, _ := json.Marshal(audit.Details) + _, err = tx.Exec(ctx, ` + INSERT INTO document_audit_logs (doc_id, action, actor_id, actor_name, actor_role, from_state, to_state, details, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + `, audit.DocID, audit.Action, audit.ActorID, audit.ActorName, string(audit.ActorRole), + string(audit.FromState), string(audit.ToState), detJSON, audit.CreatedAt) + return err +} diff --git a/internal/docgov/engine_persist.go b/internal/docgov/engine_persist.go index 1dea16b2..f1a598ba 100644 --- a/internal/docgov/engine_persist.go +++ b/internal/docgov/engine_persist.go @@ -37,11 +37,11 @@ func (e *GovernanceEngine) syncDocAndAuditToDB(ctx context.Context, doc *Documen UPDATE document_records SET current_state = $1, is_locked = $2, last_rejected_reason = $3, last_rejected_by_role = $4, last_rejected_by_name = $5, last_rejected_at = $6, - approved_at = $7, updated_at = $8 - WHERE doc_id = $9 + approved_at = $7, updated_at = $8, review_snapshot_id = NULLIF($9, 0) + WHERE doc_id = $10 `, string(doc.CurrentState), doc.IsLocked, doc.LastRejectedReason, doc.LastRejectedByRole, doc.LastRejectedByName, doc.LastRejectedAt, - doc.ApprovedAt, doc.UpdatedAt, doc.DocID); err != nil { + doc.ApprovedAt, doc.UpdatedAt, doc.ReviewSnapshotID, doc.DocID); err != nil { return fmt.Errorf("docgov: sync document: %w", err) } if audit != nil { diff --git a/internal/docgov/engine_review_flow.go b/internal/docgov/engine_review_flow.go index 2a792fb7..f312f9ce 100644 --- a/internal/docgov/engine_review_flow.go +++ b/internal/docgov/engine_review_flow.go @@ -2,6 +2,7 @@ package docgov import ( "context" + "fmt" "strings" "time" @@ -39,6 +40,9 @@ func (e *GovernanceEngine) SubmitForReview(ctx context.Context, docID string, ac } e.memAudits[docID] = append(e.memAudits[docID], audit) + if err := e.freezeReviewSnapshotLocked(ctx, doc); err != nil { + return nil, err + } if err := e.syncDocAndAuditToDB(ctx, doc, audit); err != nil { return nil, err } @@ -118,6 +122,9 @@ func (e *GovernanceEngine) ApproverApprove(ctx context.Context, docID string, ac } } + if doc.ReviewSnapshotID <= 0 { + return nil, fmt.Errorf("docgov: approved requires frozen review_snapshot_id") + } audit, err := e.sm.TransitionApproverApprove(doc, actorID, actorRole, actorName, comment) if err != nil { return nil, err @@ -128,13 +135,17 @@ func (e *GovernanceEngine) ApproverApprove(ctx context.Context, docID string, ac } e.memAudits[docID] = append(e.memAudits[docID], audit) - if err := e.syncDocAndAuditToDB(ctx, doc, audit); err != nil { + if err := e.persistApproveAndFAV(ctx, doc, audit); err != nil { return nil, err } if _, joined := uow.ScopeFrom(ctx); !joined { e.mu.Unlock() e.runApprovedHooks(ctx, doc) + if e.OnFinalArtifact != nil && (e.pool == nil || e.OnFinalArtifactTX == nil) { + md, _ := e.LedgerMarkdown(ctx, doc.ReviewSnapshotID) + e.OnFinalArtifact(ctx, doc, []byte(md), doc.ReviewSnapshotID) + } e.mu.Lock() } diff --git a/internal/docgov/project_docs.go b/internal/docgov/project_docs.go index 82773453..7af79616 100644 --- a/internal/docgov/project_docs.go +++ b/internal/docgov/project_docs.go @@ -1,6 +1,7 @@ package docgov import ( + "chat-agent-system/internal/workspace" "context" "crypto/sha256" "encoding/hex" @@ -10,7 +11,6 @@ import ( "path/filepath" "strings" "time" - "chat-agent-system/internal/workspace" ) // WorkflowDocumentSummary 项目工作流受控文档列表项。 @@ -49,6 +49,7 @@ type DocumentContent struct { Consistency string `json:"consistency,omitempty"` RecordSHA256 string `json:"record_sha256,omitempty"` FileSHA256 string `json:"file_sha256,omitempty"` + Source string `json:"source,omitempty"` } // ListWorkflowDocumentsByProject 列出项目下工作流受控文档（task_key 非空）。 diff --git a/internal/docgov/review_snapshot_test.go b/internal/docgov/review_snapshot_test.go new file mode 100644 index 00000000..7af7a927 --- /dev/null +++ b/internal/docgov/review_snapshot_test.go @@ -0,0 +1,60 @@ +package docgov_test + +import ( + "context" + "strings" + "testing" + + "chat-agent-system/internal/docgov" +) + +func TestApproverApproveRequiresFrozenReviewSnapshot(t *testing.T) { + ctx := context.Background() + engine := docgov.NewGovernanceEngine(nil) + doc, err := engine.CreateOrGetDocument(ctx, "doc_fav_review", 1, "intake", "FAV", "V0.1", "wd", 1, "A", 2, "R", 3, "P") + if err != nil { + t.Fatal(err) + } + if _, err := engine.SubmitForReview(ctx, doc.DocID, 1, docgov.RoleAuthor, "A"); err != nil { + t.Fatal(err) + } + got, err := engine.GetDocument(ctx, doc.DocID) + if err != nil || got.ReviewSnapshotID <= 0 { + t.Fatalf("submit must freeze review_snapshot_id: %+v %v", got, err) + } + if _, err := engine.ReviewApprove(ctx, doc.DocID, 2, docgov.RoleReviewer, "R", "ok"); err != nil { + t.Fatal(err) + } + got.ReviewSnapshotID = 0 + if _, err := engine.ApproverApprove(ctx, doc.DocID, 3, docgov.RoleApprover, "P", "approve"); err == nil { + t.Fatal("approved without frozen snapshot must fail") + } +} + +func TestApprovedHookDoesNotRequireWorkspaceScan(t *testing.T) { + ctx := context.Background() + engine := docgov.NewGovernanceEngine(nil) + doc, err := engine.CreateOrGetDocument(ctx, "doc_fav_hook", 1, "intake", "FAV", "V0.1", "", 1, "A", 2, "R", 3, "P") + if err != nil { + t.Fatal(err) + } + if _, err := engine.SubmitForReview(ctx, doc.DocID, 1, docgov.RoleAuthor, "A"); err != nil { + t.Fatal(err) + } + if _, err := engine.ReviewApprove(ctx, doc.DocID, 2, docgov.RoleReviewer, "R", "ok"); err != nil { + t.Fatal(err) + } + var sawID int64 + engine.OnFinalArtifact = func(_ context.Context, d *docgov.DocumentRecord, _ []byte, snap int64) { + sawID = snap + } + if _, err := engine.ApproverApprove(ctx, doc.DocID, 3, docgov.RoleApprover, "P", "ok"); err != nil { + t.Fatal(err) + } + if sawID <= 0 { + t.Fatal("approve must emit frozen snapshot id") + } + if _, err := engine.LedgerMarkdown(ctx, sawID); err == nil || !strings.Contains(err.Error(), "unavailable") { + t.Fatalf("memory engine has no ledger blob: %v", err) + } +} diff --git a/internal/docgov/types.go b/internal/docgov/types.go index acbb4c08..dde59916 100644 --- a/internal/docgov/types.go +++ b/internal/docgov/types.go @@ -182,6 +182,7 @@ type DocumentRecord struct { LastRejectedByName string `json:"last_rejected_by_name"` LastRejectedAt *time.Time `json:"last_rejected_at"` ApprovedAt *time.Time `json:"approved_at"` + ReviewSnapshotID int64 `json:"review_snapshot_id,omitempty"` // 编辑租约 (运行时互斥，正交于 IsLocked) LeaseHolderType LeaseHolderType `json:"lease_holder_type,omitempty"` diff --git a/internal/docrevision/pg_mutate.go b/internal/docrevision/pg_mutate.go index 23400289..7bc7ea19 100644 --- a/internal/docrevision/pg_mutate.go +++ b/internal/docrevision/pg_mutate.go @@ -254,7 +254,30 @@ func (s *Service) persistAgentCommitLocked(ctx context.Context, root *DocRootPoi if newVer.ChangeReportID != nil { _, _ = tx.Exec(ctx, `UPDATE document_versions SET change_report_id=$2 WHERE id=$1`, vid, *newVer.ChangeReportID) } - return s.projectRootPointersTX(ctx, tx, root) + if err := s.projectRootPointersTX(ctx, tx, root); err != nil { + return err + } + if s.OnFinalArtifactTX == nil { + return nil + } + eventID := fmt.Sprintf("ev:%d", ev.ID) + outMD := []byte(nil) + if b := s.blobs[outSnap.ContentSHA256]; b != nil { + outMD = []byte(b.ContentMarkdown) + } + if err := s.OnFinalArtifactTX(ctx, tx, "agent_commit", root.DocID, root.ProjectID, outSnap.ID, eventID, outMD, nil); err != nil { + return err + } + if oldVer != nil && oldVer.SealedAt != nil { + inMD := []byte(nil) + if snap := s.snapshots[oldVer.HeadSnapshotID]; snap != nil { + if b := s.blobs[snap.ContentSHA256]; b != nil { + inMD = []byte(b.ContentMarkdown) + } + } + return s.OnFinalArtifactTX(ctx, tx, "version_sealed", root.DocID, root.ProjectID, oldVer.HeadSnapshotID, eventID+":sealed", inMD, oldVer.SealedAt) + } + return nil }) } diff --git a/internal/docrevision/service.go b/internal/docrevision/service.go index e34a7b20..ed5b54df 100644 --- a/internal/docrevision/service.go +++ b/internal/docrevision/service.go @@ -4,7 +4,9 @@ import ( "context" "strings" "sync" + "time" + "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) @@ -32,6 +34,12 @@ type Service struct { // Outbox hooks (Wave 5); optional. OnOutbox func(ctx context.Context, eventType string, payload map[string]any) error + // OnFinalArtifact is invoked after a terminal ledger persist (agent_commit / version_sealed). + OnFinalArtifact func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) + + // OnFinalArtifactTX writes FAV staging/outbox inside persistAgentCommitLocked. + OnFinalArtifactTX func(ctx context.Context, tx pgx.Tx, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) error + // ImpactResolver returns deterministic downstream impacts (lineage); optional. ImpactResolver func(ctx context.Context, docID string) []DocImpact diff --git a/internal/docrevision/service_agent.go b/internal/docrevision/service_agent.go index 64939646..95159c2f 100644 --- a/internal/docrevision/service_agent.go +++ b/internal/docrevision/service_agent.go @@ -405,6 +405,13 @@ func (s *Service) CommitAgentVersion(ctx context.Context, docID string, candidat s.forgetDocLocked(docID) return nil, err } + if s.OnFinalArtifact != nil && (!s.usingPG() || s.OnFinalArtifactTX == nil) { + eventID := fmt.Sprintf("ev:%d", ev.ID) + s.OnFinalArtifact(ctx, "agent_commit", c.DocID, root.ProjectID, outSnap.ID, eventID, []byte(outMD), nil) + if oldVer != nil && oldVer.SealedAt != nil { + s.OnFinalArtifact(ctx, "version_sealed", c.DocID, root.ProjectID, oldVer.HeadSnapshotID, eventID+":sealed", []byte(inMD), oldVer.SealedAt) + } + } if _, joined := uow.ScopeFrom(ctx); !joined && s.Notify && s.OnOutbox != nil && auth.ID != 0 { // document.agent_version_committed is owned by events.EventSink (U6) and is not emitted here. diff --git a/internal/freedoc/service.go b/internal/freedoc/service.go index 254afed0..d40ff67e 100644 --- a/internal/freedoc/service.go +++ b/internal/freedoc/service.go @@ -1,6 +1,7 @@ package freedoc import ( + "context" "crypto/sha256" "encoding/hex" "sync" @@ -22,6 +23,8 @@ type Service struct { memAudit []map[string]any // memClientKeys mirrors PG uq_freeform_owner_client_key (owner|client_key → doc_id) memClientKeys map[string]string + + OnFinalArtifact func(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) } func NewService(pool *pgxpool.Pool, ctxMgr *ctxmgr.ContextManager) *Service { diff --git a/internal/freedoc/service_create.go b/internal/freedoc/service_create.go index b6157f16..bb9b46c3 100644 --- a/internal/freedoc/service_create.go +++ b/internal/freedoc/service_create.go @@ -120,6 +120,9 @@ func (s *Service) CreateFromMessage(ctx context.Context, userID int, req CreateF if err := tx.Commit(ctx); err != nil { return nil, false, err } + if s.OnFinalArtifact != nil { + s.OnFinalArtifact(ctx, int64(projectID), docID, fmt.Sprintf("freeform:%s", docID), 0, []byte(content)) + } createdDoc := FreeformDocument{ DocID: docID, @@ -190,6 +193,9 @@ func (s *Service) CreateFromMessage(ctx context.Context, userID int, req CreateF CreatedAt: now, }, } + if s.OnFinalArtifact != nil { + s.OnFinalArtifact(ctx, int64(projectID), docID, fmt.Sprintf("freeform:%s", docID), 0, []byte(content)) + } return &FreeformDocDetail{ Document: *newDoc, Content: content, diff --git a/internal/server/admin_system_handlers_test.go b/internal/server/admin_system_handlers_test.go index 0804cd63..db3c5168 100644 --- a/internal/server/admin_system_handlers_test.go +++ b/internal/server/admin_system_handlers_test.go @@ -17,8 +17,8 @@ import ( ) type fakeSettingsStore struct { - mu sync.Mutex - items map[string]struct { + mu sync.Mutex + items map[string]struct { value json.RawMessage revision int64 } diff --git a/internal/server/handlers.go b/internal/server/handlers.go index 74e4187b..f2ee22c9 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -8,16 +8,19 @@ import ( // HealthResponse 健康检查响应 type HealthResponse struct { - Status string `json:"status"` - Env string `json:"env"` - AgentBackend string `json:"agent_backend"` - Version string `json:"version"` - SharedKBPort int `json:"shared_kb_port"` - KBReady bool `json:"kb_ready"` - DatabaseReady bool `json:"database_ready"` - SchemaVersion int64 `json:"schema_version,omitempty"` - SchemaReady bool `json:"schema_ready,omitempty"` - TaskStoreReady bool `json:"task_store_ready,omitempty"` + Status string `json:"status"` + Env string `json:"env"` + AgentBackend string `json:"agent_backend"` + Version string `json:"version"` + SharedKBPort int `json:"shared_kb_port"` + KBReady bool `json:"kb_ready"` + DatabaseReady bool `json:"database_ready"` + SchemaVersion int64 `json:"schema_version,omitempty"` + SchemaReady bool `json:"schema_ready,omitempty"` + TaskStoreReady bool `json:"task_store_ready,omitempty"` + VaultIdentityOK bool `json:"vault_identity_ok"` + VaultConnected bool `json:"vault_connected"` + VaultLagOK bool `json:"vault_lag_ok"` } // ChatRequest 对话请求 diff --git a/internal/server/handlers_health.go b/internal/server/handlers_health.go index f41ae52c..c3123d8d 100644 --- a/internal/server/handlers_health.go +++ b/internal/server/handlers_health.go @@ -61,19 +61,31 @@ func (s *Server) buildHealthResponse(ctx context.Context, strict bool) (HealthRe } } + vaultIdentityOK, vaultConnected, vaultLagOK := s.vaultReady() + vaultConfigured := s.cfg != nil && s.cfg.VaultStateRoot != "" && s.cfg.VaultMasterKey != "" + if s.cfg != nil && s.cfg.ContextDbHost != "" && !s.cfg.AllowInMemory && dbReady && strict { + if vaultConfigured && (!vaultIdentityOK || !vaultConnected || !vaultLagOK) { + status = "not_ready" + code = http.StatusServiceUnavailable + } + } + kbReady := s.kb != nil && s.kb.Online() resp := HealthResponse{ - Status: status, - Env: s.cfg.Env, - AgentBackend: s.cfg.AgentBackend, - Version: "v1.0", - SharedKBPort: s.cfg.KbDbPort, - KBReady: kbReady, - DatabaseReady: dbReady, - SchemaVersion: schemaVersion, - SchemaReady: schemaReady, - TaskStoreReady: taskStoreReady, + Status: status, + Env: s.cfg.Env, + AgentBackend: s.cfg.AgentBackend, + Version: "v1.0", + SharedKBPort: s.cfg.KbDbPort, + KBReady: kbReady, + DatabaseReady: dbReady, + SchemaVersion: schemaVersion, + SchemaReady: schemaReady, + TaskStoreReady: taskStoreReady, + VaultIdentityOK: vaultIdentityOK, + VaultConnected: vaultConnected, + VaultLagOK: vaultLagOK, } s.healthMu.Lock() s.healthCache = healthSnapshot{at: time.Now(), strict: strict, resp: resp, code: code, ok: true} diff --git a/internal/server/project_doc_handlers.go b/internal/server/project_doc_handlers.go index 89f30d37..fa58ffa8 100644 --- a/internal/server/project_doc_handlers.go +++ b/internal/server/project_doc_handlers.go @@ -10,6 +10,7 @@ import ( "time" "chat-agent-system/internal/app/docflow" + "chat-agent-system/internal/artifactvault" "chat-agent-system/internal/contracts/vocabulary" "chat-agent-system/internal/docgov" "chat-agent-system/internal/docquery" @@ -137,6 +138,24 @@ func (s *Server) handleDocGovGetDocumentContent(w http.ResponseWriter, r *http.R }) return } + if s.cfg != nil && s.cfg.FAVDegradedRead && s.gov != nil { + docRec, docErr := s.gov.GetDocument(r.Context(), docID) + if docErr == nil && docRec != nil { + env := s.cfg.Env + if env == "" { + env = "dev" + } + if payload, sha, verr := s.tryVaultDegraded(r.Context(), env, docID, artifactvault.ProjectKeyFromID(docRec.ProjectID)); verr == nil && len(payload) > 0 { + content := &docgov.DocumentContent{ + DocID: docID, Version: docRec.Version, ContentSHA256: sha, + Markdown: string(payload), IsTemplateOnly: docRec.IsTemplateOnly, Source: "vault/degraded", + } + w.Header().Set("ETag", content.ContentSHA256) + writeJSONResponse(w, http.StatusOK, content) + return + } + } + } writeControlledContentError(w, r, err) return } diff --git a/internal/server/reply_export_api.go b/internal/server/reply_export_api.go index b7caa4aa..b127d2d4 100644 --- a/internal/server/reply_export_api.go +++ b/internal/server/reply_export_api.go @@ -6,6 +6,8 @@ import ( "strconv" "strings" + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/ctxmgr" "chat-agent-system/internal/replyexport" ) @@ -148,7 +150,21 @@ func (s *Server) handleGetExportSource(w http.ResponseWriter, r *http.Request) { } } - title := replyexport.DeriveTitle(msg.Content, msg.CreatedAt) + markdown := msg.Content + source := "ledger" + if strings.TrimSpace(markdown) == "" && s.cfg != nil && s.cfg.FAVDegradedRead { + env := s.cfg.Env + if env == "" { + env = "dev" + } + if payload, sha, verr := s.tryVaultDegraded(r.Context(), env, fmt.Sprintf("msg:%d", msg.ID), artifactvault.ProjectKeyFromID(messageProjectIDFromMeta(msg.Metadata))); verr == nil && len(payload) > 0 { + markdown = string(payload) + cap.ContentSha256 = sha + source = "vault/degraded" + } + } + + title := replyexport.DeriveTitle(markdown, msg.CreatedAt) snapshotID := fmt.Sprintf("msg:%d:rev:%d", msg.ID, cap.ContentRevision) w.Header().Set("Cache-Control", "private, no-store") @@ -158,12 +174,103 @@ func (s *Server) handleGetExportSource(w http.ResponseWriter, r *http.Request) { "content_revision": cap.ContentRevision, "content_sha256": cap.ContentSha256, "title": title, - "markdown": msg.Content, + "markdown": markdown, "capability": cap, + "source": source, + }) +} + +// handlePinExport archives a completed copy_only assistant reply after an explicit pin. +// POST /api/v1/chat/messages/{message_id}/pin-export +func (s *Server) handlePinExport(w http.ResponseWriter, r *http.Request) { + user, ok := s.requireAuthenticatedUser(w, r) + if !ok { + return + } + if s.ctxMgr == nil || s.vaultIngest == nil { + http.Error(w, "artifact vault not initialized", http.StatusServiceUnavailable) + return + } + msgID := extractMessageIDFromPath(r, "pin-export") + if msgID <= 0 { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "INVALID_MESSAGE_ID", + "message": "无效的消息ID", + }) + return + } + msg, err := s.ctxMgr.GetMessageByID(r.Context(), msgID) + if err != nil || msg == nil { + writeJSONResponse(w, http.StatusNotFound, map[string]string{ + "error_code": "MESSAGE_NOT_FOUND", + "message": "指定消息不存在或已被删除", + }) + return + } + if !s.assertMessageOwnedByUser(w, r, msg, user.ID) { + return + } + if msg.Role != "assistant" { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "MESSAGE_NOT_PINNABLE", + "message": "仅助手回复可显式 pin", + }) + return + } + turn, err := s.ctxMgr.GetChatTurnByAssistantMessageID(r.Context(), msg.SessionID, msg.ID) + if err != nil || turn == nil || turn.Status != ctxmgr.TurnStatusCompleted || turn.CompletedAt == nil { + writeJSONResponse(w, http.StatusConflict, map[string]string{ + "error_code": "TURN_NOT_COMPLETED", + "message": "未完成回合不得 pin 进保险库", + }) + return + } + cap, okCap := msg.ExportCapability.(replyexport.MessageExportCapability) + if !okCap { + _, cap = replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: msg.Role, Status: "completed", Content: msg.Content, Revision: msg.ContentRevision, + }) + } + if cap.Policy != replyexport.PolicyCopyOnly { + writeJSONResponse(w, http.StatusBadRequest, map[string]string{ + "error_code": "PIN_POLICY_REJECTED", + "message": "仅 copy_only 需要显式 pin；full 已自动归档，none 永不进入", + }) + return + } + eventID := fmt.Sprintf("pin:%d:%d", msg.ID, cap.ContentRevision) + err = s.vaultIngest.AssistantReplyPin( + r.Context(), turn.ProjectID, fmt.Sprintf("msg:%d", msg.ID), eventID, + msg.ID, "", string(cap.Policy), turn.CompletedAt, []byte(msg.Content), + ) + if err != nil { + writeJSONResponse(w, http.StatusConflict, map[string]string{ + "error_code": "PIN_REJECTED", + "message": "保险库拒绝该 pin", + }) + return + } + writeJSONResponse(w, http.StatusOK, map[string]any{ + "message_id": msg.ID, + "policy": cap.Policy, + "status": "requested", }) } // extractMessageIDFromPath 从路由参数或 URL 路径中提取 message_id +func messageProjectIDFromMeta(meta map[string]any) int64 { + if meta == nil { + return 0 + } + if pVal, ok := meta["project_id"].(int64); ok { + return pVal + } + if pVal, ok := meta["project_id"].(float64); ok { + return int64(pVal) + } + return 0 +} + func extractMessageIDFromPath(r *http.Request, actionName string) int64 { // 优先尝试 Go 1.22 PathValue val := r.PathValue("message_id") diff --git a/internal/server/reply_export_api_test.go b/internal/server/reply_export_api_test.go index 76fde094..6a8b7e81 100644 --- a/internal/server/reply_export_api_test.go +++ b/internal/server/reply_export_api_test.go @@ -178,3 +178,46 @@ func TestServer_ReplyExportAPI(t *testing.T) { t.Fatalf("expected 403 for cross-user export, got %d: %s", recIDOR.Code, recIDOR.Body.String()) } } + +func TestServer_PinExportCopyOnlyAfterCompleted(t *testing.T) { + cfg := &config.Config{ + Env: "dev", + HTTPHost: "127.0.0.1", + HTTPPort: 8894, + DataDir: t.TempDir(), + } + svc := bizdev.MustNewService(t, cfg) + srv := server.NewServer(cfg, svc) + handler := srv.Routes() + ctx := t.Context() + cm := bizdev.Snapshot(svc).Ctx + sessID := "sess_pin_export" + token := loginTestAdmin(t, handler, cfg) + if _, err := cm.EnsureChatSession(ctx, sessID, 1, 7, "pin 测试"); err != nil { + t.Fatal(err) + } + if _, err := cm.AcquireChatTurn(ctx, sessID, "turn-pin", "hash-pin", "req-pin", 1, 7); err != nil { + t.Fatal(err) + } + msgID, err := cm.RecordUserMessageWithID(ctx, sessID, 1, "assistant", "好的", "", "", nil) + if err != nil { + t.Fatal(err) + } + reqEarly := httptest.NewRequest(http.MethodPost, fmt.Sprintf("/api/v1/chat/messages/%d/pin-export", msgID), nil) + withBearerAuth(reqEarly, token) + recEarly := httptest.NewRecorder() + handler.ServeHTTP(recEarly, reqEarly) + if recEarly.Code != http.StatusConflict { + t.Fatalf("pin before completed want 409, got %d: %s", recEarly.Code, recEarly.Body.String()) + } + if err := cm.CompleteChatTurn(ctx, sessID, "turn-pin", msgID); err != nil { + t.Fatal(err) + } + req := httptest.NewRequest(http.MethodPost, fmt.Sprintf("/api/v1/chat/messages/%d/pin-export", msgID), nil) + withBearerAuth(req, token) + rec := httptest.NewRecorder() + handler.ServeHTTP(rec, req) + if rec.Code != http.StatusOK { + t.Fatalf("pin after completed want 200, got %d: %s", rec.Code, rec.Body.String()) + } +} diff --git a/internal/server/server.go b/internal/server/server.go index 9461e6da..4259e2a3 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -77,6 +77,7 @@ type Server struct { vaultReg artifactvault.Registry vaultStore *artifactvault.FilesystemStore vaultPub *artifactvault.Publisher + vaultIngest *artifactvault.Ingest vaultPublisherStarted int32 vaultStop chan struct{} reminderStop chan struct{} @@ -130,7 +131,67 @@ func NewFromRuntime(rt *app.Runtime) *Server { } s.initTaskRunExecutors() s.initArtifactVault() + s.wireFinalArtifactHooks() s.startVaultPublisher() + if s.rev != nil && s.notifySvc != nil { + rev := s.rev + rev.OnOutbox = func(ctx context.Context, eventType string, payload map[string]any) error { + if !s.notifySvc.Available() { + return nil + } + docID, _ := payload["doc_id"].(string) + projectID, _ := payload["project_id"].(int64) + risk, _ := payload["risk"].(string) + var recipient int64 + if s.gov != nil { + if doc, err := s.gov.GetDocument(ctx, docID); err == nil && doc != nil && doc.AuthorID > 0 { + recipient = doc.AuthorID + projectID = doc.ProjectID + } + } + if recipient == 0 { + return nil + } + p := notification.DocumentEventPayload{ + ProjectID: projectID, DocID: docID, RiskLevel: risk, + } + switch eventType { + case notification.EventDocAgentCandidateReady: + if v, ok := payload["candidate_id"].(int64); ok { + p.CandidateID = v + } + _, err := s.notifySvc.EmitDocumentEvent(ctx, eventType, notification.TypeDocAgentCandidateReady, + "智能体变更待确认", "文档有高风险智能体候选变更，请确认后生效。", + fmt.Sprintf("doc_candidate:%s:%d:%d", docID, p.CandidateID, recipient), recipient, p) + return err + case notification.EventDocAgentVersionCommitted: + if v, ok := payload["version_no"].(int); ok { + p.VersionNo = v + } + _, err := s.notifySvc.EmitDocumentEvent(ctx, eventType, notification.TypeDocAgentVersionCommitted, + "智能体版本已生效", "新的智能体文档版本已提交。", + fmt.Sprintf("doc_agent_version:%s:%d:%d", docID, p.VersionNo, recipient), recipient, p) + return err + case notification.EventDocDownstreamImpact: + impacted, _ := payload["impacted_doc_id"].(string) + p.ImpactedDocID = impacted + _, err := s.notifySvc.EmitDocumentEvent(ctx, eventType, notification.TypeDocDownstreamImpact, + "下游文档受影响", "上游文档变更，请复核关联文档。", + fmt.Sprintf("doc_impact:%s:%s:%d", docID, impacted, recipient), recipient, p) + return err + case notification.EventDocChangeReportDegraded: + if v, ok := payload["report_id"].(int64); ok { + p.ReportID = v + } + _, err := s.notifySvc.EmitDocumentEvent(ctx, eventType, notification.TypeDocChangeReportDegraded, + "变更报告未完整评估", "变更报告降级，受控文档暂不可送审。", + fmt.Sprintf("doc_report_degraded:%s:%d:%d", docID, p.ReportID, recipient), recipient, p) + return err + default: + return nil + } + } + } return s } @@ -278,6 +339,7 @@ func (s *Server) Routes() http.Handler { // 7.1 单次助手回复导出与快照 mux.HandleFunc("GET /api/v1/chat/messages/{message_id}/export-capability", s.handleGetExportCapability) mux.HandleFunc("GET /api/v1/chat/messages/{message_id}/export-source", s.handleGetExportSource) + mux.HandleFunc("POST /api/v1/chat/messages/{message_id}/pin-export", s.handlePinExport) // 7.2 自由私密文档与项目共享参考 mux.HandleFunc("POST /api/v1/freeform-documents", s.handleCreateFreeformDoc) @@ -594,6 +656,13 @@ func (s *Server) getOrCreateSession( IntentPlans: app.AdaptChatIntentPlans(s.intentPlans), Environment: env, } + if s.vaultIngest != nil { + ingest := s.vaultIngest + sess.OnFinalArtifact = func(ctx context.Context, projectID int64, logical, eventID string, msgID int64, policy string, payload []byte) { + now := time.Now().UTC() + _ = ingest.AssistantReply(ctx, projectID, logical, eventID, msgID, "", policy, &now, payload) + } + } // 若从持久层恢复了会话历史，回填消息列表、事实黑板与近期对话 (G3)。 // 正文注入走 ChatLog.Restore，避免把未授权消息写进 Prompt 窗口。 diff --git a/internal/server/task_finalize.go b/internal/server/task_finalize.go index ec346813..c06d2f16 100644 --- a/internal/server/task_finalize.go +++ b/internal/server/task_finalize.go @@ -32,7 +32,15 @@ func (s *Server) finalizeTaskRun(ctx context.Context, record *taskrun.RunRecord, cmds.PW = s.pwSvc cmds.Ctx = s.ctxMgr } - return cmds.Finalize(ctx, record, result) + status, err := cmds.Finalize(ctx, record, result) + if err != nil { + return status, err + } + if status == "completed" && record.AssistantMsgID > 0 { + summaryContent := result.CompletionSummary() + s.ingestAssistantReply(ctx, record, summaryContent, status) + } + return status, nil } func (s *Server) taskCommands() *taskrun.Commands { diff --git a/internal/server/task_finalize_fav_test.go b/internal/server/task_finalize_fav_test.go new file mode 100644 index 00000000..56e5c515 --- /dev/null +++ b/internal/server/task_finalize_fav_test.go @@ -0,0 +1,31 @@ +package server + +import ( + "context" + "testing" + + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/config" + "chat-agent-system/internal/taskrun" +) + +func TestIngestAssistantReplyRegistersArtifact(t *testing.T) { + cfg := &config.Config{Env: "dev", AllowInMemory: true, DataDir: t.TempDir()} + reg := artifactvault.NewMemoryRegistry() + s := &Server{cfg: cfg, vaultReg: reg, vaultIngest: &artifactvault.Ingest{Env: "dev", Reg: reg}} + ctx := context.Background() + record := &taskrun.RunRecord{ + RunID: "run-fav", SessionID: "sess-fav", ClientTurnID: "turn-fav", ProjectID: 3, AssistantMsgID: 42, + } + s.ingestAssistantReply(ctx, record, "# 任务终态\n\n结构化说明足够长。", "completed") + items, err := reg.LoadRequested(ctx) + if err != nil { + t.Fatal(err) + } + if len(items) == 0 { + t.Fatal("expected assistant reply to be requested in FAV registry") + } + if items[0].ArtifactType != artifactvault.TypeAssistantReply { + t.Fatalf("type=%s", items[0].ArtifactType) + } +} diff --git a/internal/server/task_run_handlers.go b/internal/server/task_run_handlers.go index cf4d90b1..7c1e381c 100644 --- a/internal/server/task_run_handlers.go +++ b/internal/server/task_run_handlers.go @@ -31,10 +31,10 @@ type createTaskRunRequest struct { } type createTaskRunResponse struct { - RunID string `json:"run_id"` - WorkspaceID string `json:"workspace_id,omitempty"` - Status string `json:"status"` - AssistantMsgID int64 `json:"assistant_msg_id,omitempty"` + RunID string `json:"run_id"` + WorkspaceID string `json:"workspace_id,omitempty"` + Status string `json:"status"` + AssistantMsgID int64 `json:"assistant_msg_id,omitempty"` } type taskRunResponse struct { diff --git a/internal/server/trace_handlers.go b/internal/server/trace_handlers.go index 67d9f7a6..8ac6b00f 100644 --- a/internal/server/trace_handlers.go +++ b/internal/server/trace_handlers.go @@ -39,11 +39,11 @@ func (s *Server) handleProjectTrace(w http.ResponseWriter, r *http.Request) { return } writeJSONResponse(w, http.StatusOK, map[string]any{ - "items": visible, - "hidden_count": hidden, - "index": index, - "recheck_marks": marks, - "candidate_edges": candidates, + "items": visible, + "hidden_count": hidden, + "index": index, + "recheck_marks": marks, + "candidate_edges": candidates, }) } diff --git a/internal/server/user_profile_directory_test.go b/internal/server/user_profile_directory_test.go index c13803f7..2792a783 100644 --- a/internal/server/user_profile_directory_test.go +++ b/internal/server/user_profile_directory_test.go @@ -87,9 +87,9 @@ func TestMeProfile_PatchDepartmentAndClearAdminHint(t *testing.T) { // Admin edits target profile → hint set patchBody, _ := json.Marshal(map[string]string{ - "name": "张工-管理员改名", - "department": "骨科耗材研发组", - "email": "zhang@example.com", + "name": "张工-管理员改名", + "department": "骨科耗材研发组", + "email": "zhang@example.com", "employee_id": "RD-001", }) adminReq, _ := http.NewRequest( diff --git a/internal/server/vault.go b/internal/server/vault.go index b4f2254f..88cb0b68 100644 --- a/internal/server/vault.go +++ b/internal/server/vault.go @@ -3,28 +3,37 @@ package server import ( "context" "fmt" - "log/slog" "net/http" "sync/atomic" "time" "chat-agent-system/internal/artifactvault" "chat-agent-system/internal/config" -) + "chat-agent-system/internal/docgov" + "chat-agent-system/internal/replyexport" + "chat-agent-system/internal/taskrun" -// Vault HTTP + in-process publisher stubs for U7 incremental. -// Default: FAV_PUBLISHER_ENABLED=false, FAV_DEGRADED_READ=false. -// Full ingest hooks / enumerate backfill remain on feature/final-artifact-vault. + "github.com/jackc/pgx/v5" +) func (s *Server) initArtifactVault() { if s == nil || s.cfg == nil { return } - // Publisher store stays nil unless explicitly enabled AND vault roots are set. - if !s.cfg.FAVPublisherEnabled { + env := s.cfg.Env + if env == "" { + env = "dev" + } + if s.rt != nil { + s.vaultReg = s.rt.ArtifactVaultRegistry() + } else { + s.vaultReg = artifactvault.NewMemoryRegistry() + } + s.vaultIngest = &artifactvault.Ingest{Env: env, Reg: s.vaultReg} + if s.cfg.VaultStateRoot == "" || s.cfg.VaultMasterKey == "" { return } - if s.cfg.VaultStateRoot == "" || s.cfg.VaultMasterKey == "" { + if !s.cfg.FAVPublisherEnabled { return } if err := config.ValidateIndependentVault(s.cfg.VaultStateRoot, s.cfg.DataDir); err != nil { @@ -32,25 +41,14 @@ func (s *Server) initArtifactVault() { } ident := s.cfg.VaultStorageIdentity if ident == "" { - env := s.cfg.Env - if env == "" { - env = "dev" - } ident = "vault-" + env } store, err := artifactvault.NewFilesystemStore(s.cfg.VaultStateRoot, ident, []byte(s.cfg.VaultMasterKey)) if err != nil { return } - reg := artifactvault.Registry(nil) - if s.rt != nil { - reg = s.rt.ArtifactVaultRegistry() - } else { - reg = artifactvault.NewMemoryRegistry() - } s.vaultStore = store - s.vaultReg = reg - s.vaultPub = &artifactvault.Publisher{Reg: reg, Store: store} + s.vaultPub = &artifactvault.Publisher{Reg: s.vaultReg, Store: store} } func (s *Server) startVaultPublisher() { @@ -66,15 +64,87 @@ func (s *Server) startVaultPublisher() { case <-s.vaultStop: return case <-t.C: - if err := s.vaultPub.Drain(context.Background()); err != nil { - slog.Warn("vault_drain_failed", "error", err) - } + _ = s.vaultPub.Drain(context.Background()) } } }() atomic.StoreInt32(&s.vaultPublisherStarted, 1) } +func (s *Server) vaultReady() (identityOK, connected, lagOK bool) { + if s == nil || s.vaultStore == nil { + return false, false, true + } + idOK, conn, _ := s.vaultStore.Ready() + lagOK = true + if s.vaultReg != nil { + pending, err := s.vaultReg.LoadRequested(context.Background()) + lagOK = err == nil && len(pending) < 100 + } + return idOK, conn, lagOK +} + +func (s *Server) ingestAssistantReply(ctx context.Context, record *taskrun.RunRecord, content, terminalStatus string) { + if s == nil || s.vaultIngest == nil || record == nil || terminalStatus != "completed" { + return + } + _, cap := replyexport.EvaluatePolicy(replyexport.PolicyOptions{ + Role: "assistant", Status: "completed", Content: content, + }) + now := time.Now().UTC() + logical := fmt.Sprintf("msg:%d", record.AssistantMsgID) + eventID := record.SessionID + ":" + record.ClientTurnID + if eventID == ":" { + eventID = record.RunID + } + _ = s.vaultIngest.AssistantReply(ctx, record.ProjectID, logical, eventID, record.AssistantMsgID, record.RunID, string(cap.Policy), &now, []byte(content)) +} + +func (s *Server) wireFinalArtifactHooks() { + if s == nil || s.vaultIngest == nil { + return + } + if gov := s.gov; gov != nil { + gov.OnFinalArtifact = func(ctx context.Context, doc *docgov.DocumentRecord, markdown []byte, snapshotID int64) { + if doc == nil { + return + } + _ = s.vaultIngest.Approved(ctx, doc.ProjectID, doc.DocID, snapshotID, doc.ApprovedAt, markdown) + } + gov.OnFinalArtifactTX = func(ctx context.Context, tx pgx.Tx, doc *docgov.DocumentRecord, markdown []byte, snapshotID int64) error { + if doc == nil { + return nil + } + return s.vaultIngest.ApprovedTX(ctx, tx, doc.ProjectID, doc.DocID, snapshotID, doc.ApprovedAt, markdown) + } + } + if rev := s.rev; rev != nil { + rev.OnFinalArtifact = func(ctx context.Context, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) { + switch kind { + case "agent_commit": + _ = s.vaultIngest.AgentCommit(ctx, projectID, docID, eventID, snapshotID, payload) + case "version_sealed": + _ = s.vaultIngest.SealedVersion(ctx, projectID, docID, eventID, snapshotID, sealedAt, payload) + } + } + rev.OnFinalArtifactTX = func(ctx context.Context, tx pgx.Tx, kind, docID string, projectID, snapshotID int64, eventID string, payload []byte, sealedAt *time.Time) error { + switch kind { + case "agent_commit": + return s.vaultIngest.AgentCommitTX(ctx, tx, projectID, docID, eventID, snapshotID, payload) + case "version_sealed": + return s.vaultIngest.SealedVersionTX(ctx, tx, projectID, docID, eventID, snapshotID, sealedAt, payload) + default: + return nil + } + } + } + if fd := s.freeDoc; fd != nil { + fd.OnFinalArtifact = func(ctx context.Context, projectID int64, docID, eventID string, snapshotID int64, payload []byte) { + _ = s.vaultIngest.FreeformPublish(ctx, projectID, docID, eventID, snapshotID, payload) + } + } +} + func (s *Server) tryVaultDegraded(ctx context.Context, env, logical, projectKey string) ([]byte, string, error) { if s == nil || s.vaultPub == nil || s.cfg == nil || !s.cfg.FAVDegradedRead { return nil, "", fmt.Errorf("artifactvault: degraded read disabled") @@ -108,10 +178,5 @@ func (s *Server) handleVaultStatus(w http.ResponseWriter, r *http.Request) { "would_stage": false, "would_publish": false, "live_not_opened": true, - "note": "fav consumer remains OFF until separate approval; shadow computes intended outcome only; full FAV branch not merged", }) } - -func (s *Server) vaultPublisherRunning() bool { - return atomic.LoadInt32(&s.vaultPublisherStarted) == 1 -} diff --git a/internal/server/vault_nowrite_test.go b/internal/server/vault_nowrite_test.go new file mode 100644 index 00000000..1a91b496 --- /dev/null +++ b/internal/server/vault_nowrite_test.go @@ -0,0 +1,60 @@ +package server + +import ( + "context" + "fmt" + "testing" + + "chat-agent-system/internal/artifactvault" + "chat-agent-system/internal/config" +) + +type writeFailRegistry struct { + *artifactvault.MemoryRegistry + requests int +} + +func (w *writeFailRegistry) Request(ctx context.Context, facts artifactvault.Facts, req artifactvault.PublishRequest) (*artifactvault.Artifact, *artifactvault.Staging, error) { + w.requests++ + return nil, nil, fmt.Errorf("GET must not write") +} + +func TestTryVaultDegradedDoesNotRequest(t *testing.T) { + ctx := context.Background() + mem := artifactvault.NewMemoryRegistry() + spy := &writeFailRegistry{MemoryRegistry: mem} + dir := t.TempDir() + store, err := artifactvault.NewFilesystemStore(dir, "vault-id-test", []byte("0123456789abcdef0123456789abcdef")) + if err != nil { + t.Fatal(err) + } + pub := &artifactvault.Publisher{Reg: mem, Store: store} + facts := artifactvault.Facts{ + ArtifactType: artifactvault.TypeAgentCommit, CandidateStatus: "committed", OutputSnapshotID: 7, + ChangeKind: "agent_commit", ChangeToSnapshotID: 7, Classification: artifactvault.ClassificationOK, + Payload: []byte("# ledger\n"), + } + req := artifactvault.PublishRequest{ + ArtifactType: artifactvault.TypeAgentCommit, Environment: "dev", ProjectExternalKey: "3", + LogicalResourceID: "doc_nowrite", SourceEventKind: "agent_commit", SourceEventID: "ev-nowrite", + Payload: []byte("# ledger\n"), TerminalState: "committed", + } + if _, _, err := mem.Request(ctx, facts, req); err != nil { + t.Fatal(err) + } + if err := pub.Drain(ctx); err != nil { + t.Fatal(err) + } + s := &Server{ + cfg: &config.Config{Env: "dev", FAVDegradedRead: true, DataDir: t.TempDir()}, + vaultReg: spy, + vaultPub: &artifactvault.Publisher{Reg: spy, Store: store}, + } + b, _, err := s.tryVaultDegraded(ctx, "dev", "doc_nowrite", "3") + if err != nil || string(b) != "# ledger\n" { + t.Fatalf("degraded: %v %q", err, b) + } + if spy.requests != 0 { + t.Fatalf("GET path issued %d Request writes", spy.requests) + } +} diff --git a/internal/server/vault_test.go b/internal/server/vault_test.go index de02ed4d..d17b7218 100644 --- a/internal/server/vault_test.go +++ b/internal/server/vault_test.go @@ -5,6 +5,7 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "sync/atomic" "testing" "chat-agent-system/internal/config" @@ -28,7 +29,7 @@ func TestVaultDefaultsStayDisabled(t *testing.T) { if s.vaultPub != nil { t.Fatal("vault publisher must stay nil when flag off") } - if s.vaultPublisherRunning() { + if atomic.LoadInt32(&s.vaultPublisherStarted) == 1 { t.Fatal("vault publisher must not start when flag off") } if _, _, err := s.tryVaultDegraded(context.Background(), "dev", "doc", "1"); err == nil { diff --git a/scripts/artifact-vault-preflight.sh b/scripts/artifact-vault-preflight.sh new file mode 100755 index 00000000..3926ec6d --- /dev/null +++ b/scripts/artifact-vault-preflight.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Fail-closed FAV coverage / destroy preflight. Does not start env-service. +set -euo pipefail +DSN="${APP_DB_DSN:-${TEST_DATABASE_URL:-}}" +if [[ -z "$DSN" ]]; then + echo "APP_DB_DSN or TEST_DATABASE_URL required" >&2 + exit 2 +fi +exec go run ./cmd/artifact-vault gc-preflight --dsn "$DSN" --backup-ok="${BACKUP_OK:-false}" --corpus-ok="${CORPUS_OK:-false}"