Merge origin/main into proposal branch; resolve svc-deploy docs pin
Why: Dev #422 add/add on docs/svc-deploy after U1/U4 landed; documents gitlink also diverged from main. Evidence: took main svc-deploy specs; Documents pin 9d9b152 = proposal + main merge; Backend pin 1d9e8f4 Impact: PR #422 should become mergeable; Documents #79 tip advanced. Refs: Dev pulls/422; Documents pulls/79; merged #428/#429 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Cursor
@@ -373,3 +373,7 @@ jobs:
|
||||
AGENT_PG_TEST_DSN: postgres://ci_agent:ci_job_local_only@127.0.0.1:5432/agent_test?sslmode=disable
|
||||
AGENT_REDIS_TEST_ADDR: 127.0.0.1:6399
|
||||
run: make test-backend-pg
|
||||
- name: Two-process HTTP acceptance over shared PostgreSQL (U7, required)
|
||||
env:
|
||||
AGENT_PG_TEST_DSN: postgres://ci_agent:ci_job_local_only@127.0.0.1:5432/agent_test?sslmode=disable
|
||||
run: make test-multi-instance
|
||||
|
||||
@@ -6,6 +6,7 @@ __pycache__/
|
||||
/data-test/
|
||||
.zcode/plans/
|
||||
runtime/
|
||||
runtime-backups/
|
||||
.svc.env
|
||||
|
||||
# 运行时环境文件（含密钥，不入库；svc-envfile 任务约定）
|
||||
|
||||
+8
-2
@@ -1,6 +1,6 @@
|
||||
# feature/adminzone-plugin-sections 任务卡
|
||||
|
||||
任务主记录：<PR 链接，交付前填写>；状态：进行中；owner：weichen
|
||||
任务主记录：PR #414 https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/414（merge 28af351fb3）；状态：已完成（回填）；owner：weichen
|
||||
|
||||
## 基线与授权
|
||||
|
||||
@@ -74,7 +74,7 @@
|
||||
| 时间(UTC) | 命令 | 被测HEAD | exit | 来源 | sha256(16) |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## 结果与验证（完成后填写）
|
||||
## 结果与验证
|
||||
|
||||
- 结果：
|
||||
- 验证：<结论摘要 PASS/FAIL/SKIP/NOT_RUN 逐项；逐条证据由 make evidence 写入上方证据表>
|
||||
@@ -287,3 +287,9 @@
|
||||
"receipt_sha256": "09f6e32908df5414897575986f6cafb258dbb5b466d0bc5a384615170c89a03f"
|
||||
}
|
||||
```
|
||||
|
||||
## 闭合记录（card-autoclose 回填）
|
||||
|
||||
- 依据：merge 28af351fb32185f38c0228c4c2040e854c6ecd50（PR #414）已包含于 origin/main，分支成果已落盘；本段为合入后回填。
|
||||
- 合并主题：Merge pull request 'Host plugin-declared admin-zone sections (WP3-I3/KTD4)' (#414) from feature/adminzone-plugin-sections into main
|
||||
- 归档：随闭合移入 .task/archive/；任务执行期的过程证据见原任务记录与 PR。
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,6 +1,6 @@
|
||||
# feature/genui-spec-repair 任务卡
|
||||
|
||||
任务主记录：<PR 链接，交付前填写>；状态：进行中；owner：weichen
|
||||
任务主记录：PR #416 https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/416（merge 37e3229538）；状态：已完成（回填）；owner：weichen
|
||||
|
||||
## 基线与授权
|
||||
|
||||
@@ -68,7 +68,7 @@
|
||||
| 时间(UTC) | 命令 | 被测HEAD | exit | 来源 | sha256(16) |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## 结果与验证（完成后填写）
|
||||
## 结果与验证
|
||||
|
||||
- 结果：
|
||||
- 验证：<结论摘要 PASS/FAIL/SKIP/NOT_RUN 逐项；逐条证据由 make evidence 写入上方证据表>
|
||||
@@ -203,3 +203,9 @@
|
||||
"receipt_sha256": "8f966ac79cb51f554da022215d6f1d968240b66db8754c76247d2e3a02d6d8b7"
|
||||
}
|
||||
```
|
||||
|
||||
## 闭合记录（card-autoclose 回填）
|
||||
|
||||
- 依据：merge 37e3229538d64867a6a074020a072da925244fcc（PR #416）已包含于 origin/main，分支成果已落盘；本段为合入后回填。
|
||||
- 合并主题：Merge pull request 'Bump backend pin: genui spec repair and subagent present (backend #166)' (#416) from feature/genui-spec-repair into main
|
||||
- 归档：随闭合移入 .task/archive/；任务执行期的过程证据见原任务记录与 PR。
|
||||
+27
-15
@@ -1,6 +1,6 @@
|
||||
# feature/recovery-u1-instance-identity 任务卡
|
||||
|
||||
任务主记录：方案 r2 §7 U1（modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md，已批准，随 Documents #75 合入）；设计 v0.1：docs/recovery/u1-identity-manifest.md + docs/recovery/u1-data-ownership.md；主记录 PR：Dev #415（CI 为验收 GATE）；owner：weichen
|
||||
任务主记录：PR #415 https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/415（merge af9c2de663）；设计 v0.1：docs/recovery/u1-identity-manifest.md + docs/recovery/u1-data-ownership.md；主记录 PR：Dev #415（CI 为验收 GATE）；owner：weichen
|
||||
|
||||
## 基线与授权
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
- 授权与交付终点：用户批准方案 r2（2026-10-12）并指示启动 U1；设计经“按建议办”批准后实施 I2。2026-10-12 追加授权：推送本任务分支并创建 PR（CI 为正式验收 GATE，绿前不合入；合入由 owner 执行）。
|
||||
- 交付模式：manual（逐阶段人工评审）；明确授权后可记为 authorized-continuous
|
||||
- 自动推进授权：无；授权范围：无；授权来源：无；授权终点：无；停止条件：无（未授权）
|
||||
- 当前阶段：实施与自检（I2 代码完成：manifest 层+启动校验+init+T1–T6 全绿，待正式验收与代码 Review）；设计已批准（2026-10-12“按建议办”）
|
||||
- 当前阶段：已完成（I1+I2 合入 main；GATE=PR #415 CI 双绿于 HEAD 6d453ee；剩余实施增量 I3 另开任务 feature/recovery-u1-db-identity）
|
||||
- 流程依据：仓库根目录 policy/delivery.md（归档后仍按仓库根目录定位）
|
||||
|
||||
## 目标与非目标
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
## 关联方向与规划
|
||||
|
||||
- 适用状态：服务重启与全量数据恢复方向（方案 r2 已批准，2026-10-12）
|
||||
- 适用状态：已完成（回填）
|
||||
- 主记录路径/版本：modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2；U2 首切片已由 svc-autorecover 合入 main（PR #396/#401）
|
||||
- 对应里程碑：r2 U1（批次A）；R2/R3/R9
|
||||
|
||||
@@ -47,10 +47,10 @@
|
||||
| 方案设计 | docs/recovery/u1-identity-manifest.md v0.1 | 已批准 | weichen | 2026-10-12 | 用户“按建议办”：四未决项按设计建议定案（instance_id=svc-<group>-<随机12>创建即冻结；dsn_ref 备份影响记 U6；与 .svc/<name>.json 仅引用不合并；deps required/optional 列表表达） |
|
||||
| 实施计划 | 待定 | 未评审 | — | — | — |
|
||||
| 实施与自检 | I2 @ c170939（同步后 ce78d59） | 已完成（SELF） | — | 2026-10-12 | 身份套件 21/21 + 治理套件 169/169 全绿（收据入卡）；真实 PG 专项 NOT_RUN（缺环境，按设计 §7 显式标注） |
|
||||
| 正式测试与验收 | 同步后 HEAD 重跑 | 本地 SELF 通过；GATE 待 PR CI | 待 weichen 复核 | — | 全量 make test+make build 以 PR HEAD CI 为权威（AGENTS 验证合同） |
|
||||
| 正式测试与验收 | PR #415 HEAD 6d453ee | GATE 通过（ci/test + ci/test-pg success） | weichen | 2026-10-12 | PR CI 为权威；本地全量 make test SELF exit 0 辅助 |
|
||||
| 代码 Review | 待定 | 未评审 | — | — | — |
|
||||
| 合入 | 待定 | 未评审 | — | — | — |
|
||||
| 合入核验与收口 | 待定 | 未评审 | — | — | — |
|
||||
| 合入 | merge af9c2de（含 6d453ee） | 已合入 | weichen | 2026-10-12 | PR #415 |
|
||||
| 合入核验与收口 | worktree recycled | 已收口（task-finish） | weichen | 2026-10-12 | TASK_WRITERS_STOPPED=1 make task-finish；远端分支已删 |
|
||||
|
||||
## 探索试验与迭代
|
||||
|
||||
@@ -71,11 +71,11 @@
|
||||
| 时间(UTC) | 命令 | 被测HEAD | exit | 来源 | sha256(16) |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## 结果与验证（完成后填写）
|
||||
## 结果与验证
|
||||
|
||||
- 结果：
|
||||
- 验证：<结论摘要 PASS/FAIL/SKIP/NOT_RUN 逐项；逐条证据由 make evidence 写入上方证据表>
|
||||
- 影响与剩余项：
|
||||
- 结果：I1 设计 + I2 manifest/数据 marker/init/T1–T6 已合入 main（PR #415）。
|
||||
- 验证：GATE PASS（PR #415 CI：ci/test=success、ci/test-pg=success @ HEAD 6d453ee）；真实 PG 身份表专项 NOT_RUN（属 I3）。
|
||||
- 影响与剩余项：**I3 未做**——真实库表 `agent_system_identity` + 真实 PG 用例；另开任务 `feature/recovery-u1-db-identity` 推进。
|
||||
|
||||
## 规划进展与下一步
|
||||
|
||||
@@ -89,14 +89,14 @@
|
||||
|
||||
## 规划进展与下一步
|
||||
|
||||
- 本次推进/剩余：I1+I2 完成（设计批准；svc_manifest.py 权威层、svc.sh 迁移/校验/init 接线、T1–T6 全绿）；剩余：正式验收（含真实 PG 专项，缺环境标 NOT_RUN）→Review→推送/PR 待授权
|
||||
- 首选下一任务：本任务 I2；并行可启动 U3 生命周期设计（独立任务）
|
||||
- 前置条件/待确认：设计 v0.1 人工评审（含未决四项：instance_id 生成规则、dsn_ref 备份影响、与 .svc/<name>.json 边界、optional 依赖表达）
|
||||
- 本次推进/剩余：I1+I2 已合入；**剩余 I3**（`agent_system_identity` + 真实 PG 用例）转新任务。
|
||||
- 首选下一任务：`feature/recovery-u1-db-identity`（I3）；U3 生命周期设计可并行。
|
||||
- 前置条件/待确认：无（设计 §5.2 已定案；用户已授权「继续推进」含 I3）。
|
||||
|
||||
## 合入与清理
|
||||
|
||||
- 合入记录：NOT_RUN（授权依据、PR、合入 SHA、验证来源）
|
||||
- 清理记录：NOT_RUN（授权依据、写者退出确认、worktree/分支/任务卡处置）
|
||||
- 合入记录：用户合入 PR #415（merge af9c2de）；绿 HEAD `6d453ee`；GATE=PR CI（test + test-pg success）。
|
||||
- 清理记录：2026-10-12 授权继续收尾；`TASK_WRITERS_STOPPED=1 make task-finish` 回收 worktree 与本地/远端任务分支（observed_head=6d453ee）；任务卡随闭合 PR #421 归档；**I3 残留转新任务**。
|
||||
|
||||
<!-- Local receipt: SELF. Digests detect corruption; they do not authenticate a reviewer or CI. -->
|
||||
```json
|
||||
@@ -420,3 +420,15 @@
|
||||
"receipt_sha256": "44d1f990a1f41b14c2607be01aaf52cfb6717264f29f826cca53cf56e29e0cad"
|
||||
}
|
||||
```
|
||||
|
||||
## 闭合记录（card-autoclose 回填）
|
||||
|
||||
- 依据：merge af9c2de663be5ff102dcab7698464a645832f569（PR #415）已包含于 origin/main，分支成果已落盘；本段为合入后回填。
|
||||
- 合并主题：Merge pull request 'Implement U1 instance identity: manifest authority, startup verification and init' (#415) from feature/recovery-u1-instance-identity into main
|
||||
- 归档：随闭合移入 .task/archive/；任务执行期的过程证据见原任务记录与 PR。
|
||||
|
||||
### task-finish 收口补记（2026-10-12）
|
||||
|
||||
- GATE：PR #415 @ `6d453ee`，ci/test + ci/test-pg 双 success。
|
||||
- 回收：worktree `…-wt-feature-recovery-u1-instance-identity` 已移除；远端 `feature/recovery-u1-instance-identity` 已删。
|
||||
- 剩余项：I3（真实 PG/SQLite 表 `agent_system_identity`）→ 新任务 `feature/recovery-u1-db-identity`。
|
||||
+27
-8
@@ -1,6 +1,6 @@
|
||||
# feature/service-restart-recovery-plan 任务卡
|
||||
|
||||
任务主记录：modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2（已批准 2026-10-12）；主记录 PR：Dev #403（正文与审阅记录随 Documents #75 合入，顺序 #75→#403）；状态：待合入（CI 审核中）；owner：weichen
|
||||
任务主记录：PR #403 https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/403（merge a1ac948155）；主记录 PR：Dev #403（正文与审阅记录随 Documents #75 合入，顺序 #75→#403）；状态：已完成（回填）；owner：weichen
|
||||
|
||||
## 基线与授权
|
||||
|
||||
@@ -44,11 +44,11 @@
|
||||
| 需求表确认 | r2（documents 71cd9d1） | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 方案设计 | r2（documents 71cd9d1） | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 实施计划 | r2（documents 71cd9d1） | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 实施与自检 | 待定 | 未评审 | — | — | — |
|
||||
| 正式测试与验收 | 待定 | 未评审 | — | — | — |
|
||||
| 代码 Review | 待定 | 未评审 | — | — | — |
|
||||
| 合入 | 待定 | 未评审 | — | — | — |
|
||||
| 合入核验与收口 | 待定 | 未评审 | — | — | — |
|
||||
| 实施与自检 | 不适用 | 不适用（方案卡） | — | — | 本任务只交付方案；实施由 U1–U7 增量任务承接（落地见「规划进展」） |
|
||||
| 正式测试与验收 | 不适用 | 不适用（方案卡） | — | — | 同上；业务验证在各增量任务的验收中执行 |
|
||||
| 代码 Review | 不适用 | 不适用（方案卡） | — | — | 方案审阅见过程仓 records 两轮（六角色自审+外部源码复核） |
|
||||
| 合入 | documents #75 → Dev #403（merge a1ac948155） | 已合入 | weichen | 2026-10-12 | PR #403 已包含于 origin/main（自动核验） |
|
||||
| 合入核验与收口 | 本闭合分支（card-close 2026-10-12-020646Z） | 已批准 | weichen | 2026-10-12 | 用户 2026-10-12 指示「把遗留项按建议修复」覆盖本卡落地回填与闭合 |
|
||||
|
||||
## 探索试验与迭代
|
||||
|
||||
@@ -68,7 +68,7 @@
|
||||
| 时间(UTC) | 命令 | 被测HEAD | exit | 来源 | sha256(16) |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
## 结果与验证（完成后填写）
|
||||
## 结果与验证
|
||||
|
||||
- 结果：本地待审恢复方案r2及Dev/过程仓入口（documents/父仓均已本地提交固定，未推送）；详细审阅结果见过程仓records同名review记录两轮（六角色自审+外部源码复核）
|
||||
- 验证：SELF启动precheck、task-sync、source audit和文档差异/链接检查见工具原始收据；草稿收据含未提交非任务卡改动，被check-evidence标为unverified，不作正式验收依据；业务/真实PG身份/故障/模型/迁移/部署/运行验收均NOT_RUN
|
||||
@@ -76,8 +76,21 @@
|
||||
|
||||
## 规划进展与下一步
|
||||
|
||||
- 落地进展（2026-10-12 回填，依据 origin/main 实测）：
|
||||
- **U2 首切片已合入并实测**：svc-autorecover（Dev PR #396/#400/#401）——autostart 期望态 +
|
||||
`svc.sh recover` + systemd watchdog（`agent-system-svc-recover.timer` 每分钟），kill -9 实测
|
||||
≤60s 自动拉回；dev/genui/rdm1/rdm1web 四服务已登记 autostart=1。
|
||||
- **U1 已合入**：Dev PR #415（manifest 权威、启动校验与 init）。
|
||||
- **U2 残留「agent SQLite 自动备份缺位」已补实用切片**：Dev PR #417（2026-10-12）——
|
||||
`svc.sh <name> backup/backups`，SQLite 一致性导出（.backup+integrity_check，方案 §6 约束），
|
||||
快照根在 runtime/ 之外的 `runtime-backups/<name>/`；真机 dev/genui/rdm1 已完成首备
|
||||
（dev 快照 49 表 integrity ok）。restore/组合备份合同仍属 U6。
|
||||
|
||||
|
||||
- 本次推进/剩余：形成候选r1（六角色自审）与r2（外部源码复核吸收：U3停机预算来源变更、U1程序/cwd验收、U4状态映射表、第9节svc-autorecover归属二选一），补充协作存储R7恢复合同；全部业务实现与运行验证未执行
|
||||
- 首选下一任务：U1设计细化与U3生命周期设计；U2首切片已由svc-autorecover任务合入main（PR #396/#401，CI绿；残留：postgres真实recover未实测、agent SQLite自动备份缺位），U2余量（KTD1/KTD2 manifest合同对齐、服务组/单一监管者）另立增量
|
||||
- 首选下一任务：U1 已由 #415 落地（其收尾与验收见该任务卡）；下一步为 U3 生命周期设计与
|
||||
U2 余量（KTD1/KTD2 manifest 合同对齐、服务组/单一监管者、postgres 真实 recover 实测、
|
||||
按批准 RPO 的自动快照频率）——均另立增量，未授权不启动
|
||||
- 推荐依据：先保证DB/data身份不漂移、配置更新完整和真实ready，再接监管与自动任务续跑
|
||||
- 前置条件/待确认：r2已批准、svc-autorecover归属已裁定（2026-10-12选项a：作U2实施载体）；仍待固定：目标实例/服务组、数据归属、管理权、故障档和RPO/RTO（对应实施增量启动前）
|
||||
- 预期产物/验收条件：兼容manifest合同、数据归属清单、真实PG隔离恢复回归和生命周期设计；判据见主案U1/U3
|
||||
@@ -433,3 +446,9 @@
|
||||
"receipt_sha256": "923876ca5027520d486d24e92a6c77a92a102d015fbe54c896e56f4ee079d013"
|
||||
}
|
||||
```
|
||||
|
||||
## 闭合记录（card-autoclose 回填）
|
||||
|
||||
- 依据：merge a1ac948155eb8957a87f37d8da4a9feb870dd532（PR #403）已包含于 origin/main，分支成果已落盘；本段为合入后回填。
|
||||
- 合并主题：Merge pull request 'Deliver the approved service restart and recovery plan (documents pin + task card)' (#403) from feature/service-restart-recovery-plan into main
|
||||
- 归档：随闭合移入 .task/archive/；任务执行期的过程证据见原任务记录与 PR。
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -13,7 +13,7 @@ PLUGINS := plugins
|
||||
COMBO := core=$(shell git -C $(CORE) rev-parse --short HEAD 2>/dev/null)+backend=$(shell git -C $(BACKEND) rev-parse --short HEAD 2>/dev/null)+frontend=$(shell git -C $(FRONTEND) rev-parse --short HEAD 2>/dev/null)+ui=$(shell git -C $(UI) rev-parse --short HEAD 2>/dev/null)+agentcmd=$(shell git -C deps/agent-command rev-parse --short HEAD 2>/dev/null)+documents=$(shell git -C modules/documents rev-parse --short HEAD 2>/dev/null)+native=$(shell git -C deps/agent-native rev-parse --short HEAD 2>/dev/null)+squad=$(shell git -C deps/agent-squad rev-parse --short HEAD 2>/dev/null)+intent=$(shell git -C deps/agent-intent rev-parse --short HEAD 2>/dev/null)
|
||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null)+$(COMBO)
|
||||
|
||||
.PHONY: help sync status build frontend svc svc-list svc-info svc-commits svc-history svc-update svc-recover svc-watchdog test-backend test-backend-pg test-frontend test-ui test-core test-agentcmd test-pins test-libs dev pin clean setup precheck task-start task-sync task-close task-finish facts-verify evidence version
|
||||
.PHONY: help sync status build frontend svc svc-list svc-info svc-commits svc-history svc-update svc-recover svc-watchdog test-backend test-backend-pg test-multi-instance test-frontend test-ui test-core test-agentcmd test-pins test-libs dev pin clean setup precheck task-start task-sync task-close task-finish facts-verify evidence version
|
||||
|
||||
version:
|
||||
@echo "agent-system $(VERSION)"
|
||||
@@ -112,6 +112,14 @@ test-backend-pg:
|
||||
@test -n "$${AGENT_REDIS_TEST_ADDR}" || { echo "test-backend-pg: AGENT_REDIS_TEST_ADDR 未设置（真实 Redis 必需，不静默跳过）"; exit 1; }
|
||||
cd $(BACKEND) && go test -count=1 ./...
|
||||
|
||||
# 双进程 HTTP 级验收（协作存储方案 U7）：两个真实 serve 进程共享隔离 PG。
|
||||
# 缺环境时明确失败而非静默跳过；见 tests/integration/multi-instance/README.md。
|
||||
test-multi-instance:
|
||||
@test -n "$${AGENT_PG_TEST_DSN}" || { echo "test-multi-instance: AGENT_PG_TEST_DSN 未设置（真实 PostgreSQL 必需，不静默跳过）"; exit 1; }
|
||||
mkdir -p tests/integration/multi-instance/.build
|
||||
cd $(BACKEND) && CGO_ENABLED=0 go build -o ../../tests/integration/multi-instance/.build/agent-system ./cmd/agent-system
|
||||
cd tests/integration/multi-instance && AGENT_BIN=./.build/agent-system go test -count=1 -timeout 15m .
|
||||
|
||||
eval-deliverables:
|
||||
cd $(BACKEND) && go test ./eval/ ./internal/quality/ -count=1
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# 双实例放行前检查单（协作存储 U7）
|
||||
|
||||
协作存储方案 §5 U7 的放行出口要求：**单实例团队、双进程、双主机分别出具
|
||||
结果；CI 身份/链接/SHA 或独立 REPLAY 可核验才用于放行，本地 SELF 只作辅助。**
|
||||
本检查单列出每一放行条件、其证据来源与当前状态；未全部满足前，第二实例
|
||||
不得承接写请求、双主机部署不得开工。
|
||||
|
||||
状态图例：✅ 已满足（证据可核验）｜⏳ 入口就绪、待执行窗口｜❌ 未开始。
|
||||
|
||||
## A. 正确性与故障矩阵（代码与 CI 可覆盖）
|
||||
|
||||
| # | 条件 | 证据（命令/测试） | 状态 |
|
||||
|---|---|---|---|
|
||||
| A1 | 单实例全链路（mock）：认证/项目/会话/SSE/检索/卡片 | backend `go test ./...`（CI job `test`，SQLite 档 35+ 包） | ✅ CI 绿 |
|
||||
| A2 | 真 PG/Redis 全量 | `make test-backend-pg`（CI job `test-pg`，AGENT_PG_TEST_DSN 必跑） | ✅ CI 绿 |
|
||||
| A3 | 磁盘满清楚拒绝、零残留、可恢复 | backend `TestUploadRejectsCleanlyWhenObjectDirUnwritable`、`TestArtifactOverwriteRejectsCleanlyWhenObjectDirUnwritable` | ✅ CI 绿 |
|
||||
| A4 | 恢复隔离（未知副作用不自动重跑） | backend `recovery_hold_test.go`（U3） | ✅ CI 绿 |
|
||||
| A5 | 库级多实例语义（争抢/围栏/命令/跨实例 resolve 不落卡） | backend `runmgr/multi_instance_test.go`（真库门控） | ✅ CI 绿 |
|
||||
| A6 | **双进程 HTTP 级验收**（跨实例读/SSE/409/HITL/取消/崩溃恢复/重启不变量） | `make test-multi-instance`（CI job `test-pg`；本地 `tests/integration/multi-instance/README.md`） | ✅ CI 绿 |
|
||||
| A7 | 权限矩阵重放（撤权零泄漏：请求/SSE/检索/下载/工具） | backend `resource_authz_test`、`shared_*_test`、`sse_gap_test`、`pgvector_test`（U3/U6） | ✅ CI 绿 |
|
||||
|
||||
## B. 容量与负载合同（需要实测窗口，本地入口已就绪）
|
||||
|
||||
| # | 条件 | 证据（命令） | 状态 |
|
||||
|---|---|---|---|
|
||||
| B1 | 冒烟压测通过（入口可复核、不变量零违反） | `tests/load/collaboration/run_load.sh --profile smoke` | ✅ 已跑通（SELF，2026-10-12：读 P95<1ms、SSE P95≈18ms、对账 0 违反） |
|
||||
| B2 | 阶梯负载 10→30→100 用户，读 P95≤300ms、P99≤1s、非预期失败≤0.1% | 同上 `--profile ladder`（k6 需安装：`brew install k6`） | ⏳ 待窗口 |
|
||||
| B3 | 60 分钟稳定负载 + 2 小时混合长稳 | `--profile sustained60m` / `--profile sustained2h` | ⏳ 待窗口（约 3h 机时） |
|
||||
| B4 | SSE 持久事件可见延迟 P95≤1s | 同上套件自动执行（sse_probe，断言即退出码） | ✅ 冒烟档已证机制；正式窗口随 B3 |
|
||||
| B5 | 数据量档（100 万消息/10 万知识分块/多项目交叉授权） | sustained 长跑自然积累 + seed 扩展（未实现 seed 工具，标注） | ❌ seed 工具未实施 |
|
||||
| B6 | 数据库基线（pgbench 只测 DB 能力，不替代业务验收） | `pgbench` 对隔离实例（只读参考） | ❌ 未执行 |
|
||||
| B7 | 容量数字绑定组件 SHA/配置/数据集归档 | results/<ts>/ 目录 + 报告引用 SHA | ⏳ 随 B2/B3 产出 |
|
||||
|
||||
## C. 双进程与双主机放行（分布式出口）
|
||||
|
||||
| # | 条件 | 证据 | 状态 |
|
||||
|---|---|---|---|
|
||||
| C1 | 双进程共享 PG：正确性矩阵全过（A6） | `make test-multi-instance` | ✅ |
|
||||
| C2 | 双进程**负载**下互相不拖累（热点会话/命令时延） | 双栈压测（run_load.sh 尚为单实例档，需扩展第二实例参数） | ❌ 入口未扩展 |
|
||||
| C3 | 双主机（跨机网络）验证：同套验收在两台主机执行 | A6 的 AGENT_BIN/DSN 指向远端；需用户指定第二主机 | ❌ 待资源 |
|
||||
| C4 | 故障演练：PG 断连/Redis 断连恢复（RTO≤15min） | backend ratelimit/recovery 测试 + 运维手册演练记录 | ⏳ 代码级 ✅；运维演练待窗口 |
|
||||
| C5 | 备份恢复演练（RPO≤5min/RTO≤60min，pg_dump+物理备份） | U4 迁移演练已过；定期备份恢复演练属运维，待排期 | ⏳ |
|
||||
|
||||
## 放行结论（当前）
|
||||
|
||||
- **单实例团队档**：A1–A7 全绿 + B1 冒烟通过。正式容量结论待 B2/B3 窗口；
|
||||
在此之前按「已验收功能、未承诺容量」运营。
|
||||
- **双进程档**：正确性已证（A5/A6/C1），负载与运维演练（C2/C4）未过，
|
||||
**不放行第二实例承接写请求**。
|
||||
- **双主机档**：未开始（C3），不放行。
|
||||
|
||||
## 已知边界与诚实声明
|
||||
|
||||
- U7 期间经双进程验收发现并修复三个跨实例缺陷（详见
|
||||
`tests/integration/multi-instance/README.md`）：HITL resolve 落卡短路命令、
|
||||
cancel 命令空会话 ID、CancelRun 假成功。修复均已入 CI 回归。
|
||||
- k6 首次冒烟使用临时二进制（v2.3.0）验证后删除；正式窗口前按
|
||||
`brew install k6` 装固定位。
|
||||
- B5 大数据量 seed 与 C2 双实例压测入口是下一步实施项，本检查单不替代。
|
||||
@@ -1,10 +1,10 @@
|
||||
# U1 实例—账户绑定补齐（规格 v0.1）
|
||||
|
||||
状态：设计候选，随主提案 r1 评审。主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` §7 U1。
|
||||
状态：已启动实施（分支 `feature/svc-deploy-u1-account-binding`）。主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` §7 U1。
|
||||
前置：恢复方案 recovery U1 已合入 main（Dev #415）——`docs/recovery/u1-identity-manifest.md`、`scripts/svc_manifest.py`、启动身份校验与 `svc init`。
|
||||
本增量**不重做** manifest 层；只补齐账户操作与安装入口对既有身份合同的消费缺口。
|
||||
|
||||
基线（写作时 SELF）：Dev `af9c2de`（含 recovery U1）。实施前以任务 `task-sync` 后 SHA 为准。业务实施 **NOT_RUN**。
|
||||
基线：任务卡 R0；实施中。真实 PG 绑定专项缺环境标 NOT_RUN。
|
||||
|
||||
## 1. 目标与非目标
|
||||
|
||||
|
||||
@@ -1,20 +1,4 @@
|
||||
# 草稿：feature/svc-deploy-u1-account-binding 任务卡
|
||||
# 草稿：feature/svc-deploy-u1-account-binding
|
||||
|
||||
状态：**待授权**。主提案批准并获实施授权后，在共享主检出执行：
|
||||
|
||||
```bash
|
||||
RECEIPT=/tmp/precheck-svc-deploy-u1.json CMD="make precheck" make evidence
|
||||
make task-start NAME=feature/svc-deploy-u1-account-binding PRECHECK_RECEIPT=/tmp/precheck-svc-deploy-u1.json
|
||||
```
|
||||
|
||||
将本草稿字段拷入新卡并删除本文件或改为「已启动」链接。
|
||||
|
||||
## 建议填写
|
||||
|
||||
- 主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` r1 §7 U1
|
||||
- 规格：`docs/svc-deploy/u1-instance-account-binding.md` v0.1
|
||||
- 目标：账户 CLI 绑定实例 manifest；OpenExisting；无 flock 拒绝冲突操作；不破坏 recovery U1
|
||||
- 非目标：U2 下载、U3 doctor、U4 Key 语义、U5 页面
|
||||
- 兄弟任务：可与 `feature/svc-deploy-u4-login-key` 共用同一 R0
|
||||
- 验收：规格 §6 T1–T5；治理回归；真实 PG 缺环境 NOT_RUN
|
||||
- 启动授权：未授权（须主提案阶段批准 + 明确实施授权）
|
||||
状态：**已启动** → 任务卡 `.task/feature-svc-deploy-u1-account-binding.md`；工作区
|
||||
`/home/weichen/projects/Agent-System-Dev-wt-feature-svc-deploy-u1-account-binding`。
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# U4 一次性登录凭据与认证限制（规格 v0.1）
|
||||
|
||||
状态：设计候选，随主提案 r1 评审。主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` §7 U4 / §8。
|
||||
状态：已启动实施（Backend/Dev 分支 `feature/svc-deploy-u4-login-key`）。主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` §7 U4 / §8。
|
||||
本增量在 **Backend**（及必要迁移/CLI）落地；Frontend 兑换与设密页属 **U5**，但 U4 须提供稳定 HTTP 合同供 U5 接线。
|
||||
|
||||
基线（写作时 SELF）：Backend 随 Dev `af9c2de` gitlink；关键现状见主提案 §2。业务实施 **NOT_RUN**。
|
||||
基线：任务卡 R0；Backend 实现提交见 gitlink。真实 PG 缺环境标 NOT_RUN。
|
||||
|
||||
## 1. 目标与非目标
|
||||
|
||||
|
||||
@@ -1,21 +1,4 @@
|
||||
# 草稿：feature/svc-deploy-u4-login-key 任务卡
|
||||
# 草稿：feature/svc-deploy-u4-login-key
|
||||
|
||||
状态：**待授权**。主提案批准并获实施授权后，在共享主检出执行：
|
||||
|
||||
```bash
|
||||
RECEIPT=/tmp/precheck-svc-deploy-u4.json CMD="make precheck" make evidence
|
||||
make task-start NAME=feature/svc-deploy-u4-login-key PRECHECK_RECEIPT=/tmp/precheck-svc-deploy-u4.json
|
||||
```
|
||||
|
||||
将本草稿字段拷入新卡并删除本文件或改为「已启动」链接。
|
||||
|
||||
## 建议填写
|
||||
|
||||
- 主记录：`modules/documents/docs/proposals/2026-10-12-svc-unified-deploy-and-login-key.md` r1 §7 U4
|
||||
- 规格：`docs/svc-deploy/u4-one-time-login-key.md` v0.1
|
||||
- 目标：独立 login_keys；原子兑换；受限会话中间件；设密事务；无密码开户/恢复签发；OpenExisting 最小集
|
||||
- 非目标：svc 账户接线与登录页（U5）；部署来源（U2/U3）
|
||||
- 组件：Backend 为主；Dev 仅 gitlink / 文档引用；Frontend 不强制本增量改动
|
||||
- 兄弟任务：可与 `feature/svc-deploy-u1-account-binding` 共用同一 R0
|
||||
- 验收：规格 §7 T1–T11；SQLite+PG；并发兑换；治理/Backend 测试
|
||||
- 启动授权：未授权（须主提案阶段批准 + 明确实施授权）
|
||||
状态：**已启动** → 任务卡 `.task/feature-svc-deploy-u4-login-key.md`；工作区
|
||||
`/home/weichen/projects/Agent-System-Dev-wt-feature-svc-deploy-u4-login-key`。
|
||||
|
||||
+26
@@ -583,6 +583,32 @@ scripts/svc.sh db backups pg1 # 列表（时间 / 大小 / 文
|
||||
- 完成后 `pg_restore --list` 自检归档有效性，失败即删除该文件并记 `backup fail`；
|
||||
- 备份**只增不删**；随 `rm --purge` 一并删除。
|
||||
|
||||
### 8.1.1 agent 数据目录快照（SQLite 一致性备份）
|
||||
|
||||
```bash
|
||||
scripts/svc.sh dev backup # 快照 data/ + svc.env + history.jsonl
|
||||
scripts/svc.sh dev backup --keep 3 # 备份后只保留最近 3 份
|
||||
scripts/svc.sh dev backup --out /mnt/bk/dev1 # 自定义快照目录（不参与自动清理）
|
||||
scripts/svc.sh dev backups # 快照列表（时间 / 大小 / 文件数）
|
||||
scripts/svc.sh dev backups --prune --keep 3 # 手动清理旧快照
|
||||
```
|
||||
|
||||
- 位置：`runtime-backups/<name>/<UTC时间戳>/`——**在 runtime/ 之外**。2026-10-11
|
||||
发生过 `runtime/` 被整体误删的事故（SQLite 聊天/设置数据全失，且无任何备份）；
|
||||
放在服务目录内的备份会跟着一起消失，因此快照根独立于 runtime/；
|
||||
- 一致性：每个 `*.db` 经 `sqlite3 .backup` 导出（并入 WAL 内容，服务运行中也可
|
||||
安全备份）并做 `PRAGMA integrity_check` 自检；`-wal`/`-shm` 不入快照。无 sqlite3
|
||||
时**拒绝备份**，不回退裸拷贝（裸拷会漏 WAL 得到撕裂副本）；
|
||||
- MANIFEST：快照内 `MANIFEST`（600）记录服务、来源版本与逐文件 sha256，恢复时
|
||||
可校验完整性。非 SQLite 的 `*.db` 文件（无文件头）保留原样副本并告警；
|
||||
- 保留策略：默认保留最近 7 份（`--keep N` 调整；只清理符合时间戳命名的快照目录，
|
||||
不动 `--out` 指定的自定义目录）；`rm <name> --purge` 会连带删除
|
||||
`runtime-backups/<name>/`，`rm`（不带 --purge）保留快照；
|
||||
- 边界：本命令是**数据面快照**，不含 restore（恢复到新目标、manifest 合同与
|
||||
外部效果对账属恢复方案 U6，见
|
||||
`modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md`）；
|
||||
postgres 实例请用 `db backup`，static 服务无数据目录。
|
||||
|
||||
### 8.2 恢复
|
||||
|
||||
```bash
|
||||
|
||||
+1
-1
Submodule modules/backend updated: 3d850a282f...1d9e8f4a79
+1
-1
Submodule modules/documents updated: 7df7d786d0...9d9b1523c8
+258
-11
@@ -7,6 +7,8 @@
|
||||
# scripts/svc.sh <name> [action] [参数...] 服务级：deploy(默认)|start|stop|restart|status|logs [N]
|
||||
# ｜db [pg名|--clear]（agent 数据库绑定/切换）
|
||||
# ｜config [file|--clear]（agent 配置文件更新/生效）
|
||||
# ｜backup [--out D] [--keep N]（agent 数据目录快照）
|
||||
# ｜backups [--prune]（快照列表 / 按保留数清理）
|
||||
# scripts/svc.sh list [--tag T] 服务总览（status 等价）
|
||||
# scripts/svc.sh info [name] [--tag T] [--reveal] 综合报告（只读）：服务/数据库/配置快照/最近事件
|
||||
# scripts/svc.sh set <name> [--port N --apply] [--tag T] 变更登记（换端口/标签）
|
||||
@@ -59,6 +61,7 @@
|
||||
# db backup：--db X --out F db restore：--file F --db X --force
|
||||
# db clone：--port P --host H --password PW --tag T --online
|
||||
# db copy：--db X --force set：--port N --host H --apply --tag T
|
||||
# backup：--out DIR --keep N backups：--prune --keep N（agent 数据目录快照）
|
||||
#
|
||||
# ════════════════════ 配置优先级 ════════════════════
|
||||
# 命令行 > --config/--env 文件 > svc.env 持久化（exec=/addr=/dbref=/dsn=/db=）
|
||||
@@ -77,6 +80,9 @@
|
||||
# - runtime/<name>/：bin|htdocs|data|pgdata|plugins|backups + svc.pid +
|
||||
# svc.log|pg.log + svc.env（登记+持久化配置，600）+ history.jsonl（事件流）
|
||||
# + .start.lock（生命周期互斥锁，recover 与 deploy/手动操作防竞态）
|
||||
# - runtime-backups/<name>/<UTC时间戳>/：agent 数据目录快照（svc.env+history+data；
|
||||
# *.db 经 sqlite3 .backup 一致性导出并自检，MANIFEST 记逐文件 sha256。放 runtime/
|
||||
# 之外——runtime 被整体误删时快照仍在，rm --purge 随服务一并删除）
|
||||
# - kind：agent（后端单二进制，内嵌前端）/ static（前端静态服务）/ postgres（数据库）；
|
||||
# 无 kind= 的旧登记自动视为 agent；start/restart 使用环境内产物
|
||||
#
|
||||
@@ -89,6 +95,7 @@
|
||||
# scripts/svc.sh ui-qa deploy --kind static --dir ../some/dist
|
||||
# scripts/svc.sh set pg1 --port 8895 --apply # 换端口并重启生效
|
||||
# scripts/svc.sh db backup pg1 && scripts/svc.sh db backups pg1
|
||||
# scripts/svc.sh dev backup && scripts/svc.sh dev backups # agent 数据目录快照 + 列表
|
||||
# scripts/svc.sh db clone pg1 pg2 --port 8896 # 物理克隆一套到新端口
|
||||
# scripts/svc.sh dev db pg1 --db dev # 把 dev 服务绑定到 pg1/dev 库（自动重启）
|
||||
# scripts/svc.sh dev db pg2 --no-restart # 切换但不重启（下次 start 生效）
|
||||
@@ -105,7 +112,7 @@
|
||||
# make svc ENV=dev ARGS="--env .env.dev --port 9000"
|
||||
#
|
||||
# ════════════════════ 安全约束 ════════════════════
|
||||
# - 只写 runtime/<name>/、runtime/.worktrees/ 与独立部署源仓（SVC_SRC_DIR，默认
|
||||
# - 只写 runtime/<name>/、runtime/.worktrees/、runtime-backups/<name>/ 与独立部署源仓（SVC_SRC_DIR，默认
|
||||
# ~/.cache/agent-system-svc/src，可随时删除重建）；不改主检出 git 状态、不动他人现场；
|
||||
# --from local 是唯一写主检出构建产物（bin/、node_modules/、dist/，均 gitignore）
|
||||
# 的路径，dev@sha/release/static 构建全部在隔离 worktree（自部署源仓物化）；
|
||||
@@ -113,8 +120,10 @@
|
||||
# pg_ctl -m fast 优雅停机，SIGKILL 不用于 postmaster；
|
||||
# - 端口被非本脚本进程占用时报错并给出占用者，不抢不杀；
|
||||
# - svc.env/备份文件权限 600；DSN 默认掩码密码（--reveal 才显示）；
|
||||
# - 备份/克隆只增不删；restore/copy 覆盖非空库必须显式 --force（drop 重建再恢复）；
|
||||
# 备份文件在 runtime/<name>/backups/，随 rm --purge 一并删除；
|
||||
# - 备份/克隆只增不删（backup --keep / backups --prune 的显式保留策略除外）；
|
||||
# restore/copy 覆盖非空库必须显式 --force（drop 重建再恢复）；
|
||||
# pg 备份在 runtime/<name>/backups/，agent 快照在 runtime-backups/<name>/（在
|
||||
# runtime/ 之外，防整删事故连带），均随 rm --purge 一并删除；
|
||||
# - 物理克隆期间源实例短暂停止（一致性要求），完成后自动复启；--online 不停源。
|
||||
#
|
||||
# 完整文档：docs/svc.md
|
||||
@@ -159,6 +168,8 @@ svc.sh —— 本机服务管理：后端(agent) / 前端(static) / 数据库(po
|
||||
scripts/svc.sh <name> start|stop|restart|logs [N] 生命周期与日志
|
||||
scripts/svc.sh <name> status 服务详情（版本/来源/DSN 等）
|
||||
scripts/svc.sh <name> db [pg名|--clear] agent 数据库绑定/切换（缺省查看）
|
||||
scripts/svc.sh <name> backup [--out D] [--keep N] agent 数据目录快照（SQLite 一致性备份）
|
||||
scripts/svc.sh <name> backups [--prune] 快照列表 / 按保留数清理
|
||||
scripts/svc.sh <name> config [file|--clear] agent 配置文件更新/即时生效（缺省查看）
|
||||
scripts/svc.sh db create pg1 --port 8890 建数据库
|
||||
scripts/svc.sh db dsn|backup|backups|restore|clone|copy|ls|set pg1 ... 数据库运维
|
||||
@@ -217,6 +228,12 @@ update deploy stop start restart logs help info version。
|
||||
history.jsonl 事件流（append-only，一行一个 JSON）
|
||||
.start.lock 生命周期互斥锁（recover 与 deploy/手动 start 防竞态）
|
||||
|
||||
runtime-backups/<name>/<UTC时间戳>/
|
||||
agent 数据目录快照：svc.env + history.jsonl + data/（*.db 经 sqlite3 .backup
|
||||
导出一致性副本并入 WAL 内容，integrity_check 自检；-wal/-shm 不入快照；
|
||||
MANIFEST 记逐文件 sha256）。位于 runtime/ 之外——runtime 被整体误删时快照
|
||||
仍在；rm --purge 随服务一并删除。
|
||||
|
||||
svc.env 字段：
|
||||
kind agent|static|postgres（旧登记无此键视为 agent）
|
||||
env 服务名
|
||||
@@ -245,7 +262,7 @@ history.jsonl 事件（scripts/svc.sh history 查看）：
|
||||
recover ok|fail（watchdog/手动 recover 拉回死亡服务）
|
||||
stop ok|fail
|
||||
set 登记变更（addr 旧 → 新）
|
||||
backup ok|fail（文件与大小）
|
||||
backup ok|fail（pg：库/文件/大小；agent：快照目录/大小）
|
||||
restore ok|fail（库、文件、表数）
|
||||
clone ok|fail（物理/在线）
|
||||
copy ok|fail
|
||||
@@ -340,6 +357,10 @@ EOF
|
||||
scripts/svc.sh rm <name> 停止并移除登记（目录/数据/历史保留）
|
||||
scripts/svc.sh rm <name> --purge 连 runtime/<name>/ 整个目录一起删除
|
||||
|
||||
数据目录快照（agent；SQLite 一致性导出，详见 docs/svc.md）：
|
||||
scripts/svc.sh <name> backup [--out DIR] [--keep N] 快照到 runtime-backups/<name>/
|
||||
scripts/svc.sh <name> backups [--prune] [--keep N] 列表 / 按保留数清理旧快照
|
||||
|
||||
自动恢复（崩溃 / OOM 被杀 / 机器重启后自动拉回；详见 help recovery）：
|
||||
scripts/svc.sh recover [name] [--quiet] 把 autostart=1 但已死的服务重新拉起
|
||||
scripts/svc.sh watchdog install systemd 用户 timer 每分钟执行 recover
|
||||
@@ -669,7 +690,7 @@ addr_compose() {
|
||||
}
|
||||
|
||||
# ---- 名称与端口 ----
|
||||
RESERVED_ENVS=" status list add set db rm create dsn commits history update deploy stop start restart logs help info version recover watchdog "
|
||||
RESERVED_ENVS=" status list add set db rm create dsn commits history update deploy stop start restart logs help info version recover watchdog backup backups "
|
||||
|
||||
# valid_env NAME：服务名合法性（小写字母开头、a-z0-9-、≤16 位；保留字不可用）。
|
||||
valid_env() {
|
||||
@@ -871,18 +892,25 @@ port_owner() {
|
||||
|
||||
# ---- 生命周期互斥锁与期望运行态（autostart；见 help recovery）----
|
||||
# svc_lock MODE：对当前服务取 .start.lock 互斥锁。MODE=wait 阻塞（最长 300s），
|
||||
# MODE=-n 非阻塞（抢不到返回 1）。flock 缺失时降级为无锁直通（尽力而为）。
|
||||
# MODE=-n 非阻塞（抢不到返回 1）。flock 缺失或等待超时 → 拒绝（不得直通冲突操作）。
|
||||
# 锁随进程退出自动释放（fd 关闭）；长驻循环（recover）用 svc_unlock 逐服务显式释放。
|
||||
SVC_LOCK_HELD=0
|
||||
svc_lock() {
|
||||
[ "$SVC_LOCK_HELD" = 1 ] && return 0
|
||||
command -v flock >/dev/null 2>&1 || { SVC_LOCK_HELD=1; return 0; }
|
||||
if ! command -v flock >/dev/null 2>&1; then
|
||||
c_err "[$ENV_NAME] 缺少 flock：拒绝生命周期/账户冲突操作（安装 util-linux 后重试）"
|
||||
return 1
|
||||
fi
|
||||
mkdir -p "$ENV_DIR" 2>/dev/null || return 1
|
||||
exec 9>>"$ENV_DIR/.start.lock" || return 1
|
||||
if [ "$1" = "-n" ]; then
|
||||
flock -n 9 || { exec 9>&-; return 1; }
|
||||
else
|
||||
flock -w 300 9 || c_warn "[$ENV_NAME] 生命周期锁等待超时（继续执行，注意并发）"
|
||||
if ! flock -w 300 9; then
|
||||
exec 9>&-
|
||||
c_err "[$ENV_NAME] 生命周期锁等待超时：拒绝继续（避免并发冲突）"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
SVC_LOCK_HELD=1
|
||||
return 0
|
||||
@@ -1384,6 +1412,66 @@ materialize_release() {
|
||||
install_artifacts "$bin" "$art/plugins" || return 1
|
||||
}
|
||||
|
||||
# ---- agent：账户 CLI（绑定实例存储；禁止回落默认 ./data）----
|
||||
# 用法：scripts/svc.sh <env> user <create|list|...> [--owner …]
|
||||
# 从实例最终配置 / manifest identity 解析 data/DSN，强制 --open-existing，
|
||||
# 调用 ENV_BIN 对应版本。拒绝调用方覆盖 --data/--dsn/--open-existing。
|
||||
agent_do_user() {
|
||||
local a
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--data|--dsn|--open-existing|--data=*|--dsn=*|--open-existing=*)
|
||||
c_err "[$ENV_NAME] 账户操作禁止覆盖存储绑定（勿传 --data/--dsn/--open-existing；由实例 manifest 决定）"
|
||||
return 2 ;;
|
||||
esac
|
||||
done
|
||||
svc_lock wait || { c_err "[$ENV_NAME] 无法获取生命周期锁（账户操作与启停互斥）"; return 1; }
|
||||
# 有 manifest 时以 identity.data_root 为权威，拒绝 ENV_DATA 漂移到默认 ./data。
|
||||
if [ -f "$ENV_DIR/manifest.json" ] && command -v python3 >/dev/null 2>&1; then
|
||||
local mf_root
|
||||
mf_root="$(python3 -c 'import json,sys; m=json.load(open(sys.argv[1])); print(m["identity"]["data_root"])' \
|
||||
"$ENV_DIR/manifest.json" 2>/dev/null || true)"
|
||||
if [ -n "$mf_root" ]; then
|
||||
ENV_DATA="$mf_root"
|
||||
fi
|
||||
fi
|
||||
case "${ENV_DATA:-}" in
|
||||
""|./data|data)
|
||||
c_err "[$ENV_NAME] 未绑定数据目录：拒绝账户操作（不回落默认 ./data）"
|
||||
return 1 ;;
|
||||
esac
|
||||
resolve_db_config || {
|
||||
c_err "[$ENV_NAME] 数据库绑定无效：拒绝账户操作（不回落默认 ./data）"
|
||||
return 1
|
||||
}
|
||||
if [ "$DB_MODE" = mock ]; then
|
||||
c_err "[$ENV_NAME] mock 档不支持账户操作"
|
||||
return 1
|
||||
fi
|
||||
if [ ! -x "${ENV_BIN:-}" ]; then
|
||||
c_err "[$ENV_NAME] 实例二进制不存在：$ENV_BIN（先 deploy）"
|
||||
return 1
|
||||
fi
|
||||
if [ "$DB_MODE" = pg ] && [ -z "${DB_DSN:-}" ] && [ "${DB_SOURCE:-}" != conffile ]; then
|
||||
c_err "[$ENV_NAME] PostgreSQL 档缺少 DSN：拒绝账户操作"
|
||||
return 1
|
||||
fi
|
||||
# 嵌入式档：数据文件必须已存在（OpenExisting）；PG 档由 --open-existing 在后端核验。
|
||||
if [ "$DB_MODE" != pg ]; then
|
||||
if [ ! -d "$ENV_DATA" ] || [ ! -f "$ENV_DATA/agent-system.db" ]; then
|
||||
c_err "[$ENV_NAME] 既有库不存在：$ENV_DATA/agent-system.db（OpenExisting：拒绝创建空库）"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
local bin_args=(--open-existing --data "$ENV_DATA")
|
||||
if [ "$DB_MODE" = pg ] && [ -n "${DB_DSN:-}" ]; then
|
||||
bin_args+=(--dsn "$DB_DSN")
|
||||
fi
|
||||
c_info "[$ENV_NAME] 账户 CLI → $ENV_BIN user $* （data=$ENV_DATA open-existing）"
|
||||
# 子命令参数在前，绑定 flags 在后（Go flag 不区分顺序）。
|
||||
AGENT_SYSTEM_DSN="${DB_DSN:-}" "$ENV_BIN" user "$@" "${bin_args[@]}"
|
||||
}
|
||||
|
||||
# ---- agent：部署与启动 ----
|
||||
|
||||
agent_do_deploy() {
|
||||
@@ -2940,6 +3028,150 @@ EOF
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---- agent 数据目录快照（SQLite 一致性备份；2026-10-12）----
|
||||
# 快照根在 runtime-backups/<name>/（runtime/ 之外）：2026-10-11 runtime/ 被整体
|
||||
# 误删的事故里，放在 runtime/<name>/backups/ 的任何东西都会一起消失。每个 *.db
|
||||
# 经 sqlite3 .backup 导出一致性副本（并入 WAL 内容）并 integrity_check 自检，
|
||||
# 不裸拷 .db（约束见 modules/documents/docs/proposals/
|
||||
# 2026-10-12-service-restart-data-recovery.md §6）；-wal/-shm 不入快照。
|
||||
BACKUP_KEEP_DEFAULT=7
|
||||
BACKUP_TS_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'
|
||||
|
||||
agent_backup_root() { printf '%s/runtime-backups/%s' "$ROOT" "$ENV_NAME"; }
|
||||
|
||||
agent_backup_guard() { # 公共前置：agent 已登记（backups 列表也走这里）
|
||||
if [ "$SVC_KIND" = postgres ]; then
|
||||
c_err "[$ENV_NAME] postgres 用 db backup（pg_dump）；backup 是 agent 数据目录快照"
|
||||
exit 2
|
||||
elif [ "$SVC_KIND" != agent ]; then
|
||||
c_err "[$ENV_NAME] kind=$SVC_KIND 无数据目录，不适用 backup"
|
||||
exit 2
|
||||
fi
|
||||
[ -f "$ENV_DIR/svc.env" ] || { c_err "[$ENV_NAME] 未登记（无 svc.env）；先 deploy"; exit 2; }
|
||||
}
|
||||
|
||||
agent_backup_snapshot() { # $1=dest：svc.env + history.jsonl + data/（*.db 一致性化）
|
||||
local dest="$1" rel
|
||||
cp -p "$ENV_DIR/svc.env" "$dest/svc.env" || return 1
|
||||
[ -f "$ENV_DIR/history.jsonl" ] && cp -p "$ENV_DIR/history.jsonl" "$dest/history.jsonl"
|
||||
mkdir -p "$dest/data" || return 1
|
||||
# 先整树拷贝（非 db 内容的原样副本），再对 *.db 用一致性导出覆盖：cp 得到的
|
||||
# 撕裂 db/-wal 副本在下一步被丢弃替换，不影响最终一致性。
|
||||
cp -a "$ENV_DATA/." "$dest/data/" || return 1
|
||||
while IFS= read -r rel; do
|
||||
[ -f "$ENV_DATA/$rel" ] || continue
|
||||
case "$rel" in
|
||||
*"'"*|*'"'*)
|
||||
if head -c 15 "$ENV_DATA/$rel" | grep -qa '^SQLite format 3$'; then
|
||||
c_err "[$ENV_NAME] 路径含引号，无法走 sqlite 一致性导出：$rel"
|
||||
return 1
|
||||
fi
|
||||
continue ;;
|
||||
esac
|
||||
if sqlite3 "$ENV_DATA/$rel" ".backup '$dest/data/$rel.bk'" 2>/dev/null \
|
||||
&& [ "$(sqlite3 "$dest/data/$rel.bk" 'PRAGMA integrity_check;' 2>/dev/null)" = ok ]; then
|
||||
mv -f "$dest/data/$rel.bk" "$dest/data/$rel" || return 1
|
||||
chmod 600 "$dest/data/$rel"
|
||||
rm -f "$dest/data/$rel-wal" "$dest/data/$rel-shm" # .backup 已并入 WAL 内容
|
||||
else
|
||||
rm -f "$dest/data/$rel.bk" # 失败尝试的半成品不留在快照里
|
||||
if head -c 15 "$ENV_DATA/$rel" | grep -qa '^SQLite format 3$'; then
|
||||
c_err "[$ENV_NAME] SQLite 一致性备份/自检失败：$rel"
|
||||
return 1
|
||||
fi
|
||||
c_warn "[$ENV_NAME] $rel 不是 SQLite 库（无文件头），保留原样副本"
|
||||
fi
|
||||
done < <(cd "$dest/data" 2>/dev/null && find . -type f -name '*.db' | sort)
|
||||
return 0
|
||||
}
|
||||
|
||||
agent_backup_manifest() { # $1=dest：快照自描述（600）——逐文件 sha256 供恢复时校验
|
||||
local dest="$1"
|
||||
{
|
||||
printf 'service=%s\nkind=%s\ncreated=%s\nsource=%s\nversion=%s\ndata_dir=%s\n' \
|
||||
"$ENV_NAME" "$SVC_KIND" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
|
||||
"$(svc_get source)" "$(svc_get version)" "$ENV_DATA"
|
||||
( cd "$dest" && find . -type f ! -name MANIFEST -print0 | sort -z | xargs -0 -r sha256sum )
|
||||
} > "$dest/MANIFEST"
|
||||
chmod 600 "$dest/MANIFEST"
|
||||
}
|
||||
|
||||
agent_backup_prune() { # $1=root $2=keep：只清理符合时间戳命名的快照目录（最新在前保留 keep 份）
|
||||
local root="$1" keep="$2" n=0 d
|
||||
[ -d "$root" ] || return 0
|
||||
while IFS= read -r d; do
|
||||
n=$((n + 1))
|
||||
[ "$n" -le "$keep" ] && continue
|
||||
rm -rf "$root/$d"
|
||||
c_info "[$ENV_NAME] 清理旧快照：$d（保留最近 $keep 份）"
|
||||
done < <(cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB" | sort -r)
|
||||
return 0
|
||||
}
|
||||
|
||||
svc_agent_backup() { # [--out DIR] [--keep N]
|
||||
local out="" keep="$BACKUP_KEEP_DEFAULT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--out) need_value "$@"; out="$2"; shift 2 ;;
|
||||
--keep) need_value "$@"; keep="$2"; shift 2
|
||||
case "$keep" in ''|*[!0-9]*) c_err "--keep 须为非负整数"; exit 2 ;; esac ;;
|
||||
*) c_err "用法：scripts/svc.sh <name> backup [--out DIR] [--keep N]（未知参数：$1）"; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
agent_backup_guard
|
||||
[ -d "$ENV_DATA" ] || { c_err "[$ENV_NAME] 数据目录不存在：$ENV_DATA"; exit 2; }
|
||||
command -v sqlite3 >/dev/null 2>&1 \
|
||||
|| { c_err "缺少 sqlite3——一致性备份必需，不回退裸拷贝（会漏 WAL 得到撕裂副本）"; exit 1; }
|
||||
local ts dest size
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
dest="${out:-$(agent_backup_root)/$ts}"
|
||||
case "$dest" in /*) ;; *) dest="$PWD/$dest" ;; esac
|
||||
[ ! -e "$dest" ] || { c_err "备份目标已存在：$dest"; exit 1; }
|
||||
mkdir -p "$dest" || { c_err "无法创建备份目录：$dest"; exit 1; }
|
||||
if ! agent_backup_snapshot "$dest" || ! agent_backup_manifest "$dest"; then
|
||||
rm -rf "$dest"
|
||||
hist_event backup fail "$(read_pid)" "dir=$dest"
|
||||
c_err "[$ENV_NAME] 备份失败，半成品已清理：$dest"
|
||||
exit 1
|
||||
fi
|
||||
chmod 700 "$dest" 2>/dev/null || true
|
||||
size="$(du -sh "$dest" 2>/dev/null | cut -f1)"
|
||||
hist_event backup ok "$(read_pid)" "dir=$dest size=$size"
|
||||
[ -n "$out" ] || agent_backup_prune "$(agent_backup_root)" "$keep"
|
||||
c_ok "[$ENV_NAME] 数据目录快照完成：$dest（$size；svc.env+history+data，db 已一致性自检）"
|
||||
}
|
||||
|
||||
svc_agent_backups() { # [--prune] [--keep N]
|
||||
local prune=0 keep="$BACKUP_KEEP_DEFAULT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--prune) prune=1; shift ;;
|
||||
--keep) need_value "$@"; keep="$2"; shift 2
|
||||
case "$keep" in ''|*[!0-9]*) c_err "--keep 须为非负整数"; exit 2 ;; esac ;;
|
||||
*) c_err "用法：scripts/svc.sh <name> backups [--prune] [--keep N]（未知参数：$1）"; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
agent_backup_guard
|
||||
local root n=0
|
||||
root="$(agent_backup_root)"
|
||||
if [ -d "$root" ]; then
|
||||
while IFS= read -r _; do n=$((n + 1)); done \
|
||||
< <(cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB")
|
||||
fi
|
||||
if [ "$n" -eq 0 ]; then
|
||||
c_info "[$ENV_NAME] 暂无快照（root=$root；scripts/svc.sh $ENV_NAME backup）"
|
||||
return 0
|
||||
fi
|
||||
c_info "[$ENV_NAME] 快照列表（$root，最新在前）"
|
||||
( cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB" | sort -r ) \
|
||||
| while IFS= read -r d; do
|
||||
printf ' %s %6s %s\n' "${d#./}" "$(du -sh "$root/$d" | cut -f1)" \
|
||||
"$(find "$root/$d/data" -type f 2>/dev/null | wc -l) 个文件"
|
||||
done
|
||||
if [ "$prune" = 1 ]; then agent_backup_prune "$root" "$keep"; fi
|
||||
return 0
|
||||
}
|
||||
|
||||
cmd_set() {
|
||||
local name="${1:-}"
|
||||
[ -n "$name" ] || { c_err "用法：scripts/svc.sh set <name> [--port N] [--host H] [--apply] [--tag T...]"; exit 2; }
|
||||
@@ -3078,6 +3310,11 @@ cmd_rm() {
|
||||
rm -f "$ENV_DIR/svc.env" "$ENV_DIR/svc.pid"
|
||||
if [ "$CLI_PURGE" = 1 ]; then
|
||||
rm -rf "${ENV_DIR:?}"
|
||||
local bkroot="$ROOT/runtime-backups/$name"
|
||||
if [ -d "$bkroot" ]; then
|
||||
rm -rf "$bkroot"
|
||||
c_ok "[$ENV_NAME] 数据目录快照已删除：$bkroot"
|
||||
fi
|
||||
c_ok "[$ENV_NAME] 已注销并删除整个目录：$ENV_DIR（含数据与历史）"
|
||||
else
|
||||
c_ok "[$ENV_NAME] 已注销登记；目录保留（数据/历史在 $ENV_DIR；彻底删除用 --purge）"
|
||||
@@ -3954,17 +4191,27 @@ main() {
|
||||
# 动作词可省略（省略 = deploy）；flags 可直接跟随服务名。
|
||||
local action="deploy"
|
||||
case "${1:-}" in
|
||||
deploy|stop|start|restart|status|logs|db|config) action="$1"; shift ;;
|
||||
deploy|stop|start|restart|status|logs|db|config|init|user|backup|backups) action="$1"; shift ;;
|
||||
esac
|
||||
if [ "$action" = logs ] && [ "${1:-}" != "" ] && [[ "$1" =~ ^[0-9]+$ ]]; then
|
||||
CLI_LOGS_N="$1"; shift
|
||||
fi
|
||||
# db / config 子命令：自带参数解析（位置参数不走 parse_args；未知 flags 由其自行拒绝）。
|
||||
if [ "$action" = db ] || [ "$action" = config ]; then
|
||||
# db / config / user / backup / backups：自带参数解析（位置参数不走 parse_args；
|
||||
# 未知 flags 由其自行拒绝）。
|
||||
if [ "$action" = db ] || [ "$action" = config ] || [ "$action" = user ]; then
|
||||
env_config "$ENV_NAME"
|
||||
case "$action" in
|
||||
db) svc_db_bind "$@" || exit $? ;;
|
||||
config) svc_config "$@" || exit $? ;;
|
||||
user) agent_do_user "$@" || exit $? ;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
if [ "$action" = backup ] || [ "$action" = backups ]; then
|
||||
env_config "$ENV_NAME"
|
||||
case "$action" in
|
||||
backup) svc_agent_backup "$@" || exit $? ;;
|
||||
backups) svc_agent_backups "$@" || exit $? ;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
/.build/
|
||||
/data/
|
||||
@@ -0,0 +1,45 @@
|
||||
# 双进程 HTTP 级验收（协作存储 U7）
|
||||
|
||||
两个**真实 backend serve 进程**共享同一 PostgreSQL 的黑盒验收，覆盖方案 §7
|
||||
最小验收矩阵的进程级行（库级语义已在 backend `internal/runmgr/multi_instance_test.go`
|
||||
覆盖；本目录验证 HTTP/SSE/进程生命周期层面的同一套不变量）：
|
||||
|
||||
| 子场景 | 断言（矩阵行） |
|
||||
|---|---|
|
||||
| `cross_read` | A 建的项目/会话在 B 立即可见（共享权威） |
|
||||
| `cross_event_sse` | A 写入的事件在 B 的 SSE 可见（R5 跨实例传播） |
|
||||
| `same_session_conflict_then_cross_hitl` | 同会话并发提交明确 409 不误写；B 跨实例 resolve A 的 HITL 经命令总线到达执行者（R3/R4） |
|
||||
| `cross_cancel` | B 跨实例取消 A 的 run：命令持久、唯一消费、run 收口 cancelled（R4） |
|
||||
| `owner_crash_takeover` | 执行实例被 SIGKILL：租约期内 B 明确 409 不误抢；死实例不拖累其他会话；原实例重启对账收口遗留 run 后恢复推进（R4/R7/R8） |
|
||||
| `restart_invariants` | 实例重启后消息俱在、SSE 从 0 重放 seq 连续无洞、写入可推进（R5/R7） |
|
||||
|
||||
## 运行
|
||||
|
||||
```bash
|
||||
# 1) 隔离 PostgreSQL（示例：svc.sh 独立实例；已有实例则直接给 DSN）
|
||||
scripts/svc.sh db create mi7 --port 18890
|
||||
scripts/svc.sh mi7 start
|
||||
DSN="$(scripts/svc.sh db dsn mi7 --reveal)"
|
||||
|
||||
# 2) 构建 + 运行（Make 目标自动构建二进制并注入）
|
||||
make test-multi-instance AGENT_PG_TEST_DSN="$DSN?sslmode=disable"
|
||||
|
||||
# 3) 回收
|
||||
scripts/svc.sh mi7 stop && scripts/svc.sh db rm mi7
|
||||
```
|
||||
|
||||
门控：`AGENT_PG_TEST_DSN`（真实 PostgreSQL，CI test-pg job 必跑不跳过）与
|
||||
`AGENT_BIN`（由 Make 目标构建注入）。测试自建临时数据库、自起进程并在
|
||||
退出时回收；不打任何生产服务。
|
||||
|
||||
## 已实测发现并修复的缺陷（历史）
|
||||
|
||||
本验收在 U7 落地时抓出三个跨实例真实缺陷（均已有回归测试）：
|
||||
|
||||
1. **HITL resolve 落卡短路命令路径**：非执行实例先把卡标记 resolved，
|
||||
命令 fallback 的 `!card.Resolved` 判定被跳过，执行者永远等不到答案
|
||||
（backend `TestCrossInstanceResolveDoesNotSettleCard`）。
|
||||
2. **跨实例 cancel 命令 conversation_id 为空**：消费侧按执行权 JOIN
|
||||
匹配，空串永远无法送达。
|
||||
3. **CancelRun 假成功**：非执行实例无句柄时 `CancelRun` 返回 nil，HTTP
|
||||
层 200 受理但 run 永远停不下（命令路径被短路）。
|
||||
@@ -0,0 +1,13 @@
|
||||
module agent-system-dev/tests/integration/multi-instance
|
||||
|
||||
go 1.27
|
||||
|
||||
require github.com/jackc/pgx/v5 v5.9.2
|
||||
|
||||
require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
golang.org/x/sync v0.17.0 // indirect
|
||||
golang.org/x/text v0.29.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,26 @@
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
|
||||
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
||||
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,620 @@
|
||||
// 协作存储 U7：双进程（真实 backend serve ×2 共享同一 PG）黑盒验收。
|
||||
// 对应方案 §7 最小验收矩阵的进程级行：
|
||||
// - 两实例同会话不同请求 → 仅一个主执行，另一个明确 409（不误写）
|
||||
// - 在 B 提交 A 的 HITL / 取消 → 命令持久、唯一消费、到达执行者
|
||||
// - 执行实例被杀 → 租约期内不误抢（明确 409），过期后另一实例受控接管
|
||||
// - 实例重启 → 数据不变量保持（消息在、事件 seq 连续无洞）
|
||||
// - 跨实例读一致与事件传播（A 写 → B 的 SSE 可见）
|
||||
//
|
||||
// 资源边界：测试自建临时数据库、自起进程、自回收（方案 §5 U7 的隔离测试
|
||||
// 入口要求）。门控：AGENT_PG_TEST_DSN（真 PG；CI test-pg job 必跑）与
|
||||
// AGENT_BIN（backend 二进制；由 make test-multi-instance 构建并传入）。
|
||||
// 不打任何生产服务。
|
||||
package multiinstance
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
const (
|
||||
aliceOwner = "mi_alice"
|
||||
alicePass0 = "mi-alice-pass-0"
|
||||
alicePass = "mi-alice-pass-1"
|
||||
httpWait = 15 * time.Second
|
||||
)
|
||||
|
||||
type instance struct {
|
||||
name string
|
||||
base string
|
||||
cmd *exec.Cmd
|
||||
log *lockedBuffer
|
||||
}
|
||||
|
||||
type lockedBuffer struct {
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
func (b *lockedBuffer) Write(p []byte) (int, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return b.buf.Write(p)
|
||||
}
|
||||
|
||||
func (b *lockedBuffer) String() string {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return b.buf.String()
|
||||
}
|
||||
|
||||
type hitlCard struct{ id, runID string }
|
||||
|
||||
// userClient 持 alice 在两实例的会话（A：api；B：apiB/streamB）。
|
||||
// 会话表在共享 PG，同账号可在两实例同时登录。
|
||||
type userClient struct {
|
||||
api *http.Client // alice@A 普通请求（带超时）
|
||||
apiB *http.Client // alice@B 普通请求
|
||||
streamB *http.Client // alice@B 流式（SSE：无整体超时，chan 背压控制）
|
||||
}
|
||||
|
||||
type env struct {
|
||||
t *testing.T
|
||||
dbName string
|
||||
dbDSN string
|
||||
artsDir string
|
||||
bin string
|
||||
a, b *instance
|
||||
uc *userClient
|
||||
}
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("AGENT_PG_TEST_DSN") == "" {
|
||||
fmt.Println("SKIP: AGENT_PG_TEST_DSN 未设置（双进程验收由 CI test-pg job 必跑覆盖）")
|
||||
return
|
||||
}
|
||||
if os.Getenv("AGENT_BIN") == "" {
|
||||
fmt.Println("SKIP: AGENT_BIN 未设置（由 make test-multi-instance 构建并传入 agent-system 二进制路径）")
|
||||
return
|
||||
}
|
||||
// 子进程用独立 cwd（避免 data/ 落测试目录），二进制路径必须先转绝对。
|
||||
bin, err := filepath.Abs(os.Getenv("AGENT_BIN"))
|
||||
if err != nil {
|
||||
fmt.Println("AGENT_BIN abs:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
os.Setenv("AGENT_BIN", bin)
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func TestTwoInstancesOverSharedPG(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
e.startPair()
|
||||
|
||||
uc := e.loginAlice(t)
|
||||
e.uc = uc
|
||||
projID, convID := e.setupConversation(t, uc)
|
||||
|
||||
t.Run("cross_read", func(t *testing.T) {
|
||||
// A 建的资源在 B 立即可见（共享权威，无实例本地视图缺口）。
|
||||
var detail map[string]any
|
||||
e.do(t, uc.apiB, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
if conv, _ := detail["conversation"].(map[string]any); conv == nil || conv["id"] != convID {
|
||||
t.Fatalf("B cannot read conversation created via A: %v", detail)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("cross_event_sse", func(t *testing.T) {
|
||||
lastSeq := e.lastSeq(t, uc.apiB, convID)
|
||||
events := e.subscribe(t, uc.streamB, convID, lastSeq)
|
||||
e.postMessage(t, uc.api, convID, "hello from A")
|
||||
select {
|
||||
case ev, ok := <-events:
|
||||
if !ok {
|
||||
t.Fatal("SSE stream closed before any event")
|
||||
}
|
||||
if ev.seq <= lastSeq {
|
||||
t.Fatalf("stale event replayed: seq %d <= since %d", ev.seq, lastSeq)
|
||||
}
|
||||
case <-time.After(8 * time.Second):
|
||||
t.Fatal("event written via A not visible on B SSE within 8s")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("same_session_conflict_then_cross_hitl", func(t *testing.T) {
|
||||
// A 发 #hitl：mock runner 挂起 run（waiting_hitl），会话被 A 的执行租约占用。
|
||||
e.postMessage(t, uc.api, convID, "#hitl")
|
||||
card := e.waitActiveHITL(t, uc.apiB, convID)
|
||||
|
||||
// B 同会话并发提交 → 明确 409，而不是排队后误写或静默接受。
|
||||
code, body := e.raw(t, uc.apiB, "POST", "/api/conversations/"+convID+"/messages", map[string]any{"text": "from B while A holds"})
|
||||
if code != http.StatusConflict {
|
||||
t.Fatalf("concurrent submit on busy session: status %d (want 409), body %.200s", code, body)
|
||||
}
|
||||
|
||||
// B 跨实例 resolve A 的 HITL → 持久命令转交执行者（202 via command；
|
||||
// 执行实例本地 resolve 才是 200），run 随后走完。
|
||||
code, body = e.raw(t, uc.apiB, "POST", "/api/hitl/"+card.id+"/resolve", map[string]any{"choiceId": "md"})
|
||||
if code != http.StatusOK && code != http.StatusAccepted {
|
||||
t.Fatalf("cross-instance resolve: status %d, body %.200s", code, body)
|
||||
}
|
||||
e.waitRunStatus(t, uc.apiB, card.runID, "completed")
|
||||
// run 收口后会话释放：B 可推进。
|
||||
e.postMessage(t, uc.apiB, convID, "B proceeds after release")
|
||||
})
|
||||
|
||||
t.Run("cross_cancel", func(t *testing.T) {
|
||||
e.postMessage(t, uc.api, convID, "#hitl")
|
||||
card := e.waitActiveHITL(t, uc.apiB, convID)
|
||||
// 跨实例取消经命令总线受理（202 via command；执行实例本地取消为 200）。
|
||||
code, body := e.raw(t, uc.apiB, "POST", "/api/runs/"+card.runID+"/cancel", nil)
|
||||
if code != http.StatusOK && code != http.StatusAccepted {
|
||||
t.Fatalf("cross-instance cancel: status %d, body %.200s", code, body)
|
||||
}
|
||||
e.waitRunStatus(t, uc.apiB, card.runID, "cancelled")
|
||||
e.postMessage(t, uc.apiB, convID, "B proceeds after cancel")
|
||||
})
|
||||
|
||||
t.Run("owner_crash_takeover", func(t *testing.T) {
|
||||
e.postMessage(t, uc.api, convID, "#hitl")
|
||||
_ = e.waitActiveHITL(t, uc.api, convID)
|
||||
|
||||
e.killNine(t, e.a)
|
||||
|
||||
// 会话执行租约（15 分钟档）期内不误抢：B 对该会话的提交得到明确
|
||||
// 409（有界失败，不是写坏或挂死）；过期接管语义由库级围栏测试覆盖
|
||||
// （backend runmgr multi_instance_test），进程级不空等 15 分钟。
|
||||
code, body := e.raw(t, uc.apiB, "POST", "/api/conversations/"+convID+"/messages", map[string]any{"text": "too early"})
|
||||
if code != http.StatusConflict {
|
||||
t.Fatalf("submit inside dead owner's lease: status %d (want 409), body %.200s", code, body)
|
||||
}
|
||||
// 死实例不拖累其他会话：B 在新会话上正常推进（公平有界，R8）。
|
||||
var conv2 map[string]any
|
||||
e.do(t, uc.apiB, "POST", "/api/projects/"+projID+"/conversations", map[string]any{"title": "不受拖累"}, &conv2)
|
||||
e.do(t, uc.apiB, "POST", "/api/conversations/"+conv2["id"].(string)+"/messages", map[string]any{"text": "B unaffected"}, nil)
|
||||
// 整机失效恢复剧本：原实例以同一身份重启，启动对账把遗留 run 收口
|
||||
// （interrupted），随后原实例重新占用会话并推进。
|
||||
e.a = e.startInstance(t, e.a.name, strings.TrimPrefix(e.a.base, "http://"))
|
||||
var detail map[string]any
|
||||
deadline := time.Now().Add(httpWait)
|
||||
for time.Now().Before(deadline) {
|
||||
e.do(t, uc.api, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
if runs, ok := detail["runs"].([]any); ok && len(runs) > 0 {
|
||||
last, _ := runs[len(runs)-1].(map[string]any)
|
||||
if last["status"] == "interrupted" {
|
||||
break
|
||||
}
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
e.do(t, uc.api, "POST", "/api/conversations/"+convID+"/messages", map[string]any{"text": "A resumes after restart"}, nil)
|
||||
})
|
||||
|
||||
t.Run("restart_invariants", func(t *testing.T) {
|
||||
e.killNine(t, e.b)
|
||||
e.b = e.startInstance(t, e.b.name, strings.TrimPrefix(e.b.base, "http://"))
|
||||
|
||||
var detail map[string]any
|
||||
e.do(t, uc.apiB, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
msgs, _ := detail["messages"].([]any)
|
||||
if len(msgs) < 5 {
|
||||
t.Fatalf("messages lost after restart: %d", len(msgs))
|
||||
}
|
||||
lastSeq := e.lastSeq(t, uc.apiB, convID)
|
||||
if lastSeq == 0 {
|
||||
t.Fatal("event seq lost after restart")
|
||||
}
|
||||
// SSE 从 0 全量重放：seq 严格连续，无洞（R5 不变量在进程失效后的视图）。
|
||||
events := e.subscribe(t, uc.streamB, convID, 0)
|
||||
seen := int64(0)
|
||||
deadline := time.After(15 * time.Second)
|
||||
for seen < lastSeq {
|
||||
select {
|
||||
case ev, ok := <-events:
|
||||
if !ok {
|
||||
t.Fatalf("replay stream closed early at seq %d (want up to %d)", seen, lastSeq)
|
||||
}
|
||||
if ev.seq != seen+1 {
|
||||
t.Fatalf("replay gap: got seq %d after %d", ev.seq, seen)
|
||||
}
|
||||
seen = ev.seq
|
||||
case <-deadline:
|
||||
t.Fatalf("replay stalled at seq %d / %d", seen, lastSeq)
|
||||
}
|
||||
}
|
||||
// 重启后写入仍可推进。
|
||||
e.postMessage(t, uc.apiB, convID, "after restart")
|
||||
})
|
||||
}
|
||||
|
||||
// ---- 环境与进程 ----
|
||||
|
||||
func newEnv(t *testing.T) *env {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("AGENT_PG_TEST_DSN")
|
||||
admin, err := sql.Open("pgx", dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dbName := "mi_" + strconv.Itoa(os.Getpid()) + "_" + fmt.Sprint(time.Now().UnixNano()%1e6)
|
||||
if _, err := admin.Exec("CREATE DATABASE " + dbName); err != nil {
|
||||
admin.Close()
|
||||
t.Fatalf("create test database: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = admin.Exec("DROP DATABASE IF EXISTS " + dbName + " WITH (FORCE)")
|
||||
admin.Close()
|
||||
})
|
||||
testDSN, err := swapDB(dsn, dbName)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
return &env{t: t, dbName: dbName, dbDSN: testDSN, artsDir: filepath.Join(root, "artifacts"), bin: os.Getenv("AGENT_BIN")}
|
||||
}
|
||||
|
||||
// swapDB 替换 DSN 的库名，保留其余参数（sslmode 等）。
|
||||
func swapDB(dsn, dbName string) (string, error) {
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
u.Path = "/" + dbName
|
||||
return u.String(), nil
|
||||
}
|
||||
|
||||
func freePort(t *testing.T) string {
|
||||
t.Helper()
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer l.Close()
|
||||
_, port, _ := net.SplitHostPort(l.Addr().String())
|
||||
return port
|
||||
}
|
||||
|
||||
func (e *env) startPair() {
|
||||
e.t.Helper()
|
||||
e.a = e.startInstance(e.t, "miA", "127.0.0.1:"+freePort(e.t))
|
||||
e.b = e.startInstance(e.t, "miB", "127.0.0.1:"+freePort(e.t))
|
||||
}
|
||||
|
||||
func (e *env) startInstance(t *testing.T, name, addr string) *instance {
|
||||
t.Helper()
|
||||
cmd := exec.Command(e.bin, "serve",
|
||||
"--dsn", e.dbDSN,
|
||||
"--addr", addr,
|
||||
"--tier", "team",
|
||||
"--auth", "on",
|
||||
"--mock",
|
||||
"--instance", name,
|
||||
"--artifacts", e.artsDir,
|
||||
"--disable-exec-tools",
|
||||
)
|
||||
log := &lockedBuffer{}
|
||||
cmd.Stdout, cmd.Stderr = log, log
|
||||
// 独立 cwd：serve 会向 ./data 写 skills/SQLite 等，不能落在测试目录。
|
||||
cmd.Dir = t.TempDir()
|
||||
if err := cmd.Start(); err != nil {
|
||||
t.Fatalf("start %s: %v", name, err)
|
||||
}
|
||||
inst := &instance{name: name, base: "http://" + addr, cmd: cmd, log: log}
|
||||
t.Cleanup(func() { e.stopInstance(inst) })
|
||||
e.waitHealthy(t, inst)
|
||||
return inst
|
||||
}
|
||||
|
||||
func (e *env) waitHealthy(t *testing.T, inst *instance) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(60 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
resp, err := http.Get(inst.base + "/api/health")
|
||||
if err == nil {
|
||||
resp.Body.Close()
|
||||
// auth=on 时 health 同样要求鉴权：401 = 服务已就绪并正确应答挑战。
|
||||
if resp.StatusCode == http.StatusOK || resp.StatusCode == http.StatusUnauthorized {
|
||||
return
|
||||
}
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("instance %s not healthy in 60s; log:\n%s", inst.name, inst.log.String())
|
||||
}
|
||||
|
||||
func (e *env) stopInstance(inst *instance) {
|
||||
if inst.cmd.Process == nil || inst.cmd.ProcessState != nil {
|
||||
return
|
||||
}
|
||||
_ = inst.cmd.Process.Signal(syscall.SIGTERM)
|
||||
done := make(chan struct{})
|
||||
go func() { _ = inst.cmd.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(10 * time.Second):
|
||||
_ = inst.cmd.Process.Kill()
|
||||
<-done
|
||||
}
|
||||
}
|
||||
|
||||
func (e *env) killNine(t *testing.T, inst *instance) {
|
||||
t.Helper()
|
||||
if inst.cmd.Process == nil {
|
||||
t.Fatal("no process to kill")
|
||||
}
|
||||
if err := inst.cmd.Process.Signal(syscall.SIGKILL); err != nil {
|
||||
t.Fatalf("SIGKILL %s: %v", inst.name, err)
|
||||
}
|
||||
_ = inst.cmd.Wait()
|
||||
}
|
||||
|
||||
// ---- HTTP 客户端 ----
|
||||
|
||||
// loginAlice 引导 alice 账号并发放 API key（CLI 直连测试库；key 存共享 PG，
|
||||
// 对两实例同时有效——程序化验收的首选凭证，SSE 亦可用 ?key= 兜底）。
|
||||
func (e *env) loginAlice(t *testing.T) *userClient {
|
||||
t.Helper()
|
||||
e.cli(t, "user", "create", "--dsn", e.dbDSN, "--owner", aliceOwner, "--name", "Alice", "--password", alicePass)
|
||||
key := e.cliOut(t, "apikey", "create", "--dsn", e.dbDSN, "--owner", aliceOwner, "--name", "mi-acceptance")
|
||||
uc := &userClient{
|
||||
api: &http.Client{Timeout: httpWait, Transport: bearerRT{key: key}},
|
||||
apiB: &http.Client{Timeout: httpWait, Transport: bearerRT{key: key}},
|
||||
streamB: &http.Client{Transport: bearerRT{key: key}}, // SSE 无整体超时；chan 背压控制
|
||||
}
|
||||
// --mock 只写 OwnerDefault 的设置；alice 是独立租户，须显式开 mock
|
||||
// （selectConversationRunner 按 EffectiveSettings 分流），否则 run 走
|
||||
// LLMRunner 且无模型配置直接失败。
|
||||
e.do(t, uc.api, "PUT", "/api/settings/user", map[string]any{"runtime": map[string]any{"mock": true}}, nil)
|
||||
return uc
|
||||
}
|
||||
|
||||
// bearerRT 为所有请求附加 Authorization: Bearer <key>。
|
||||
type bearerRT struct{ key string }
|
||||
|
||||
func (b bearerRT) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
r2 := r.Clone(r.Context())
|
||||
r2.Header.Set("Authorization", "Bearer "+b.key)
|
||||
return http.DefaultTransport.RoundTrip(r2)
|
||||
}
|
||||
|
||||
// cliOut 运行 CLI 并取 stdout 首行（apikey create 的明文仅此一次输出）。
|
||||
func (e *env) cliOut(t *testing.T, args ...string) string {
|
||||
t.Helper()
|
||||
cmd := exec.Command(e.bin, args...)
|
||||
cmd.Dir = t.TempDir()
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
t.Fatalf("cli %v…: %v\n%s", args[:2], err, stderr.String())
|
||||
}
|
||||
line := strings.TrimSpace(stdout.String())
|
||||
if line == "" {
|
||||
t.Fatalf("cli %v…: empty stdout", args[:2])
|
||||
}
|
||||
return strings.Fields(line)[0]
|
||||
}
|
||||
|
||||
func (e *env) cli(t *testing.T, args ...string) {
|
||||
t.Helper()
|
||||
cmd := exec.Command(e.bin, args...)
|
||||
cmd.Dir = t.TempDir()
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("cli %v…: %v\n%s", args[:2], err, out)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *env) do(t *testing.T, c *http.Client, method, path string, body any, out any) {
|
||||
t.Helper()
|
||||
code, raw := e.raw(t, c, method, path, body)
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("%s %s: status %d, body %.300s", method, path, code, raw)
|
||||
}
|
||||
if out != nil {
|
||||
if err := json.Unmarshal([]byte(raw), out); err != nil {
|
||||
t.Fatalf("%s %s: decode %v", method, path, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// raw 以 JSON body 发起请求，返回状态码与响应体文本。请求地址取
|
||||
// client 所绑定实例（由调用方选择 api / apiB / streamB 及对应 base）。
|
||||
func (e *env) raw(t *testing.T, c *http.Client, method, path string, body any) (int, string) {
|
||||
t.Helper()
|
||||
return e.rawBase(t, c, e.baseFor(c), method, path, body)
|
||||
}
|
||||
|
||||
func (e *env) rawBase(t *testing.T, c *http.Client, base, method, path string, body any) (int, string) {
|
||||
t.Helper()
|
||||
var rd io.Reader
|
||||
if body != nil {
|
||||
buf, _ := json.Marshal(body)
|
||||
rd = bytes.NewReader(buf)
|
||||
}
|
||||
req, err := http.NewRequest(method, base+path, rd)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", method, path, err)
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", method, path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
out, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
return resp.StatusCode, string(out)
|
||||
}
|
||||
|
||||
// baseFor 按 client 身份路由实例：B 侧（apiB/streamB）→ B；其余 → A。
|
||||
func (e *env) baseFor(c *http.Client) string {
|
||||
if e.uc != nil && (c == e.uc.apiB || c == e.uc.streamB) {
|
||||
return e.b.base
|
||||
}
|
||||
return e.a.base
|
||||
}
|
||||
|
||||
// ---- 场景辅助 ----
|
||||
|
||||
type sseEvent struct {
|
||||
seq int64
|
||||
event string
|
||||
data string
|
||||
}
|
||||
|
||||
func (e *env) setupConversation(t *testing.T, uc *userClient) (string, string) {
|
||||
t.Helper()
|
||||
var proj map[string]any
|
||||
e.do(t, uc.api, "POST", "/api/projects", map[string]any{"name": "mi-验收"}, &proj)
|
||||
projID, _ := proj["id"].(string)
|
||||
var conv map[string]any
|
||||
e.do(t, uc.api, "POST", "/api/projects/"+projID+"/conversations", map[string]any{"title": "双进程"}, &conv)
|
||||
convID, _ := conv["id"].(string)
|
||||
return projID, convID
|
||||
}
|
||||
|
||||
// postMessage 先等会话空闲（上一 run 终态），再提交消息——#hitl 需要一个
|
||||
// 可占用的空闲会话；普通消息同样避免撞上上一 run 的收口窗口。
|
||||
func (e *env) postMessage(t *testing.T, c *http.Client, convID, text string) {
|
||||
t.Helper()
|
||||
e.waitConversationIdle(t, c, convID, 20*time.Second)
|
||||
e.do(t, c, "POST", "/api/conversations/"+convID+"/messages", map[string]any{"text": text}, nil)
|
||||
}
|
||||
|
||||
func (e *env) waitConversationIdle(t *testing.T, c *http.Client, convID string, within time.Duration) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(within)
|
||||
for time.Now().Before(deadline) {
|
||||
var detail map[string]any
|
||||
e.do(t, c, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
busy := false
|
||||
if runs, ok := detail["runs"].([]any); ok {
|
||||
for _, r := range runs {
|
||||
m, _ := r.(map[string]any)
|
||||
switch m["status"] {
|
||||
case "pending", "running", "waiting_hitl":
|
||||
busy = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !busy {
|
||||
return
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("conversation %s still busy after %v", convID, within)
|
||||
}
|
||||
|
||||
func (e *env) lastSeq(t *testing.T, c *http.Client, convID string) int64 {
|
||||
t.Helper()
|
||||
var detail map[string]any
|
||||
e.do(t, c, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
n, _ := detail["lastSeq"].(float64)
|
||||
return int64(n)
|
||||
}
|
||||
|
||||
// subscribe 在 B 实例打开会话 SSE（since 起），帧 seq > since 才投递；
|
||||
// 返回通道在流结束/出错时关闭。调用方通过关闭底层响应取消。
|
||||
func (e *env) subscribe(t *testing.T, c *http.Client, convID string, since int64) <-chan sseEvent {
|
||||
t.Helper()
|
||||
u := e.b.base + "/api/events?conversationId=" + convID + "&since=" + strconv.FormatInt(since, 10)
|
||||
req, err := http.NewRequest(http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("subscribe SSE: %v", err)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
resp.Body.Close()
|
||||
t.Fatalf("subscribe SSE: status %d", resp.StatusCode)
|
||||
}
|
||||
ch := make(chan sseEvent, 1024)
|
||||
go func() {
|
||||
defer close(ch)
|
||||
defer resp.Body.Close()
|
||||
sc := bufio.NewScanner(resp.Body)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 4<<20)
|
||||
var seq int64
|
||||
var event, data string
|
||||
flush := func() {
|
||||
if seq > since {
|
||||
// 阻塞发送：restart 场景依赖全量重放断言，不允许丢帧；
|
||||
// 消费停滞时读取 goroutine 挂起（进程随测试退出回收）。
|
||||
ch <- sseEvent{seq: seq, event: event, data: data}
|
||||
}
|
||||
seq, event, data = 0, "", ""
|
||||
}
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
switch {
|
||||
case line == "":
|
||||
if seq > 0 {
|
||||
flush()
|
||||
}
|
||||
case strings.HasPrefix(line, "id: "):
|
||||
seq, _ = strconv.ParseInt(strings.TrimPrefix(line, "id: "), 10, 64)
|
||||
case strings.HasPrefix(line, "event: "):
|
||||
event = strings.TrimPrefix(line, "event: ")
|
||||
case strings.HasPrefix(line, "data: "):
|
||||
data = strings.TrimPrefix(line, "data: ")
|
||||
}
|
||||
}
|
||||
}()
|
||||
return ch
|
||||
}
|
||||
|
||||
func (e *env) waitActiveHITL(t *testing.T, c *http.Client, convID string) hitlCard {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(15 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
var detail map[string]any
|
||||
e.do(t, c, "GET", "/api/conversations/"+convID, nil, &detail)
|
||||
if cards, ok := detail["activeHitl"].([]any); ok && len(cards) > 0 {
|
||||
m, _ := cards[0].(map[string]any)
|
||||
id, _ := m["id"].(string)
|
||||
runID, _ := m["runId"].(string)
|
||||
if id != "" && runID != "" {
|
||||
return hitlCard{id: id, runID: runID}
|
||||
}
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("no active HITL card in 15s; logA:\n%s", e.a.log.String())
|
||||
return hitlCard{}
|
||||
}
|
||||
|
||||
func (e *env) waitRunStatus(t *testing.T, c *http.Client, runID, status string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(httpWait)
|
||||
for time.Now().Before(deadline) {
|
||||
var out map[string]any
|
||||
e.do(t, c, "GET", "/api/runs/"+runID, nil, &out)
|
||||
run, _ := out["run"].(map[string]any)
|
||||
if s, _ := run["status"].(string); s == status {
|
||||
return
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("run %s not %s within %v", runID, status, httpWait)
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
/.build/
|
||||
/results/
|
||||
@@ -0,0 +1,50 @@
|
||||
# 协作存储压测套件（U7）
|
||||
|
||||
方案 §3 负载合同 / §5 U7 的业务压测入口。**只打隔离栈**（自建 PG 实例 +
|
||||
mock 模式 backend），绝不打生产服务；模拟模型（mock）用于容量基线，真实
|
||||
模型的全链路验证按方案另行小流量执行。
|
||||
|
||||
## 组成
|
||||
|
||||
| 文件 | 职责 |
|
||||
|---|---|
|
||||
| `run_load.sh` | 编排入口：起隔离栈 → 引导账号 → k6 阶梯 → SSE 探针 → 不变量对账 → 回收 |
|
||||
| `k6/collab_api.js` | 非模型普通 API 的混合读写负载（登录/项目/会话/卡片/设置/消息，requestId 幂等重放抽样；预期 409/429 单列计数） |
|
||||
| `sse_probe/` | SSE 持久事件可见延迟探针（Go）：k6 核心不支持 SSE 流式读取，P95≤1s 指标由此承担，含 seq 单调性断言 |
|
||||
| `results/<ts>/` | 每次运行的原始输出：`k6-summary.json`、`k6-output.txt`、`sse-probe.json`、`invariants.txt`、`app.log` |
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
# 冒烟（~2 分钟，无需 k6——缺失时自动跳过 k6 段并给出安装指引）
|
||||
tests/load/collaboration/run_load.sh --profile smoke
|
||||
|
||||
# 阶梯（10→30→100 VU，~12 分钟；k6: brew install k6）
|
||||
tests/load/collaboration/run_load.sh --profile ladder --users 30
|
||||
|
||||
# 长稳（60 分钟 / 2 小时混合，负载合同的两个正式窗口）
|
||||
tests/load/collaboration/run_load.sh --profile sustained60m
|
||||
tests/load/collaboration/run_load.sh --profile sustained2h
|
||||
|
||||
# 复用已有隔离 PG；保留栈便于复查
|
||||
tests/load/collaboration/run_load.sh --dsn 'postgresql://…/agent' --keep
|
||||
```
|
||||
|
||||
## 阈值与口径（负载合同，待实测校准）
|
||||
|
||||
- 非模型普通 API：读路径 P95≤300ms、P99≤1s（k6 threshold 按 `read` 组单列）。
|
||||
- 非预期失败率≤0.1%（`http_req_failed<0.001`）；预期拥塞（会话忙 409 /
|
||||
队列满 429）记入 `expected_contention` 计数器，**不**计入失败。
|
||||
- SSE 持久事件可见延迟 P95≤1s + seq 零乱序（探针退出码即断言）。
|
||||
- 数据不变量零违反：events 每会话 seq 无重复（SQL 对账，违反即 exit 1）；
|
||||
各表计数与孤儿 run 报告落 `invariants.txt`。
|
||||
|
||||
## 边界（诚实声明）
|
||||
|
||||
- 首期为**单实例**压测（mock 模式、执行原语关闭）；分布式限流窗口与双实例
|
||||
放行指标在双栈就位后扩展（`--dsn` 复用 + 第二实例待加）。
|
||||
- 容量数字（连接等待、CPU/内存/IO、pgbench 数据库基线）须绑定组件 SHA、
|
||||
配置与数据集记录在结果目录；`pgbench` 只测数据库能力，不替代本套业务
|
||||
验收（方案 §5 U7）。
|
||||
- 正式 60min/2h 窗口的执行与双主机验证需要用户指定资源与时间；本仓提供
|
||||
的是可复核入口，不预填容量结论。
|
||||
@@ -0,0 +1,137 @@
|
||||
// 协作存储 U7 压测（方案 §3 负载合同 / §5 U7）：k6 覆盖非模型普通 API
|
||||
// 的吞吐与延迟阶梯；SSE 可见延迟 P95 由 Go 探针（../sse_probe）承担——
|
||||
// k6 核心不支持 SSE 流式读取，两者分工记录在同一份结果目录。
|
||||
//
|
||||
// 目标实例：run_load.sh 起的隔离栈（mock 模式 backend + 隔离 PG），绝不打
|
||||
// 生产服务。预期限流/冲突（409/429）单列计数，不计入失败。
|
||||
//
|
||||
// 阶梯与环境变量（由 run_load.sh 注入）：
|
||||
// BASE_URL 目标基址（默认 http://127.0.0.1:18980）
|
||||
// ADMIN_* 引导管理员（run_load.sh 预建；setup 里登录后建压测用户）
|
||||
// USERS 压测用户数（默认 30；每 VU 绑定一个用户）
|
||||
// PROFILE smoke | ladder | sustained60m | sustained2h（默认 smoke）
|
||||
//
|
||||
// 阈值（负载合同，待实测后校准）：
|
||||
// 读路径 p(95)<300ms、全量 p(99)<1s、非预期失败率<0.1%。
|
||||
|
||||
import http from 'k6/http';
|
||||
import { check, group, sleep } from 'k6';
|
||||
import { Counter, Trend } from 'k6/metrics';
|
||||
import exec from 'k6/execution';
|
||||
|
||||
const BASE = __ENV.BASE_URL || 'http://127.0.0.1:18980';
|
||||
const ADMIN_OWNER = __ENV.ADMIN_OWNER || 'load_admin';
|
||||
const ADMIN_PASS = __ENV.ADMIN_PASS || 'load-admin-pass';
|
||||
const USERS = parseInt(__ENV.USERS || '30', 10);
|
||||
const PROFILE = __ENV.PROFILE || 'smoke';
|
||||
|
||||
// 预期拥塞（负载合同要求单列，不算失败）：409 会话忙 / 429 队列满。
|
||||
const expectedContention = new Counter('expected_contention');
|
||||
const msgLatency = new Trend('message_roundtrip', true);
|
||||
|
||||
const stages = {
|
||||
smoke: [{ duration: '30s', target: 10 }],
|
||||
ladder: [
|
||||
{ duration: '2m', target: 10 },
|
||||
{ duration: '3m', target: 30 },
|
||||
{ duration: '5m', target: 100 },
|
||||
{ duration: '2m', target: 0 },
|
||||
],
|
||||
sustained60m: [
|
||||
{ duration: '2m', target: 30 },
|
||||
{ duration: '56m', target: 30 },
|
||||
{ duration: '2m', target: 0 },
|
||||
],
|
||||
sustained2h: [
|
||||
{ duration: '3m', target: 30 },
|
||||
{ duration: '114m', target: 30 },
|
||||
{ duration: '3m', target: 0 },
|
||||
],
|
||||
};
|
||||
|
||||
export const options = {
|
||||
stages: stages[PROFILE] || stages.smoke,
|
||||
thresholds: {
|
||||
// 非预期失败：2xx 之外的（409/429 已在 expected_contention 单列）。
|
||||
http_req_failed: ['rate<0.001'],
|
||||
'http_req_duration{group:::read}': ['p(95)<300', 'p(99)<1000'],
|
||||
http_req_duration: ['p(99)<1000'],
|
||||
},
|
||||
summaryTrendStats: ['avg', 'p(50)', 'p(95)', 'p(99)', 'max'],
|
||||
};
|
||||
|
||||
// ---- setup：管理员建压测用户，各自建项目与会话（数据集随用户数扩展） ----
|
||||
|
||||
export function setup() {
|
||||
// setup 上下文的 cookie jar 自动附着（k6 语义：jar 不作为请求器使用）。
|
||||
const login = http.post(`${BASE}/api/auth/login`, JSON.stringify({ owner: ADMIN_OWNER, password: ADMIN_PASS }), { headers: { 'Content-Type': 'application/json' } });
|
||||
if (login.status !== 200) throw new Error(`admin login: ${login.status} ${login.body}`);
|
||||
// 首登强制改密由 run_load.sh 预处理（CLI 建号即带终态口令），此处直接可用。
|
||||
|
||||
const users = [];
|
||||
for (let i = 0; i < USERS; i++) {
|
||||
const owner = `load_u${i}`;
|
||||
const password = `load-user-pass-${i}-Aa9`;
|
||||
const created = http.post(`${BASE}/api/admin/users`, JSON.stringify({ owner, name: `压测用户${i}`, password, isAdmin: false }), { headers: { 'Content-Type': 'application/json' } });
|
||||
// 409 = 已存在（重复压测同一栈），接受。
|
||||
if (created.status !== 200 && created.status !== 409) throw new Error(`create user ${owner}: ${created.status} ${created.body}`);
|
||||
users.push({ owner, password });
|
||||
}
|
||||
return { users };
|
||||
}
|
||||
|
||||
// ---- default：每 VU 一个用户身份，混合读写 ----
|
||||
|
||||
export default function (data) {
|
||||
const u = data.users[exec.scenario.iterationInTest % data.users.length];
|
||||
// VU 上下文的 cookie jar 自动附着；每 VU 绑定一个用户身份。
|
||||
const login = http.post(`${BASE}/api/auth/login`, JSON.stringify({ owner: u.owner, password: u.password }), { headers: { 'Content-Type': 'application/json' } });
|
||||
if (login.status !== 200) {
|
||||
check(null, { 'user login': () => false });
|
||||
return;
|
||||
}
|
||||
// 会话/项目绑定 VU（VU 内多次迭代复用同一身份的会话，形成热点会话）。
|
||||
if (!u.__ctx) {
|
||||
const proj = http.post(`${BASE}/api/projects`, JSON.stringify({ name: `压测-${u.owner}` }), { headers: { 'Content-Type': 'application/json' } });
|
||||
const conv = http.post(`${BASE}/api/projects/${proj.json().id}/conversations`, JSON.stringify({ title: '压测会话' }), { headers: { 'Content-Type': 'application/json' } });
|
||||
u.__ctx = { projectID: proj.json().id, convID: conv.json().id };
|
||||
}
|
||||
const { projectID, convID } = u.__ctx;
|
||||
|
||||
group('read', () => {
|
||||
const projs = http.get(`${BASE}/api/projects`);
|
||||
check(projs, { 'list projects 2xx': (r) => r.status === 200 });
|
||||
const detail = http.get(`${BASE}/api/conversations/${convID}`);
|
||||
check(detail, { 'conversation detail 2xx': (r) => r.status === 200 });
|
||||
const cards = http.get(`${BASE}/api/conversations/${convID}/cards`);
|
||||
check(cards, { 'cards 2xx': (r) => r.status === 200 });
|
||||
const settings = http.get(`${BASE}/api/settings`);
|
||||
check(settings, { 'settings 2xx': (r) => r.status === 200 });
|
||||
});
|
||||
|
||||
group('write', () => {
|
||||
const requestId = `req-${u.owner}-${exec.scenario.iterationInTest}`;
|
||||
const body = JSON.stringify({ text: `压测消息 ${Date.now()}`, requestId });
|
||||
const t0 = Date.now();
|
||||
const sent = http.post(`${BASE}/api/conversations/${convID}/messages`, body, { headers: { 'Content-Type': 'application/json' } });
|
||||
if (sent.status === 200) {
|
||||
msgLatency.add(Date.now() - t0);
|
||||
const id = sent.json().message && sent.json().message.id;
|
||||
check(sent, { 'message has id': () => !!id });
|
||||
// 幂等重放（矩阵行：同 requestId 返回同一逻辑结果）：抽样 10%。
|
||||
if (exec.scenario.iterationInTest % 10 === 0) {
|
||||
const replay = http.post(`${BASE}/api/conversations/${convID}/messages`, body, { headers: { 'Content-Type': 'application/json' } });
|
||||
check(replay, {
|
||||
'requestId replay same message': (r) => r.status === 200 && r.json().message && r.json().message.id === id,
|
||||
});
|
||||
}
|
||||
} else if (sent.status === 409 || sent.status === 429) {
|
||||
// 热点会话忙/队列满：预期拥塞，单列。
|
||||
expectedContention.add(1);
|
||||
} else {
|
||||
check(sent, { 'message send 2xx': () => false });
|
||||
}
|
||||
});
|
||||
|
||||
sleep(1);
|
||||
}
|
||||
Executable
+143
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env bash
|
||||
# 协作存储 U7 压测入口（方案 §3 负载合同 / §5 U7）。
|
||||
#
|
||||
# 起一个隔离栈（backend mock 单实例 + 隔离 PG）→ 预建引导账号 → k6 阶梯
|
||||
# → SSE 可见延迟探针 → 数据不变量对账 → 结果落 results/<ts>/ → 回收。
|
||||
# 绝不打生产服务：默认自建 PG 实例（svc.sh）；--dsn 复用已有隔离实例。
|
||||
#
|
||||
# 用法：
|
||||
# tests/load/collaboration/run_load.sh [--profile smoke|ladder|sustained60m|sustained2h]
|
||||
# [--users N] [--dsn <PG DSN>] [--port <app port>]
|
||||
# [--keep] [--skip-k6] [--skip-probe] [--help]
|
||||
# 依赖：bash4、go（构建 backend/探针）、psql（对账）、curl；
|
||||
# k6 可选（缺失时跳过 k6 段并给出安装指引，探针与对账照跑）。
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
HERE="$ROOT/tests/load/collaboration"
|
||||
PROFILE=smoke USERS=30 APP_PORT=18980 PG_PORT=18981 KEEP=0 SKIP_K6=0 SKIP_PROBE=0
|
||||
DSN=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--profile) PROFILE="$2"; shift 2 ;;
|
||||
--users) USERS="$2"; shift 2 ;;
|
||||
--dsn) DSN="$2"; shift 2 ;;
|
||||
--port) APP_PORT="$2"; shift 2 ;;
|
||||
--keep) KEEP=1; shift ;;
|
||||
--skip-k6) SKIP_K6=1; shift ;;
|
||||
--skip-probe) SKIP_PROBE=1; shift ;;
|
||||
-h|--help) sed -n '2,16p' "$0"; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
TS="$(date +%Y%m%d-%H%M%S)"
|
||||
RESULTS="$HERE/results/$TS"
|
||||
mkdir -p "$RESULTS"
|
||||
OWNED_PG=0
|
||||
APP_PID=""
|
||||
cleanup() {
|
||||
[[ -n "$APP_PID" ]] && kill "$APP_PID" 2>/dev/null || true
|
||||
if [[ $OWNED_PG -eq 1 && $KEEP -eq 0 ]]; then
|
||||
"$ROOT/scripts/svc.sh" u7load stop >/dev/null 2>&1 || true
|
||||
"$ROOT/scripts/svc.sh" rm u7load --purge >/dev/null 2>&1 || true
|
||||
fi
|
||||
if [[ $KEEP -eq 1 ]]; then
|
||||
echo "[load] --keep：栈保留（PG=u7load:$PG_PORT，app=127.0.0.1:$APP_PORT，结果 $RESULTS）"
|
||||
fi
|
||||
return 0 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# ---- 1) PG：自建隔离实例（或复用 --dsn） ----
|
||||
if [[ -z "$DSN" ]]; then
|
||||
echo "[load] 创建隔离 PG 实例 u7load（端口 $PG_PORT）"
|
||||
"$ROOT/scripts/svc.sh" db create u7load --port "$PG_PORT" >/dev/null
|
||||
OWNED_PG=1
|
||||
"$ROOT/scripts/svc.sh" u7load start >/dev/null
|
||||
DSN="$("$ROOT/scripts/svc.sh" db dsn u7load --reveal)"
|
||||
fi
|
||||
echo "[load] PG DSN 已就绪（密码不回显）"
|
||||
|
||||
# ---- 2) 构建 backend 与探针 ----
|
||||
echo "[load] 构建 backend（worktree 当前源码）与 SSE 探针"
|
||||
BIN="$HERE/.build/agent-system"
|
||||
mkdir -p "$HERE/.build"
|
||||
( cd "$ROOT/modules/backend" && CGO_ENABLED=0 go build -o "$BIN" ./cmd/agent-system )
|
||||
( cd "$HERE/sse_probe" && go build -o "$HERE/.build/sse_probe" . )
|
||||
|
||||
# ---- 3) 压测库 + 引导账号（CLI 直连，终态口令避免首登改密分支） ----
|
||||
# DSN 规范化：库名换 u7load_app、query 补 sslmode=disable（密码不回显）。
|
||||
ADMIN_DSN="$(python3 - "$DSN" <<'PY'
|
||||
import sys
|
||||
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
|
||||
u = urlsplit(sys.argv[1])
|
||||
q = dict(parse_qsl(u.query))
|
||||
q.setdefault("sslmode", "disable")
|
||||
print(urlunsplit((u.scheme, u.netloc, "/" + (u.path.lstrip("/") or "postgres"), urlencode(q), "")))
|
||||
PY
|
||||
)"
|
||||
psql "$ADMIN_DSN" -v ON_ERROR_STOP=1 -c "DROP DATABASE IF EXISTS u7load_app WITH (FORCE)" -c "CREATE DATABASE u7load_app" >/dev/null
|
||||
APP_DSN="$(python3 - "$ADMIN_DSN" <<'PY'
|
||||
import sys
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
u = urlsplit(sys.argv[1])
|
||||
print(urlunsplit((u.scheme, u.netloc, "/u7load_app", u.query, "")))
|
||||
PY
|
||||
)"
|
||||
"$BIN" user create --dsn "$APP_DSN" --admin --owner load_admin --password load-admin-pass-Aa1 >/dev/null
|
||||
"$BIN" user create --dsn "$APP_DSN" --owner probe_user --password probe-user-pass-Aa9 >/dev/null
|
||||
|
||||
# ---- 4) 起被测实例（mock、单实例首期；执行原语关闭——压测不打执行面） ----
|
||||
echo "[load] 启动被测实例 127.0.0.1:$APP_PORT（mock / tier=team / auth=on）"
|
||||
APP_LOG="$RESULTS/app.log"
|
||||
"$BIN" serve --dsn "$APP_DSN" --addr "127.0.0.1:$APP_PORT" --tier team --auth on --mock \
|
||||
--instance u7load --disable-exec-tools >"$APP_LOG" 2>&1 &
|
||||
APP_PID=$!
|
||||
# auth=on 时 /api/health 同样要求鉴权：200/401 都视为已就绪（服务在正确应答）。
|
||||
app_ready() { code=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$APP_PORT/api/health"); [[ "$code" == 200 || "$code" == 401 ]]; }
|
||||
for _ in $(seq 1 60); do app_ready && break; sleep 0.5; done
|
||||
app_ready || { echo "[load] app 未就绪，日志 $APP_LOG" >&2; exit 1; }
|
||||
|
||||
BASE="http://127.0.0.1:$APP_PORT"
|
||||
|
||||
# ---- 5) k6 阶梯 ----
|
||||
if [[ $SKIP_K6 -eq 1 ]]; then
|
||||
echo "[load] --skip-k6：跳过 k6 段"
|
||||
elif ! command -v k6 >/dev/null 2>&1; then
|
||||
cat >"$RESULTS/k6-skipped.txt" <<'EOF'
|
||||
k6 未安装：本段跳过（探针与对账照跑）。
|
||||
安装（本机 linuxbrew）：brew install k6
|
||||
安装后重跑：tests/load/collaboration/run_load.sh --profile <同参数>
|
||||
EOF
|
||||
echo "[load] k6 未安装，跳过（见 $RESULTS/k6-skipped.txt）"
|
||||
else
|
||||
echo "[load] k6 profile=$PROFILE users=$USERS"
|
||||
( cd "$HERE" && BASE_URL="$BASE" ADMIN_OWNER=load_admin ADMIN_PASS=load-admin-pass-Aa1 \
|
||||
USERS="$USERS" PROFILE="$PROFILE" \
|
||||
k6 run --summary-export "$RESULTS/k6-summary.json" k6/collab_api.js ) | tee "$RESULTS/k6-output.txt"
|
||||
fi
|
||||
|
||||
# ---- 6) SSE 可见延迟探针 ----
|
||||
if [[ $SKIP_PROBE -eq 0 ]]; then
|
||||
echo "[load] SSE 可见延迟探针（60s，10 会话）"
|
||||
"$HERE/.build/sse_probe" -base "$BASE" -dur 60s -convs 10 -out "$RESULTS/sse-probe.json"
|
||||
fi
|
||||
|
||||
# ---- 7) 数据不变量对账（方案 §3：零违反） ----
|
||||
echo "[load] 数据不变量对账"
|
||||
psql "$APP_DSN" -v ON_ERROR_STOP=1 \
|
||||
-c "SELECT 'users' k, COUNT(*) v FROM users UNION ALL SELECT 'projects', COUNT(*) FROM projects UNION ALL SELECT 'conversations', COUNT(*) FROM conversations UNION ALL SELECT 'messages', COUNT(*) FROM messages UNION ALL SELECT 'events', COUNT(*) FROM events UNION ALL SELECT 'runs', COUNT(*) FROM runs" \
|
||||
-c "SELECT 'dup_seq_per_conv' k, COUNT(*) v FROM (SELECT conversation_id, seq FROM events GROUP BY 1,2 HAVING COUNT(*)>1) d" \
|
||||
-c "SELECT 'nonterminal_orphan_runs' k, COUNT(*) v FROM runs WHERE status IN ('pending','running','waiting_hitl')" \
|
||||
| tee "$RESULTS/invariants.txt"
|
||||
|
||||
DUP=$(psql "$APP_DSN" -tAc "SELECT COUNT(*) FROM (SELECT conversation_id, seq FROM events GROUP BY 1,2 HAVING COUNT(*)>1) d")
|
||||
if [[ "$DUP" != "0" ]]; then
|
||||
echo "[load] 对账失败：events 每会话 seq 出现重复（$DUP 组）" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "[load] 对账通过：无重复 seq；孤儿 run 报告见 invariants.txt（探针正常收口应为 0）"
|
||||
|
||||
echo "[load] 完成。结果目录：$RESULTS"
|
||||
@@ -0,0 +1,3 @@
|
||||
module agent-system-dev/tests/load/collaboration/sse_probe
|
||||
|
||||
go 1.27.1
|
||||
@@ -0,0 +1,241 @@
|
||||
// 协作存储 U7：SSE 持久事件可见延迟探针（负载合同：P95≤1s）。
|
||||
// k6 核心不支持 SSE 流式读取，此项由本 Go 探针承担；与 k6 结果记录在
|
||||
// 同一目录（run_load.sh 编排）。
|
||||
//
|
||||
// 行为：并发 C 个会话各挂一个 SSE 订阅（共享登录 cookie）；写者按固定
|
||||
// 间隔轮转 POST 消息（mock 即时回合）；统计「POST 前基准 seq → 会话内
|
||||
// 新帧到达」延迟分布与 seq 单调性，输出 JSON 报告并以退出码执行
|
||||
// P95≤1s、零乱序断言（数据不变量零违反，方案 §3）。
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/cookiejar"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type sample struct {
|
||||
Conversation string `json:"conversation"`
|
||||
SendSeq int64 `json:"sendSeq"`
|
||||
LatencyMS float64 `json:"latencyMs"`
|
||||
}
|
||||
|
||||
type report struct {
|
||||
Target string `json:"target"`
|
||||
Duration string `json:"duration"`
|
||||
Subscribers int `json:"subscribers"`
|
||||
Messages int `json:"messages"`
|
||||
Samples int `json:"samples"`
|
||||
P50MS float64 `json:"p50Ms"`
|
||||
P95MS float64 `json:"p95Ms"`
|
||||
P99MS float64 `json:"p99Ms"`
|
||||
MaxMS float64 `json:"maxMs"`
|
||||
OutOfOrder int `json:"outOfOrder"` // seq 乱序/回退（不变量违反）
|
||||
ThresholdMS float64 `json:"thresholdMs"`
|
||||
Pass bool `json:"pass"`
|
||||
Detail []sample `json:"samplesDetail"`
|
||||
}
|
||||
|
||||
var (
|
||||
jar, _ = cookiejar.New(nil)
|
||||
client = &http.Client{Timeout: 10 * time.Second, Jar: jar}
|
||||
streamClient = &http.Client{Jar: jar} // SSE：无整体超时，stop 控制
|
||||
pendingMu sync.Mutex
|
||||
pending = map[string]pendingSend{}
|
||||
outOfOrderMu sync.Mutex
|
||||
outOfOrderN int
|
||||
samplesMu sync.Mutex
|
||||
samples []sample
|
||||
)
|
||||
|
||||
type pendingSend struct {
|
||||
seq int64
|
||||
sent time.Time
|
||||
}
|
||||
|
||||
func main() {
|
||||
base := flag.String("base", "http://127.0.0.1:18980", "目标实例基址")
|
||||
owner := flag.String("owner", "probe_user", "探针用户")
|
||||
password := flag.String("password", "probe-user-pass-Aa9", "探针用户口令")
|
||||
convs := flag.Int("convs", 10, "并发订阅会话数")
|
||||
every := flag.Duration("every", 500*time.Millisecond, "写者间隔（轮转各会话）")
|
||||
dur := flag.Duration("dur", 60*time.Second, "采样时长")
|
||||
out := flag.String("out", "", "JSON 报告输出路径（空 = stdout）")
|
||||
flag.Parse()
|
||||
|
||||
login(*base, *owner, *password)
|
||||
|
||||
projectID := apiJSON("POST", *base, "/api/projects", map[string]any{"name": "sse-probe"})["id"].(string)
|
||||
convIDs := make([]string, *convs)
|
||||
for i := range convIDs {
|
||||
convIDs[i] = apiJSON("POST", *base, "/api/projects/"+projectID+"/conversations", map[string]any{"title": fmt.Sprintf("probe-%d", i)})["id"].(string)
|
||||
}
|
||||
|
||||
stop := make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
for _, cid := range convIDs {
|
||||
wg.Add(1)
|
||||
go func(cid string) {
|
||||
defer wg.Done()
|
||||
subscribe(*base, cid, stop)
|
||||
}(cid)
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond) // 订阅就绪
|
||||
|
||||
deadline := time.Now().Add(*dur)
|
||||
sent := 0
|
||||
for i := 0; time.Now().Before(deadline); i++ {
|
||||
cid := convIDs[i%len(convIDs)]
|
||||
detail := apiJSON("GET", *base, "/api/conversations/"+cid, nil)
|
||||
seq, _ := detail["lastSeq"].(float64)
|
||||
pendingMu.Lock()
|
||||
pending[cid] = pendingSend{seq: int64(seq), sent: time.Now()}
|
||||
pendingMu.Unlock()
|
||||
apiJSON("POST", *base, "/api/conversations/"+cid+"/messages", map[string]any{"text": fmt.Sprintf("probe %d", i)})
|
||||
sent++
|
||||
time.Sleep(*every)
|
||||
}
|
||||
|
||||
time.Sleep(3 * time.Second) // 最后一批帧在途余量
|
||||
close(stop)
|
||||
wg.Wait()
|
||||
|
||||
r := buildReport(*base, *dur, *convs, sent)
|
||||
data, _ := json.MarshalIndent(r, "", " ")
|
||||
if *out != "" {
|
||||
if err := os.WriteFile(*out, data, 0o644); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "write report:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
fmt.Println("report:", *out)
|
||||
} else {
|
||||
fmt.Println(string(data))
|
||||
}
|
||||
if !r.Pass {
|
||||
fmt.Fprintf(os.Stderr, "sse probe FAIL: p95=%.1fms (≤%.0fms) samples=%d outOfOrder=%d\n", r.P95MS, r.ThresholdMS, r.Samples, r.OutOfOrder)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("sse probe PASS: p95=%.1fms samples=%d\n", r.P95MS, r.Samples)
|
||||
}
|
||||
|
||||
func login(base, owner, pass string) {
|
||||
body, _ := json.Marshal(map[string]string{"owner": owner, "password": pass})
|
||||
resp, err := client.Post(base+"/api/auth/login", "application/json", bytes.NewReader(body))
|
||||
if err != nil || resp.StatusCode != 200 {
|
||||
fmt.Fprintf(os.Stderr, "probe login: err=%v status=%v\n", err, statusOf(resp))
|
||||
os.Exit(2)
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
|
||||
func statusOf(resp *http.Response) string {
|
||||
if resp == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
return strconv.Itoa(resp.StatusCode)
|
||||
}
|
||||
|
||||
func apiJSON(method, base, path string, body any) map[string]any {
|
||||
var rd io.Reader
|
||||
_ = rd
|
||||
if body != nil {
|
||||
buf, _ := json.Marshal(body)
|
||||
rd = bytes.NewReader(buf)
|
||||
}
|
||||
req, _ := http.NewRequest(method, base+path, rd)
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "%s %s: %v\n", method, path, err)
|
||||
os.Exit(2)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
out := map[string]any{}
|
||||
if resp.StatusCode != 200 {
|
||||
fmt.Fprintf(os.Stderr, "%s %s: status %d\n", method, path, resp.StatusCode)
|
||||
os.Exit(2)
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "%s %s: decode %v\n", method, path, err)
|
||||
os.Exit(2)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// subscribe 读会话 SSE：seq 严格递增校验；帧到达时与写者基准配对成延迟样本。
|
||||
func subscribe(base, cid string, stop <-chan struct{}) {
|
||||
req, _ := http.NewRequest(http.MethodGet, base+"/api/events?conversationId="+cid, nil)
|
||||
resp, err := streamClient.Do(req)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
sc := bufio.NewScanner(resp.Body)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 4<<20)
|
||||
last := int64(-1)
|
||||
for sc.Scan() {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
default:
|
||||
}
|
||||
line := sc.Text()
|
||||
if !strings.HasPrefix(line, "id: ") {
|
||||
continue
|
||||
}
|
||||
seq, _ := strconv.ParseInt(strings.TrimPrefix(line, "id: "), 10, 64)
|
||||
arrived := time.Now()
|
||||
if last != -1 && seq <= last {
|
||||
outOfOrderMu.Lock()
|
||||
outOfOrderN++
|
||||
outOfOrderMu.Unlock()
|
||||
}
|
||||
last = seq
|
||||
pendingMu.Lock()
|
||||
p, ok := pending[cid]
|
||||
pendingMu.Unlock()
|
||||
if ok && seq > p.seq {
|
||||
pendingMu.Lock()
|
||||
delete(pending, cid)
|
||||
pendingMu.Unlock()
|
||||
samplesMu.Lock()
|
||||
samples = append(samples, sample{Conversation: cid, SendSeq: seq, LatencyMS: float64(arrived.Sub(p.sent)) / float64(time.Millisecond)})
|
||||
samplesMu.Unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func buildReport(base string, dur time.Duration, subs, msgs int) report {
|
||||
samplesMu.Lock()
|
||||
detail := append([]sample(nil), samples...)
|
||||
samplesMu.Unlock()
|
||||
outOfOrderMu.Lock()
|
||||
ooo := outOfOrderN
|
||||
outOfOrderMu.Unlock()
|
||||
|
||||
r := report{Target: base, Duration: dur.String(), Subscribers: subs, Messages: msgs, Samples: len(detail), OutOfOrder: ooo, ThresholdMS: 1000, Detail: detail}
|
||||
if len(detail) > 0 {
|
||||
vals := make([]float64, len(detail))
|
||||
for i, s := range detail {
|
||||
vals[i] = s.LatencyMS
|
||||
}
|
||||
sort.Float64s(vals)
|
||||
pct := func(p float64) float64 { return vals[int(float64(len(vals)-1)*p)] }
|
||||
r.P50MS, r.P95MS, r.P99MS, r.MaxMS = pct(0.50), pct(0.95), pct(0.99), vals[len(vals)-1]
|
||||
}
|
||||
r.Pass = len(detail) > 0 && r.P95MS <= r.ThresholdMS && ooo == 0
|
||||
return r
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
"""svc.sh agent 数据目录快照回归（2026-10-12）。
|
||||
|
||||
验证 backup/backups：活跃 WAL 写入者下 SQLite 一致性导出（.backup + integrity_check，
|
||||
不裸拷 .db/-wal）、快照落在 runtime/ 之外的 runtime-backups/<name>/、MANIFEST 逐文件
|
||||
sha256、--keep/--prune 保留策略只清理时间戳目录、--out 自定义目录不参与清理、
|
||||
kind 守卫（postgres 引导 db backup）、缺 sqlite3 拒绝备份、rm --purge 连带删除快照。
|
||||
全部使用临时目录 + 手工登记 svc.env（backup 不触碰 bin/进程，无需桩二进制）。
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SOURCE = Path(__file__).resolve().parents[1]
|
||||
SVC_SCRIPT = SOURCE / "scripts" / "svc.sh"
|
||||
|
||||
SQLITE3 = shutil.which("sqlite3")
|
||||
|
||||
# 活跃写入者：后台进程持有 WAL 连接（提交后不关闭），保证备份时 -wal 真实存在——
|
||||
# 裸拷 .db 会丢这部分已提交内容，.backup 不会。
|
||||
WRITER = r"""
|
||||
import sqlite3, sys, time
|
||||
db = sys.argv[1]
|
||||
c = sqlite3.connect(db)
|
||||
c.execute("pragma journal_mode=wal")
|
||||
c.execute("create table if not exists t(x)")
|
||||
c.execute("insert into t values (42)")
|
||||
c.commit()
|
||||
print("ready", flush=True)
|
||||
time.sleep(float(sys.argv[2]))
|
||||
"""
|
||||
|
||||
|
||||
@unittest.skipUnless(SQLITE3, "agent 备份依赖 sqlite3 CLI（一致性导出不回退裸拷贝）")
|
||||
class SvcAgentBackupTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.mkdtemp(prefix="svc-backup-")
|
||||
self.addCleanup(shutil.rmtree, tmp, True)
|
||||
self.root = Path(tmp)
|
||||
(self.root / "scripts").mkdir(parents=True)
|
||||
shutil.copy2(SVC_SCRIPT, self.root / "scripts" / "svc.sh")
|
||||
self.writer_proc = None
|
||||
self.env = os.environ.copy()
|
||||
self.env.update({
|
||||
"GIT_CONFIG_NOSYSTEM": "1", "GIT_CONFIG_GLOBAL": "/dev/null",
|
||||
"SVC_SRC_DIR": str(self.root / "srccache"),
|
||||
"XDG_CONFIG_HOME": str(self.root / "xdg"),
|
||||
})
|
||||
for key in list(self.env):
|
||||
if key.startswith("SVC_") and key != "SVC_SRC_DIR":
|
||||
self.env.pop(key)
|
||||
for key in ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE"):
|
||||
self.env.pop(key, None)
|
||||
|
||||
# ---- 夹具 ----
|
||||
|
||||
def svc(self, *args, check=True, env=None):
|
||||
proc = subprocess.run(
|
||||
["bash", str(self.root / "scripts" / "svc.sh"), *map(str, args)],
|
||||
cwd=self.root, env=env if env is not None else self.env, text=True,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60)
|
||||
if check:
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout)
|
||||
return proc
|
||||
|
||||
def register_agent(self, name="t1", port=8899):
|
||||
d = self.root / "runtime" / name
|
||||
(d / "data").mkdir(parents=True, exist_ok=True)
|
||||
(d / "svc.env").write_text(
|
||||
"kind=agent\nenv=%s\npid=\naddr=127.0.0.1:%d\ntags=\n"
|
||||
"source=dev@test123\nversion=stub\nstarted=\npidcheck=\n"
|
||||
"exec=--addr 127.0.0.1:%d --data %s\n" % (name, port, port, d / "data"))
|
||||
return d
|
||||
|
||||
def seed_db(self, name="t1", live_writer=True, hold_seconds=30):
|
||||
"""建 WAL 库并提交一行；live_writer=True 时留后台连接持有 WAL。"""
|
||||
db = self.root / "runtime" / name / "data" / "agent-system.db"
|
||||
if live_writer:
|
||||
self.writer_proc = subprocess.Popen(
|
||||
["python3", "-c", WRITER, str(db), str(hold_seconds)],
|
||||
stdout=subprocess.PIPE, text=True)
|
||||
self.assertEqual(self.writer_proc.stdout.readline().strip(), "ready")
|
||||
self.addCleanup(self.stop_writer)
|
||||
else:
|
||||
c = sqlite3.connect(db)
|
||||
c.execute("pragma journal_mode=wal")
|
||||
c.execute("create table t(x)")
|
||||
c.execute("insert into t values (42)")
|
||||
c.commit()
|
||||
c.close()
|
||||
return db
|
||||
|
||||
def stop_writer(self):
|
||||
if self.writer_proc and self.writer_proc.poll() is None:
|
||||
self.writer_proc.kill()
|
||||
self.writer_proc.wait(timeout=10)
|
||||
|
||||
def snapshots(self, name="t1"):
|
||||
root = self.root / "runtime-backups" / name
|
||||
if not root.is_dir():
|
||||
return []
|
||||
return sorted(p.name for p in root.iterdir() if p.is_dir())
|
||||
|
||||
# ---- 用例 ----
|
||||
|
||||
def test_backup_live_writer_consistent_and_outside_runtime(self):
|
||||
"""活跃 WAL 写入者：快照可读、行在、integrity ok；-wal/-shm 不入快照。"""
|
||||
self.register_agent()
|
||||
db = self.seed_db()
|
||||
self.assertTrue(db.with_name("agent-system.db-wal").exists()) # 前置：WAL 真实存在
|
||||
proc = self.svc("t1", "backup")
|
||||
self.assertIn("runtime-backups/t1/", proc.stdout)
|
||||
snaps = self.snapshots()
|
||||
self.assertEqual(len(snaps), 1)
|
||||
snap = self.root / "runtime-backups" / "t1" / snaps[0]
|
||||
# 位置在 runtime/ 之外（runtime 整删不波及）
|
||||
self.assertTrue(snap.exists())
|
||||
self.assertFalse((self.root / "runtime" / "t1" / "backups").exists())
|
||||
# 内容：登记 + 数据库一致性副本
|
||||
self.assertTrue((snap / "svc.env").exists())
|
||||
bak = snap / "data" / "agent-system.db"
|
||||
self.assertTrue(bak.exists())
|
||||
self.assertFalse((snap / "data" / "agent-system.db-wal").exists())
|
||||
self.assertFalse((snap / "data" / "agent-system.db-shm").exists())
|
||||
out = subprocess.run([SQLITE3, str(bak), "select x from t; pragma integrity_check;"],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(out.stdout.split(), ["42", "ok"], out.stderr)
|
||||
|
||||
def test_backup_manifest_sha_matches(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
snap = self.root / "runtime-backups" / "t1" / self.snapshots()[0]
|
||||
manifest = (snap / "MANIFEST").read_text()
|
||||
self.assertIn("service=t1\n", manifest)
|
||||
self.assertIn("source=dev@test123\n", manifest)
|
||||
entries = [l.split() for l in manifest.splitlines() if " ./" in l]
|
||||
self.assertTrue(entries)
|
||||
for sha, rel in entries:
|
||||
self.assertRegex(sha, r"^[0-9a-f]{64}$")
|
||||
self.assertEqual(
|
||||
subprocess.run(["sha256sum", str(snap / rel)],
|
||||
capture_output=True, text=True).stdout.split()[0], sha)
|
||||
|
||||
def test_backup_non_sqlite_db_kept_raw_without_leftovers(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
junk = self.root / "runtime" / "t1" / "data" / "blob.db"
|
||||
junk.write_bytes(b"plain-bytes-not-a-sqlite-file")
|
||||
proc = self.svc("t1", "backup")
|
||||
self.assertIn("blob.db", proc.stdout) # 告警原样保留
|
||||
snap = self.root / "runtime-backups" / "t1" / self.snapshots()[0]
|
||||
self.assertEqual((snap / "data" / "blob.db").read_bytes(), b"plain-bytes-not-a-sqlite-file")
|
||||
self.assertFalse(list(snap.rglob("*.bk"))) # 失败尝试的半成品不残留
|
||||
|
||||
def test_backup_target_exists_refused(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
dest = self.root / "runtime-backups" / "t1" / "20260101T000000Z"
|
||||
dest.mkdir(parents=True)
|
||||
(dest / "marker").write_text("x")
|
||||
proc = self.svc("t1", "backup", "--out", dest, check=False)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout)
|
||||
self.assertIn("已存在", proc.stdout)
|
||||
self.assertTrue((dest / "marker").exists()) # 不覆盖既有内容
|
||||
|
||||
def test_backup_custom_out_not_pruned(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
outdir = self.root / "custom-bk"
|
||||
self.svc("t1", "backup", "--out", outdir)
|
||||
self.assertTrue((outdir / "data" / "agent-system.db").exists())
|
||||
# 默认 root 无快照；再走默认路径备份 --keep 0 也不应动自定义目录
|
||||
self.svc("t1", "backup", "--keep", "0")
|
||||
self.assertEqual(self.snapshots(), [])
|
||||
self.assertTrue(outdir.exists())
|
||||
|
||||
def test_retention_and_listing(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
for _ in range(3):
|
||||
self.svc("t1", "backup")
|
||||
time.sleep(1.1) # 时间戳目录名唯一（UTC 秒级）
|
||||
self.assertEqual(len(self.snapshots()), 3)
|
||||
proc = self.svc("t1", "backups")
|
||||
for name in self.snapshots():
|
||||
self.assertIn(name, proc.stdout)
|
||||
# backup --keep 2 → 剩 2 份最新
|
||||
self.svc("t1", "backup", "--keep", "2")
|
||||
self.assertEqual(len(self.snapshots()), 2)
|
||||
# backups --prune --keep 1 → 剩最新 1 份
|
||||
self.svc("t1", "backups", "--prune", "--keep", "1")
|
||||
self.assertEqual(len(self.snapshots()), 1)
|
||||
|
||||
def test_backup_event_recorded(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
hist = (self.root / "runtime" / "t1" / "history.jsonl").read_text()
|
||||
self.assertIn('"event":"backup"', hist)
|
||||
self.assertIn('"result":"ok"', hist)
|
||||
self.assertIn("runtime-backups/t1/", hist)
|
||||
|
||||
def test_backup_rejects_postgres_with_db_hint(self):
|
||||
d = self.root / "runtime" / "pgx"
|
||||
d.mkdir(parents=True)
|
||||
(d / "svc.env").write_text("kind=postgres\nenv=pgx\naddr=127.0.0.1:8890\n")
|
||||
proc = self.svc("pgx", "backup", check=False)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout)
|
||||
self.assertIn("db backup", proc.stdout)
|
||||
|
||||
def test_backup_requires_registration(self):
|
||||
proc = self.svc("ghost", "backup", check=False)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout)
|
||||
|
||||
def test_backup_without_sqlite3_refuses(self):
|
||||
"""PATH 无 sqlite3 → 拒绝（不回退裸拷贝）。重建与真实 PATH 等价的工具目录、
|
||||
唯独不链 sqlite3——只测 command -v 失败这一条分支，其余行为不变形。"""
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
bindir = self.root / "bin-nosqlite3"
|
||||
bindir.mkdir()
|
||||
seen = set()
|
||||
for d in self.env.get("PATH", os.environ["PATH"]).split(os.pathsep):
|
||||
if not d or not os.path.isdir(d):
|
||||
continue
|
||||
for f in os.listdir(d):
|
||||
if f == "sqlite3" or f in seen or (bindir / f).exists():
|
||||
continue
|
||||
seen.add(f)
|
||||
try:
|
||||
os.symlink(os.path.join(d, f), bindir / f)
|
||||
except OSError:
|
||||
pass
|
||||
self.assertFalse((bindir / "sqlite3").exists())
|
||||
env = dict(self.env)
|
||||
env["PATH"] = str(bindir)
|
||||
proc = self.svc("t1", "backup", check=False, env=env)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout)
|
||||
self.assertIn("sqlite3", proc.stdout)
|
||||
self.assertEqual(self.snapshots(), []) # 无半成品快照
|
||||
|
||||
def test_rm_purge_removes_snapshots(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
self.assertEqual(len(self.snapshots()), 1)
|
||||
self.svc("rm", "t1", "--purge")
|
||||
self.assertFalse((self.root / "runtime-backups" / "t1").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -95,6 +95,13 @@ PYSRV
|
||||
child=$!
|
||||
wait "$child"
|
||||
;;
|
||||
user)
|
||||
printf 'user %s\n' "$*" >> "$trace_dir/cli.txt"
|
||||
case "${2:-}" in
|
||||
list) echo '[{"owner":"op","name":"op","isAdmin":true,"mustChangePassword":false}]' ;;
|
||||
create) echo '{"owner":"new","issued":true}' ;;
|
||||
esac
|
||||
exit 0 ;;
|
||||
*)
|
||||
exit 0 ;;
|
||||
esac
|
||||
@@ -156,10 +163,10 @@ class SvcAutoRecoverTests(unittest.TestCase):
|
||||
|
||||
# ---- 夹具 ----
|
||||
|
||||
def svc(self, *args, check=True):
|
||||
def svc(self, *args, check=True, env=None):
|
||||
proc = subprocess.run(
|
||||
["bash", str(self.root / "scripts" / "svc.sh"), *map(str, args)],
|
||||
cwd=self.root, env=self.env, text=True,
|
||||
cwd=self.root, env=env if env is not None else self.env, text=True,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=120)
|
||||
if check:
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout)
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
"""svc.sh 账户 CLI 实例绑定（U1 规格 §6 T1–T5）。
|
||||
|
||||
覆盖：合法 manifest 下 user list 命中 data_root+open-existing（T1）、禁止覆盖
|
||||
--data（T2）、删除 data 根后 OpenExisting 拒绝且不建空库（T3）、无 flock 拒绝
|
||||
user/deploy（T4）。T5 身份回归由 test_svc_identity 同发现执行；此处仅冒烟。
|
||||
桩与夹具复用 tests/test_svc_recover.py；manifest 辅助对齐 test_svc_identity。
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SOURCE = Path(__file__).resolve().parents[1]
|
||||
|
||||
_spec = importlib.util.spec_from_file_location("tsr", SOURCE / "tests" / "test_svc_recover.py")
|
||||
tsr = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(tsr)
|
||||
|
||||
|
||||
class UserBindingTests(tsr.SvcAutoRecoverTests):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
shutil.copy2(SOURCE / "scripts" / "svc_manifest.py",
|
||||
self.root / "scripts" / "svc_manifest.py")
|
||||
self.trace.mkdir(exist_ok=True)
|
||||
|
||||
def manifest(self, name):
|
||||
p = self.runtime(name) / "manifest.json"
|
||||
return json.loads(p.read_text(encoding="utf-8")) if p.exists() else None
|
||||
|
||||
def running(self, name):
|
||||
pid = self.pid_of(name)
|
||||
if not pid:
|
||||
return False
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def touch_db(self, name):
|
||||
data = Path(self.manifest(name)["identity"]["data_root"])
|
||||
data.mkdir(parents=True, exist_ok=True)
|
||||
(data / "agent-system.db").write_bytes(b"sqlite-stub")
|
||||
|
||||
def cli_trace(self):
|
||||
f = self.trace / "cli.txt"
|
||||
return f.read_text(encoding="utf-8") if f.exists() else ""
|
||||
|
||||
def clear_cli_trace(self):
|
||||
f = self.trace / "cli.txt"
|
||||
if f.exists():
|
||||
f.unlink()
|
||||
|
||||
# ---- T1 合法 manifest：user list 绑定 data_root + --open-existing ----
|
||||
|
||||
def test_T1_user_list_binds_manifest_data(self):
|
||||
self.seed_agent("ub1", 9441)
|
||||
self.start_ok("ub1")
|
||||
self.svc("ub1", "stop")
|
||||
root = self.manifest("ub1")["identity"]["data_root"]
|
||||
self.touch_db("ub1")
|
||||
self.clear_cli_trace()
|
||||
result = self.svc("ub1", "user", "list")
|
||||
self.assertIn("op", result.stdout)
|
||||
trace = self.cli_trace()
|
||||
self.assertIn("user list", trace)
|
||||
self.assertIn("--open-existing", trace)
|
||||
self.assertIn("--data %s" % root, trace)
|
||||
self.assertNotIn("./data", trace)
|
||||
|
||||
# ---- T2 禁止覆盖 --data；未绑定默认目录拒绝 ----
|
||||
|
||||
def test_T2_reject_override_data_flag(self):
|
||||
self.seed_agent("ub2", 9442)
|
||||
self.start_ok("ub2")
|
||||
self.svc("ub2", "stop")
|
||||
self.touch_db("ub2")
|
||||
other = self.root / "evil-data"
|
||||
other.mkdir()
|
||||
(other / "agent-system.db").write_bytes(b"x")
|
||||
result = self.svc("ub2", "user", "list", "--data", str(other), check=False)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertTrue(any(k in result.stdout for k in ("禁止覆盖", "勿传")), result.stdout)
|
||||
self.assertFalse((self.trace / "cli.txt").exists(), "拒绝后不得调用二进制")
|
||||
|
||||
def test_T2b_reject_literal_default_data(self):
|
||||
d = self.runtime("ub2b")
|
||||
(d / "bin").mkdir(parents=True)
|
||||
shutil.copy2(self.stub, d / "bin" / "agent-system")
|
||||
(d / "svc.env").write_text("".join("%s=%s\n" % kv for kv in [
|
||||
("kind", "agent"), ("env", "ub2b"), ("pid", ""),
|
||||
("addr", "127.0.0.1:9443"), ("tags", ""),
|
||||
("source", "local@deadbee"), ("version", "stub"), ("started", ""),
|
||||
("pidcheck", str(d / "bin" / "agent-system")),
|
||||
("exec", "--addr 127.0.0.1:9443 --data ./data --plugins %s" % (d / "plugins")),
|
||||
]))
|
||||
env = dict(self.env, SVC_UB2B_DATA="./data")
|
||||
result = self.svc("ub2b", "user", "list", check=False, env=env)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertTrue(any(k in result.stdout for k in ("未绑定数据目录", "不回落")),
|
||||
result.stdout)
|
||||
|
||||
# ---- T3 删除 data 根：OpenExisting 失败，不建空库 ----
|
||||
|
||||
def test_T3_missing_data_root_rejected(self):
|
||||
self.seed_agent("ub3", 9444)
|
||||
self.start_ok("ub3")
|
||||
self.svc("ub3", "stop")
|
||||
root = Path(self.manifest("ub3")["identity"]["data_root"])
|
||||
self.touch_db("ub3")
|
||||
shutil.rmtree(root)
|
||||
self.assertFalse(root.exists())
|
||||
result = self.svc("ub3", "user", "create", "--owner", "x", "--name", "x",
|
||||
check=False)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertTrue(any(k in result.stdout for k in ("既有库不存在", "OpenExisting")),
|
||||
result.stdout)
|
||||
self.assertFalse(root.exists(), "拒绝后不得重建数据根")
|
||||
self.assertFalse((root / "agent-system.db").exists())
|
||||
|
||||
# ---- T4 隐藏 flock：user/deploy 拒绝；已运行服务仍在 ----
|
||||
|
||||
def test_T4_no_flock_rejects_user_and_deploy(self):
|
||||
self.seed_agent("ub4", 9445)
|
||||
self.start_ok("ub4")
|
||||
self.assertTrue(self.running("ub4"))
|
||||
self.touch_db("ub4")
|
||||
# 构造不含 flock 的 PATH：链入常用外部命令，刻意排除 flock。
|
||||
bin_dir = self.root / "nobin"
|
||||
bin_dir.mkdir()
|
||||
for name in (
|
||||
"bash", "python3", "python", "dirname", "basename", "tr", "cksum",
|
||||
"cut", "mkdir", "cat", "sed", "grep", "head", "rm", "mv", "cp",
|
||||
"ls", "chmod", "uname", "date", "sleep", "kill", "ps", "awk",
|
||||
"sort", "uniq", "env", "printf", "realpath", "readlink", "stat",
|
||||
"tee", "wc", "find", "xargs", "touch", "ln", "mktemp", "id",
|
||||
"git", "make", "tar", "gzip", "which", "command",
|
||||
):
|
||||
src = shutil.which(name)
|
||||
if src:
|
||||
dest = bin_dir / name
|
||||
if not dest.exists():
|
||||
dest.symlink_to(src)
|
||||
self.assertIsNone(shutil.which("flock", path=str(bin_dir)))
|
||||
env = dict(self.env, PATH=str(bin_dir))
|
||||
r_user = self.svc("ub4", "user", "list", check=False, env=env)
|
||||
self.assertNotEqual(r_user.returncode, 0)
|
||||
self.assertIn("缺少 flock", r_user.stdout)
|
||||
r_dep = self.svc("ub4", "deploy", "--dry-run", check=False, env=env)
|
||||
self.assertNotEqual(r_dep.returncode, 0, r_dep.stdout)
|
||||
self.assertIn("缺少 flock", r_dep.stdout)
|
||||
self.assertTrue(self.running("ub4"), "拒绝后旧服务应仍在跑")
|
||||
self.svc("ub4", "stop")
|
||||
|
||||
# ---- T5 recovery U1 身份冒烟（完整套件见 test_svc_identity）----
|
||||
|
||||
def test_T5_identity_regression_smoke(self):
|
||||
self.seed_agent("ub5", 9446)
|
||||
self.start_ok("ub5")
|
||||
iid = self.manifest("ub5")["instance_id"]
|
||||
self.svc("ub5", "stop")
|
||||
self.start_ok("ub5")
|
||||
self.assertEqual(self.manifest("ub5")["instance_id"], iid)
|
||||
self.svc("ub5", "stop")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user