Add svc.sh agent data-dir snapshots with SQLite-consistent backup
Why: the 2026-10-11 runtime/ deletion lost all agent SQLite data with no backup; the svc-autorecover closing report left "agent SQLite backup" as a follow-up and the user asked to land it (recovery plan U2 residue). Evidence: tests/test_svc_backup.py 11/11 green locally (live WAL writer, manifest sha, retention, guards); make test-governance full suite green (159 tests, rerun twice after one unreproduced flake). Impact: new service-level actions backup/backups; snapshots live in runtime-backups/<name>/ outside runtime/; rm --purge removes them; pg backup and other flows unchanged. Refs: modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2 (sec 6, U2 residue); task card .task/feature-svc-agent-backup.md.
This commit is contained in:
@@ -6,6 +6,7 @@ __pycache__/
|
||||
/data-test/
|
||||
.zcode/plans/
|
||||
runtime/
|
||||
runtime-backups/
|
||||
.svc.env
|
||||
|
||||
# 运行时环境文件(含密钥,不入库;svc-envfile 任务约定)
|
||||
|
||||
@@ -7,48 +7,67 @@
|
||||
- 权威仓:ssh://git@git-ssh.u1680832.nyat.app:31411/weichen/Agent-System-Dev.git;目标分支:main;R0(initial_base_sha)= 475654363de3ec72df68d246ffab8217b1cfeab6
|
||||
- 工作区:/home/weichen/projects/Agent-System-Dev-wt-feature-svc-agent-backup(worktree);分支:feature/svc-agent-backup
|
||||
|
||||
- 授权与交付终点:<本次允许的动作与终点;未授权事项写明>
|
||||
- 交付模式:manual(逐阶段人工评审);明确授权后可记为 authorized-continuous
|
||||
- 自动推进授权:无;授权范围:无;授权来源:无;授权终点:无;停止条件:无(未授权)
|
||||
- 当前阶段:需求拆解;当前增量:I1(待定义)
|
||||
- 授权与交付终点:用户 2026-10-12 指示「把遗留项按建议修复」——即落地前次 svc-autorecover
|
||||
收尾报告的两条遗留建议。允许:独立工作区内实施/测试/证据、推送分支、建 PR 并合入 main
|
||||
(沿用本工作区既有交付模式);本任务覆盖建议第 1 条(agent 数据目录 backup)与第 2 条的
|
||||
落地参考回填(规划卡 feature/service-restart-recovery-plan,随本任务收口时闭合)。
|
||||
未授权:restore/灾难恢复语义(属恢复方案 U6)、UI、跨机异地副本。
|
||||
- 交付模式:manual;用户指示即本次范围的小任务连续推进授权(阶段产物保留)
|
||||
- 自动推进授权:用户对话指示(2026-10-12);授权范围:上述全部;授权来源:用户;授权终点:合入+两卡闭合;停止条件:CI 红或偏离建议范围
|
||||
- 当前阶段:实施与自检;当前增量:I1(backup 能力 + 回归)
|
||||
- 流程依据:仓库根目录 policy/delivery.md(归档后仍按仓库根目录定位)
|
||||
|
||||
## 目标与非目标
|
||||
|
||||
<预期结果、受影响入口/模块/契约;明确不做什么。>
|
||||
目标:`svc.sh <name> backup [--out DIR] [--keep N]` / `backups [--prune] [--keep N]`——
|
||||
agent 服务数据目录(svc.env + history.jsonl + data/)一致性快照:每个 `*.db` 经
|
||||
`sqlite3 .backup` 导出并 integrity_check 自检(活跃 WAL 写入者下安全),MANIFEST 记逐文件
|
||||
sha256;快照根在 `runtime-backups/<name>/`(runtime/ 之外,防 2026-10-11 式 runtime 整删
|
||||
事故);保留策略默认 7 份;`rm --purge` 连带删除。
|
||||
|
||||
非目标:restore/恢复到新目标与外部效果对账(恢复方案 U6);自动定时备份(频率须按批准的
|
||||
RPO 固定,属方案 §6 后续);postgres(已有 `db backup`)与 static(无数据目录)。
|
||||
|
||||
## 关联方向与规划
|
||||
|
||||
- 适用状态:待核对(无适用规划时说明原因)
|
||||
- 主记录路径/版本:待核对(路径按仓库根目录定位,归档后不变;只引用唯一正文)
|
||||
- 对应里程碑:待核对
|
||||
- 适用状态:恢复方案的实施增量
|
||||
- 主记录路径/版本:modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2(U2 残留「agent SQLite 自动备份缺位」的实用切片;SQLite 一致性接口约束见其 §6/KTD)
|
||||
- 对应里程碑:恢复合同 U2(首切片 svc-autorecover PR #396/#401 已合入);组合备份完整合同属 U6
|
||||
- 关联任务卡:feature/service-restart-recovery-plan(本任务落地后回填其进展并闭合)
|
||||
|
||||
## 需求分析
|
||||
|
||||
| ID | 描述 | 项目符合度及依据 | 优先级 | 验收条件 | 依赖 | 结论 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| R1 | 待填写 | 待评估 | 待定 | 待定义 | 待确认 | 待评审 |
|
||||
| R1 | agent 数据目录一键快照(svc.env+history+data) | 符合;svc.sh 既有 pg backup/backups 语法与事件模型可平移 | P0 | 活跃 WAL 写入者下快照内 db 可查询、integrity ok、无 -wal/-shm;事件 backup ok 入 history | — | 纳入 |
|
||||
| R2 | 快照列表与保留策略 | 符合;沿用 --keep/prune 惯例 | P1 | backups 列表正确;--keep N 只清理时间戳目录,--out 自定义目录不被清理 | R1 | 纳入 |
|
||||
| R3 | 一致性与安全约束 | 需调整后符合;方案 §6 禁止裸拷 .db | P0 | 无 sqlite3 拒绝备份;快照根在 runtime/ 之外;MANIFEST sha256 与文件一致;非 SQLite 的 *.db 原样保留并告警;rm --purge 连带删除;postgres/static 拒绝并引导 | R1 | 纳入 |
|
||||
|
||||
## 方案与计划
|
||||
|
||||
- 方案引用:待填写(含对象版本)
|
||||
- 实施计划引用:待填写(含增量与需求 ID)
|
||||
- 测试计划引用:待填写(含需求 ID 与验收条件)
|
||||
- 方案:本卡内联(小任务)。快照三步:整树 cp -a → 逐 *.db 用 sqlite3 .backup 覆盖(丢弃
|
||||
cp 得到的撕裂 db/-wal 副本)+ integrity_check → MANIFEST。快照根 `runtime-backups/<name>/
|
||||
<UTC时间戳>/`(700 目录 / 600 敏感文件)。backup/backups 作为服务级动作进主分发
|
||||
(与 db/config 同款自带参数解析),backup/backups 列入保留字防歧义。
|
||||
- 实施计划:I1 = svc.sh(命令+帮助+rm purge)+ .gitignore + docs/svc.md §8.1.1 +
|
||||
tests/test_svc_backup.py(覆盖 R1–R3)。
|
||||
- 测试计划:unittest 沙箱(真实 sqlite3 CLI + 后台 WAL 写入者进程);验收条件逐条映射
|
||||
R1–R3;治理套件 `make test-governance` 全量回归。
|
||||
|
||||
## 阶段评审
|
||||
|
||||
| 阶段 | 对象版本 | 结论 | 评审人 | 时间 | 依据 |
|
||||
|---|---|---|---|---|---|
|
||||
| 需求拆解 | 待定 | 未评审 | — | — | — |
|
||||
| 需求评估 | 待定 | 未评审 | — | — | — |
|
||||
| 需求表确认 | 待定 | 未评审 | — | — | — |
|
||||
| 方案设计 | 待定 | 未评审 | — | — | — |
|
||||
| 实施计划 | 待定 | 未评审 | — | — | — |
|
||||
| 实施与自检 | 待定 | 未评审 | — | — | — |
|
||||
| 正式测试与验收 | 待定 | 未评审 | — | — | — |
|
||||
| 代码 Review | 待定 | 未评审 | — | — | — |
|
||||
| 合入 | 待定 | 未评审 | — | — | — |
|
||||
| 合入核验与收口 | 待定 | 未评审 | — | — | — |
|
||||
| 需求拆解 | 收尾报告遗留建议(2026-10-12 用户会话) | 已批准 | weichen | 2026-10-12 | 用户指示「把遗留项按建议修复」 |
|
||||
| 需求评估 | 同上 | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 需求表确认 | 本卡 R1–R3 | 已批准 | weichen | 2026-10-12 | 同上(小任务五阶段一并) |
|
||||
| 方案设计 | 本卡方案节 | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 实施计划 | 本卡 I1 | 已批准 | weichen | 2026-10-12 | 同上 |
|
||||
| 实施与自检 | 待回填 | 未评审 | — | — | — |
|
||||
| 正式测试与验收 | 待回填 | 未评审 | — | — | — |
|
||||
| 代码 Review | 待回填 | 未评审 | — | — | — |
|
||||
| 合入 | 待回填 | 未评审 | — | — | — |
|
||||
| 合入核验与收口 | 待回填 | 未评审 | — | — | — |
|
||||
|
||||
## 探索试验与迭代
|
||||
|
||||
|
||||
+26
@@ -583,6 +583,32 @@ scripts/svc.sh db backups pg1 # 列表(时间 / 大小 / 文
|
||||
- 完成后 `pg_restore --list` 自检归档有效性,失败即删除该文件并记 `backup fail`;
|
||||
- 备份**只增不删**;随 `rm --purge` 一并删除。
|
||||
|
||||
### 8.1.1 agent 数据目录快照(SQLite 一致性备份)
|
||||
|
||||
```bash
|
||||
scripts/svc.sh dev backup # 快照 data/ + svc.env + history.jsonl
|
||||
scripts/svc.sh dev backup --keep 3 # 备份后只保留最近 3 份
|
||||
scripts/svc.sh dev backup --out /mnt/bk/dev1 # 自定义快照目录(不参与自动清理)
|
||||
scripts/svc.sh dev backups # 快照列表(时间 / 大小 / 文件数)
|
||||
scripts/svc.sh dev backups --prune --keep 3 # 手动清理旧快照
|
||||
```
|
||||
|
||||
- 位置:`runtime-backups/<name>/<UTC时间戳>/`——**在 runtime/ 之外**。2026-10-11
|
||||
发生过 `runtime/` 被整体误删的事故(SQLite 聊天/设置数据全失,且无任何备份);
|
||||
放在服务目录内的备份会跟着一起消失,因此快照根独立于 runtime/;
|
||||
- 一致性:每个 `*.db` 经 `sqlite3 .backup` 导出(并入 WAL 内容,服务运行中也可
|
||||
安全备份)并做 `PRAGMA integrity_check` 自检;`-wal`/`-shm` 不入快照。无 sqlite3
|
||||
时**拒绝备份**,不回退裸拷贝(裸拷会漏 WAL 得到撕裂副本);
|
||||
- MANIFEST:快照内 `MANIFEST`(600)记录服务、来源版本与逐文件 sha256,恢复时
|
||||
可校验完整性。非 SQLite 的 `*.db` 文件(无文件头)保留原样副本并告警;
|
||||
- 保留策略:默认保留最近 7 份(`--keep N` 调整;只清理符合时间戳命名的快照目录,
|
||||
不动 `--out` 指定的自定义目录);`rm <name> --purge` 会连带删除
|
||||
`runtime-backups/<name>/`,`rm`(不带 --purge)保留快照;
|
||||
- 边界:本命令是**数据面快照**,不含 restore(恢复到新目标、manifest 合同与
|
||||
外部效果对账属恢复方案 U6,见
|
||||
`modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md`);
|
||||
postgres 实例请用 `db backup`,static 服务无数据目录。
|
||||
|
||||
### 8.2 恢复
|
||||
|
||||
```bash
|
||||
|
||||
+186
-7
@@ -7,6 +7,8 @@
|
||||
# scripts/svc.sh <name> [action] [参数...] 服务级:deploy(默认)|start|stop|restart|status|logs [N]
|
||||
# |db [pg名|--clear](agent 数据库绑定/切换)
|
||||
# |config [file|--clear](agent 配置文件更新/生效)
|
||||
# |backup [--out D] [--keep N](agent 数据目录快照)
|
||||
# |backups [--prune](快照列表 / 按保留数清理)
|
||||
# scripts/svc.sh list [--tag T] 服务总览(status 等价)
|
||||
# scripts/svc.sh info [name] [--tag T] [--reveal] 综合报告(只读):服务/数据库/配置快照/最近事件
|
||||
# scripts/svc.sh set <name> [--port N --apply] [--tag T] 变更登记(换端口/标签)
|
||||
@@ -58,6 +60,7 @@
|
||||
# db backup:--db X --out F db restore:--file F --db X --force
|
||||
# db clone:--port P --host H --password PW --tag T --online
|
||||
# db copy:--db X --force set:--port N --host H --apply --tag T
|
||||
# backup:--out DIR --keep N backups:--prune --keep N(agent 数据目录快照)
|
||||
#
|
||||
# ════════════════════ 配置优先级 ════════════════════
|
||||
# 命令行 > --config/--env 文件 > svc.env 持久化(exec=/addr=/dbref=/dsn=/db=)
|
||||
@@ -76,6 +79,9 @@
|
||||
# - runtime/<name>/:bin|htdocs|data|pgdata|plugins|backups + svc.pid +
|
||||
# svc.log|pg.log + svc.env(登记+持久化配置,600)+ history.jsonl(事件流)
|
||||
# + .start.lock(生命周期互斥锁,recover 与 deploy/手动操作防竞态)
|
||||
# - runtime-backups/<name>/<UTC时间戳>/:agent 数据目录快照(svc.env+history+data;
|
||||
# *.db 经 sqlite3 .backup 一致性导出并自检,MANIFEST 记逐文件 sha256。放 runtime/
|
||||
# 之外——runtime 被整体误删时快照仍在,rm --purge 随服务一并删除)
|
||||
# - kind:agent(后端单二进制,内嵌前端)/ static(前端静态服务)/ postgres(数据库);
|
||||
# 无 kind= 的旧登记自动视为 agent;start/restart 使用环境内产物
|
||||
#
|
||||
@@ -88,6 +94,7 @@
|
||||
# scripts/svc.sh ui-qa deploy --kind static --dir ../some/dist
|
||||
# scripts/svc.sh set pg1 --port 8895 --apply # 换端口并重启生效
|
||||
# scripts/svc.sh db backup pg1 && scripts/svc.sh db backups pg1
|
||||
# scripts/svc.sh dev backup && scripts/svc.sh dev backups # agent 数据目录快照 + 列表
|
||||
# scripts/svc.sh db clone pg1 pg2 --port 8896 # 物理克隆一套到新端口
|
||||
# scripts/svc.sh dev db pg1 --db dev # 把 dev 服务绑定到 pg1/dev 库(自动重启)
|
||||
# scripts/svc.sh dev db pg2 --no-restart # 切换但不重启(下次 start 生效)
|
||||
@@ -104,7 +111,7 @@
|
||||
# make svc ENV=dev ARGS="--env .env.dev --port 9000"
|
||||
#
|
||||
# ════════════════════ 安全约束 ════════════════════
|
||||
# - 只写 runtime/<name>/、runtime/.worktrees/ 与独立部署源仓(SVC_SRC_DIR,默认
|
||||
# - 只写 runtime/<name>/、runtime/.worktrees/、runtime-backups/<name>/ 与独立部署源仓(SVC_SRC_DIR,默认
|
||||
# ~/.cache/agent-system-svc/src,可随时删除重建);不改主检出 git 状态、不动他人现场;
|
||||
# --from local 是唯一写主检出构建产物(bin/、node_modules/、dist/,均 gitignore)
|
||||
# 的路径,dev@sha/release/static 构建全部在隔离 worktree(自部署源仓物化);
|
||||
@@ -112,8 +119,10 @@
|
||||
# pg_ctl -m fast 优雅停机,SIGKILL 不用于 postmaster;
|
||||
# - 端口被非本脚本进程占用时报错并给出占用者,不抢不杀;
|
||||
# - svc.env/备份文件权限 600;DSN 默认掩码密码(--reveal 才显示);
|
||||
# - 备份/克隆只增不删;restore/copy 覆盖非空库必须显式 --force(drop 重建再恢复);
|
||||
# 备份文件在 runtime/<name>/backups/,随 rm --purge 一并删除;
|
||||
# - 备份/克隆只增不删(backup --keep / backups --prune 的显式保留策略除外);
|
||||
# restore/copy 覆盖非空库必须显式 --force(drop 重建再恢复);
|
||||
# pg 备份在 runtime/<name>/backups/,agent 快照在 runtime-backups/<name>/(在
|
||||
# runtime/ 之外,防整删事故连带),均随 rm --purge 一并删除;
|
||||
# - 物理克隆期间源实例短暂停止(一致性要求),完成后自动复启;--online 不停源。
|
||||
#
|
||||
# 完整文档:docs/svc.md
|
||||
@@ -158,6 +167,8 @@ svc.sh —— 本机服务管理:后端(agent) / 前端(static) / 数据库(po
|
||||
scripts/svc.sh <name> start|stop|restart|logs [N] 生命周期与日志
|
||||
scripts/svc.sh <name> status 服务详情(版本/来源/DSN 等)
|
||||
scripts/svc.sh <name> db [pg名|--clear] agent 数据库绑定/切换(缺省查看)
|
||||
scripts/svc.sh <name> backup [--out D] [--keep N] agent 数据目录快照(SQLite 一致性备份)
|
||||
scripts/svc.sh <name> backups [--prune] 快照列表 / 按保留数清理
|
||||
scripts/svc.sh <name> config [file|--clear] agent 配置文件更新/即时生效(缺省查看)
|
||||
scripts/svc.sh db create pg1 --port 8890 建数据库
|
||||
scripts/svc.sh db dsn|backup|backups|restore|clone|copy|ls|set pg1 ... 数据库运维
|
||||
@@ -216,6 +227,12 @@ update deploy stop start restart logs help info version。
|
||||
history.jsonl 事件流(append-only,一行一个 JSON)
|
||||
.start.lock 生命周期互斥锁(recover 与 deploy/手动 start 防竞态)
|
||||
|
||||
runtime-backups/<name>/<UTC时间戳>/
|
||||
agent 数据目录快照:svc.env + history.jsonl + data/(*.db 经 sqlite3 .backup
|
||||
导出一致性副本并入 WAL 内容,integrity_check 自检;-wal/-shm 不入快照;
|
||||
MANIFEST 记逐文件 sha256)。位于 runtime/ 之外——runtime 被整体误删时快照
|
||||
仍在;rm --purge 随服务一并删除。
|
||||
|
||||
svc.env 字段:
|
||||
kind agent|static|postgres(旧登记无此键视为 agent)
|
||||
env 服务名
|
||||
@@ -244,7 +261,7 @@ history.jsonl 事件(scripts/svc.sh history 查看):
|
||||
recover ok|fail(watchdog/手动 recover 拉回死亡服务)
|
||||
stop ok|fail
|
||||
set 登记变更(addr 旧 → 新)
|
||||
backup ok|fail(文件与大小)
|
||||
backup ok|fail(pg:库/文件/大小;agent:快照目录/大小)
|
||||
restore ok|fail(库、文件、表数)
|
||||
clone ok|fail(物理/在线)
|
||||
copy ok|fail
|
||||
@@ -339,6 +356,10 @@ EOF
|
||||
scripts/svc.sh rm <name> 停止并移除登记(目录/数据/历史保留)
|
||||
scripts/svc.sh rm <name> --purge 连 runtime/<name>/ 整个目录一起删除
|
||||
|
||||
数据目录快照(agent;SQLite 一致性导出,详见 docs/svc.md):
|
||||
scripts/svc.sh <name> backup [--out DIR] [--keep N] 快照到 runtime-backups/<name>/
|
||||
scripts/svc.sh <name> backups [--prune] [--keep N] 列表 / 按保留数清理旧快照
|
||||
|
||||
自动恢复(崩溃 / OOM 被杀 / 机器重启后自动拉回;详见 help recovery):
|
||||
scripts/svc.sh recover [name] [--quiet] 把 autostart=1 但已死的服务重新拉起
|
||||
scripts/svc.sh watchdog install systemd 用户 timer 每分钟执行 recover
|
||||
@@ -668,7 +689,7 @@ addr_compose() {
|
||||
}
|
||||
|
||||
# ---- 名称与端口 ----
|
||||
RESERVED_ENVS=" status list add set db rm create dsn commits history update deploy stop start restart logs help info version recover watchdog "
|
||||
RESERVED_ENVS=" status list add set db rm create dsn commits history update deploy stop start restart logs help info version recover watchdog backup backups "
|
||||
|
||||
# valid_env NAME:服务名合法性(小写字母开头、a-z0-9-、≤16 位;保留字不可用)。
|
||||
valid_env() {
|
||||
@@ -2871,6 +2892,150 @@ EOF
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---- agent 数据目录快照(SQLite 一致性备份;2026-10-12)----
|
||||
# 快照根在 runtime-backups/<name>/(runtime/ 之外):2026-10-11 runtime/ 被整体
|
||||
# 误删的事故里,放在 runtime/<name>/backups/ 的任何东西都会一起消失。每个 *.db
|
||||
# 经 sqlite3 .backup 导出一致性副本(并入 WAL 内容)并 integrity_check 自检,
|
||||
# 不裸拷 .db(约束见 modules/documents/docs/proposals/
|
||||
# 2026-10-12-service-restart-data-recovery.md §6);-wal/-shm 不入快照。
|
||||
BACKUP_KEEP_DEFAULT=7
|
||||
BACKUP_TS_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'
|
||||
|
||||
agent_backup_root() { printf '%s/runtime-backups/%s' "$ROOT" "$ENV_NAME"; }
|
||||
|
||||
agent_backup_guard() { # 公共前置:agent 已登记(backups 列表也走这里)
|
||||
if [ "$SVC_KIND" = postgres ]; then
|
||||
c_err "[$ENV_NAME] postgres 用 db backup(pg_dump);backup 是 agent 数据目录快照"
|
||||
exit 2
|
||||
elif [ "$SVC_KIND" != agent ]; then
|
||||
c_err "[$ENV_NAME] kind=$SVC_KIND 无数据目录,不适用 backup"
|
||||
exit 2
|
||||
fi
|
||||
[ -f "$ENV_DIR/svc.env" ] || { c_err "[$ENV_NAME] 未登记(无 svc.env);先 deploy"; exit 2; }
|
||||
}
|
||||
|
||||
agent_backup_snapshot() { # $1=dest:svc.env + history.jsonl + data/(*.db 一致性化)
|
||||
local dest="$1" rel
|
||||
cp -p "$ENV_DIR/svc.env" "$dest/svc.env" || return 1
|
||||
[ -f "$ENV_DIR/history.jsonl" ] && cp -p "$ENV_DIR/history.jsonl" "$dest/history.jsonl"
|
||||
mkdir -p "$dest/data" || return 1
|
||||
# 先整树拷贝(非 db 内容的原样副本),再对 *.db 用一致性导出覆盖:cp 得到的
|
||||
# 撕裂 db/-wal 副本在下一步被丢弃替换,不影响最终一致性。
|
||||
cp -a "$ENV_DATA/." "$dest/data/" || return 1
|
||||
while IFS= read -r rel; do
|
||||
[ -f "$ENV_DATA/$rel" ] || continue
|
||||
case "$rel" in
|
||||
*"'"*|*'"'*)
|
||||
if head -c 15 "$ENV_DATA/$rel" | grep -qa '^SQLite format 3$'; then
|
||||
c_err "[$ENV_NAME] 路径含引号,无法走 sqlite 一致性导出:$rel"
|
||||
return 1
|
||||
fi
|
||||
continue ;;
|
||||
esac
|
||||
if sqlite3 "$ENV_DATA/$rel" ".backup '$dest/data/$rel.bk'" 2>/dev/null \
|
||||
&& [ "$(sqlite3 "$dest/data/$rel.bk" 'PRAGMA integrity_check;' 2>/dev/null)" = ok ]; then
|
||||
mv -f "$dest/data/$rel.bk" "$dest/data/$rel" || return 1
|
||||
chmod 600 "$dest/data/$rel"
|
||||
rm -f "$dest/data/$rel-wal" "$dest/data/$rel-shm" # .backup 已并入 WAL 内容
|
||||
else
|
||||
rm -f "$dest/data/$rel.bk" # 失败尝试的半成品不留在快照里
|
||||
if head -c 15 "$ENV_DATA/$rel" | grep -qa '^SQLite format 3$'; then
|
||||
c_err "[$ENV_NAME] SQLite 一致性备份/自检失败:$rel"
|
||||
return 1
|
||||
fi
|
||||
c_warn "[$ENV_NAME] $rel 不是 SQLite 库(无文件头),保留原样副本"
|
||||
fi
|
||||
done < <(cd "$dest/data" 2>/dev/null && find . -type f -name '*.db' | sort)
|
||||
return 0
|
||||
}
|
||||
|
||||
agent_backup_manifest() { # $1=dest:快照自描述(600)——逐文件 sha256 供恢复时校验
|
||||
local dest="$1"
|
||||
{
|
||||
printf 'service=%s\nkind=%s\ncreated=%s\nsource=%s\nversion=%s\ndata_dir=%s\n' \
|
||||
"$ENV_NAME" "$SVC_KIND" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
|
||||
"$(svc_get source)" "$(svc_get version)" "$ENV_DATA"
|
||||
( cd "$dest" && find . -type f ! -name MANIFEST -print0 | sort -z | xargs -0 -r sha256sum )
|
||||
} > "$dest/MANIFEST"
|
||||
chmod 600 "$dest/MANIFEST"
|
||||
}
|
||||
|
||||
agent_backup_prune() { # $1=root $2=keep:只清理符合时间戳命名的快照目录(最新在前保留 keep 份)
|
||||
local root="$1" keep="$2" n=0 d
|
||||
[ -d "$root" ] || return 0
|
||||
while IFS= read -r d; do
|
||||
n=$((n + 1))
|
||||
[ "$n" -le "$keep" ] && continue
|
||||
rm -rf "$root/$d"
|
||||
c_info "[$ENV_NAME] 清理旧快照:$d(保留最近 $keep 份)"
|
||||
done < <(cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB" | sort -r)
|
||||
return 0
|
||||
}
|
||||
|
||||
svc_agent_backup() { # [--out DIR] [--keep N]
|
||||
local out="" keep="$BACKUP_KEEP_DEFAULT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--out) need_value "$@"; out="$2"; shift 2 ;;
|
||||
--keep) need_value "$@"; keep="$2"; shift 2
|
||||
case "$keep" in ''|*[!0-9]*) c_err "--keep 须为非负整数"; exit 2 ;; esac ;;
|
||||
*) c_err "用法:scripts/svc.sh <name> backup [--out DIR] [--keep N](未知参数:$1)"; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
agent_backup_guard
|
||||
[ -d "$ENV_DATA" ] || { c_err "[$ENV_NAME] 数据目录不存在:$ENV_DATA"; exit 2; }
|
||||
command -v sqlite3 >/dev/null 2>&1 \
|
||||
|| { c_err "缺少 sqlite3——一致性备份必需,不回退裸拷贝(会漏 WAL 得到撕裂副本)"; exit 1; }
|
||||
local ts dest size
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
dest="${out:-$(agent_backup_root)/$ts}"
|
||||
case "$dest" in /*) ;; *) dest="$PWD/$dest" ;; esac
|
||||
[ ! -e "$dest" ] || { c_err "备份目标已存在:$dest"; exit 1; }
|
||||
mkdir -p "$dest" || { c_err "无法创建备份目录:$dest"; exit 1; }
|
||||
if ! agent_backup_snapshot "$dest" || ! agent_backup_manifest "$dest"; then
|
||||
rm -rf "$dest"
|
||||
hist_event backup fail "$(read_pid)" "dir=$dest"
|
||||
c_err "[$ENV_NAME] 备份失败,半成品已清理:$dest"
|
||||
exit 1
|
||||
fi
|
||||
chmod 700 "$dest" 2>/dev/null || true
|
||||
size="$(du -sh "$dest" 2>/dev/null | cut -f1)"
|
||||
hist_event backup ok "$(read_pid)" "dir=$dest size=$size"
|
||||
[ -n "$out" ] || agent_backup_prune "$(agent_backup_root)" "$keep"
|
||||
c_ok "[$ENV_NAME] 数据目录快照完成:$dest($size;svc.env+history+data,db 已一致性自检)"
|
||||
}
|
||||
|
||||
svc_agent_backups() { # [--prune] [--keep N]
|
||||
local prune=0 keep="$BACKUP_KEEP_DEFAULT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--prune) prune=1; shift ;;
|
||||
--keep) need_value "$@"; keep="$2"; shift 2
|
||||
case "$keep" in ''|*[!0-9]*) c_err "--keep 须为非负整数"; exit 2 ;; esac ;;
|
||||
*) c_err "用法:scripts/svc.sh <name> backups [--prune] [--keep N](未知参数:$1)"; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
agent_backup_guard
|
||||
local root n=0
|
||||
root="$(agent_backup_root)"
|
||||
if [ -d "$root" ]; then
|
||||
while IFS= read -r _; do n=$((n + 1)); done \
|
||||
< <(cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB")
|
||||
fi
|
||||
if [ "$n" -eq 0 ]; then
|
||||
c_info "[$ENV_NAME] 暂无快照(root=$root;scripts/svc.sh $ENV_NAME backup)"
|
||||
return 0
|
||||
fi
|
||||
c_info "[$ENV_NAME] 快照列表($root,最新在前)"
|
||||
( cd "$root" && find . -maxdepth 1 -mindepth 1 -type d -name "$BACKUP_TS_GLOB" | sort -r ) \
|
||||
| while IFS= read -r d; do
|
||||
printf ' %s %6s %s\n' "${d#./}" "$(du -sh "$root/$d" | cut -f1)" \
|
||||
"$(find "$root/$d/data" -type f 2>/dev/null | wc -l) 个文件"
|
||||
done
|
||||
if [ "$prune" = 1 ]; then agent_backup_prune "$root" "$keep"; fi
|
||||
return 0
|
||||
}
|
||||
|
||||
cmd_set() {
|
||||
local name="${1:-}"
|
||||
[ -n "$name" ] || { c_err "用法:scripts/svc.sh set <name> [--port N] [--host H] [--apply] [--tag T...]"; exit 2; }
|
||||
@@ -3009,6 +3174,11 @@ cmd_rm() {
|
||||
rm -f "$ENV_DIR/svc.env" "$ENV_DIR/svc.pid"
|
||||
if [ "$CLI_PURGE" = 1 ]; then
|
||||
rm -rf "${ENV_DIR:?}"
|
||||
local bkroot="$ROOT/runtime-backups/$name"
|
||||
if [ -d "$bkroot" ]; then
|
||||
rm -rf "$bkroot"
|
||||
c_ok "[$ENV_NAME] 数据目录快照已删除:$bkroot"
|
||||
fi
|
||||
c_ok "[$ENV_NAME] 已注销并删除整个目录:$ENV_DIR(含数据与历史)"
|
||||
else
|
||||
c_ok "[$ENV_NAME] 已注销登记;目录保留(数据/历史在 $ENV_DIR;彻底删除用 --purge)"
|
||||
@@ -3885,12 +4055,13 @@ main() {
|
||||
# 动作词可省略(省略 = deploy);flags 可直接跟随服务名。
|
||||
local action="deploy"
|
||||
case "${1:-}" in
|
||||
deploy|stop|start|restart|status|logs|db|config) action="$1"; shift ;;
|
||||
deploy|stop|start|restart|status|logs|db|config|backup|backups) action="$1"; shift ;;
|
||||
esac
|
||||
if [ "$action" = logs ] && [ "${1:-}" != "" ] && [[ "$1" =~ ^[0-9]+$ ]]; then
|
||||
CLI_LOGS_N="$1"; shift
|
||||
fi
|
||||
# db / config 子命令:自带参数解析(位置参数不走 parse_args;未知 flags 由其自行拒绝)。
|
||||
# db / config / backup / backups 子命令:自带参数解析(位置参数不走 parse_args;
|
||||
# 未知 flags 由其自行拒绝)。
|
||||
if [ "$action" = db ] || [ "$action" = config ]; then
|
||||
env_config "$ENV_NAME"
|
||||
case "$action" in
|
||||
@@ -3899,6 +4070,14 @@ main() {
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
if [ "$action" = backup ] || [ "$action" = backups ]; then
|
||||
env_config "$ENV_NAME"
|
||||
case "$action" in
|
||||
backup) svc_agent_backup "$@" || exit $? ;;
|
||||
backups) svc_agent_backups "$@" || exit $? ;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
parse_args "$@"
|
||||
if { [ -n "$CLI_FROM" ] || [ -n "$CLI_PICK" ] || [ "$CLI_DRYRUN" = 1 ] || [ -n "$CLI_DIR" ]; } && [ "$action" != deploy ]; then
|
||||
c_err "--from/--pick/--dry-run/--dir 仅用于 deploy 动作"
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
"""svc.sh agent 数据目录快照回归(2026-10-12)。
|
||||
|
||||
验证 backup/backups:活跃 WAL 写入者下 SQLite 一致性导出(.backup + integrity_check,
|
||||
不裸拷 .db/-wal)、快照落在 runtime/ 之外的 runtime-backups/<name>/、MANIFEST 逐文件
|
||||
sha256、--keep/--prune 保留策略只清理时间戳目录、--out 自定义目录不参与清理、
|
||||
kind 守卫(postgres 引导 db backup)、缺 sqlite3 拒绝备份、rm --purge 连带删除快照。
|
||||
全部使用临时目录 + 手工登记 svc.env(backup 不触碰 bin/进程,无需桩二进制)。
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SOURCE = Path(__file__).resolve().parents[1]
|
||||
SVC_SCRIPT = SOURCE / "scripts" / "svc.sh"
|
||||
|
||||
SQLITE3 = shutil.which("sqlite3")
|
||||
|
||||
# 活跃写入者:后台进程持有 WAL 连接(提交后不关闭),保证备份时 -wal 真实存在——
|
||||
# 裸拷 .db 会丢这部分已提交内容,.backup 不会。
|
||||
WRITER = r"""
|
||||
import sqlite3, sys, time
|
||||
db = sys.argv[1]
|
||||
c = sqlite3.connect(db)
|
||||
c.execute("pragma journal_mode=wal")
|
||||
c.execute("create table if not exists t(x)")
|
||||
c.execute("insert into t values (42)")
|
||||
c.commit()
|
||||
print("ready", flush=True)
|
||||
time.sleep(float(sys.argv[2]))
|
||||
"""
|
||||
|
||||
|
||||
@unittest.skipUnless(SQLITE3, "agent 备份依赖 sqlite3 CLI(一致性导出不回退裸拷贝)")
|
||||
class SvcAgentBackupTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.mkdtemp(prefix="svc-backup-")
|
||||
self.addCleanup(shutil.rmtree, tmp, True)
|
||||
self.root = Path(tmp)
|
||||
(self.root / "scripts").mkdir(parents=True)
|
||||
shutil.copy2(SVC_SCRIPT, self.root / "scripts" / "svc.sh")
|
||||
self.writer_proc = None
|
||||
self.env = os.environ.copy()
|
||||
self.env.update({
|
||||
"GIT_CONFIG_NOSYSTEM": "1", "GIT_CONFIG_GLOBAL": "/dev/null",
|
||||
"SVC_SRC_DIR": str(self.root / "srccache"),
|
||||
"XDG_CONFIG_HOME": str(self.root / "xdg"),
|
||||
})
|
||||
for key in list(self.env):
|
||||
if key.startswith("SVC_") and key != "SVC_SRC_DIR":
|
||||
self.env.pop(key)
|
||||
for key in ("GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE"):
|
||||
self.env.pop(key, None)
|
||||
|
||||
# ---- 夹具 ----
|
||||
|
||||
def svc(self, *args, check=True, env=None):
|
||||
proc = subprocess.run(
|
||||
["bash", str(self.root / "scripts" / "svc.sh"), *map(str, args)],
|
||||
cwd=self.root, env=env if env is not None else self.env, text=True,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60)
|
||||
if check:
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout)
|
||||
return proc
|
||||
|
||||
def register_agent(self, name="t1", port=8899):
|
||||
d = self.root / "runtime" / name
|
||||
(d / "data").mkdir(parents=True, exist_ok=True)
|
||||
(d / "svc.env").write_text(
|
||||
"kind=agent\nenv=%s\npid=\naddr=127.0.0.1:%d\ntags=\n"
|
||||
"source=dev@test123\nversion=stub\nstarted=\npidcheck=\n"
|
||||
"exec=--addr 127.0.0.1:%d --data %s\n" % (name, port, port, d / "data"))
|
||||
return d
|
||||
|
||||
def seed_db(self, name="t1", live_writer=True, hold_seconds=30):
|
||||
"""建 WAL 库并提交一行;live_writer=True 时留后台连接持有 WAL。"""
|
||||
db = self.root / "runtime" / name / "data" / "agent-system.db"
|
||||
if live_writer:
|
||||
self.writer_proc = subprocess.Popen(
|
||||
["python3", "-c", WRITER, str(db), str(hold_seconds)],
|
||||
stdout=subprocess.PIPE, text=True)
|
||||
self.assertEqual(self.writer_proc.stdout.readline().strip(), "ready")
|
||||
self.addCleanup(self.stop_writer)
|
||||
else:
|
||||
c = sqlite3.connect(db)
|
||||
c.execute("pragma journal_mode=wal")
|
||||
c.execute("create table t(x)")
|
||||
c.execute("insert into t values (42)")
|
||||
c.commit()
|
||||
c.close()
|
||||
return db
|
||||
|
||||
def stop_writer(self):
|
||||
if self.writer_proc and self.writer_proc.poll() is None:
|
||||
self.writer_proc.kill()
|
||||
self.writer_proc.wait(timeout=10)
|
||||
|
||||
def snapshots(self, name="t1"):
|
||||
root = self.root / "runtime-backups" / name
|
||||
if not root.is_dir():
|
||||
return []
|
||||
return sorted(p.name for p in root.iterdir() if p.is_dir())
|
||||
|
||||
# ---- 用例 ----
|
||||
|
||||
def test_backup_live_writer_consistent_and_outside_runtime(self):
|
||||
"""活跃 WAL 写入者:快照可读、行在、integrity ok;-wal/-shm 不入快照。"""
|
||||
self.register_agent()
|
||||
db = self.seed_db()
|
||||
self.assertTrue(db.with_name("agent-system.db-wal").exists()) # 前置:WAL 真实存在
|
||||
proc = self.svc("t1", "backup")
|
||||
self.assertIn("runtime-backups/t1/", proc.stdout)
|
||||
snaps = self.snapshots()
|
||||
self.assertEqual(len(snaps), 1)
|
||||
snap = self.root / "runtime-backups" / "t1" / snaps[0]
|
||||
# 位置在 runtime/ 之外(runtime 整删不波及)
|
||||
self.assertTrue(snap.exists())
|
||||
self.assertFalse((self.root / "runtime" / "t1" / "backups").exists())
|
||||
# 内容:登记 + 数据库一致性副本
|
||||
self.assertTrue((snap / "svc.env").exists())
|
||||
bak = snap / "data" / "agent-system.db"
|
||||
self.assertTrue(bak.exists())
|
||||
self.assertFalse((snap / "data" / "agent-system.db-wal").exists())
|
||||
self.assertFalse((snap / "data" / "agent-system.db-shm").exists())
|
||||
out = subprocess.run([SQLITE3, str(bak), "select x from t; pragma integrity_check;"],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(out.stdout.split(), ["42", "ok"], out.stderr)
|
||||
|
||||
def test_backup_manifest_sha_matches(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
snap = self.root / "runtime-backups" / "t1" / self.snapshots()[0]
|
||||
manifest = (snap / "MANIFEST").read_text()
|
||||
self.assertIn("service=t1\n", manifest)
|
||||
self.assertIn("source=dev@test123\n", manifest)
|
||||
entries = [l.split() for l in manifest.splitlines() if " ./" in l]
|
||||
self.assertTrue(entries)
|
||||
for sha, rel in entries:
|
||||
self.assertRegex(sha, r"^[0-9a-f]{64}$")
|
||||
self.assertEqual(
|
||||
subprocess.run(["sha256sum", str(snap / rel)],
|
||||
capture_output=True, text=True).stdout.split()[0], sha)
|
||||
|
||||
def test_backup_non_sqlite_db_kept_raw_without_leftovers(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
junk = self.root / "runtime" / "t1" / "data" / "blob.db"
|
||||
junk.write_bytes(b"plain-bytes-not-a-sqlite-file")
|
||||
proc = self.svc("t1", "backup")
|
||||
self.assertIn("blob.db", proc.stdout) # 告警原样保留
|
||||
snap = self.root / "runtime-backups" / "t1" / self.snapshots()[0]
|
||||
self.assertEqual((snap / "data" / "blob.db").read_bytes(), b"plain-bytes-not-a-sqlite-file")
|
||||
self.assertFalse(list(snap.rglob("*.bk"))) # 失败尝试的半成品不残留
|
||||
|
||||
def test_backup_target_exists_refused(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
dest = self.root / "runtime-backups" / "t1" / "20260101T000000Z"
|
||||
dest.mkdir(parents=True)
|
||||
(dest / "marker").write_text("x")
|
||||
proc = self.svc("t1", "backup", "--out", dest, check=False)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout)
|
||||
self.assertIn("已存在", proc.stdout)
|
||||
self.assertTrue((dest / "marker").exists()) # 不覆盖既有内容
|
||||
|
||||
def test_backup_custom_out_not_pruned(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
outdir = self.root / "custom-bk"
|
||||
self.svc("t1", "backup", "--out", outdir)
|
||||
self.assertTrue((outdir / "data" / "agent-system.db").exists())
|
||||
# 默认 root 无快照;再走默认路径备份 --keep 0 也不应动自定义目录
|
||||
self.svc("t1", "backup", "--keep", "0")
|
||||
self.assertEqual(self.snapshots(), [])
|
||||
self.assertTrue(outdir.exists())
|
||||
|
||||
def test_retention_and_listing(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
for _ in range(3):
|
||||
self.svc("t1", "backup")
|
||||
time.sleep(1.1) # 时间戳目录名唯一(UTC 秒级)
|
||||
self.assertEqual(len(self.snapshots()), 3)
|
||||
proc = self.svc("t1", "backups")
|
||||
for name in self.snapshots():
|
||||
self.assertIn(name, proc.stdout)
|
||||
# backup --keep 2 → 剩 2 份最新
|
||||
self.svc("t1", "backup", "--keep", "2")
|
||||
self.assertEqual(len(self.snapshots()), 2)
|
||||
# backups --prune --keep 1 → 剩最新 1 份
|
||||
self.svc("t1", "backups", "--prune", "--keep", "1")
|
||||
self.assertEqual(len(self.snapshots()), 1)
|
||||
|
||||
def test_backup_event_recorded(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
hist = (self.root / "runtime" / "t1" / "history.jsonl").read_text()
|
||||
self.assertIn('"event":"backup"', hist)
|
||||
self.assertIn('"result":"ok"', hist)
|
||||
self.assertIn("runtime-backups/t1/", hist)
|
||||
|
||||
def test_backup_rejects_postgres_with_db_hint(self):
|
||||
d = self.root / "runtime" / "pgx"
|
||||
d.mkdir(parents=True)
|
||||
(d / "svc.env").write_text("kind=postgres\nenv=pgx\naddr=127.0.0.1:8890\n")
|
||||
proc = self.svc("pgx", "backup", check=False)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout)
|
||||
self.assertIn("db backup", proc.stdout)
|
||||
|
||||
def test_backup_requires_registration(self):
|
||||
proc = self.svc("ghost", "backup", check=False)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout)
|
||||
|
||||
def test_backup_without_sqlite3_refuses(self):
|
||||
"""PATH 无 sqlite3 → 拒绝(不回退裸拷贝)。重建与真实 PATH 等价的工具目录、
|
||||
唯独不链 sqlite3——只测 command -v 失败这一条分支,其余行为不变形。"""
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
bindir = self.root / "bin-nosqlite3"
|
||||
bindir.mkdir()
|
||||
seen = set()
|
||||
for d in self.env.get("PATH", os.environ["PATH"]).split(os.pathsep):
|
||||
if not d or not os.path.isdir(d):
|
||||
continue
|
||||
for f in os.listdir(d):
|
||||
if f == "sqlite3" or f in seen or (bindir / f).exists():
|
||||
continue
|
||||
seen.add(f)
|
||||
try:
|
||||
os.symlink(os.path.join(d, f), bindir / f)
|
||||
except OSError:
|
||||
pass
|
||||
self.assertFalse((bindir / "sqlite3").exists())
|
||||
env = dict(self.env)
|
||||
env["PATH"] = str(bindir)
|
||||
proc = self.svc("t1", "backup", check=False, env=env)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout)
|
||||
self.assertIn("sqlite3", proc.stdout)
|
||||
self.assertEqual(self.snapshots(), []) # 无半成品快照
|
||||
|
||||
def test_rm_purge_removes_snapshots(self):
|
||||
self.register_agent()
|
||||
self.seed_db(live_writer=False)
|
||||
self.svc("t1", "backup")
|
||||
self.assertEqual(len(self.snapshots()), 1)
|
||||
self.svc("rm", "t1", "--purge")
|
||||
self.assertFalse((self.root / "runtime-backups" / "t1").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user