Add svc auto-recovery: autostart desired-state, recover, watchdog #396

Merged
weichen merged 12 commits from feature/svc-autorecover into main 2026-10-11 22:07:40 +00:00
Owner

svc 自恢复:autostart 期望态 + recover + watchdog

任务卡(同一份文档):.task/feature-svc-autorecover.md(分支内,证据表含 precheck/task-sync/测试三段,均由 make evidence 写入)。

背景(2026-10-11 事故)

  • svc 服务是裸 nohup 进程:崩溃 / OOM 被杀 / 重启后无人拉回;
  • 另 runtime/ 目录当日 15:38 被整删(非 OOM 行为,人工/其他终端所为),SQLite 数据随删丢失——登记与数据其实一直持久化在 runtime//,缺的是自动恢复与备份意识。

变更

  • autostart 期望态(svc.env):start/restart/deploy/update 健康检查通过置 1;显式 stop(含前置停机)移除;进程自行死亡保留。stop 后不被复活,deploy 失败不盲目重启。
  • svc.sh recover [name] [--quiet]:postgres 优先扫描,对 autostart=1 且已死的服务裸 start 拉回(复用持久化 exec= 与数据库绑定);逐服务非阻塞持 .start.lock;写 recover ok/fail 事件。
  • svc.sh watchdog install|remove|status:systemd 用户 timer 每分钟 recover(~/.config/systemd/user/agent-system-svc-recover.*),无用户会话环境给 cron 降级指引;linger 提示。
  • 生命周期互斥锁:start/stop/restart/deploy/update 持锁,防 recover 在停机-换产物间隙抢跑;修复两个实测 bug——服务进程继承锁 fd(9>&- 显式关闭)与 exec 重定向吞掉整个脚本 stderr。
  • Makefile:svc-recover / svc-watchdog;docs/svc.md §6.5/§6.6 与脚本帮助同步。

验证

  • 新增 tests/test_svc_recover.py:10 用例(SIGKILL→recover 拉回、stop 不复活、锁占用跳过、端口被抢 fail 事件、watchdog 单元文本),本地 10/10;
  • make test-governance 146/146(sync 后重跑,证据在任务卡);
  • CI(PR HEAD make test + make build)为权威。

关联

  • 并行规划卡 feature/service-restart-recovery-plan(同一问题域,方案阶段):本 PR 即其可落地的实现参考。

Why: services needed crash/OOM auto-recovery.
Evidence: task card evidence table (SELF) + CI on PR HEAD (GATE).
Impact: scripts/svc.sh, Makefile, docs/svc.md, tests/test_svc_recover.py.

## svc 自恢复:autostart 期望态 + recover + watchdog 任务卡(同一份文档):`.task/feature-svc-autorecover.md`(分支内,证据表含 precheck/task-sync/测试三段,均由 make evidence 写入)。 ### 背景(2026-10-11 事故) - svc 服务是裸 nohup 进程:崩溃 / OOM 被杀 / 重启后无人拉回; - 另 runtime/ 目录当日 15:38 被整删(非 OOM 行为,人工/其他终端所为),SQLite 数据随删丢失——登记与数据其实一直持久化在 runtime/<name>/,缺的是自动恢复与备份意识。 ### 变更 - **autostart 期望态**(svc.env):start/restart/deploy/update 健康检查通过置 1;显式 stop(含前置停机)移除;进程自行死亡保留。stop 后不被复活,deploy 失败不盲目重启。 - **`svc.sh recover [name] [--quiet]`**:postgres 优先扫描,对 autostart=1 且已死的服务裸 start 拉回(复用持久化 exec= 与数据库绑定);逐服务非阻塞持 `.start.lock`;写 recover ok/fail 事件。 - **`svc.sh watchdog install|remove|status`**:systemd 用户 timer 每分钟 recover(~/.config/systemd/user/agent-system-svc-recover.*),无用户会话环境给 cron 降级指引;linger 提示。 - **生命周期互斥锁**:start/stop/restart/deploy/update 持锁,防 recover 在停机-换产物间隙抢跑;修复两个实测 bug——服务进程继承锁 fd(9>&- 显式关闭)与 exec 重定向吞掉整个脚本 stderr。 - Makefile:svc-recover / svc-watchdog;docs/svc.md §6.5/§6.6 与脚本帮助同步。 ### 验证 - 新增 tests/test_svc_recover.py:10 用例(SIGKILL→recover 拉回、stop 不复活、锁占用跳过、端口被抢 fail 事件、watchdog 单元文本),本地 10/10; - make test-governance 146/146(sync 后重跑,证据在任务卡); - CI(PR HEAD make test + make build)为权威。 ### 关联 - 并行规划卡 feature/service-restart-recovery-plan(同一问题域,方案阶段):本 PR 即其可落地的实现参考。 Why: services needed crash/OOM auto-recovery. Evidence: task card evidence table (SELF) + CI on PR HEAD (GATE). Impact: scripts/svc.sh, Makefile, docs/svc.md, tests/test_svc_recover.py.
weichen added 6 commits 2026-10-11 20:23:41 +00:00
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-svc-autorecover.md; R0=5e57029b439e5690f28dc6002f9784f0b08920eb.

Impact: task metadata only; the feature workspace starts at 5e57029b43.

Refs: .task/feature-svc-autorecover.md
Why: svc services are bare nohup'd processes with persisted registration
but no supervision -- a crash, an OOM kill, or a reboot left them dead
until manual restart (2026-10-11 outage: dev service lost, runtime/ also
wiped by hand, nothing brought services back).

Evidence: tests/test_svc_recover.py 10 cases green (start records
autostart, stop clears it incl. dead-process stop, SIGKILL then recover
revives with persisted exec=, lock-held skips, port-stolen fails with
recover fail event, watchdog unit text via --print); make test-governance
146/146 OK locally; real-machine watchdog install + kill -9 revival to be
recorded on the task card.

Impact: svc.sh gains autostart key in svc.env, `recover [name]
[--quiet]` (postgres-first, per-service .start.lock, recover history
events), `watchdog install|remove|status` (systemd user timer, cron
fallback hint), lifecycle lock across start/stop/restart/deploy/update;
spawned services no longer inherit the lock fd (9>&-), and lock release
no longer clobbers script stderr. Makefile adds svc-recover/svc-watchdog;
docs/svc.md documents semantics and boundaries.

Refs: task card .task/feature-svc-autorecover.md; docs/svc.md 6.5/6.6;
parallel planning card feature/service-restart-recovery-plan (same
problem space, plan-stage).
Why: the completion claim needs tool-written evidence per the evidence gate.

Evidence: make evidence wrote the make test-governance run (146/146 OK) into the card.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
Why: include the current integration baseline before verification.

Evidence: fetched origin/main=acc8bf124c04b89ed05093467ebce865934bdddc.

Impact: integrate upstream changes without rewriting task history.

Refs: task-sync
Why: bind the task validation checkpoint to the latest integration baseline.

Evidence: SELF recursive component checks; sync_base=acc8bf124c04b89ed05093467ebce865934bdddc; candidate=59109e589c8a942628d2eac88e57faeddae7e882.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
Record post-sync governance rerun and sync checkpoint
ci / test-pg (pull_request) Failing after 1m13s
ci / test (pull_request) Failing after 1m29s
fe4027c46c
Why: task-sync advanced the base; key checks re-run at the merged head per the evidence gate.

Evidence: make evidence appended the post-sync make test-governance run (146/146 OK); sync row recorded by task-sync.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
weichen added 1 commit 2026-10-11 20:37:41 +00:00
Treat SIGKILL zombies as dead in recover tests
ci / test-pg (pull_request) Successful in 59s
ci / test (pull_request) Failing after 1m2s
cd43a62a87
Why: CI containers run a PID 1 that does not reap orphans, so the
killed stub lingers as a zombie and os.kill(pid, 0) keeps succeeding;
local dev machines reap instantly, which hid the difference.

Evidence: tests/test_svc_recover 10/10 after the fix; make
test-governance re-run recorded on the task card; CI on the new head
is the authoritative gate.

Impact: test helper only (svc.sh already treats zombies as dead via
empty-cmdline pidcheck).

Refs: .task/feature-svc-autorecover.md
weichen added 1 commit 2026-10-11 20:49:51 +00:00
Wait for the port thief to bind before recovering
ci / test-pg (pull_request) Successful in 1m0s
ci / test (pull_request) Failing after 1m3s
b12e94dcd9
Why: Popen returns before python http.server finishes binding, so on
slower CI runners recover won the race and started the service, turning
the port-stolen case into a flake.

Evidence: single-case rerun OK locally; make test-governance rerun
recorded on the task card; CI on the new head is authoritative.

Impact: test synchronization only.

Refs: .task/feature-svc-autorecover.md
weichen added 1 commit 2026-10-11 21:12:19 +00:00
Detect listening ports without ss via /dev/tcp fallback
ci / test-pg (pull_request) Successful in 1m0s
ci / test (pull_request) Failing after 1m6s
9b5ecd57a2
Why: the CI runner container has no iproute2, so port_listening went
blind: recover started a service onto a thief-held port and the health
probe hit the thief (HTTP 200), reporting a bogus start success.

Evidence: no-ss PATH simulation shows the fallback detects a bound port
and passes free ports; make test-governance rerun recorded on the task
card; CI on the new head is authoritative.

Impact: scripts/svc.sh port_listening only (ss stays primary when
present).

Refs: .task/feature-svc-autorecover.md
weichen added 1 commit 2026-10-11 21:34:15 +00:00
Guard start paths with the ss-free port probe too
ci / test-pg (pull_request) Successful in 1m1s
ci / test (pull_request) Successful in 2m34s
e08ceebb95
Why: check_port_free consulted port_owner (ss -ltnp) only, so the
/dev/tcp fallback added to port_listening never ran on the start path;
CI (no iproute2) still started onto a thief-held port.

Evidence: no-ss simulation refuses a bound port and passes a free one;
full suite 10/10 and make test-governance rerun recorded on the card;
CI on the new head is authoritative.

Impact: scripts/svc.sh check_port_free only.

Refs: .task/feature-svc-autorecover.md
weichen added 2 commits 2026-10-11 21:56:46 +00:00
Why: include the current integration baseline before verification.

Evidence: fetched origin/main=3de5791e0841e9c7c40c3dc236be6d95b3969827.

Impact: integrate upstream changes without rewriting task history.

Refs: task-sync
Record the sync checkpoint
ci / test-pg (pull_request) Successful in 1m1s
ci / test (pull_request) Successful in 2m39s
e0b49d1b90
Why: bind the task validation checkpoint to the latest integration baseline.

Evidence: SELF recursive component checks; sync_base=3de5791e0841e9c7c40c3dc236be6d95b3969827; candidate=c514bc4f4d8f3236d6d001fe9046fc2a6a72705a.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
weichen merged commit bbce8db905 into main 2026-10-11 22:07:40 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Dev#396