Add svc auto-recovery: autostart desired-state, recover, watchdog #396

Merged
weichen merged 12 commits from feature/svc-autorecover into main 2026-10-11 22:07:40 +00:00
12 Commits
Author SHA1 Message Date
weichen e0b49d1b90 Record the sync checkpoint
ci / test-pg (pull_request) Successful in 1m1s
ci / test (pull_request) Successful in 2m39s
Why: bind the task validation checkpoint to the latest integration baseline.

Evidence: SELF recursive component checks; sync_base=3de5791e0841e9c7c40c3dc236be6d95b3969827; candidate=c514bc4f4d8f3236d6d001fe9046fc2a6a72705a.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 21:56:41 +00:00
weichen c514bc4f4d Merge authoritative main into the task branch
Why: include the current integration baseline before verification.

Evidence: fetched origin/main=3de5791e0841e9c7c40c3dc236be6d95b3969827.

Impact: integrate upstream changes without rewriting task history.

Refs: task-sync
2026-10-11 21:56:13 +00:00
weichen e08ceebb95 Guard start paths with the ss-free port probe too
ci / test-pg (pull_request) Successful in 1m1s
ci / test (pull_request) Successful in 2m34s
Why: check_port_free consulted port_owner (ss -ltnp) only, so the
/dev/tcp fallback added to port_listening never ran on the start path;
CI (no iproute2) still started onto a thief-held port.

Evidence: no-ss simulation refuses a bound port and passes a free one;
full suite 10/10 and make test-governance rerun recorded on the card;
CI on the new head is authoritative.

Impact: scripts/svc.sh check_port_free only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 21:34:12 +00:00
weichen 9b5ecd57a2 Detect listening ports without ss via /dev/tcp fallback
ci / test-pg (pull_request) Successful in 1m0s
ci / test (pull_request) Failing after 1m6s
Why: the CI runner container has no iproute2, so port_listening went
blind: recover started a service onto a thief-held port and the health
probe hit the thief (HTTP 200), reporting a bogus start success.

Evidence: no-ss PATH simulation shows the fallback detects a bound port
and passes free ports; make test-governance rerun recorded on the task
card; CI on the new head is authoritative.

Impact: scripts/svc.sh port_listening only (ss stays primary when
present).

Refs: .task/feature-svc-autorecover.md
2026-10-11 21:12:16 +00:00
weichen b12e94dcd9 Wait for the port thief to bind before recovering
ci / test-pg (pull_request) Successful in 1m0s
ci / test (pull_request) Failing after 1m3s
Why: Popen returns before python http.server finishes binding, so on
slower CI runners recover won the race and started the service, turning
the port-stolen case into a flake.

Evidence: single-case rerun OK locally; make test-governance rerun
recorded on the task card; CI on the new head is authoritative.

Impact: test synchronization only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 20:49:50 +00:00
weichen cd43a62a87 Treat SIGKILL zombies as dead in recover tests
ci / test-pg (pull_request) Successful in 59s
ci / test (pull_request) Failing after 1m2s
Why: CI containers run a PID 1 that does not reap orphans, so the
killed stub lingers as a zombie and os.kill(pid, 0) keeps succeeding;
local dev machines reap instantly, which hid the difference.

Evidence: tests/test_svc_recover 10/10 after the fix; make
test-governance re-run recorded on the task card; CI on the new head
is the authoritative gate.

Impact: test helper only (svc.sh already treats zombies as dead via
empty-cmdline pidcheck).

Refs: .task/feature-svc-autorecover.md
2026-10-11 20:37:40 +00:00
weichen fe4027c46c Record post-sync governance rerun and sync checkpoint
ci / test-pg (pull_request) Failing after 1m13s
ci / test (pull_request) Failing after 1m29s
Why: task-sync advanced the base; key checks re-run at the merged head per the evidence gate.

Evidence: make evidence appended the post-sync make test-governance run (146/146 OK); sync row recorded by task-sync.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 20:23:22 +00:00
weichen ab39c796ea Record the sync checkpoint
Why: bind the task validation checkpoint to the latest integration baseline.

Evidence: SELF recursive component checks; sync_base=acc8bf124c04b89ed05093467ebce865934bdddc; candidate=59109e589c8a942628d2eac88e57faeddae7e882.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 20:22:09 +00:00
weichen 59109e589c Merge authoritative main into the task branch
Why: include the current integration baseline before verification.

Evidence: fetched origin/main=acc8bf124c04b89ed05093467ebce865934bdddc.

Impact: integrate upstream changes without rewriting task history.

Refs: task-sync
2026-10-11 20:21:43 +00:00
weichen 9bb7eb579a Record governance test evidence on the task card
Why: the completion claim needs tool-written evidence per the evidence gate.

Evidence: make evidence wrote the make test-governance run (146/146 OK) into the card.

Impact: task metadata only.

Refs: .task/feature-svc-autorecover.md
2026-10-11 20:21:41 +00:00
weichen eda942523c Add svc auto-recovery: autostart desired-state, recover, watchdog
Why: svc services are bare nohup'd processes with persisted registration
but no supervision -- a crash, an OOM kill, or a reboot left them dead
until manual restart (2026-10-11 outage: dev service lost, runtime/ also
wiped by hand, nothing brought services back).

Evidence: tests/test_svc_recover.py 10 cases green (start records
autostart, stop clears it incl. dead-process stop, SIGKILL then recover
revives with persisted exec=, lock-held skips, port-stolen fails with
recover fail event, watchdog unit text via --print); make test-governance
146/146 OK locally; real-machine watchdog install + kill -9 revival to be
recorded on the task card.

Impact: svc.sh gains autostart key in svc.env, `recover [name]
[--quiet]` (postgres-first, per-service .start.lock, recover history
events), `watchdog install|remove|status` (systemd user timer, cron
fallback hint), lifecycle lock across start/stop/restart/deploy/update;
spawned services no longer inherit the lock fd (9>&-), and lock release
no longer clobbers script stderr. Makefile adds svc-recover/svc-watchdog;
docs/svc.md documents semantics and boundaries.

Refs: task card .task/feature-svc-autorecover.md; docs/svc.md 6.5/6.6;
parallel planning card feature/service-restart-recovery-plan (same
problem space, plan-stage).
2026-10-11 20:20:27 +00:00
weichen 552b28e307 Record the task card for feature-svc-autorecover
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-svc-autorecover.md; R0=5e57029b439e5690f28dc6002f9784f0b08920eb.

Impact: task metadata only; the feature workspace starts at 5e57029b43.

Refs: .task/feature-svc-autorecover.md
2026-10-11 16:09:04 +00:00