Why: Batch update --all left SVC_PIDCHECK from the prior service in
memory, so pid_alive/reclaim_own_port misclassified the next service's
process as dead/foreign, deleted svc.pid, and failed start on a still-
occupied port (genui/rdm1/rdmcand after successful dev).
Evidence: make test-governance (SELF; receipt /tmp/svc-pidcheck-gov.json);
new test_batch_update_stops_second_service_despite_prior_pidcheck +
SVC_PIDCHECK ratchet in test_svc_precheck.
Impact: env_config clears SVC_PIDCHECK/EXEC/VERSION/SOURCE per service;
batch update correctly stops and restarts each agent. After merge:
restore genui/rdm1/rdmcand and move rdmcand to its own data dir.
Refs: .task/feature-svc-pidcheck-leak.md; complements #456 reclaim_own_port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: genui update cleared svc.pid without killing the process, then
overwrote the binary and failed start with a false "non-ours" port error.
Evidence: make test-governance (receipt /tmp/svc-reclaim-governance.json)
Impact: reclaim_own_port walks ss listener ancestry; stop/check_port/install
wire it; orphans matching pidcheck are stopped, foreign occupants still refused
Refs: fix/svc-reclaim-own-port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: backend U4 password gate returns 403 on business APIs until must_change clears; CLI-created load passwords are terminal.
Evidence: dual smoke failed POST /api/projects 403 after login_keys pin.
Impact: load suite works on schema v26+; EXIT trap disables re-entry and waits for serve.
Refs: server enforcePasswordGate; CreateAccount must_change
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: background subshell PID was recorded instead of serve, so cleanup left orphans that stole ports and broke the next smoke.
Evidence: prior dual evidence failed with bind address already in use + probe 401 against stale instance.
Impact: trap kill now terminates the real serve process.
Refs: tests/load/collaboration/run_load.sh
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Backend login_keys gate blocks Key traffic while
must_change_password is set; U7 failed on PUT settings/user 403.
Evidence: go test -c multi-instance compiles; CI test-pg will GATE.
Impact: loginAlice does session password change alicePass0→alicePass.
Refs: Dev PR #426; Backend #170 pin e7b70c2
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b431777c9fa9c74f096d7f14fce40d1beba53fe2.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Dev #429 conflicted on scripts/svc.sh (U1 user vs main backup) and Backend gitlink.
Evidence: conflict hunks resolved; actions include init|user|backup|backups
Impact: PR #429 should become mergeable after push; both U1 binding and agent backup entrypoints preserved.
Refs: Dev pulls/429; main backup #417; Backend #171
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: task-sync hit a submodule conflict; pin must track Backend #170
HEAD after rebase onto main (copyOrder fix).
Evidence: backend 0c79618; origin/main merge completed in worktree.
Impact: modules/backend gitlink; sync with main.
Refs: Backend #170; Dev #426
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=ae465d24441ab349c961a55e9b04f8c4ccaf6de8.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3e4a79e4133202b537c888a8beeb7c10d82595f4.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: the 2026-10-11 runtime/ deletion lost all agent SQLite data with no
backup; the svc-autorecover closing report left "agent SQLite backup" as
a follow-up and the user asked to land it (recovery plan U2 residue).
Evidence: tests/test_svc_backup.py 11/11 green locally (live WAL writer,
manifest sha, retention, guards); make test-governance full suite green
(159 tests, rerun twice after one unreproduced flake).
Impact: new service-level actions backup/backups; snapshots live in
runtime-backups/<name>/ outside runtime/; rm --purge removes them; pg
backup and other flows unchanged.
Refs: modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2 (sec 6, U2 residue); task card .task/feature-svc-agent-backup.md.
Why: collaboration storage U7 (proposal sections 3/5/7) requires two-process
HTTP-level acceptance over shared PG, a reproducible load-test entry honoring
the load contract, and an auditable pre-release checklist; capacity windows and
dual-host runs need user-scheduled resources, so the deliverable here is the
verifiable entry, not capacity claims.
Evidence: make test green (composite); make test-multi-instance green locally
against an isolated PG (five scenarios incl. crash-takeover and restart
invariants); run_load.sh --profile smoke end-to-end green (SSE probe p95~18ms,
zero seq duplicates, stack auto-recycled); backend pinned to feature branch
with the three cross-instance fixes.
Impact: new tests/integration/multi-instance (CI test-pg job step), new
tests/load/collaboration (k6 + SSE probe + run_load.sh), release checklist at
docs/release-multi-instance.md; no runtime behavior change in Dev repo itself.
Refs: backend PR #164; task card feature/u7-load-fault-release.
Why: systemd's default KillMode=control-group kills every process in
the unit's cgroup when the oneshot finishes -- the daemon recover just
started died seconds after the recover ok event (observed live on the
host: revived at 22:13:40, gone by 22:13:42).
Evidence: unit text now carries KillMode=process (asserted in
test_watchdog_print_plan); suite 10/10 and make test-governance rerun
recorded on the card; live re-demo on the host after merge.
Impact: scripts/svc.sh wd_service_text only.
Refs: .task/feature-svc-autorecover.md
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3de5791e0841e9c7c40c3dc236be6d95b3969827.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Popen returns before python http.server finishes binding, so on
slower CI runners recover won the race and started the service, turning
the port-stolen case into a flake.
Evidence: single-case rerun OK locally; make test-governance rerun
recorded on the task card; CI on the new head is authoritative.
Impact: test synchronization only.
Refs: .task/feature-svc-autorecover.md
Why: RDM capabilities (two-trees-one-net, rdm.* trace-audit command face)
only exist in the downstream Agent-System-RDM composite binary; svc's three
existing sources all produce the Dev binary, so none of it could be deployed
or lifecycle-managed. This adds --from rdm[@<sha>] with the same isolated
source-repo semantics as the dev source.
Evidence: make test-governance 138/138 (two new stub cases: deploy E2E and
commits/update track); real deployment rdm@2aef3fc recorded in the task card
(pins materialized, pure-Go build, web workbench to htdocs, serve health OK,
rdm.ping/rdm.inspect.capabilities committed, update --check reports latest);
make test full suite exit=0. All segments written by make evidence.
Impact: scripts/svc.sh (rdm source resolution/materialization/deploy/update/
commits, probe timeouts, rdm seed skip, ELF env-DSN evidence fallback),
docs/svc.md 5.4, tests/test_svc.py. RDM-side CLI alignment landed as RDM
PR #228; no base submodules touched.
Refs: task card .task/feature-svc-rdm-source.md; RDM PR #228/#229.
Why: CI containers run a PID 1 that does not reap orphans, so the
killed stub lingers as a zombie and os.kill(pid, 0) keeps succeeding;
local dev machines reap instantly, which hid the difference.
Evidence: tests/test_svc_recover 10/10 after the fix; make
test-governance re-run recorded on the task card; CI on the new head
is the authoritative gate.
Impact: test helper only (svc.sh already treats zombies as dead via
empty-cmdline pidcheck).
Refs: .task/feature-svc-autorecover.md
Why: svc services are bare nohup'd processes with persisted registration
but no supervision -- a crash, an OOM kill, or a reboot left them dead
until manual restart (2026-10-11 outage: dev service lost, runtime/ also
wiped by hand, nothing brought services back).
Evidence: tests/test_svc_recover.py 10 cases green (start records
autostart, stop clears it incl. dead-process stop, SIGKILL then recover
revives with persisted exec=, lock-held skips, port-stolen fails with
recover fail event, watchdog unit text via --print); make test-governance
146/146 OK locally; real-machine watchdog install + kill -9 revival to be
recorded on the task card.
Impact: svc.sh gains autostart key in svc.env, `recover [name]
[--quiet]` (postgres-first, per-service .start.lock, recover history
events), `watchdog install|remove|status` (systemd user timer, cron
fallback hint), lifecycle lock across start/stop/restart/deploy/update;
spawned services no longer inherit the lock fd (9>&-), and lock release
no longer clobbers script stderr. Makefile adds svc-recover/svc-watchdog;
docs/svc.md documents semantics and boundaries.
Refs: task card .task/feature-svc-autorecover.md; docs/svc.md 6.5/6.6;
parallel planning card feature/service-restart-recovery-plan (same
problem space, plan-stage).
Why: seed_settings PUTs /api/settings/user without credentials, so under the PG tier's always-on auth the seed always 401s and model config silently never lands (reproduced on the dev instance 2026-10-11; settings had to be written by hand).
Evidence: make test-governance && make test exit=0 (receipt 8ffc3a89); tests/test_svc.py adds three stub-based cases - auth-on seeds with a Bearer header and create/revoke around the PUT, --seed-owner skips account lookup, auth-off stays credential-free; 14/14 svc tests green.
Impact: deployments that pass SVC_MODEL_*/SVC_TAVILY_KEYS now seed reliably regardless of auth mode; a temporary apikey (name=svc-seed) is minted for the first admin (or --seed-owner/SVC_SEED_OWNER) and revoked right after; auth-off path unchanged.
Refs: task card feature-svc-seed-auth.
Why: main landed fix-svc-dsn-reuse-dbref (reuse-path dbref restore, env-DSN
aware precheck, DSN_ENV scoping) for the same restart-DSN incident; the
approved recovery plan is its superset and replaces those mechanisms with
resolve_db_config + recoverable svc.env keys + refuse-on-invalid semantics.
Evidence: tests/test_svc_precheck.py ported to the new units (18 cases green,
same scenario matrix; broken-dbref expectation updated to refusal per the
approved plan); tests/test_svc.py 11 cases green; bash -n OK.
Impact: conflict hunks resolved toward the unified resolver; main's source
guard at the script tail restored; agent_listen_rejected kept as a pure
predicate (used by the start precheck and the governance tests).
Refs: main 7acf318 (PR #317); task card .task/feature-svc-db-config-recover.md.
Why: the exec= reuse path stripped --dsn into the process environment (G1-7)
and record_state then persisted a DSN-less exec=, while the dbref fallback only
existed in the explicit-rebuild branch - the second bare restart after a
rebind silently dropped the database (SQLite fallback on loopback). Auth
preflight also matched --dsn on the command line only, so env-delivered DSN
with auth=auto was misjudged as unauthenticated.
Evidence: bash -n OK; make test-governance 115 tests OK (new tests/test_svc.py
11 cases; 10/11 fail against the pre-fix script); make test green; make build
OK; isolated real PostgreSQL 18.6 + real binary validation 27/27 (consecutive
restarts/deploy keep binding, psql-verified database identity, DB-down restart
fails without SQLite fallback or config loss, no credential leakage) - all
recorded via make evidence in the task card.
Impact: svc.env gains db=/dbref=/dsn= recovery keys (atomic 0600 writes);
start/restart/deploy/update/db share one resolver; invalid bindings, config
conflicts, mock coexistence and env-DSN-incapable binaries now refuse to
start instead of silently downgrading; legacy exec= DSN migrates idempotently;
docs/svc.md and script help updated.
Refs: task card .task/feature-svc-db-config-recover.md; user-approved plan
(2026-10-11 session message); AGENTS.md evidence gates.
Why: G1-7 moved the DSN into AGENT_SYSTEM_DSN, but the precheck only read
the arg string, so a non-loopback restart with auth auto was wrongly
rejected (2026-10-10 dev incident); the reuse path no longer restored the
dbref-bound DSN and would silently fall back to SQLite; and DSN_ENV leaked
across same-process starts (update --all) into other services' env.
Evidence: tests/test_svc_precheck.py 19 cases green (auth matrix vs
config.go AuthEnabled, reuse migration/dbref restore, scope isolation);
full governance suite 123 green; incident-scenario replay allowed + DSN
carried. Recorded via make evidence.
Impact: scripts/svc.sh precheck and reuse-start DSN handling only; no
binary change, no persistence-format change.
Refs: .task/fix-svc-dsn-reuse-dbref.md; config.go AuthEnabled (on || (not
off && DSN non-empty), env DSN counts).
Why: git receive-pack runs 'gc --auto' in the background after a push;
the fixture's TemporaryDirectory teardown then races the gc writer and
fails with OSError 39 (remote.git not empty) — observed on the PR #285
runner (job 1967). The fixture remote holds no durable value, so auto-gc
is pure flake surface.
Evidence: the failing test passes 5/5 locally after the fix; full
make test-governance green (104 tests).
Impact: test fixture only; no governance-script behavior change.
Refs: Dev CI run 1438 job 1967; same race family as the upload-host
TempDir flake fixed earlier.
Why: preserve project direction across tasks and require grounded next-task recommendations.
Evidence: the generated task-card regression fails before the template update and passes afterward; original SELF receipts are in the task card.
Impact: document the planning entrypoint and extend task records without granting new-task execution rights.
Refs: .task/feature-goal-continuity.md; policy/delivery.md
Why: archived task cards change directory and must retain an unambiguous policy reference.
Evidence: review finding 1, focused red-green proof and 104 passing governance tests are preserved in the task card.
Impact: use a repository-root text reference; record the completed review and its resolved finding.
Refs: .task/feature-iterative-delivery.md; .task/reviews/iterative-delivery.json
Why: main advanced with merged-task-name and legacy evidence guards during this task.
Evidence: SELF 44 lifecycle regressions pass after resolving both hunks; independent review repairs are recorded in the task card.
Impact: retains both task-start protections and the stronger JSON/legacy receipt reader, without rewriting either branch history.
Refs: Dev PR #211 and #212; .task/feature-governance-reliability.md
Why: independent review found shared submodule metadata risk, quoted header leakage, stale JSON receipt blindness and missing abandon coverage.
Evidence: SELF 92 governance regressions pass; the original review receipt and replay output are recorded in the task card.
Impact: preserves external component worktrees, handles both evidence formats and adds targeted failure-path coverage; no product behavior changes.
Refs: .task/feature-governance-reliability.md; .task/reviews/governance-reliability.json
Why: cleanup, component synchronization, receipt recording and credential handling could contradict the documented development boundaries.
Evidence: SELF 75 isolated governance tests and source checks are captured in .task/feature-governance-reliability.md; combination and CI verification follow.
Impact: hardens local task and CI tooling, aligns policy and component documentation, and adds continuous regression tests without changing product behavior.
Refs: .task/feature-governance-reliability.md; policy/governance.md; policy/platform-admission.md
weichenandCursor