Why: The restored postmaster refused to start because the results-directory socket path exceeded the 107-byte Unix socket limit.
Evidence: restore/postgres.log from backup-20261012-150713 says the socket path is too long (SELF). bash -n passed.
Impact: The restored instance listens on /tmp/u7bk-18997, and cleanup removes that directory.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The checklist had no measured restore path for a physical backup, WAL archive, or pg_dump onto a new target.
Evidence: bash -n, go test in tests/ops/drill_driver, and run_backup_drill.sh --self-check passed locally (SELF). The timed run is recorded separately.
Impact: Isolated u7bk only. Upload responses keep artifactId so the restore check can download the same objects.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The first pg-stop window counted an in-flight 200 at shutdown, and five writes on the timeline conversation returned 409 while a mock run held it. That measured the probe, not a proven database failure.
Evidence: go test ./... in tests/ops/drill_driver passed locally (SELF).
Impact: The outage clock starts 2s after stop or kill. RTO is five writes on new conversations. Idempotency replays only the window and retries 409 for up to 2 minutes. The Redis phase no longer uses local outside a function.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The release checklist cannot mark disconnect recovery until an isolated drill records bounded failures, no SQLite fallback, and idempotent writes.
Evidence: bash -n passed, drill_driver go test passed, unknown arguments exit 2. The timed drill is recorded next.
Impact: New drill entry only. No product change and no release decision.
Refs: docs/release-multi-instance.md C4
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: C4 and C5 need a shared way to start, refuse, and tear down stacks without touching production ports or processes.
Evidence: tests/ops/self_check.sh passed (help, unknown arg, foreign name, production port, busy port, SIGINT cleanup). drill_driver go test passed.
Impact: New tests/ops library and HTTP driver. No drill result and no release-gate change yet.
Refs: docs/release-multi-instance.md C4 C5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A timed window that starts under load average above 16 is not a usable baseline, and creating the instance first only to pause wastes a startup.
Evidence: The rerun at load 24.03 exited 3 before any timed window and removed the instance.
Impact: The pause now happens before initdb. No change to the benchmark itself.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The percentile reader also opened the pgbench stdout file, so the first run exited 1 after six successful windows.
Evidence: Parser check on results/pgbench-20261012-131229 select-only-10 gave p50 0.018 ms, matching the reported average. The timed rerun is recorded next.
Impact: Percentile parsing only. The six windows themselves had zero failed transactions.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b4b042630a352eca2a399a7a64931c74978bca7f.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: make the six instance-bound user commands discoverable (T5-1) and pin Documents r2/r3.
Evidence: python3 -m unittest tests.test_svc_user_help; existing test_svc_user_binding still green.
Impact: scripts/svc.sh help accounts|user; docs/svc.md topic list; documents gitlink to plan branch tip.
Refs: Documents #87; feature/svc-account-durability-plan
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=c6fefa3435dc5d9524fc35483f1d34fa1a6e0d59.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
# Conflicts:
# tests/test_svc.py
Why: Batch update --all left SVC_PIDCHECK from the prior service in
memory, so pid_alive/reclaim_own_port misclassified the next service's
process as dead/foreign, deleted svc.pid, and failed start on a still-
occupied port (genui/rdm1/rdmcand after successful dev).
Evidence: make test-governance (SELF; receipt /tmp/svc-pidcheck-gov.json);
new test_batch_update_stops_second_service_despite_prior_pidcheck +
SVC_PIDCHECK ratchet in test_svc_precheck.
Impact: env_config clears SVC_PIDCHECK/EXEC/VERSION/SOURCE per service;
batch update correctly stops and restarts each agent. After merge:
restore genui/rdm1/rdmcand and move rdmcand to its own data dir.
Refs: .task/feature-svc-pidcheck-leak.md; complements #456 reclaim_own_port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: genui update cleared svc.pid without killing the process, then
overwrote the binary and failed start with a false "non-ours" port error.
Evidence: make test-governance (receipt /tmp/svc-reclaim-governance.json)
Impact: reclaim_own_port walks ss listener ancestry; stop/check_port/install
wire it; orphans matching pidcheck are stopped, foreign occupants still refused
Refs: fix/svc-reclaim-own-port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: the agent stub left python http.server children when tearDown
SIGKILL'd the bash parent, leaking ports 1880x into recover/identity
suites that share the same range.
Evidence: local suite re-run pending after this fix.
Impact: stub HTTP child exits when ppid changes (same as recover stub).
Refs: .task/feature-svc-rdm-ui-gate.md
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: RDM agent deploy left workbench htdocs unserved (prompt-only), so
environments like dev silently offered API-only 404 pages after a source
switch; static first-start also misclassified http.server as dead when
pidcheck was unset.
Evidence: CMD="make test-governance" make evidence (200 ok, 1 skipped; SELF).
Impact: rdm deploy auto-starts <name>web and asserts UI 200; cross-kind
redeploy needs --allow-source-switch; list/info flag missing workbench.
Refs: .task/feature-svc-rdm-ui-gate.md; docs/svc.md §5.4
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: backend U4 password gate returns 403 on business APIs until must_change clears; CLI-created load passwords are terminal.
Evidence: dual smoke failed POST /api/projects 403 after login_keys pin.
Impact: load suite works on schema v26+; EXIT trap disables re-entry and waits for serve.
Refs: server enforcePasswordGate; CreateAccount must_change
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: background subshell PID was recorded instead of serve, so cleanup left orphans that stole ports and broke the next smoke.
Evidence: prior dual evidence failed with bind address already in use + probe 401 against stale instance.
Impact: trap kill now terminates the real serve process.
Refs: tests/load/collaboration/run_load.sh
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Backend login_keys gate blocks Key traffic while
must_change_password is set; U7 failed on PUT settings/user 403.
Evidence: go test -c multi-instance compiles; CI test-pg will GATE.
Impact: loginAlice does session password change alicePass0→alicePass.
Refs: Dev PR #426; Backend #170 pin e7b70c2
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b431777c9fa9c74f096d7f14fce40d1beba53fe2.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Dev #429 conflicted on scripts/svc.sh (U1 user vs main backup) and Backend gitlink.
Evidence: conflict hunks resolved; actions include init|user|backup|backups
Impact: PR #429 should become mergeable after push; both U1 binding and agent backup entrypoints preserved.
Refs: Dev pulls/429; main backup #417; Backend #171
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: task-sync hit a submodule conflict; pin must track Backend #170
HEAD after rebase onto main (copyOrder fix).
Evidence: backend 0c79618; origin/main merge completed in worktree.
Impact: modules/backend gitlink; sync with main.
Refs: Backend #170; Dev #426
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=ae465d24441ab349c961a55e9b04f8c4ccaf6de8.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3e4a79e4133202b537c888a8beeb7c10d82595f4.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: the 2026-10-11 runtime/ deletion lost all agent SQLite data with no
backup; the svc-autorecover closing report left "agent SQLite backup" as
a follow-up and the user asked to land it (recovery plan U2 residue).
Evidence: tests/test_svc_backup.py 11/11 green locally (live WAL writer,
manifest sha, retention, guards); make test-governance full suite green
(159 tests, rerun twice after one unreproduced flake).
Impact: new service-level actions backup/backups; snapshots live in
runtime-backups/<name>/ outside runtime/; rm --purge removes them; pg
backup and other flows unchanged.
Refs: modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2 (sec 6, U2 residue); task card .task/feature-svc-agent-backup.md.
Why: collaboration storage U7 (proposal sections 3/5/7) requires two-process
HTTP-level acceptance over shared PG, a reproducible load-test entry honoring
the load contract, and an auditable pre-release checklist; capacity windows and
dual-host runs need user-scheduled resources, so the deliverable here is the
verifiable entry, not capacity claims.
Evidence: make test green (composite); make test-multi-instance green locally
against an isolated PG (five scenarios incl. crash-takeover and restart
invariants); run_load.sh --profile smoke end-to-end green (SSE probe p95~18ms,
zero seq duplicates, stack auto-recycled); backend pinned to feature branch
with the three cross-instance fixes.
Impact: new tests/integration/multi-instance (CI test-pg job step), new
tests/load/collaboration (k6 + SSE probe + run_load.sh), release checklist at
docs/release-multi-instance.md; no runtime behavior change in Dev repo itself.
Refs: backend PR #164; task card feature/u7-load-fault-release.
Why: systemd's default KillMode=control-group kills every process in
the unit's cgroup when the oneshot finishes -- the daemon recover just
started died seconds after the recover ok event (observed live on the
host: revived at 22:13:40, gone by 22:13:42).
Evidence: unit text now carries KillMode=process (asserted in
test_watchdog_print_plan); suite 10/10 and make test-governance rerun
recorded on the card; live re-demo on the host after merge.
Impact: scripts/svc.sh wd_service_text only.
Refs: .task/feature-svc-autorecover.md
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3de5791e0841e9c7c40c3dc236be6d95b3969827.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Popen returns before python http.server finishes binding, so on
slower CI runners recover won the race and started the service, turning
the port-stolen case into a flake.
Evidence: single-case rerun OK locally; make test-governance rerun
recorded on the task card; CI on the new head is authoritative.
Impact: test synchronization only.
Refs: .task/feature-svc-autorecover.md
Why: RDM capabilities (two-trees-one-net, rdm.* trace-audit command face)
only exist in the downstream Agent-System-RDM composite binary; svc's three
existing sources all produce the Dev binary, so none of it could be deployed
or lifecycle-managed. This adds --from rdm[@<sha>] with the same isolated
source-repo semantics as the dev source.
Evidence: make test-governance 138/138 (two new stub cases: deploy E2E and
commits/update track); real deployment rdm@2aef3fc recorded in the task card
(pins materialized, pure-Go build, web workbench to htdocs, serve health OK,
rdm.ping/rdm.inspect.capabilities committed, update --check reports latest);
make test full suite exit=0. All segments written by make evidence.
Impact: scripts/svc.sh (rdm source resolution/materialization/deploy/update/
commits, probe timeouts, rdm seed skip, ELF env-DSN evidence fallback),
docs/svc.md 5.4, tests/test_svc.py. RDM-side CLI alignment landed as RDM
PR #228; no base submodules touched.
Refs: task card .task/feature-svc-rdm-source.md; RDM PR #228/#229.
Why: CI containers run a PID 1 that does not reap orphans, so the
killed stub lingers as a zombie and os.kill(pid, 0) keeps succeeding;
local dev machines reap instantly, which hid the difference.
Evidence: tests/test_svc_recover 10/10 after the fix; make
test-governance re-run recorded on the task card; CI on the new head
is the authoritative gate.
Impact: test helper only (svc.sh already treats zombies as dead via
empty-cmdline pidcheck).
Refs: .task/feature-svc-autorecover.md
Why: svc services are bare nohup'd processes with persisted registration
but no supervision -- a crash, an OOM kill, or a reboot left them dead
until manual restart (2026-10-11 outage: dev service lost, runtime/ also
wiped by hand, nothing brought services back).
Evidence: tests/test_svc_recover.py 10 cases green (start records
autostart, stop clears it incl. dead-process stop, SIGKILL then recover
revives with persisted exec=, lock-held skips, port-stolen fails with
recover fail event, watchdog unit text via --print); make test-governance
146/146 OK locally; real-machine watchdog install + kill -9 revival to be
recorded on the task card.
Impact: svc.sh gains autostart key in svc.env, `recover [name]
[--quiet]` (postgres-first, per-service .start.lock, recover history
events), `watchdog install|remove|status` (systemd user timer, cron
fallback hint), lifecycle lock across start/stop/restart/deploy/update;
spawned services no longer inherit the lock fd (9>&-), and lock release
no longer clobbers script stderr. Makefile adds svc-recover/svc-watchdog;
docs/svc.md documents semantics and boundaries.
Refs: task card .task/feature-svc-autorecover.md; docs/svc.md 6.5/6.6;
parallel planning card feature/service-restart-recovery-plan (same
problem space, plan-stage).
Why: seed_settings PUTs /api/settings/user without credentials, so under the PG tier's always-on auth the seed always 401s and model config silently never lands (reproduced on the dev instance 2026-10-11; settings had to be written by hand).
Evidence: make test-governance && make test exit=0 (receipt 8ffc3a89); tests/test_svc.py adds three stub-based cases - auth-on seeds with a Bearer header and create/revoke around the PUT, --seed-owner skips account lookup, auth-off stays credential-free; 14/14 svc tests green.
Impact: deployments that pass SVC_MODEL_*/SVC_TAVILY_KEYS now seed reliably regardless of auth mode; a temporary apikey (name=svc-seed) is minted for the first admin (or --seed-owner/SVC_SEED_OWNER) and revoked right after; auth-off path unchanged.
Refs: task card feature-svc-seed-auth.
Why: main landed fix-svc-dsn-reuse-dbref (reuse-path dbref restore, env-DSN
aware precheck, DSN_ENV scoping) for the same restart-DSN incident; the
approved recovery plan is its superset and replaces those mechanisms with
resolve_db_config + recoverable svc.env keys + refuse-on-invalid semantics.
Evidence: tests/test_svc_precheck.py ported to the new units (18 cases green,
same scenario matrix; broken-dbref expectation updated to refusal per the
approved plan); tests/test_svc.py 11 cases green; bash -n OK.
Impact: conflict hunks resolved toward the unified resolver; main's source
guard at the script tail restored; agent_listen_rejected kept as a pure
predicate (used by the start precheck and the governance tests).
Refs: main 7acf318 (PR #317); task card .task/feature-svc-db-config-recover.md.
Why: the exec= reuse path stripped --dsn into the process environment (G1-7)
and record_state then persisted a DSN-less exec=, while the dbref fallback only
existed in the explicit-rebuild branch - the second bare restart after a
rebind silently dropped the database (SQLite fallback on loopback). Auth
preflight also matched --dsn on the command line only, so env-delivered DSN
with auth=auto was misjudged as unauthenticated.
Evidence: bash -n OK; make test-governance 115 tests OK (new tests/test_svc.py
11 cases; 10/11 fail against the pre-fix script); make test green; make build
OK; isolated real PostgreSQL 18.6 + real binary validation 27/27 (consecutive
restarts/deploy keep binding, psql-verified database identity, DB-down restart
fails without SQLite fallback or config loss, no credential leakage) - all
recorded via make evidence in the task card.
Impact: svc.env gains db=/dbref=/dsn= recovery keys (atomic 0600 writes);
start/restart/deploy/update/db share one resolver; invalid bindings, config
conflicts, mock coexistence and env-DSN-incapable binaries now refuse to
start instead of silently downgrading; legacy exec= DSN migrates idempotently;
docs/svc.md and script help updated.
Refs: task card .task/feature-svc-db-config-recover.md; user-approved plan
(2026-10-11 session message); AGENTS.md evidence gates.
Why: G1-7 moved the DSN into AGENT_SYSTEM_DSN, but the precheck only read
the arg string, so a non-loopback restart with auth auto was wrongly
rejected (2026-10-10 dev incident); the reuse path no longer restored the
dbref-bound DSN and would silently fall back to SQLite; and DSN_ENV leaked
across same-process starts (update --all) into other services' env.
Evidence: tests/test_svc_precheck.py 19 cases green (auth matrix vs
config.go AuthEnabled, reuse migration/dbref restore, scope isolation);
full governance suite 123 green; incident-scenario replay allowed + DSN
carried. Recorded via make evidence.
Impact: scripts/svc.sh precheck and reuse-start DSN handling only; no
binary change, no persistence-format change.
Refs: .task/fix-svc-dsn-reuse-dbref.md; config.go AuthEnabled (on || (not
off && DSN non-empty), env DSN counts).
Why: git receive-pack runs 'gc --auto' in the background after a push;
the fixture's TemporaryDirectory teardown then races the gc writer and
fails with OSError 39 (remote.git not empty) — observed on the PR #285
runner (job 1967). The fixture remote holds no durable value, so auto-gc
is pure flake surface.
Evidence: the failing test passes 5/5 locally after the fix; full
make test-governance green (104 tests).
Impact: test fixture only; no governance-script behavior change.
Refs: Dev CI run 1438 job 1967; same race family as the upload-host
TempDir flake fixed earlier.
weichenandCursor