Why: only the deploy invocation carried RDM_DISABLE_EXEC_TOOLS; the
trailing restart ran without it and the serve died on keyless runners
under the bwrap fail-closed policy (run 3765).
Evidence: local suite green; the restart now inherits the hard-disable.
Impact: tests only.
Why: the CI-only serve-exit failure left no server-side evidence in the
assertion output; the tail pinpoints the refuse reason on the runner.
Evidence: local rerun green (diagnostic inert on success).
Impact: test diagnostics only.
Why: on keyless runners the RDM cache clone and pins materialization
inside svc.sh fetched nested pins over raw SSH URLs (run 2608: host key
verification failed); the suite now injects the same URL-scoped token
header and insteadOf rewrite via environment when SVC_CI_GIT_* is set.
Evidence: local rerun stays green (env unset locally); CI run applies it.
Impact: tests + ci.yaml only; token travels via environment only.
Why: CI runners lack bwrap, so the real RDM binary refuses to start
under the sandbox fail-closed policy (run 2607: serve exits right after
launch); the product provides RDM_DISABLE_EXEC_TOOLS as the top-level
fallback and lifecycle semantics under test are unchanged.
Evidence: local rerun with the env set stays green.
Impact: tests only.
Why: stub suites proved the script logic but not real artifact
behavior (the 2026-10-12 incidents passed all stub tests); the report's
U3 requires real-combination gates, and realcombo dry-runs exposed two
real gaps: fresh deploys closed identity governance one lifecycle late,
and pg-bound identity calls put the DSN on the command line.
Evidence: make test exit 0; local realcombo dry-run 16 OK (rows in the
task card); CI adds svc-realcombo (SQLite, both real binaries, full RDM
track) and a real-PG identity step on test-pg with env-delivered DSN.
Impact: tests/test_svc_realcombo.py (SVC_TEST_REAL gate, explicit
NOT_RUN otherwise); ci.yaml new job + test-pg step (RDM clone via
trusted-base URL-scoped token header, allowlist addition reviewed in
this PR); svc.sh delivers identity DSN via AGENT_SYSTEM_DSN and runs
manifest adoption in the same deploy that first registers svc.env;
modules/backend pinned to main (env fallback, backend #182).
Refs: .task/feature-svc-u3-realcombo.md; user goal directive 2026-10-12
(U3); backend #182; RDM #285.
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3b9cf82cd6fb53f973f26e2039275f7858b0c654.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: deploy overwrote live artifacts after stopping the old service, so
a failed start or workbench check left instances in a mixed
registration-vs-artifact state with no recovery path (2026-10-12
rdm1/rdmcand); the report's U2 requires restorable updates.
Evidence: make test-governance 250 OK; five svc suites 105 OK (rows in
the task card); new tests cover layout/pidcheck stability, rollback on
start failure with desired-state restore, build failure keeping the old
service running, interrupted-switch journal resolution, and schema-
advance blocking rollback without touching the database.
Impact: agent deploys stage immutable releases/<id>/ and switch the
current symlink atomically; deploy.journal persists the switch and is
resolved at start/restart/deploy/update/recover entries; legacy layouts
are adopted on first deploy and remain readable; release/rdm tracks
verify sha256 before staging.
Refs: .task/feature-svc-u2-release-recovery.md; user goal directive
2026-10-12; Dev #474 (U1 gate); Documents unified-deploy plan r2.
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=6a494d3977e6557e7bebe0e5bb1c4a12091296e0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: batch update leaked AGENT_SYSTEM_INSTANCE_ID/DBREF from one service
to the next, and RDM artifacts without the identity CLI were stopped and
overwritten before their startup identity refusal surfaced, leaving
rdm1/rdmcand in a mixed registration-vs-artifact state (2026-10-12).
Evidence: make test-governance 217 OK; tests.test_svc_identity and
tests.test_svc_precheck green; rows recorded in the task card.
Impact: env_config resets identity env per service; capability probe is
timeout-guarded and rejects serve-shaped output; identity ensure moves
to adoption/init (bare start is verify-only); deploy builds and gates
candidates before stopping the old service (refuses identityless
candidates, completes missing identity rows via the candidate); update
runs svc_manifest_migrate like every other lifecycle entry.
Refs: .task/feature-svc-identity-u1.md; user root-cause report 2026-10-12.
Why: make the agent responsible for merging reviewed PRs without another human merge step.
Evidence: SELF governance and generated-card checks are captured in .task/feature-agent-pr-merge.md; CI remains required on the final PR HEAD.
Impact: update project delivery policy, agent entry points, task metadata and closure guidance; preserve platform protection and scoped authorization.
Refs: .task/feature-agent-pr-merge.md; user instruction on 2026-10-12
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b4b042630a352eca2a399a7a64931c74978bca7f.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: make the six instance-bound user commands discoverable (T5-1) and pin Documents r2/r3.
Evidence: python3 -m unittest tests.test_svc_user_help; existing test_svc_user_binding still green.
Impact: scripts/svc.sh help accounts|user; docs/svc.md topic list; documents gitlink to plan branch tip.
Refs: Documents #87; feature/svc-account-durability-plan
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=c6fefa3435dc5d9524fc35483f1d34fa1a6e0d59.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
# Conflicts:
# tests/test_svc.py
Why: Batch update --all left SVC_PIDCHECK from the prior service in
memory, so pid_alive/reclaim_own_port misclassified the next service's
process as dead/foreign, deleted svc.pid, and failed start on a still-
occupied port (genui/rdm1/rdmcand after successful dev).
Evidence: make test-governance (SELF; receipt /tmp/svc-pidcheck-gov.json);
new test_batch_update_stops_second_service_despite_prior_pidcheck +
SVC_PIDCHECK ratchet in test_svc_precheck.
Impact: env_config clears SVC_PIDCHECK/EXEC/VERSION/SOURCE per service;
batch update correctly stops and restarts each agent. After merge:
restore genui/rdm1/rdmcand and move rdmcand to its own data dir.
Refs: .task/feature-svc-pidcheck-leak.md; complements #456 reclaim_own_port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: genui update cleared svc.pid without killing the process, then
overwrote the binary and failed start with a false "non-ours" port error.
Evidence: make test-governance (receipt /tmp/svc-reclaim-governance.json)
Impact: reclaim_own_port walks ss listener ancestry; stop/check_port/install
wire it; orphans matching pidcheck are stopped, foreign occupants still refused
Refs: fix/svc-reclaim-own-port
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: the agent stub left python http.server children when tearDown
SIGKILL'd the bash parent, leaking ports 1880x into recover/identity
suites that share the same range.
Evidence: local suite re-run pending after this fix.
Impact: stub HTTP child exits when ppid changes (same as recover stub).
Refs: .task/feature-svc-rdm-ui-gate.md
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: RDM agent deploy left workbench htdocs unserved (prompt-only), so
environments like dev silently offered API-only 404 pages after a source
switch; static first-start also misclassified http.server as dead when
pidcheck was unset.
Evidence: CMD="make test-governance" make evidence (200 ok, 1 skipped; SELF).
Impact: rdm deploy auto-starts <name>web and asserts UI 200; cross-kind
redeploy needs --allow-source-switch; list/info flag missing workbench.
Refs: .task/feature-svc-rdm-ui-gate.md; docs/svc.md §5.4
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: backend U4 password gate returns 403 on business APIs until must_change clears; CLI-created load passwords are terminal.
Evidence: dual smoke failed POST /api/projects 403 after login_keys pin.
Impact: load suite works on schema v26+; EXIT trap disables re-entry and waits for serve.
Refs: server enforcePasswordGate; CreateAccount must_change
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: background subshell PID was recorded instead of serve, so cleanup left orphans that stole ports and broke the next smoke.
Evidence: prior dual evidence failed with bind address already in use + probe 401 against stale instance.
Impact: trap kill now terminates the real serve process.
Refs: tests/load/collaboration/run_load.sh
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=673e687c5c85b9af3ce13ca30845d5e68de5dfa0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Backend login_keys gate blocks Key traffic while
must_change_password is set; U7 failed on PUT settings/user 403.
Evidence: go test -c multi-instance compiles; CI test-pg will GATE.
Impact: loginAlice does session password change alicePass0→alicePass.
Refs: Dev PR #426; Backend #170 pin e7b70c2
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b431777c9fa9c74f096d7f14fce40d1beba53fe2.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Dev #429 conflicted on scripts/svc.sh (U1 user vs main backup) and Backend gitlink.
Evidence: conflict hunks resolved; actions include init|user|backup|backups
Impact: PR #429 should become mergeable after push; both U1 binding and agent backup entrypoints preserved.
Refs: Dev pulls/429; main backup #417; Backend #171
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: task-sync hit a submodule conflict; pin must track Backend #170
HEAD after rebase onto main (copyOrder fix).
Evidence: backend 0c79618; origin/main merge completed in worktree.
Impact: modules/backend gitlink; sync with main.
Refs: Backend #170; Dev #426
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=ae465d24441ab349c961a55e9b04f8c4ccaf6de8.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3e4a79e4133202b537c888a8beeb7c10d82595f4.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=af9c2de663be5ff102dcab7698464a645832f569.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: the 2026-10-11 runtime/ deletion lost all agent SQLite data with no
backup; the svc-autorecover closing report left "agent SQLite backup" as
a follow-up and the user asked to land it (recovery plan U2 residue).
Evidence: tests/test_svc_backup.py 11/11 green locally (live WAL writer,
manifest sha, retention, guards); make test-governance full suite green
(159 tests, rerun twice after one unreproduced flake).
Impact: new service-level actions backup/backups; snapshots live in
runtime-backups/<name>/ outside runtime/; rm --purge removes them; pg
backup and other flows unchanged.
Refs: modules/documents/docs/proposals/2026-10-12-service-restart-data-recovery.md r2 (sec 6, U2 residue); task card .task/feature-svc-agent-backup.md.
Why: collaboration storage U7 (proposal sections 3/5/7) requires two-process
HTTP-level acceptance over shared PG, a reproducible load-test entry honoring
the load contract, and an auditable pre-release checklist; capacity windows and
dual-host runs need user-scheduled resources, so the deliverable here is the
verifiable entry, not capacity claims.
Evidence: make test green (composite); make test-multi-instance green locally
against an isolated PG (five scenarios incl. crash-takeover and restart
invariants); run_load.sh --profile smoke end-to-end green (SSE probe p95~18ms,
zero seq duplicates, stack auto-recycled); backend pinned to feature branch
with the three cross-instance fixes.
Impact: new tests/integration/multi-instance (CI test-pg job step), new
tests/load/collaboration (k6 + SSE probe + run_load.sh), release checklist at
docs/release-multi-instance.md; no runtime behavior change in Dev repo itself.
Refs: backend PR #164; task card feature/u7-load-fault-release.
Why: systemd's default KillMode=control-group kills every process in
the unit's cgroup when the oneshot finishes -- the daemon recover just
started died seconds after the recover ok event (observed live on the
host: revived at 22:13:40, gone by 22:13:42).
Evidence: unit text now carries KillMode=process (asserted in
test_watchdog_print_plan); suite 10/10 and make test-governance rerun
recorded on the card; live re-demo on the host after merge.
Impact: scripts/svc.sh wd_service_text only.
Refs: .task/feature-svc-autorecover.md
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3de5791e0841e9c7c40c3dc236be6d95b3969827.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Popen returns before python http.server finishes binding, so on
slower CI runners recover won the race and started the service, turning
the port-stolen case into a flake.
Evidence: single-case rerun OK locally; make test-governance rerun
recorded on the task card; CI on the new head is authoritative.
Impact: test synchronization only.
Refs: .task/feature-svc-autorecover.md
Why: RDM capabilities (two-trees-one-net, rdm.* trace-audit command face)
only exist in the downstream Agent-System-RDM composite binary; svc's three
existing sources all produce the Dev binary, so none of it could be deployed
or lifecycle-managed. This adds --from rdm[@<sha>] with the same isolated
source-repo semantics as the dev source.
Evidence: make test-governance 138/138 (two new stub cases: deploy E2E and
commits/update track); real deployment rdm@2aef3fc recorded in the task card
(pins materialized, pure-Go build, web workbench to htdocs, serve health OK,
rdm.ping/rdm.inspect.capabilities committed, update --check reports latest);
make test full suite exit=0. All segments written by make evidence.
Impact: scripts/svc.sh (rdm source resolution/materialization/deploy/update/
commits, probe timeouts, rdm seed skip, ELF env-DSN evidence fallback),
docs/svc.md 5.4, tests/test_svc.py. RDM-side CLI alignment landed as RDM
PR #228; no base submodules touched.
Refs: task card .task/feature-svc-rdm-source.md; RDM PR #228/#229.
Why: CI containers run a PID 1 that does not reap orphans, so the
killed stub lingers as a zombie and os.kill(pid, 0) keeps succeeding;
local dev machines reap instantly, which hid the difference.
Evidence: tests/test_svc_recover 10/10 after the fix; make
test-governance re-run recorded on the task card; CI on the new head
is the authoritative gate.
Impact: test helper only (svc.sh already treats zombies as dead via
empty-cmdline pidcheck).
Refs: .task/feature-svc-autorecover.md
Why: svc services are bare nohup'd processes with persisted registration
but no supervision -- a crash, an OOM kill, or a reboot left them dead
until manual restart (2026-10-11 outage: dev service lost, runtime/ also
wiped by hand, nothing brought services back).
Evidence: tests/test_svc_recover.py 10 cases green (start records
autostart, stop clears it incl. dead-process stop, SIGKILL then recover
revives with persisted exec=, lock-held skips, port-stolen fails with
recover fail event, watchdog unit text via --print); make test-governance
146/146 OK locally; real-machine watchdog install + kill -9 revival to be
recorded on the task card.
Impact: svc.sh gains autostart key in svc.env, `recover [name]
[--quiet]` (postgres-first, per-service .start.lock, recover history
events), `watchdog install|remove|status` (systemd user timer, cron
fallback hint), lifecycle lock across start/stop/restart/deploy/update;
spawned services no longer inherit the lock fd (9>&-), and lock release
no longer clobbers script stderr. Makefile adds svc-recover/svc-watchdog;
docs/svc.md documents semantics and boundaries.
Refs: task card .task/feature-svc-autorecover.md; docs/svc.md 6.5/6.6;
parallel planning card feature/service-restart-recovery-plan (same
problem space, plan-stage).
Why: seed_settings PUTs /api/settings/user without credentials, so under the PG tier's always-on auth the seed always 401s and model config silently never lands (reproduced on the dev instance 2026-10-11; settings had to be written by hand).
Evidence: make test-governance && make test exit=0 (receipt 8ffc3a89); tests/test_svc.py adds three stub-based cases - auth-on seeds with a Bearer header and create/revoke around the PUT, --seed-owner skips account lookup, auth-off stays credential-free; 14/14 svc tests green.
Impact: deployments that pass SVC_MODEL_*/SVC_TAVILY_KEYS now seed reliably regardless of auth mode; a temporary apikey (name=svc-seed) is minted for the first admin (or --seed-owner/SVC_SEED_OWNER) and revoked right after; auth-off path unchanged.
Refs: task card feature-svc-seed-auth.
Why: main landed fix-svc-dsn-reuse-dbref (reuse-path dbref restore, env-DSN
aware precheck, DSN_ENV scoping) for the same restart-DSN incident; the
approved recovery plan is its superset and replaces those mechanisms with
resolve_db_config + recoverable svc.env keys + refuse-on-invalid semantics.
Evidence: tests/test_svc_precheck.py ported to the new units (18 cases green,
same scenario matrix; broken-dbref expectation updated to refusal per the
approved plan); tests/test_svc.py 11 cases green; bash -n OK.
Impact: conflict hunks resolved toward the unified resolver; main's source
guard at the script tail restored; agent_listen_rejected kept as a pure
predicate (used by the start precheck and the governance tests).
Refs: main 7acf318 (PR #317); task card .task/feature-svc-db-config-recover.md.
Why: the exec= reuse path stripped --dsn into the process environment (G1-7)
and record_state then persisted a DSN-less exec=, while the dbref fallback only
existed in the explicit-rebuild branch - the second bare restart after a
rebind silently dropped the database (SQLite fallback on loopback). Auth
preflight also matched --dsn on the command line only, so env-delivered DSN
with auth=auto was misjudged as unauthenticated.
Evidence: bash -n OK; make test-governance 115 tests OK (new tests/test_svc.py
11 cases; 10/11 fail against the pre-fix script); make test green; make build
OK; isolated real PostgreSQL 18.6 + real binary validation 27/27 (consecutive
restarts/deploy keep binding, psql-verified database identity, DB-down restart
fails without SQLite fallback or config loss, no credential leakage) - all
recorded via make evidence in the task card.
Impact: svc.env gains db=/dbref=/dsn= recovery keys (atomic 0600 writes);
start/restart/deploy/update/db share one resolver; invalid bindings, config
conflicts, mock coexistence and env-DSN-incapable binaries now refuse to
start instead of silently downgrading; legacy exec= DSN migrates idempotently;
docs/svc.md and script help updated.
Refs: task card .task/feature-svc-db-config-recover.md; user-approved plan
(2026-10-11 session message); AGENTS.md evidence gates.
weichen
