Why: The follow-up CI exposed that backend #188 broke the Dev
multi-instance suite: serve --mock seeded runtime.mock at the user
scope, which system-managed settings now reject, so instances never
became healthy. Backend #190 seeds the startup flag into the system
scope; this PR repins and drops the now-invalid alice user-scope write.
Evidence: Backend #190 CI success at head 0af1ebc (GATE); this PR's
composite CI re-runs make test-multi-instance (GATE for the fix).
Impact: backend pin 741da4a; multi-instance test comments updated to
the system-managed regime.
Refs: Backend #188, #190.
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3fb8210f6495a78e70a67af3661bdfe9c00fb9de.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: test-pg only cloned the default branch, so prev_sha and legacy_sha were never built or attested.
Evidence: SELF python3 -m unittest tests.test_ci_credentials tests.test_lifecycle_pin (11 tests, exit 0).
Impact: the lifecycle suite prints the combination inventory and keeps login and rdm.ping as NOT_RUN.
Refs: .task/feature-svc-d3-real-scenarios.md
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A real process must not keep the caller DSN in its command line, environment, log, history, or svc.env. Domain events written by -bootstrap have to survive a later pin deploy.
Evidence: SELF unittest of the two new LifecycleRealTests methods exited 0.
Impact: The scan uses a sentinel that is not a live database password. Login and rdm.ping stay NOT_RUN.
Refs: tests/integration/lifecycle/test_lifecycle_real.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A static dev deploy deleted the worktree before copying dist, so a mixed batch could not update the workbench. Missing manifests and a second instance on the same Postgres database must fail closed the same way on real artifacts.
Evidence: SELF unittest of the mixed batch, the three-entry manifest-loss case, and the PG cross-instance case exited 0. The PG case used a temporary cluster.
Impact: static dev deploys copy dist before recycling the worktree. Login and rdm.ping stay NOT_RUN.
Refs: tests/integration/lifecycle/test_lifecycle_real.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Stub journals do not prove that a real RDM binary rolls back after a switched fault, a failed build, or a candidate that exits on serve.
Evidence: SELF unittest of the three new LifecycleRealTests methods exited 0. Staging leaves the old pid and release in place; the next start clears that journal. Switched and an immediate serve exit both restore the previous real binary.
Impact: The crash candidate delegates version and identity to the real binary so the gate still sees a capable artifact. Login and rdm.ping stay NOT_RUN.
Refs: tests/integration/lifecycle/test_lifecycle_real.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The lifecycle gate only ran four realcombo cases. A legacy candidate and a missing or mismatched identity row must be refused before the old process stops, and the prev/sha pins must switch without changing the identity row.
Evidence: SELF unittest of the four LifecycleRealTests methods exited 0 against a private RDM clone. The pin moved to the identity merge and a later commit on the same backend pin, because the previous card-close SHAs build a binary with no ensure/verify.
Impact: make test still skips this class unless SVC_TEST_REAL, SVC_RDM_SRC_DIR and SVC_REAL_RDM_BIN are set. make test-lifecycle now runs the module. Login and rdm.ping stay NOT_RUN.
Refs: tests/integration/lifecycle/rdm.pin
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=6193db1b548b964ab3133557ddbb9d83a3643d31.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=5aebedebeb963b5c01f39fd9122ec60dbe6d34d3.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=5aebedebeb963b5c01f39fd9122ec60dbe6d34d3.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: U7 acceptance bootstrapped u7-admin into the shared dev runtime's
singleton rdm_admins ledger, locking the operator out of RDM admin pages
(rdm.role.list 400); recovery required forensics. KTD4 keeps base isAdmin
to audit-view only, so an operator-side trusted channel is needed.
Evidence: python3 -m unittest tests.test_svc_rdm_admin -v (25 OK, SELF);
live smoke on runtime/dev: show/status/typo-guard outputs verified.
Impact: new svc.sh <env> rdm-admin [show|transfer --to x] command, help
topic, status ledger line with stale-owner warning; no behavior change
for existing commands.
Refs: task card .task/feature-rdm-admin-lockout-recovery.md R1/R2
Why: an unset object store must keep artifact bytes available to the other process, which the store-level test does not execute.
Evidence: cross_artifact_roundtrip failed with EOF on backend c6fb130 and passed on 46073d30 (SELF, isolated u7drill-k4).
Impact: Dev now builds the fixed backend. Second-instance write traffic stays closed.
Refs: modules/backend tests/integration/multi-instance/multi_instance_test.go
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The live deploy journal never called fault injection, and test-lifecycle failed closed on login so CI could not require the real combination cases.
Evidence: SELF python3 -m unittest tests.test_svc_u2.U2ReleaseTests -q exit 0 (35 ok, 1 skipped) on 3602a87; receipt sha256 9b5785577751e8e9e7947cf9700fb8854b1c80142f34273ae25f3097ee4c9aa8.
Impact: test-pg builds the Dev binary and runs make test-lifecycle. Login and full-track deploy without SVC_RDM_ORIGIN stay NOT_RUN. No live data move and no account CLI.
Refs: recovery plan D2/D3; Dev #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=2c016931044bebdc4a114c1ca1127d1b40d4ab17.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: The task card still said C4 and C5 had not run, which would make the pull request describe the wrong result.
Evidence: B6, C4, and C5 receipts already on the card (SELF). drill_driver tests pass after grouping the imports.
Impact: The card now states K4 failed and second-instance writes stay closed. No product code change.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=cd61bbb112eea6a210a64e84f2cef00b0d64bfc1.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: Main D1 now shares the same explicit fixture initialization contract.
Evidence: Both real SQLite and temporary PostgreSQL initialization paths
were independently replayed; capabilities JSON assertions are preserved.
Impact: Resolve duplicate PG setup through the tested helper; preserve
all main lifecycle changes and the rollback directory protection.
Refs: Dev #483/#485; D1 existing-store identity contract
Why: Current RDM identity ensure/verify intentionally refuse missing
databases and schema; the old fixture invoked them on empty stores.
Evidence: Actual RDM red failure, real SQLite and isolated PostgreSQL
green replays, and independent review receipts in the task card.
Impact: Test setup only; normal serve initializes temporary stores and
is reaped before the unchanged identity/lifecycle assertions.
Refs: Dev #483/#484; current RDM D1 existing-store contract
Why: identity ensure no longer creates the schema, so the probe has to follow a real serve that applies migrations.
Evidence: SELF python3 -m py_compile tests/test_svc_realcombo.py. GATE run 2663 job 3839 failed on the previous head with identity table missing.
Impact: the PostgreSQL check still passes the DSN only through the environment, after serve has migrated.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: env -i dropped RDM_DISABLE_EXEC_TOOLS, so the real RDM binary exited immediately on runners without bwrap, and the capability probe now prints the JSON document.
Evidence: SELF bash -n scripts/svc.sh; python3 -m py_compile tests/test_svc_realcombo.py. GATE run 2656 job 3831 failed on the previous head before this fix.
Impact: serve and the lifecycle child forward that one switch; identity capabilities is checked as protocol 1 JSON.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A missing artifact id aborted the drill before recovery counts, the settle window, and the logical restore.
Evidence: backup-20261012-151951 reached promoted=1 and smoke 200, then StopIteration on an empty id (SELF). bash -n passed.
Impact: K4 stays failed when uploads do not return an id. The rest of the verdict is still written.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: main gained the real-combination CI gate after the previous sync, and this branch still has to carry that gate with the isolated identity calls.
Evidence: SELF python3 -m unittest recovery/identity/u2/deploy/pin/isolation — 100 tests OK (2 skipped); bash -n scripts/svc.sh.
Impact: identity ensure/verify still pass DSN only through the environment; stop waits for the port and can keep the desired state. CI keeps the lifecycle pin step and the svc-realcombo job.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The restored postmaster refused to start because the results-directory socket path exceeded the 107-byte Unix socket limit.
Evidence: restore/postgres.log from backup-20261012-150713 says the socket path is too long (SELF). bash -n passed.
Impact: The restored instance listens on /tmp/u7bk-18997, and cleanup removes that directory.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=99048dda1f48bccb3d5d798050d7beeea2907177.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: main already has one release transaction, so this branch keeps that path and only isolates update and recover in a clean child.
Evidence: SELF bash -n scripts/svc.sh; python3 -m unittest identity/recover/deploy/u2/lifecycle pin 92 tests OK (2 skipped), then 66 tests OK after dropping the obsolete journal hook.
Impact: deploy uses releases/ and deploy.journal; the lifecycle child reconciles that journal before deploy or recover. Backend pin is 0f192a4.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The checklist had no measured restore path for a physical backup, WAL archive, or pg_dump onto a new target.
Evidence: bash -n, go test in tests/ops/drill_driver, and run_backup_drill.sh --self-check passed locally (SELF). The timed run is recorded separately.
Impact: Isolated u7bk only. Upload responses keep artifactId so the restore check can download the same objects.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Second-resolution release IDs let staging delete a previous release
when a same-source deployment ran within the same clock tick.
Evidence: Fixed-clock rollback regression fails before and passes after;
250 governance checks pass. Independent review replays collision and
copy-failure protection; receipts are in the primary task card.
Impact: Add invocation identity and exclusive directory creation; keep
old artifacts intact. Also retain the latest compatible frontend pointer.
Refs: User-approved I2; Dev #483; Frontend #71/#72
Why: only the deploy invocation carried RDM_DISABLE_EXEC_TOOLS; the
trailing restart ran without it and the serve died on keyless runners
under the bwrap fail-closed policy (run 3765).
Evidence: local suite green; the restart now inherits the hard-disable.
Impact: tests only.
Why: the CI-only serve-exit failure left no server-side evidence in the
assertion output; the tail pinpoints the refuse reason on the runner.
Evidence: local rerun green (diagnostic inert on success).
Impact: test diagnostics only.
Why: an update that stops the old process and overwrites the binary cannot
get the previous version back, and a migration the old binary cannot read
must be refused before that stop. CI also had no fixed RDM SHA to build from.
Evidence: python3 -m unittest tests.test_svc_identity tests.test_svc_recover tests.test_svc_deploy_txn tests.test_lifecycle_pin passed. make test-lifecycle exits 2 when AGENT_PG_TEST_DSN is unset.
Impact: installs land in releases/<rid> and current switches with mv -T.
Deploy keeps the desired state and rolls back a non-terminal journal.
Schema compatFloor above the installed target is refused unless
--migrate-schema --confirm is paired with a backup. test-lifecycle fails
closed without PostgreSQL and an RDM checkout. Login stays NOT_RUN.
Refs: recovery plan r3.1 D2 and D3; Backend feature/d1-identity-open
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: on keyless runners the RDM cache clone and pins materialization
inside svc.sh fetched nested pins over raw SSH URLs (run 2608: host key
verification failed); the suite now injects the same URL-scoped token
header and insteadOf rewrite via environment when SVC_CI_GIT_* is set.
Evidence: local rerun stays green (env unset locally); CI run applies it.
Impact: tests + ci.yaml only; token travels via environment only.
Why: CI runners lack bwrap, so the real RDM binary refuses to start
under the sandbox fail-closed policy (run 2607: serve exits right after
launch); the product provides RDM_DISABLE_EXEC_TOOLS as the top-level
fallback and lifecycle semantics under test are unchanged.
Evidence: local rerun with the env set stays green.
Impact: tests only.
Why: stub suites proved the script logic but not real artifact
behavior (the 2026-10-12 incidents passed all stub tests); the report's
U3 requires real-combination gates, and realcombo dry-runs exposed two
real gaps: fresh deploys closed identity governance one lifecycle late,
and pg-bound identity calls put the DSN on the command line.
Evidence: make test exit 0; local realcombo dry-run 16 OK (rows in the
task card); CI adds svc-realcombo (SQLite, both real binaries, full RDM
track) and a real-PG identity step on test-pg with env-delivered DSN.
Impact: tests/test_svc_realcombo.py (SVC_TEST_REAL gate, explicit
NOT_RUN otherwise); ci.yaml new job + test-pg step (RDM clone via
trusted-base URL-scoped token header, allowlist addition reviewed in
this PR); svc.sh delivers identity DSN via AGENT_SYSTEM_DSN and runs
manifest adoption in the same deploy that first registers svc.env;
modules/backend pinned to main (env fallback, backend #182).
Refs: .task/feature-svc-u3-realcombo.md; user goal directive 2026-10-12
(U3); backend #182; RDM #285.
Why: The first pg-stop window counted an in-flight 200 at shutdown, and five writes on the timeline conversation returned 409 while a mock run held it. That measured the probe, not a proven database failure.
Evidence: go test ./... in tests/ops/drill_driver passed locally (SELF).
Impact: The outage clock starts 2s after stop or kill. RTO is five writes on new conversations. Idempotency replays only the window and retries 409 for up to 2 minutes. The Redis phase no longer uses local outside a function.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: probing identity by running the subcommand still launched old RDM
builds, and ensure on a missing database created an empty one. A batch
update also inherited the caller's AGENT_SYSTEM_* variables.
Evidence: python3 -m unittest tests.test_svc_identity tests.test_svc_recover passed (41 tests, 1 skipped). tests.test_svc_lifecycle_isolation caller-env and restart cases passed.
Impact: capability probes use version --json only. Established instances
refuse a missing identity row until adopt --confirm. update and recover
run each service in a clean environment. serve reads a 0600 serve.env
and does not inherit the caller environment. Backend and documents pins
move to the D1 identity and recovery-plan commits.
Refs: recovery plan r3.1 D1; Backend feature/d1-identity-open; RDM feature/d1-version-json
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=3b9cf82cd6fb53f973f26e2039275f7858b0c654.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: deploy overwrote live artifacts after stopping the old service, so
a failed start or workbench check left instances in a mixed
registration-vs-artifact state with no recovery path (2026-10-12
rdm1/rdmcand); the report's U2 requires restorable updates.
Evidence: make test-governance 250 OK; five svc suites 105 OK (rows in
the task card); new tests cover layout/pidcheck stability, rollback on
start failure with desired-state restore, build failure keeping the old
service running, interrupted-switch journal resolution, and schema-
advance blocking rollback without touching the database.
Impact: agent deploys stage immutable releases/<id>/ and switch the
current symlink atomically; deploy.journal persists the switch and is
resolved at start/restart/deploy/update/recover entries; legacy layouts
are adopted on first deploy and remain readable; release/rdm tracks
verify sha256 before staging.
Refs: .task/feature-svc-u2-release-recovery.md; user goal directive
2026-10-12; Dev #474 (U1 gate); Documents unified-deploy plan r2.
Why: The release checklist cannot mark disconnect recovery until an isolated drill records bounded failures, no SQLite fallback, and idempotent writes.
Evidence: bash -n passed, drill_driver go test passed, unknown arguments exit 2. The timed drill is recorded next.
Impact: New drill entry only. No product change and no release decision.
Refs: docs/release-multi-instance.md C4
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: C4 and C5 need a shared way to start, refuse, and tear down stacks without touching production ports or processes.
Evidence: tests/ops/self_check.sh passed (help, unknown arg, foreign name, production port, busy port, SIGINT cleanup). drill_driver go test passed.
Impact: New tests/ops library and HTTP driver. No drill result and no release-gate change yet.
Refs: docs/release-multi-instance.md C4 C5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=6a494d3977e6557e7bebe0e5bb1c4a12091296e0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: A timed window that starts under load average above 16 is not a usable baseline, and creating the instance first only to pause wastes a startup.
Evidence: The rerun at load 24.03 exited 3 before any timed window and removed the instance.
Impact: The pause now happens before initdb. No change to the benchmark itself.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The percentile reader also opened the pgbench stdout file, so the first run exited 1 after six successful windows.
Evidence: Parser check on results/pgbench-20261012-131229 select-only-10 gave p50 0.018 ms, matching the reported average. The timed rerun is recorded next.
Impact: Percentile parsing only. The six windows themselves had zero failed transactions.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: batch update leaked AGENT_SYSTEM_INSTANCE_ID/DBREF from one service
to the next, and RDM artifacts without the identity CLI were stopped and
overwritten before their startup identity refusal surfaced, leaving
rdm1/rdmcand in a mixed registration-vs-artifact state (2026-10-12).
Evidence: make test-governance 217 OK; tests.test_svc_identity and
tests.test_svc_precheck green; rows recorded in the task card.
Impact: env_config resets identity env per service; capability probe is
timeout-guarded and rejects serve-shaped output; identity ensure moves
to adoption/init (bare start is verify-only); deploy builds and gates
candidates before stopping the old service (refuses identityless
candidates, completes missing identity rows via the candidate); update
runs svc_manifest_migrate like every other lifecycle entry.
Refs: .task/feature-svc-identity-u1.md; user root-cause report 2026-10-12.
Why: make the agent responsible for merging reviewed PRs without another human merge step.
Evidence: SELF governance and generated-card checks are captured in .task/feature-agent-pr-merge.md; CI remains required on the final PR HEAD.
Impact: update project delivery policy, agent entry points, task metadata and closure guidance; preserve platform protection and scoped authorization.
Refs: .task/feature-agent-pr-merge.md; user instruction on 2026-10-12
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=b4b042630a352eca2a399a7a64931c74978bca7f.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: make the six instance-bound user commands discoverable (T5-1) and pin Documents r2/r3.
Evidence: python3 -m unittest tests.test_svc_user_help; existing test_svc_user_binding still green.
Impact: scripts/svc.sh help accounts|user; docs/svc.md topic list; documents gitlink to plan branch tip.
Refs: Documents #87; feature/svc-account-durability-plan
Co-authored-by: Cursor <cursoragent@cursor.com>
weichen
