Why: Policy requires a task-sync immediately before requesting review.
Evidence: make task-sync exit 0 via make evidence (SELF). origin/main was already contained; tested_head is 2eff856.
Impact: The branch stays aligned with authoritative main. Older drill receipts still predate the earlier main merge; the post-sync make test does not.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The acceptance sync merged main, so the earlier make test receipt predates those changes.
Evidence: make test exit 0 via make evidence (SELF) on 794e51b. CI on the pull request remains the GATE.
Impact: The local suite passed again on the synced head, including worktree, Python, frontend, UI, and agent-command checks.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The branch had to contain authoritative main before acceptance, and that sync is part of the evidence.
Evidence: make task-sync exit 0 via make evidence (SELF). It merged origin/main and warned that the earlier make test predates the merge.
Impact: Backend pin moved with main. The combined test must be rerun on this head.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=cd61bbb112eea6a210a64e84f2cef00b0d64bfc1.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: The drill, handbook, and Documents pin need a combined test result before review.
Evidence: make test exit 0 via make evidence (SELF). CI on the pull request remains the GATE.
Impact: Local suite passed, including worktree, Python, frontend, UI, and agent-command checks.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The drill report and storage plan r2.2 are on Documents main, and this repo still pointed at the previous pin.
Evidence: Documents PR https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Documents/pulls/90 merged as f76cce9 (SELF API merge, no CI statuses on that repo).
Impact: The gitlink moves from 3cc6438 to f76cce9. Second-instance writes stay closed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The measured backup, PITR, and disconnect steps were only in the drill scripts, and the checklist still said those rows were not run.
Evidence: Results pgbench-20261012-131948, fault-20261012-142734, and backup-20261012-153441 (SELF). K4 stays failed.
Impact: Second-instance write traffic stays closed. The handbook records the upload panic and the short socket path.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The timed drill finished with a verdict for every criterion, including the upload failure.
Evidence: make evidence exit 1 because K4 failed; results tests/ops/results/backup-20261012-153441 (SELF). K1-K3 and K5-K7 passed. RPO 64s, RTO 4s.
Impact: Physical backup, WAL replay, and pg_dump restore are measured. Artifact upload panics when S3 is unset, so K4 is FAIL and product code was not changed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: deploys need a side-effect-free identity probe, an isolated lifecycle child, and the release transaction already on main.
Evidence: GATE run 2667 jobs 3845 test, 3846 test-pg, 3847 svc-realcombo all success on b4eb3a814c.
Impact: update and recover run in a clean child; serve keeps the exec-tool kill switch; Backend pin is 0f192a4.
Refs: #485
Why: identity ensure no longer creates the schema, so the probe has to follow a real serve that applies migrations.
Evidence: SELF python3 -m py_compile tests/test_svc_realcombo.py. GATE run 2663 job 3839 failed on the previous head with identity table missing.
Impact: the PostgreSQL check still passes the DSN only through the environment, after serve has migrated.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The backup drill again refused to create an instance because the 1-minute load average was 18.23.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: env -i dropped RDM_DISABLE_EXEC_TOOLS, so the real RDM binary exited immediately on runners without bwrap, and the capability probe now prints the JSON document.
Evidence: SELF bash -n scripts/svc.sh; python3 -m py_compile tests/test_svc_realcombo.py. GATE run 2656 job 3831 failed on the previous head before this fix.
Impact: serve and the lifecycle child forward that one switch; identity capabilities is checked as protocol 1 JSON.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The restored server promoted and the smoke check passed, then the drill stopped while reading artifact ids.
Evidence: make evidence exit 1, results tests/ops/results/backup-20261012-151951 (SELF). rto_s=3, smoke login/projects/conversation 200.
Impact: The instance was cleaned up. The next run records a full verdict, including a failed upload check.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A missing artifact id aborted the drill before recovery counts, the settle window, and the logical restore.
Evidence: backup-20261012-151951 reached promoted=1 and smoke 200, then StopIteration on an empty id (SELF). bash -n passed.
Impact: K4 stays failed when uploads do not return an id. The rest of the verdict is still written.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: main gained the real-combination CI gate after the previous sync, and this branch still has to carry that gate with the isolated identity calls.
Evidence: SELF python3 -m unittest recovery/identity/u2/deploy/pin/isolation — 100 tests OK (2 skipped); bash -n scripts/svc.sh.
Impact: identity ensure/verify still pass DSN only through the environment; stop waits for the port and can keep the desired state. CI keeps the lifecycle pin step and the svc-realcombo job.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: pg_ctl could not start the restored server, so the drill exited before a verdict.
Evidence: make evidence exit 1, results tests/ops/results/backup-20261012-150713 (SELF). The log says the Unix socket path is too long.
Impact: Cleanup removed the instance. The rerun uses a short socket path.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The restored postmaster refused to start because the results-directory socket path exceeded the 107-byte Unix socket limit.
Evidence: restore/postgres.log from backup-20261012-150713 says the socket path is too long (SELF). bash -n passed.
Impact: The restored instance listens on /tmp/u7bk-18997, and cleanup removes that directory.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: env -i dropped SVC_STUB_*, so recovery tests saw an empty serve record even though the bound DSN was still passed.
Evidence: SELF python3 -m unittest tests.test_svc.SvcDbRecoveryTests tests.test_svc_lifecycle_isolation — 34 tests OK.
Impact: only SVC_STUB_* is added at launch; serve.env still omits them, and caller AGENT_SYSTEM_* stays out of the process.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: main already has one release transaction, so this branch keeps that path and only isolates update and recover in a clean child.
Evidence: SELF bash -n scripts/svc.sh; python3 -m unittest identity/recover/deploy/u2/lifecycle pin 92 tests OK (2 skipped), then 66 tests OK after dropping the obsolete journal hook.
Impact: deploy uses releases/ and deploy.journal; the lifecycle child reconciles that journal before deploy or recover. Backend pin is 0f192a4.
Refs: #485
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The backup drill refused to create an instance because the 1-minute load average was 17.72.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created. The rerun waits until load is at or below 16.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: branches merged into main had left their task cards open; backfill status and archive to prevent bookkeeping debt.
Evidence: feature-svc-u3-realcombo.md(PR#484) — merges verified via git log --merges against origin/main.
Impact: .task/*.md moved to .task/archive/ with status backfilled; no code or submodule change.
Refs: task.sh close (card-autoclose-r1)
Why: C4 evidence is recorded, and the next measured run is the backup and PITR drill.
Evidence: The increment line is the only change.
Impact: No runtime change.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: on slow runners the orphaned http child of a stopped stub lingers
a few hundred milliseconds past the parent's exit (ppid-watch suicide),
and the immediately following start - notably the U2 rollback restart -
classified the unparented listener as a foreign occupant and refused to
start (CI run 3774).
Evidence: test_svc_u2/identity/svc suites 82 OK locally with the wait
in both stop paths.
Impact: adds <=3s bounded wait in do_stop/do_stop_quiet after SIGTERM
cleanup; no behavior change when the port is already free.
Why: The checklist had no measured restore path for a physical backup, WAL archive, or pg_dump onto a new target.
Evidence: bash -n, go test in tests/ops/drill_driver, and run_backup_drill.sh --self-check passed locally (SELF). The timed run is recorded separately.
Impact: Isolated u7bk only. Upload responses keep artifactId so the restore check can download the same objects.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The corrected probe measured bounded failures, sub-second recovery without restarting the apps, and Redis shared limits across both instances.
Evidence: make evidence exit 0, results tests/ops/results/fault-20261012-142734 (SELF). F1-F5 and G1-G3 passed. G4 and G5 are the expected findings.
Impact: C4 is SELF on this host. Second-instance write traffic stays closed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: only the deploy invocation carried RDM_DISABLE_EXEC_TOOLS; the
trailing restart ran without it and the serve died on keyless runners
under the bwrap fail-closed policy (run 3765).
Evidence: local suite green; the restart now inherits the hard-disable.
Impact: tests only.
Why: the branches are ready for review but are not merged, so the card
must stay open and say what was verified on the live instances.
Evidence: local commits listed in the card. Companion pages titled RDM 工作台 returned HTTP 200. rdm.ping did not return a domain result. No data was moved.
Impact: documentation only. Login and the real lifecycle suite stay NOT_RUN.
Refs: recovery plan r3.1
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: the CI-only serve-exit failure left no server-side evidence in the
assertion output; the tail pinpoints the refuse reason on the runner.
Evidence: local rerun green (diagnostic inert on success).
Impact: test diagnostics only.
Why: an update that stops the old process and overwrites the binary cannot
get the previous version back, and a migration the old binary cannot read
must be refused before that stop. CI also had no fixed RDM SHA to build from.
Evidence: python3 -m unittest tests.test_svc_identity tests.test_svc_recover tests.test_svc_deploy_txn tests.test_lifecycle_pin passed. make test-lifecycle exits 2 when AGENT_PG_TEST_DSN is unset.
Impact: installs land in releases/<rid> and current switches with mv -T.
Deploy keeps the desired state and rolls back a non-terminal journal.
Schema compatFloor above the installed target is refused unless
--migrate-schema --confirm is paired with a backup. test-lifecycle fails
closed without PostgreSQL and an RDM checkout. Login stays NOT_RUN.
Refs: recovery plan r3.1 D2 and D3; Backend feature/d1-identity-open
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: on keyless runners the RDM cache clone and pins materialization
inside svc.sh fetched nested pins over raw SSH URLs (run 2608: host key
verification failed); the suite now injects the same URL-scoped token
header and insteadOf rewrite via environment when SVC_CI_GIT_* is set.
Evidence: local rerun stays green (env unset locally); CI run applies it.
Impact: tests + ci.yaml only; token travels via environment only.
Why: CI runners lack bwrap, so the real RDM binary refuses to start
under the sandbox fail-closed policy (run 2607: serve exits right after
launch); the product provides RDM_DISABLE_EXEC_TOOLS as the top-level
fallback and lifecycle semantics under test are unchanged.
Evidence: local rerun with the env set stays green.
Impact: tests only.
Why: YAML env: values do not shell-expand, so $PWD-based binary paths
reached the suite as literal strings (run 2606: SVC_REAL_BIN not
executable).
Evidence: job log run 2606; paths now exported in the run script.
Impact: ci.yaml only.
Why: the realcombo steps unset the URL-scoped token env before
'submodule update', so nested pins fell back to raw SSH URLs and failed
host-key verification on the runner (first svc-realcombo run 2605).
Evidence: job log run 2605 shows ssh clone failures; reordered steps
keep insteadOf + token header active through the pins fetch.
Impact: ci.yaml ordering only.
Why: stub suites proved the script logic but not real artifact
behavior (the 2026-10-12 incidents passed all stub tests); the report's
U3 requires real-combination gates, and realcombo dry-runs exposed two
real gaps: fresh deploys closed identity governance one lifecycle late,
and pg-bound identity calls put the DSN on the command line.
Evidence: make test exit 0; local realcombo dry-run 16 OK (rows in the
task card); CI adds svc-realcombo (SQLite, both real binaries, full RDM
track) and a real-PG identity step on test-pg with env-delivered DSN.
Impact: tests/test_svc_realcombo.py (SVC_TEST_REAL gate, explicit
NOT_RUN otherwise); ci.yaml new job + test-pg step (RDM clone via
trusted-base URL-scoped token header, allowlist addition reviewed in
this PR); svc.sh delivers identity DSN via AGENT_SYSTEM_DSN and runs
manifest adoption in the same deploy that first registers svc.env;
modules/backend pinned to main (env fallback, backend #182).
Refs: .task/feature-svc-u3-realcombo.md; user goal directive 2026-10-12
(U3); backend #182; RDM #285.
Why: The corrected drill refused to create an instance because the 1-minute load average was 16.98.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created. The rerun waits until load is at or below 16.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A second evidence run found port 18991 still held by the in-flight drill and exited before creating an instance.
Evidence: make evidence exit 2, receipt on the task card (SELF).
Impact: Nothing was created. The in-flight drill was stopped so the corrected probe can run.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The first pg-stop window counted an in-flight 200 at shutdown, and five writes on the timeline conversation returned 409 while a mock run held it. That measured the probe, not a proven database failure.
Evidence: go test ./... in tests/ops/drill_driver passed locally (SELF).
Impact: The outage clock starts 2s after stop or kill. RTO is five writes on new conversations. Idempotency replays only the window and retries 409 for up to 2 minutes. The Redis phase no longer uses local outside a function.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: probing identity by running the subcommand still launched old RDM
builds, and ensure on a missing database created an empty one. A batch
update also inherited the caller's AGENT_SYSTEM_* variables.
Evidence: python3 -m unittest tests.test_svc_identity tests.test_svc_recover passed (41 tests, 1 skipped). tests.test_svc_lifecycle_isolation caller-env and restart cases passed.
Impact: capability probes use version --json only. Established instances
refuse a missing identity row until adopt --confirm. update and recover
run each service in a clean environment. serve reads a 0600 serve.env
and does not inherit the caller environment. Backend and documents pins
move to the D1 identity and recovery-plan commits.
Refs: recovery plan r3.1 D1; Backend feature/d1-identity-open; RDM feature/d1-version-json
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: establish the authorized task workspace and its initial baseline.
Evidence: preserved precheck receipt in .task/feature-svc-u3-realcombo.md; R0=4d362537d0e8755a03b1757b5bad92779a064b33.
Impact: task metadata only; the feature workspace starts at 4d362537d0.
Refs: .task/feature-svc-u3-realcombo.md
Why: branches merged into main had left their task cards open; backfill status and archive to prevent bookkeeping debt.
Evidence: feature-svc-u2-release-recovery.md(PR#480) — merges verified via git log --merges against origin/main.
Impact: .task/*.md moved to .task/archive/ with status backfilled; no code or submodule change.
Refs: task.sh close (card-autoclose-r1)
weichenandCursor