Why: origin/main moved during CI by archiving an unrelated task card, so the branch had to include it before merge.
Evidence: make task-sync exit 0 via make evidence (SELF). sync_base=df5f8761ea tested_head=bb6b9877d4. The delta is that card archive only.
Impact: The earlier make test on afdf43e still covers the code. CI must go green on this new head before merge.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=df5f8761ea9e2fb20a4b5e3f20318d748a83fbb0.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: The pre-merge sync brought in a newer main, so the previous make test receipt no longer covers this head.
Evidence: make test exit 0 via make evidence (SELF) with before=afdf43e. CI on the pull request remains the GATE.
Impact: Frontend, UI, Python, and agent-command checks passed on the synced head. Drill verdicts are unchanged.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: origin/main moved before the pull request, so the branch had to take that baseline.
Evidence: make task-sync exit 0 via make evidence (SELF). sync_base=2c01693104 tested_head=9526d35f9c.
Impact: Frontend pin and release-directory uniqueness from main are now in the branch. Drill results stay bound to their measurement SHAs.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=2c016931044bebdc4a114c1ca1127d1b40d4ab17.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: The task card still said C4 and C5 had not run, which would make the pull request describe the wrong result.
Evidence: B6, C4, and C5 receipts already on the card (SELF). drill_driver tests pass after grouping the imports.
Impact: The card now states K4 failed and second-instance writes stay closed. No product code change.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Policy requires a task-sync immediately before requesting review.
Evidence: make task-sync exit 0 via make evidence (SELF). origin/main was already contained; tested_head is 2eff856.
Impact: The branch stays aligned with authoritative main. Older drill receipts still predate the earlier main merge; the post-sync make test does not.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The acceptance sync merged main, so the earlier make test receipt predates those changes.
Evidence: make test exit 0 via make evidence (SELF) on 794e51b. CI on the pull request remains the GATE.
Impact: The local suite passed again on the synced head, including worktree, Python, frontend, UI, and agent-command checks.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The branch had to contain authoritative main before acceptance, and that sync is part of the evidence.
Evidence: make task-sync exit 0 via make evidence (SELF). It merged origin/main and warned that the earlier make test predates the merge.
Impact: Backend pin moved with main. The combined test must be rerun on this head.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: include the current integration baseline before verification.
Evidence: fetched origin/main=cd61bbb112eea6a210a64e84f2cef00b0d64bfc1.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: The drill, handbook, and Documents pin need a combined test result before review.
Evidence: make test exit 0 via make evidence (SELF). CI on the pull request remains the GATE.
Impact: Local suite passed, including worktree, Python, frontend, UI, and agent-command checks.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The drill report and storage plan r2.2 are on Documents main, and this repo still pointed at the previous pin.
Evidence: Documents PR https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Documents/pulls/90 merged as f76cce9 (SELF API merge, no CI statuses on that repo).
Impact: The gitlink moves from 3cc6438 to f76cce9. Second-instance writes stay closed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The measured backup, PITR, and disconnect steps were only in the drill scripts, and the checklist still said those rows were not run.
Evidence: Results pgbench-20261012-131948, fault-20261012-142734, and backup-20261012-153441 (SELF). K4 stays failed.
Impact: Second-instance write traffic stays closed. The handbook records the upload panic and the short socket path.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The timed drill finished with a verdict for every criterion, including the upload failure.
Evidence: make evidence exit 1 because K4 failed; results tests/ops/results/backup-20261012-153441 (SELF). K1-K3 and K5-K7 passed. RPO 64s, RTO 4s.
Impact: Physical backup, WAL replay, and pg_dump restore are measured. Artifact upload panics when S3 is unset, so K4 is FAIL and product code was not changed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The backup drill again refused to create an instance because the 1-minute load average was 18.23.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The restored server promoted and the smoke check passed, then the drill stopped while reading artifact ids.
Evidence: make evidence exit 1, results tests/ops/results/backup-20261012-151951 (SELF). rto_s=3, smoke login/projects/conversation 200.
Impact: The instance was cleaned up. The next run records a full verdict, including a failed upload check.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A missing artifact id aborted the drill before recovery counts, the settle window, and the logical restore.
Evidence: backup-20261012-151951 reached promoted=1 and smoke 200, then StopIteration on an empty id (SELF). bash -n passed.
Impact: K4 stays failed when uploads do not return an id. The rest of the verdict is still written.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: pg_ctl could not start the restored server, so the drill exited before a verdict.
Evidence: make evidence exit 1, results tests/ops/results/backup-20261012-150713 (SELF). The log says the Unix socket path is too long.
Impact: Cleanup removed the instance. The rerun uses a short socket path.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The restored postmaster refused to start because the results-directory socket path exceeded the 107-byte Unix socket limit.
Evidence: restore/postgres.log from backup-20261012-150713 says the socket path is too long (SELF). bash -n passed.
Impact: The restored instance listens on /tmp/u7bk-18997, and cleanup removes that directory.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The backup drill refused to create an instance because the 1-minute load average was 17.72.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created. The rerun waits until load is at or below 16.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: C4 evidence is recorded, and the next measured run is the backup and PITR drill.
Evidence: The increment line is the only change.
Impact: No runtime change.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The checklist had no measured restore path for a physical backup, WAL archive, or pg_dump onto a new target.
Evidence: bash -n, go test in tests/ops/drill_driver, and run_backup_drill.sh --self-check passed locally (SELF). The timed run is recorded separately.
Impact: Isolated u7bk only. Upload responses keep artifactId so the restore check can download the same objects.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The corrected probe measured bounded failures, sub-second recovery without restarting the apps, and Redis shared limits across both instances.
Evidence: make evidence exit 0, results tests/ops/results/fault-20261012-142734 (SELF). F1-F5 and G1-G3 passed. G4 and G5 are the expected findings.
Impact: C4 is SELF on this host. Second-instance write traffic stays closed.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The corrected drill refused to create an instance because the 1-minute load average was 16.98.
Evidence: make evidence exit 3, receipt on the task card (SELF).
Impact: No PostgreSQL instance was created. The rerun waits until load is at or below 16.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A second evidence run found port 18991 still held by the in-flight drill and exited before creating an instance.
Evidence: make evidence exit 2, receipt on the task card (SELF).
Impact: Nothing was created. The in-flight drill was stopped so the corrected probe can run.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The first pg-stop window counted an in-flight 200 at shutdown, and five writes on the timeline conversation returned 409 while a mock run held it. That measured the probe, not a proven database failure.
Evidence: go test ./... in tests/ops/drill_driver passed locally (SELF).
Impact: The outage clock starts 2s after stop or kill. RTO is five writes on new conversations. Idempotency replays only the window and retries 409 for up to 2 minutes. The Redis phase no longer uses local outside a function.
Refs: feature/u7-ops-drills
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The release checklist cannot mark disconnect recovery until an isolated drill records bounded failures, no SQLite fallback, and idempotent writes.
Evidence: bash -n passed, drill_driver go test passed, unknown arguments exit 2. The timed drill is recorded next.
Impact: New drill entry only. No product change and no release decision.
Refs: docs/release-multi-instance.md C4
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: C4 and C5 need a shared way to start, refuse, and tear down stacks without touching production ports or processes.
Evidence: tests/ops/self_check.sh passed (help, unknown arg, foreign name, production port, busy port, SIGINT cleanup). drill_driver go test passed.
Impact: New tests/ops library and HTTP driver. No drill result and no release-gate change yet.
Refs: docs/release-multi-instance.md C4 C5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The database-only baseline is usable only with the command, object SHA, and full output kept on the task card.
Evidence: CMD=tests/load/collaboration/run_pgbench.sh make evidence exit 0 at b2e129a (SELF). Six windows, zero failed transactions. An earlier parse failure and a load-gate pause stay on the card.
Impact: Evidence only. Checklist B6 is not flipped until the drill report and checklist edit land. Second-instance writes stay closed.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: A timed window that starts under load average above 16 is not a usable baseline, and creating the instance first only to pause wastes a startup.
Evidence: The rerun at load 24.03 exited 3 before any timed window and removed the instance.
Impact: The pause now happens before initdb. No change to the benchmark itself.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: The percentile reader also opened the pgbench stdout file, so the first run exited 1 after six successful windows.
Evidence: Parser check on results/pgbench-20261012-131229 select-only-10 gave p50 0.018 ms, matching the reported average. The timed rerun is recorded next.
Impact: Percentile parsing only. The six windows themselves had zero failed transactions.
Refs: docs/release-multi-instance.md B6
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Continuous delivery needs the approved scope, stop conditions, and B6/C4/C5 acceptance on the task card before any drill runs.
Evidence: Precheck receipt imported at task-start (SELF, R0 8ebf502146); plan approved by the user on 2026-10-12.
Impact: Records authorization only. No drill, product, or release-gate change.
Refs: docs/release-multi-instance.md B6/C4/C5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: establish the authorized task workspace and its initial baseline.
Evidence: preserved precheck receipt in .task/feature-u7-ops-drills.md; R0=8ebf5021468a1301ccbe2fe0568508e16d8bb708.
Impact: task metadata only; the feature workspace starts at 8ebf502146.
Refs: .task/feature-u7-ops-drills.md
weichenandCursor