Agent-System-Backend/.task/feature-rdm-two-trees-project-audit.md

W9 Backend project audit candidate

• Scope: TT-A01/A02/A04/U04, approved stage 5 W9. Stage 6 local implementation; human review, formal acceptance, PR, CI, tag, and deployment are pending.
• Branch/worktree: feature/rdm-two-trees-project-audit at /home/weichen/projects/Agent-System-Backend-wt-feature-rdm-two-trees-project-audit.
• R0: Backend authoritative origin/main 5d5fc9891c0eb8994545ccb8b740535d9b99fb0e; main was clean and equal to the fetched remote before this worktree was created. Core gitlink in this worktree is 5773e74086791731059730bd3726dc8c9cd1381f.
• Startup receipt: /tmp/backend-w9-precheck-20261014-01.json (SELF, exit 2, SYNC_REQUIRED for the separate Dev checkout feature/rdm-increment-acceptance-note, six commits behind Dev origin/main). This Backend task uses the independent-development path in Backend AGENTS.md; it does not write that Dev checkout or its task card. The receipt is preserved as a failure, not relabeled READY.
• Approved affected design: rdm-two-trees-stage6-audit-link-design-adjustment-r1.md, SHA256 385d9d7570217ddcce9fe60a60c99dc58c9a4875ad790584a724d09a210a6e52; user approval in the task session covers this local Backend increment, not tag publication or Stage 7 acceptance.
• Approved PostgreSQL cutoff adjustment: rdm-two-trees-stage6-w9-pg-audit-cutoff-adjustment-r1.md, SHA256 2357b715a7c8f3451b34cc25a578d655e8a48f3f26d9cf7561ea7a13998ca857; scope is isolated Backend implementation/testing, with no real database migration or release authorization.
• Approved all-writers affected design: rdm-two-trees-stage6-w9-all-writers-cutoff-adjustment-r1.md, SHA256 44f6f7cd34bf95a9c1eb844bfdd70a5c2233e1b5ffc89b3091359dcb5fc3b5d8; the direct user approval allows the local v34 correction and isolated tests only.

Implemented local slice

• Optional ProjectAuditHost and domain audit.read registration. A different plugin, absent registration, or unbound actor cannot read.
• Store write snapshots the project from the owner-matched conversation; query requires the authenticated actor's owner and that immutable project. This deliberately excludes other members' private conversations and unbound historical rows, and does not move old receipts when a conversation changes projects.
• Command audit stores the current verified account display name at write time; old rows expose not-recorded. Public receipts omit raw payload and parameter digest. Cursor uses project-and-owner scoped source cutoff.
• Initial commit c8dd169a1fdc1451fab3b475c14f9636a7f5bf6c exposed only the actor-owned project slice. The next local increment fixes button actor propagation and adds a trusted domain-commit link publisher, verifier callback, exact linked metadata read, current audit.read and per-object metadata checks, and durable idempotency. Unlinked personal conversations remain private.
• Trusted command dispatch now creates an operation ID before calling the compiled handler and passes that ID through extension.Request. The handler can commit the same ID with its domain event/outbox; the final Backend audit/receipt uses it. Client params.operationId is ignored. An outbox link request with HostAuditOffset=0 resolves a unique committed host row by owner, fixed project, and operation ID; a missing row returns retryable ErrProjectAuditPending, while duplicate rows fail closed. The separate idempotency key still controls retry equivalence.
• Independent review of a8fa2fe found two P2 security defects: clock-based operation IDs could collide; and NewScopedHost(owner,"rdm") let another compiled plugin self-report the publisher's plugin ID. The new local increment uses 192 bits from crypto/rand for each host-generated operation ID. Both zero-offset and explicit-offset publication require a unique committed row for owner/project/operation; a historical duplicate rejects publication, including attempts to link another private command. Verifier registration returns a private ProjectAuditPublisherFactory for the registering plugin's background outbox; a generic scoped Host has no publisher method. Fixed-clock concurrent IDs, forged params, ambiguous private offset, forged scoped Host, and legitimate registered publisher each have focused tests.
• The minted publisher exposes only ProjectAuditPublisher, not the full owner-scoped Host; tests reject attempts to use NewScopedHost with a self-reported rdm string as publisher authority. A publisher is a registration-time capability, while IdempotencyKey remains the separate retry identity.
• Exact-source linked reads do not enumerate hidden links or reveal raw payload, private conversation ID, or content. Body disclosure and a broad project link list are not implemented; the public coverage reports the narrower capability.
• A PostgreSQL-only v33 default function takes a table-specific transaction advisory lock before assigning command_audit.id, making supported implicit-ID writes commit in source-offset order. Migration and each PG append/project page verify CACHE 1, ascending non-cyclic positive IDs, the exact guarded default expression, and the next ID above the committed maximum; unsafe drift fails closed. SQLite remains single-writer. The owner-only project page reports committed-prefix only after runtime checks. v34 below closes the explicit-ID bypass; online sequence maintenance remains prohibited.
• Isolated PG RED receipt /tmp/backend-w9-explicit-low-red-20261014-01.json proved v33 still misreported committed-prefix: explicit ID 3 committed after normal ID 4 and disappeared across pages. Local v34 adds ENABLE ALWAYS insert and append-only mutation triggers. Every online INSERT checks READ COMMITTED, obtains the same transaction lock, and rejects a nonpositive, nonmonotone, or beyond-reserved ID. UPDATE/DELETE reject. PG append and every strong project page verify both trigger identities, exact event shape, and enablement; the existing default allocator also checks this guard. This is a local candidate, not yet independently reviewed or deployed.
• storage-migrate now copies command_audit within one ACCESS EXCLUSIVE transaction: it disables only the named insert guard, copies explicit IDs, verifies count and checksum, advances the owned sequence only forward, restores and verifies the guard, and records done before commit. Failure rolls back table data, trigger state, and done; nontransactional setval may leave a safe forward gap. Idempotent done checks guard health before reporting success. Other tables retain their existing path.

Verification and remaining work

• The session saw RED for the missing store and host interfaces, unscoped cutoff, cross-member private disclosure, and conversation-project reassignment before the respective fixes. These RED outputs were not captured by make evidence, so this card does not use them as formal evidence.
• SELF receipt /tmp/backend-w9-go-test-20261014-01.json: go test ./..., exit 0, run on the pre-commit worktree based on R0.
• SELF receipt /tmp/backend-w9-build-20261014-01.json: CGO_ENABLED=0 go build ./cmd/agent-system, exit 0, run on that pre-commit tree. Generated local binary removed after the build.
• Historical SELF PostgreSQL receipt /tmp/backend-w9-pg-cutoff-20261014-01.json: isolated PostgreSQL 18.6 instance on port 55442 reproduced a low-ID transaction committing after a higher visible ID. /tmp/backend-w9-pg-link-20261014-02.json verifies the link table, retry/conflict behavior, and the pre-v33 cutoff counterexample. These receipts describe the old offset-upper-bound-only candidate, not the new v33 guarantee.
• SELF PostgreSQL receipt /tmp/backend-w9-pg-migrate-20261014-01.json: isolated SQLite→PostgreSQL migration preserved one linked receipt and advanced its target sequence; the same migration ID replayed idempotently. The temporary PG instance was stopped and removed after verification.
• SELF PostgreSQL receipt /tmp/backend-w9-pg-operation-20261014-01.json: a fresh isolated PG instance reran the link lookup, idempotency/conflict, and low-ID late-commit checks after trusted operation-offset resolution was added. The instance was stopped and removed.
• New PG cross-page RED receipt /tmp/backend-w9-pg-crosspage-red-20261014-01.json: before v33, page 1 returned offsets 4 and 2, page 2 returned 1 after offset 3 committed, and the paged view omitted 3. The same test now passes with v33. Isolated PG tests also cover CACHE 10 counterexample/rejection, CACHE 1, sequence rewind into a vacant low ID, down/up with unchanged link IDs, advisory-lock timeout and rollback gaps, and SQLite→PG explicit-ID migration.
• Historical pre-commit SELF load receipt /tmp/backend-w9-pg-load-precommit-20261014-01.json captured an initial v33 variant whose unlocked status function later produced false cutoff unsafe errors under concurrency; it is not valid performance acceptance evidence. The corrected status check reads committed MAX before non-MVCC sequence state and does not lock readers; the default ID allocator alone holds the transaction advisory lock and rechecks after nextval. TestProjectAuditConcurrentReadWriteNeverReportsFalseDrift and repeated PG load/cross-page runs protect that ordering. Three-condition pre-commit SELF raw receipts: /tmp/backend-w9-pg-load-readlock-precommit-20261014-01.json uses a test-only locked status wrapper, while /tmp/backend-w9-pg-load-unlocked2-precommit-20261014-01.json uses the final MAX-first unlocked status; each run also logs its v32 direct-INSERT baseline, PostgreSQL 18.6 version, 192 writes per mode/8 workers, elapsed time, p95, and throughput. The wrapper adds overhead and all measurements vary with hardware/load; no performance threshold has been approved, so they are Stage 6 cost evidence only.
• Pre-commit SELF /tmp/backend-w9-default-drift-red-20261014-01.json showed that a replacement default containing the guarded function name but executing unguarded nextval was accepted; /tmp/backend-w9-default-drift-green-20261014-01.json verifies exact-expression rejection. Pre-commit SELF /tmp/backend-w9-precommit-all-20261014-02.json and /tmp/backend-w9-precommit-pg-20261014-02.json passed before this last guard tightening; fixed-HEAD runs are required below.
• After the guard tightening, pre-commit SELF receipts /tmp/backend-w9-precommit-all-final-20261014-01.json (go test ./... -count=1), /tmp/backend-w9-precommit-pg-final-20261014-01.json (isolated PG store/storage-migrate), and /tmp/backend-w9-precommit-pg-race-20261014-01.json (cross-page/concurrent read-write with race detector) all exited 0. These still identify the dirty pre-commit tree, so they do not replace fixed-HEAD verification.
• v34 pre-commit SELF /tmp/backend-w9-allwriters-load-precommit-20261014-01.json records isolated PostgreSQL 18.6, three runs of 192 writes per mode with 8 workers, comparing v32 direct INSERT with v34 guarded Store.AppendAudit; it also logs three mixed runs of 180 writes with 6 writers and 160 200-row pages with 4 readers. Protected write throughput ranged about 2.7–3.0k/s, write p95 6.4–6.6ms, and mixed large-page p95 1.2–1.9ms on this one host. These are SELF cost signals with no approved threshold and no Stage 7 acceptance meaning; fixed-HEAD tests still follow.
• Integration constraint: this branch uses schema v31 (audit attribution), v32 (links), v33 (PG implicit cutoff), and v34 (all-writer guard), while the independent W6 Backend branch also has a local v31. Neither branch has consumed the other's unmerged work. After the first branch lands, the later branch owner must sync the new main, renumber its unpublished migrations without rewriting an already released version, and rerun fresh/old SQLite and PG migration plus combined regression on the actual merged source. These isolated tests do not prove W6+W9 compatibility.
• Required next W9 work: real RDM committed-event verifier/outbox consumer, Backend project-event source, run ID propagation, model/tool/background receipts, body/detail disclosure, broad authorized linked-operation pagination, history/export, and combined product verification remain unfinished. A domain commit followed by a lost Backend audit append leaves the link pending; recovery must retain and reconcile that unknown state without claiming a host receipt. Exact-link and owner-only paging do not satisfy Stage 7 audit acceptance. PostgreSQL v33 adds a global command-audit append lock; its measured wait/throughput cost has no approved performance threshold and needs human Stage 6 review.