Why: The separately approved r0.4 trial must use its own bounded authorization label and output file, without reusing exhausted r0.3 artifacts.
Evidence: Both new label tests failed before the change and passed afterward; full Backend vet/Go suite and selected zero-paid live-tag budget/evaluation tests passed. No real provider call was made.
Impact: Evaluation-only selector and output name; production model routing, budgets and runtime behavior unchanged. Trial remains gated on fixed Dev CI and preflight.
Refs: U3 r0.4 one-time A01 resource approval in this conversation; Dev PR #512
Why: A future separately approved model trial must not reuse the consumed r0.2 label or overwrite its raw result.
Evidence: New label/output and 30000-token guard tests failed before implementation, then selected live-tag tests and Backend full Go suite passed.
Impact: Zero-paid evaluation entry only; its name does not grant resource approval or issue provider requests.
Refs: Dev task feature/fast-slow-u2-u3-plan; r0.3 plan Task 5
Why: Reusing an exhausted authorization label would overwrite or misattribute the next controlled quality sample.
Evidence: New label/output and 30000-token guarded-run tests failed before implementation and passed after; full Backend Go suite passed.
Impact: The r0.2 A01 harness can be used only after separate resource approval, with an independent output name and existing two-run guard.
Refs: U3 quality implementation plan r0.2, Task 6 preparation
Why: A 250ms card poll missed a checkpoint saved about one millisecond before failure.
Evidence: The persistence regression failed before the fix, passed afterward, and independent read-only replay isolated the current conversation and running state.
Impact: Evaluation reads committed card events without changing model output or re-running the paid sample; U3 quality remains failed.
Refs: Agent-System-Backend#193; Agent-System-Dev#512
Why: The prior one-shot A01 result failed and its resource approval is exhausted.
Evidence: Follow-up authorization tests failed before the change and pass after; full Backend suite passes.
Impact: Only live evaluation harness gains a separate one-shot result and 30000-token run budget; production validation remains strict.
Refs: Agent-System-Backend#193; Agent-System-Dev#512
Why: The user approved one additional A01 sample after the original six-case batch failed its quality gate.
Evidence: Live fixture tests preserve the production output cap and reject paid requests when a third actual run exists.
Impact: Evaluation-only case selection and resource guard; no production behavior changes, no automatic replay.
Refs: User approval 2026-10-13; reported token target 60000
Why: The first real-model sample batched discoveries, overcalled scope differences and exposed process chatter as a terminal conclusion.
Evidence: Preserve the failed A01 sample; process-text regression failed before the fix, then complete Backend and focused runtime checks passed.
Impact: Bump recipe identity, qualify quantities and approvals, exclude Process parts from canonical results and capture independent historical evaluation snapshots.
Refs: Approved U2/U3 plan; independent model review and native SELF receipts
Why: Text with short attachments previously completed without starting the intended background material task.
Evidence: SELF real HTTP/browser reproduction shows the absent second task; runtime, manager and server regressions pass with the existing optional port connected.
Impact: Preserves dispatch timing and makes deferred intake observable; comparison records the baseline gap without rerunning a sample.
Refs: fast-slow-u2-u3 A02 and reading acceptance
Why: Real source-backed discoveries should reach the same task card before the slow channel completes.
Evidence: SELF receipts record runtime and manager checkpoint red-green tests, scoped source/version checks, privacy filtering, quota corrections, and SQLite/PG regressions; fixed composition and live comparison follow.
Impact: Adds a local material-only checkpoint sink with no summary model calls, uses version witnesses for source commits, and preserves existing reading and notification paths.
Refs: fast-slow-u2-u3 Tasks 6 and 7; test-only live HTTP harness bounds shared provider requests without approving HITL
Why: The E1 live comparison was the only metric that got worse
(unverifiedFacts 7 -> 12): the gate gave one correction round and then
shipped bare numbers. The design (E3) calls for number sourcing with a
mechanically checkable marking escape: numbers labeled as
targets/assumptions are honest planning quantities, not fabricated
facts, so the gate must let them pass - and the model must be told that
marking is the sanctioned way out.
Why: The marking decision is clause-scoped, not a fixed window - a
window wrongly blessed "3,500" because a neighboring clause said
"target 85%" (caught by the existing test). The second round only
fires when the unverified set actually changed, because the
continuation budget is shared across all final gates and a repeat
block on identical tokens starves the downstream gates (also caught by
an existing test).
Evidence: go test ./... green; new anchors TestMarkedAsEstimate,
TestUnverifiedFactsMarksPass, TestFactCheckGateRounds; the live eval
gains an unverifiedUnmarkedFacts field sharing the same marking rule.
Impact: base.deliverables v6 adds the hard number-sourcing rule;
research/explore/general sub-agent profiles v+1 require citations for
key numbers; fact gate: marked-in-clause numbers pass, round 2
escalates to "last interception" with the marking instruction, then
passes through with the check line. Eval report keeps the strict
series and adds the E3-series field.
Refs: proposal 2026-10-10-agent-delegation-routing-sandbox.md (E3,
D4); Dev task feature/agent-e3-evidence-gate.
Why: The live report's model field came back empty because the settings
endpoint nests models under the user scope ({defaults, project, schema, user}),
not at the top level; the before-baseline report had to be annotated by hand.
Evidence: SELF - vet with -tags live passes; the scope shape verified against
GET /api/settings on the qa instance.
Impact: Future live reports carry the served model without manual annotation.
Why: No task-level measurement of real agent behavior existed, so the
failure chain documented in the diagnosis was invisible to CI. The harness
drives a configured backend over its public HTTP surface (upload, message,
poll, HITL auto-resolve), computes per-turn metrics (tool failures, full
regenerations, API-discovery calls, original-material reads, plan usage,
gate checks) and outcome checks (unverified facts vs material text, final
claims vs artifact headings), and writes a JSON + Markdown report.
Evidence: SELF - built behind the live build tag (CI skips); vet passes with
-tags live; drove the qa instance (remote main 1ebd46e) through the four
golden tasks end to end; report referenced on the phase01 task card.
Impact: New eval/live package (tasks.json + runner). Materials stay outside
the repo per eval/README; runs are opt-in via AGENT_EVAL_BASE.
Refs: Dev task feature/agent-quality-phase01; documents proposal
2026-10-09-agent-quality-improvement-plan.md §4 P0.
Why: Q3 WP5 requires a same-batch dual-run harness (direct vs squad
compiled graph) that records differences without demanding byte-identical
output, feeding DecideSwitch evidence for the runtime.entry default-switch
decision.
Evidence: eval/ab_test.go mechanics (theme-based scoring, status/theme/
latency diff notes, P50, DecideSwitch wiring, RunFunc error tolerance,
empty batch); wfcompile TestABDemoBriefDualRun runs the demo.brief scenario
on both entries (direct = scripted single-agent answer; squad = real
RunCompiled with stub backend and judge) and logs the report: both sides
100/100, summary-form difference recorded, decision '切换条件已满足…合入前
需要产品确认'. Full backend suite green (SELF); CI authoritative (GATE).
Impact: new eval/ab.go harness (pure mechanics, no runtime dependency);
no product default is flipped - runtime.entry stays direct until product
confirmation per the DecideSwitch report semantics (evidence recorded in
the task card and PR).
Refs: Q3 plan WP5 (2026-10-07-agent-stack-q2q3-plans.md); task card
.task/feature-agent-stack-q3.md
Why: Squad steps previously ran without tools, and the product had no way to make the squad the conversation entry without losing streaming, cards, or the deliverable guard.
Evidence: go test ./... -count=1 passed, including streaming, cardflow, false-deliverable rejection, tool allowlist, model routing, usage, cancel, and turn and tool budgets.
Impact: runtime.entry defaults to direct. squad uses one kernel step for simple work and the leader for the rest. squad_orchestrate is hidden in that mode. The default is not switched.
Refs: deliverable-quality-system B3-B5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Layout rules alone do not produce conclusion-first documents, extra views, or infographics, and nothing scored a deliverable before it was returned.
Evidence: go test ./internal/prompts/ ./internal/quality/ ./internal/tools/ ./internal/squadadapter/ ./eval/ -count=1 and the house-style unittest passed in the previous session.
Impact: Every task inherits content rules. Meetings, reports, and comparisons can name a view family. Long documents without a visual summary warn. quality_review and the squad checker share one scorecard.
Refs: deliverable-quality-system A1-A5
Co-authored-by: Cursor <cursoragent@cursor.com>
weichen
