Why: The live report's model field came back empty because the settings
endpoint nests models under the user scope ({defaults, project, schema, user}),
not at the top level; the before-baseline report had to be annotated by hand.
Evidence: SELF - vet with -tags live passes; the scope shape verified against
GET /api/settings on the qa instance.
Impact: Future live reports carry the served model without manual annotation.
Why: No task-level measurement of real agent behavior existed, so the
failure chain documented in the diagnosis was invisible to CI. The harness
drives a configured backend over its public HTTP surface (upload, message,
poll, HITL auto-resolve), computes per-turn metrics (tool failures, full
regenerations, API-discovery calls, original-material reads, plan usage,
gate checks) and outcome checks (unverified facts vs material text, final
claims vs artifact headings), and writes a JSON + Markdown report.
Evidence: SELF - built behind the live build tag (CI skips); vet passes with
-tags live; drove the qa instance (remote main 1ebd46e) through the four
golden tasks end to end; report referenced on the phase01 task card.
Impact: New eval/live package (tasks.json + runner). Materials stay outside
the repo per eval/README; runs are opt-in via AGENT_EVAL_BASE.
Refs: Dev task feature/agent-quality-phase01; documents proposal
2026-10-09-agent-quality-improvement-plan.md §4 P0.
Why: Q3 WP5 requires a same-batch dual-run harness (direct vs squad
compiled graph) that records differences without demanding byte-identical
output, feeding DecideSwitch evidence for the runtime.entry default-switch
decision.
Evidence: eval/ab_test.go mechanics (theme-based scoring, status/theme/
latency diff notes, P50, DecideSwitch wiring, RunFunc error tolerance,
empty batch); wfcompile TestABDemoBriefDualRun runs the demo.brief scenario
on both entries (direct = scripted single-agent answer; squad = real
RunCompiled with stub backend and judge) and logs the report: both sides
100/100, summary-form difference recorded, decision '切换条件已满足…合入前
需要产品确认'. Full backend suite green (SELF); CI authoritative (GATE).
Impact: new eval/ab.go harness (pure mechanics, no runtime dependency);
no product default is flipped - runtime.entry stays direct until product
confirmation per the DecideSwitch report semantics (evidence recorded in
the task card and PR).
Refs: Q3 plan WP5 (2026-10-07-agent-stack-q2q3-plans.md); task card
.task/feature-agent-stack-q3.md
Why: Squad steps previously ran without tools, and the product had no way to make the squad the conversation entry without losing streaming, cards, or the deliverable guard.
Evidence: go test ./... -count=1 passed, including streaming, cardflow, false-deliverable rejection, tool allowlist, model routing, usage, cancel, and turn and tool budgets.
Impact: runtime.entry defaults to direct. squad uses one kernel step for simple work and the leader for the rest. squad_orchestrate is hidden in that mode. The default is not switched.
Refs: deliverable-quality-system B3-B5
Co-authored-by: Cursor <cursoragent@cursor.com>
Why: Layout rules alone do not produce conclusion-first documents, extra views, or infographics, and nothing scored a deliverable before it was returned.
Evidence: go test ./internal/prompts/ ./internal/quality/ ./internal/tools/ ./internal/squadadapter/ ./eval/ -count=1 and the house-style unittest passed in the previous session.
Impact: Every task inherits content rules. Meetings, reports, and comparisons can name a view family. Long documents without a visual summary warn. quality_review and the squad checker share one scorecard.
Refs: deliverable-quality-system A1-A5
Co-authored-by: Cursor <cursoragent@cursor.com>
weichen
