3 Commits

Include renames

Author	SHA1	Message	Date
weichen	06f5835211	Add estimate-marking awareness and a second fact-check round (E3) ... Why: The E1 live comparison was the only metric that got worse (unverifiedFacts 7 -> 12): the gate gave one correction round and then shipped bare numbers. The design (E3) calls for number sourcing with a mechanically checkable marking escape: numbers labeled as targets/assumptions are honest planning quantities, not fabricated facts, so the gate must let them pass - and the model must be told that marking is the sanctioned way out. Why: The marking decision is clause-scoped, not a fixed window - a window wrongly blessed "3,500" because a neighboring clause said "target 85%" (caught by the existing test). The second round only fires when the unverified set actually changed, because the continuation budget is shared across all final gates and a repeat block on identical tokens starves the downstream gates (also caught by an existing test). Evidence: go test ./... green; new anchors TestMarkedAsEstimate, TestUnverifiedFactsMarksPass, TestFactCheckGateRounds; the live eval gains an unverifiedUnmarkedFacts field sharing the same marking rule. Impact: base.deliverables v6 adds the hard number-sourcing rule; research/explore/general sub-agent profiles v+1 require citations for key numbers; fact gate: marked-in-clause numbers pass, round 2 escalates to "last interception" with the marking instruction, then passes through with the check line. Eval report keeps the strict series and adds the E3-series field. Refs: proposal 2026-10-10-agent-delegation-routing-sandbox.md (E3, D4); Dev task feature/agent-e3-evidence-gate.	2026-10-10 16:08:19 +00:00
weichen	b68833f320	Read the eval target model from the user settings scope ... Why: The live report's model field came back empty because the settings endpoint nests models under the user scope ({defaults, project, schema, user}), not at the top level; the before-baseline report had to be annotated by hand. Evidence: SELF - vet with -tags live passes; the scope shape verified against GET /api/settings on the qa instance. Impact: Future live reports carry the served model without manual annotation.	2026-10-09 13:04:08 +00:00
weichen	613fb27ba0	Add the live-model task eval harness (phase 0 baseline) ... Why: No task-level measurement of real agent behavior existed, so the failure chain documented in the diagnosis was invisible to CI. The harness drives a configured backend over its public HTTP surface (upload, message, poll, HITL auto-resolve), computes per-turn metrics (tool failures, full regenerations, API-discovery calls, original-material reads, plan usage, gate checks) and outcome checks (unverified facts vs material text, final claims vs artifact headings), and writes a JSON + Markdown report. Evidence: SELF - built behind the live build tag (CI skips); vet passes with -tags live; drove the qa instance (remote main 1ebd46e) through the four golden tasks end to end; report referenced on the phase01 task card. Impact: New eval/live package (tasks.json + runner). Materials stay outside the repo per eval/README; runs are opt-in via AGENT_EVAL_BASE. Refs: Dev task feature/agent-quality-phase01; documents proposal 2026-10-09-agent-quality-improvement-plan.md §4 P0.	2026-10-09 12:49:44 +00:00