weichen/Agent-System-Backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Repair the quality-gate failure chain and add the live-model eval baseline #111

Merged
weichen merged 4 commits from feature/agent-quality-phase01 into main 2026-10-09 14:12:47 +00:00
Conversation 0 Commits 4 Files Changed 21
+1264 -34
4 Commits
Author SHA1 Message Date
weichen b68833f320 Read the eval target model from the user settings scope
ci / vet-and-test (pull_request) Successful in 47s
Details 
Why: The live report's model field came back empty because the settings
endpoint nests models under the user scope ({defaults, project, schema, user}),
not at the top level; the before-baseline report had to be annotated by hand.

Evidence: SELF - vet with -tags live passes; the scope shape verified against
GET /api/settings on the qa instance.

Impact: Future live reports carry the served model without manual annotation.
 2026-10-09 13:04:08 +00:00
weichen d88dcb999f Wire the lint skills root from the runner instead of the app root 
Why: lint-boundaries rejects new app->quality edges (architecture ratchet);
runtime already depends on quality for the scorer and holds SkillsDir, so the
injection belongs there. Same behavior, allowed dependency direction.

Evidence: SELF - go vet + go test ./internal/runtime ./internal/app pass;
lint-boundaries edge list unchanged.

Impact: quality.SetLintEnvRoot now called from LLMRunner.Execute when
SkillsDir is configured (idempotent, process-wide).
 2026-10-09 12:55:54 +00:00
weichen 613fb27ba0 Add the live-model task eval harness (phase 0 baseline) 
Why: No task-level measurement of real agent behavior existed, so the
failure chain documented in the diagnosis was invisible to CI. The harness
drives a configured backend over its public HTTP surface (upload, message,
poll, HITL auto-resolve), computes per-turn metrics (tool failures, full
regenerations, API-discovery calls, original-material reads, plan usage,
gate checks) and outcome checks (unverified facts vs material text, final
claims vs artifact headings), and writes a JSON + Markdown report.

Evidence: SELF - built behind the live build tag (CI skips); vet passes with
-tags live; drove the qa instance (remote main 1ebd46e) through the four
golden tasks end to end; report referenced on the phase01 task card.

Impact: New eval/live package (tasks.json + runner). Materials stay outside
the repo per eval/README; runs are opt-in via AGENT_EVAL_BASE.

Refs: Dev task feature/agent-quality-phase01; documents proposal
2026-10-09-agent-quality-improvement-plan.md §4 P0.
 2026-10-09 12:49:44 +00:00
weichen 989379e924 Repair the quality-gate failure chain observed on real runs 
Why: Real transcripts (dev instance, 62 assistant turns) showed the artifact
gate silently off and self-deadlocking. lintFile ran python3 -m housestyle.lint
without the skills PYTHONPATH, so the import failed and nil was treated as "no
issues" (same docx: 9 lint errors with PYTHONPATH, 0 without). stop() re-scored
every artifact registered in the run, including superseded versions, so a fixed
document could never pass and the composed reply reported the old verdict.
ctxasm rendered history artifacts as "[产出工件 …]" - the exact pattern
ClaimGate treats as a forged tool marker, so models copying the format were
rejected. scoreReadable flagged table cells like "项目概述撰写" as column
headings. Prompts and recipes said "python -m" (no python on PATH; exit 127)
and housestyle.summarize was invoked as a module that does not exist.

Evidence: SELF - new tests TestLintUnavailableFailsVisibleWhenSkillsRootSet,
TestLintCommandEnvCarriesSkillsRoot, TestDocxHeadingMarkedAndTableCellNotFlagged,
TestLatestDeliverablesKeepsLastPerName, TestRecipesUseRunnablePythonCommands,
TestLibraryUsesRunnablePythonCommands; go test ./internal/... passes; manual:
python3 -m housestyle.outline_cli reads a real artifact docx cleanly, lint
without PYTHONPATH exits 1 with ModuleNotFoundError and is now surfaced as
自检未运行 when the skills root is configured.

Impact: compose root injects quality.SetLintEnvRoot(<data>/skills). Final-turn
checks score only the newest artifact per name; deliverable lists and reply
cards follow the same dedupe. History artifact placeholders no longer collide
with the forged-marker regex. docx text extraction marks Title/HeadingN styles
so readability only judges real headings, with line numbers in the reason.
Check reminders no longer invite the model to ignore failed checks. Oversized
tool output is truncated by runes, not bytes.

Refs: Dev task feature/agent-quality-phase01 (quality gap diagnosis 2026-10-09).
 2026-10-09 12:34:21 +00:00