Quality phase01: gate repairs, live eval baseline, policy clause (Backend #111, Documents #51) #277

Merged
weichen merged 11 commits from feature/agent-quality-phase01 into main 2026-10-09 14:13:07 +00:00
Owner

摘要

智能体工作流质量改进首个增量(P0 评测基线 + P1 缺陷修复)。主记录与六项裁定:
Documents #51(2026-10-09-agent-quality-improvement-plan.md)。

组件 PR:

  • Backend #111:七项检查缺陷修复(lint 环境+fail-visible、同名工件死循环、
    渲染格式与伪造标记同形、标题误报、python3 命令、检查提醒措辞、rune 截断)
    • eval/live 真实模型评测基座
  • Documents #51:改进主记录(六项裁定、P0–P5 分期与验收)

本 PR:

  • gitlink 指向上述两分支 head
  • policy/delivery.md 新增「智能体行为面的真实验证」(裁定 6:行为面 PR 须附真实模型
    前后对照,无条件跑时显式 NOT_RUN)
  • 任务卡 .task/feature-agent-quality-phase01.md(R1–R4、证据行、结果与剩余项)
  • before 基线报告入库:.task/phase01-eval/before-main-1ebd46e.json

before 基线(旧 main 1ebd46e,真实模型,两轮)

任务 工具 失败 整篇重跑 读原件 检查
minutes-basic 14/33 3/6 2/5 11/21 栏目式标题死循环×4
plan-followup 39/51 13/15 9/11 0/0 证据维度死循环×4 + 8–10 个未核实数字
qa-citation 0 0 0 0 零工具轮仍跑验证器
minutes-pm-tool 20/19 5/6 3/5 0 死循环×4 + 内部检查文案泄漏

(两轮数字为 首跑/复跑,失败链稳定复现。)

Evidence

  • make test(组合,最终 head):PASS — 任务卡 evidence 行(SELF)
  • Backend #111 CI vet-and-test:success(GATE)
  • before 基线 evidence 行(SELF,复跑 1240s)+ 报告文件
  • 单测新增 6 个守护测试全绿(Backend #111)

合入顺序与剩余项

Backend #111 → Documents #51 → 本 PR。after 对照在修复版部署后按同一任务集补跑
(已知未解:改名重登记仍会死循环 → 裁定 4 + P5 处理;tavily 播种后需重启才注册 → 另行修复)。

## 摘要 智能体工作流质量改进首个增量(P0 评测基线 + P1 缺陷修复)。主记录与六项裁定: Documents #51(`2026-10-09-agent-quality-improvement-plan.md`)。 组件 PR: - **Backend #111**:七项检查缺陷修复(lint 环境+fail-visible、同名工件死循环、 渲染格式与伪造标记同形、标题误报、python3 命令、检查提醒措辞、rune 截断) + `eval/live` 真实模型评测基座 - **Documents #51**:改进主记录(六项裁定、P0–P5 分期与验收) 本 PR: - gitlink 指向上述两分支 head - policy/delivery.md 新增「智能体行为面的真实验证」(裁定 6:行为面 PR 须附真实模型 前后对照,无条件跑时显式 NOT_RUN) - 任务卡 `.task/feature-agent-quality-phase01.md`(R1–R4、证据行、结果与剩余项) - before 基线报告入库:`.task/phase01-eval/before-main-1ebd46e.json` ## before 基线(旧 main 1ebd46e,真实模型,两轮) | 任务 | 工具 | 失败 | 整篇重跑 | 读原件 | 检查 | |---|---|---|---|---|---| | minutes-basic | 14/33 | 3/6 | 2/5 | 11/21 | 栏目式标题死循环×4 | | plan-followup | 39/51 | 13/15 | 9/11 | **0/0** | 证据维度死循环×4 + 8–10 个未核实数字 | | qa-citation | 0 | 0 | 0 | 0 | 零工具轮仍跑验证器 | | minutes-pm-tool | 20/19 | 5/6 | 3/5 | 0 | 死循环×4 + 内部检查文案泄漏 | (两轮数字为 首跑/复跑,失败链稳定复现。) ## Evidence - make test(组合,最终 head):PASS — 任务卡 evidence 行(SELF) - Backend #111 CI vet-and-test:success(GATE) - before 基线 evidence 行(SELF,复跑 1240s)+ 报告文件 - 单测新增 6 个守护测试全绿(Backend #111) ## 合入顺序与剩余项 Backend #111 → Documents #51 → 本 PR。after 对照在修复版部署后按同一任务集补跑 (已知未解:改名重登记仍会死循环 → 裁定 4 + P5 处理;tavily 播种后需重启才注册 → 另行修复)。
weichen added 10 commits 2026-10-09 13:30:12 +00:00
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-agent-quality-phase01.md; R0=1ebd46edc5239d0d87418e9a8dd9aa225955e1dd.

Impact: task metadata only; the feature workspace starts at 1ebd46edc5.

Refs: .task/feature-agent-quality-phase01.md
Why: The 2026-10-09 diagnosis showed behavior-surface regressions (gate
deadlocks, silently broken lint, wrong artifact verdicts) passing every unit
test because no task-level check ran the real model. User ruling 6 approved
making before/after live-eval evidence a delivery requirement for prompt,
skill, tool-description, context-assembly and gate changes.

Evidence: SELF - clause drafted to reference modules/backend/eval/live and the
GATE/REPLAY/SELF sourcing already defined in AGENTS.md; task card for
feature/agent-quality-phase01 filled with the P0+P1 scope and requirements.

Impact: policy/delivery.md gains the 智能体行为面的真实验证 subsection;
NOT_RUN stays legal but must be explicit. Phase01 task card records scope,
requirements R1-R4 and the before/after eval protocol.

Refs: modules/documents proposal 2026-10-09-agent-quality-improvement-plan.md (ruling 6).
Why: Backend carries the seven quality-gate repairs with the eval/live
harness; documents carries the improvement-plan main record with the six user
rulings. The Dev combination must test these heads together.

Evidence: SELF - gitlink pointers match the pushed branch heads; make test
recorded on the task card after this commit.

Impact: Combined test surface now includes the phase01 backend and documents
content; no behavior change from the pointers alone.

Refs: feature/agent-quality-phase01 (backend), feature/agent-quality-plan (documents).
Why: The previous pointer predated the eval/live package commit; the
combination must test the full phase01 backend branch.

Evidence: SELF - gitlink equals backend feature/agent-quality-phase01 head.

Impact: Pointer only.
Why: The lint skills-root injection moved from the app root to the runner,
keeping the architecture ratchet green; the Dev combination must test that head.

Evidence: SELF - gitlink equals backend feature/agent-quality-phase01 head;
make test rerun recorded on the task card.

Impact: Pointer only.
Why: The backend branch gained the user-scope model capture fix, and the
live eval produced the before-baseline report on remote main (1ebd46e) that
the task card and the improvement-plan record reference as evidence.

Evidence: SELF - report .task/phase01-eval/before-main-1ebd46e.json (model
MiniMax-M2.7-highspeed, 4 golden tasks, 17 min run): reproduces the diagnosed
failure chain on the pre-fix build - readability dead-loop x4 on two tasks,
9 full regens with 0 original-material reads in plan-followup, 8 unverified
percentages, forged-marker false positive, internal check text leaked to the
user.

Impact: Report committed for the after-comparison once this PR merges; gitlink
points at the harness-inclusive backend head.
Why: bind the task validation checkpoint to the latest integration baseline.

Evidence: SELF recursive component checks; sync_base=1ebd46edc5239d0d87418e9a8dd9aa225955e1dd; candidate=3d9285c1f332d01f48329549359c5c7d231679d8.

Impact: task metadata only.

Refs: .task/feature-agent-quality-phase01.md
Fill the phase01 result summary and remaining items
ci / test (pull_request) Successful in 3m51s
e0fc10d829
weichen added 1 commit 2026-10-09 14:04:08 +00:00
Why: Ruling 6 requires real-model before/after evidence. The after run (fixed
build b68833f, same golden tasks) proves the repairs landed: lint now actually
executes (real layout/lead findings appear vs silent empties before), the
pm-tool readability dead-loop is gone (4/0 -> 0), plan-followup reads original
materials 0 -> 49 times. Remaining failures are the documented P5/P2 scope
(rename-revision loops via scorer semantics, unverified numbers, verifier
parse hiccups), not regressions.

Evidence: SELF - report .task/phase01-eval/after-phase01-b68833f.json +
evidence row (1749s run); comparison table in the card.

Impact: PR #277 now carries the complete before/after pair for review.
weichen merged commit 4deddf458d into main 2026-10-09 14:13:07 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Dev#277