Narrow the blocking gate to hard facts and add fact-consistency checks (quality P5) #123

Merged
weichen merged 2 commits from feature/agent-quality-phase05 into main 2026-10-10 00:34:26 +00:00
Owner

摘要

P5(裁定 4/5 收口):门禁精简 + 事实一致性核查 + 异模型验证器 + tavily 惰性注册。Dev 任务卡 feature/agent-quality-phase05;主记录 §4 P5。

  1. 拦截只留硬事实:quality.HardFacts(lint 实际运行且零 error + 工件存在/非空/引用存在)替换全量评分投影;关键词维度(栏目式标题/可执行性等)移出拦截路径——它们评的是写作偏好,实测把正常中文章节标题判负、造成 P2-P4 各阶段反复出现的死循环。评分整体保留在 ScoreBytes 供评测观测。
  2. 事实一致性核查(runtime.factCheck 默认开):抽取交付物里的百分比/日期/大额数字,回溯材料(附件解析文本+工具证据);找不到出处的给恰好一轮修正(改值/删除/标注目标值),不循环。
  3. 异模型验证器:轻量档回落到与主模型相同时默认跳过同源"独立"验证(硬事实+事实核查仍执行;runtime.verifierSameModel 可显式开启)——比假独立诚实。
  4. tavily 惰性注册(顺带修复登记缺陷):搜索工具始终注册、密钥按调用读取——真机验证播种后无需重启即出现 web_search。

真实模型对照(before=P4 后 main,6/6 回合 completed)

  • 终轮失败原因不再含语义拦截(栏目式标题等清零);现存失败=事实核查一轮修正(设计行为)+ 同源跳过提示(notice 非拦截)
  • plan-followup 未核实数字 10→6(事实修正轮生效;余量为模型遵循度)
  • qa6 实测:播种后未重启即有 web_search

Evidence

  • 单测:TestHardFacts*、TestUnverifiedFactsAndNotice、TestFactCheckContinuationOnce 全绿;go vet + go test ./... 通过
  • 组合 make test PASS(evidence 行 SELF);真实模型对照 evidence 行 + 报告 .task/phase05-eval/after-p5.json

剩余

模型 A/B(裁定 5,需用户提供第二家强模型 key;同源跳过提示也指向该配置空缺)。

## 摘要 P5(裁定 4/5 收口):门禁精简 + 事实一致性核查 + 异模型验证器 + tavily 惰性注册。Dev 任务卡 feature/agent-quality-phase05;主记录 §4 P5。 1. **拦截只留硬事实**:`quality.HardFacts`(lint 实际运行且零 error + 工件存在/非空/引用存在)替换全量评分投影;关键词维度(栏目式标题/可执行性等)移出拦截路径——它们评的是写作偏好,实测把正常中文章节标题判负、造成 P2-P4 各阶段反复出现的死循环。评分整体保留在 ScoreBytes 供评测观测。 2. **事实一致性核查**(runtime.factCheck 默认开):抽取交付物里的百分比/日期/大额数字,回溯材料(附件解析文本+工具证据);找不到出处的给**恰好一轮**修正(改值/删除/标注目标值),不循环。 3. **异模型验证器**:轻量档回落到与主模型相同时默认跳过同源"独立"验证(硬事实+事实核查仍执行;runtime.verifierSameModel 可显式开启)——比假独立诚实。 4. **tavily 惰性注册**(顺带修复登记缺陷):搜索工具始终注册、密钥按调用读取——真机验证播种后**无需重启**即出现 web_search。 ## 真实模型对照(before=P4 后 main,6/6 回合 completed) - 终轮失败原因不再含语义拦截(栏目式标题等清零);现存失败=事实核查一轮修正(设计行为)+ 同源跳过提示(notice 非拦截) - plan-followup 未核实数字 10→6(事实修正轮生效;余量为模型遵循度) - qa6 实测:播种后未重启即有 web_search ## Evidence - 单测:TestHardFacts*、TestUnverifiedFactsAndNotice、TestFactCheckContinuationOnce 全绿;go vet + go test ./... 通过 - 组合 make test PASS(evidence 行 SELF);真实模型对照 evidence 行 + 报告 `.task/phase05-eval/after-p5.json` ## 剩余 模型 A/B(裁定 5,需用户提供第二家强模型 key;同源跳过提示也指向该配置空缺)。
weichen added 1 commit 2026-10-10 00:25:37 +00:00
Why: User rulings 4/5 plus the P2-P4 live evidence. Keyword scoring dimensions
(readability, actionability) kept failing valid documents - normal Chinese
section headings flagged as column titles, loops of 4 corrections - because
they grade writing preferences, not verifiable facts. Unverified numbers
(~10-13 per run) remained the largest quality leak through every phase.
And the verifier runs on the same model as the worker (lightweight falls back
to chat), so "independent" verification was self-review. Separately, svc
seeding after boot never registered the tavily tools until a restart because
keys were snapshotted at startup.

Changes: quality.HardFacts (lint actually ran + zero errors) replaces the
full ScoreBytes projection in CheckArtifacts - keyword dimensions remain in
ScoreBytes for eval visibility only. New fact-consistency check
(runtime.factCheck, default on): extract percentages/dates/large numbers from
deliverable text, trace each against the material corpus (attachment text +
tool evidence); misses get exactly one correction round (fix, drop, or mark
as target/estimate), never a loop. runVerifier skips same-source model
verification unless runtime.verifierSameModel is set (hard facts + fact check
still run). Search tools now register lazily, reading tavily keys per call
(settings/seeding take effect immediately).

Evidence: SELF - TestHardFactsIgnoresKeywordDimensions,
TestHardFactsKeepsLintUnavailableBlocking, TestUnverifiedFactsAndNotice,
TestFactCheckContinuationOnce (fact note fires, corrected run passes);
go vet + go test ./... pass; live before/after on the golden task set
recorded on the phase05 task card per the policy clause.

Impact: Gate stops rejecting valid documents on stylistic grounds; invented
figures get one forced correction round; same-model verification no longer
pretends to be independent; tavily config is hot. No contract or storage
changes.
weichen added 1 commit 2026-10-10 00:30:07 +00:00
Merge origin/main into feature/agent-quality-phase05
ci / vet-and-test (pull_request) Successful in 40s
c802be812d
weichen merged commit e03d3ac6c6 into main 2026-10-10 00:34:26 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Backend#123