Model A/B: opus-5-5 eliminates invented numbers (selection report) #296

Merged
weichen merged 5 commits from feature/agent-model-ab into main 2026-10-10 02:35:15 +00:00
Owner

摘要

模型 A/B(裁定 5 收口,整个质量改进计划的最后一项)。任务卡 feature-agent-model-ab;主记录 §4 模型 A/B。

同 golden 任务集、同 P5 构建双模型对照:A=MiniMax-M2.7-highspeed;B=claude-opus-5-5(用户提供的 lmuai 端点)。

指标 A B
未核实数字(编造) 7 0
回合完成 6/6 6/6
检查失败 9(含同源跳过) 2(事实核查一轮修正)
独立验证器 同源跳过 生效(lightweight=deepseek-v4-flash)
总时长 ~10 分钟 ~47 分钟

建议:默认 chat=claude-opus-5-5 + lightweight=deepseek-v4-flash(独立验证恢复);MiniMax 留作快速/回退档。

本 PR 仅 .task 变更(选型报告 + 三份 A/B 报告 + evidence 行),无代码改动。

Evidence

make test PASS(evidence 行 SELF);B 侧全量 evidence 行(2793s);两轮 quota-blocked 报告如实留档。

运维注意

换供应商时设置页密钥被密钥表活动密钥遮蔽(overlayKey 优先)——经 /api/secrets 新增并激活新密钥(实测踩中并修复)。

## 摘要 模型 A/B(裁定 5 收口,**整个质量改进计划的最后一项**)。任务卡 feature-agent-model-ab;主记录 §4 模型 A/B。 同 golden 任务集、同 P5 构建双模型对照:A=MiniMax-M2.7-highspeed;B=claude-opus-5-5(用户提供的 lmuai 端点)。 | 指标 | A | **B** | |---|---|---| | 未核实数字(编造) | 7 | **0** | | 回合完成 | 6/6 | 6/6 | | 检查失败 | 9(含同源跳过) | 2(事实核查一轮修正) | | 独立验证器 | 同源跳过 | **生效**(lightweight=deepseek-v4-flash) | | 总时长 | ~10 分钟 | ~47 分钟 | **建议**:默认 chat=claude-opus-5-5 + lightweight=deepseek-v4-flash(独立验证恢复);MiniMax 留作快速/回退档。 本 PR 仅 .task 变更(选型报告 + 三份 A/B 报告 + evidence 行),无代码改动。 ## Evidence make test PASS(evidence 行 SELF);B 侧全量 evidence 行(2793s);两轮 quota-blocked 报告如实留档。 ## 运维注意 换供应商时设置页密钥被密钥表活动密钥遮蔽(overlayKey 优先)——经 /api/secrets 新增并激活新密钥(实测踩中并修复)。
weichen added 5 commits 2026-10-10 02:26:18 +00:00
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-agent-model-ab.md; R0=43f8e27b74e5e2213e2099ba04e5a69fe398586f.

Impact: task metadata only; the feature workspace starts at 43f8e27b74.

Refs: .task/feature-agent-model-ab.md
Why: Ruling 5 authorized the strong-model comparison with uncapped budget.
The A/B infrastructure is proven end to end (qa7 on remote main, seeded
claude-sonnet-4-6 via the user's existing dmxapi aggregator key, eval/live
ran the full golden set), but the account holds CNY 0.49 against a CNY 0.62
per-request pre-deduction - all six turns failed with insufficient quota,
captured honestly in the report.

Evidence: SELF - .task/model-ab/ab-sonnet-quota-blocked.json + evidence row;
smoke test of the dmxapi streaming endpoint succeeded before the run.

Impact: Task blocked on user input only (top up dmxapi or provide another
provider key). No code changes; worktree retained for immediate resumption.
Why: A small max_tokens probe passed (small pre-deduction), suggesting the
account might have been topped up; redeploying with chat=claude-sonnet-4-6
and lightweight=deepseek-v4-flash showed full-turn pre-deduction (CNY 0.62)
still exceeds the CNY 0.48 balance - 6/6 turns quota-failed again.

Evidence: SELF - .task/model-ab/ab-sonnet-quota-blocked-r2.json + evidence
row; settings confirmed chat=claude-sonnet-4-6, lightweight=deepseek-v4-flash
on the instance before the run.

Impact: Still blocked on user-only payment input. Estimated CNY 50-100 covers
multiple full comparison runs on dmxapi; alternatively any OpenAI-compatible
provider key via ~/.secrets/abmodel or .env.dev SVC_MODEL_*.
Why: Ruling 5's final item. With the user-provided lmuai endpoint
(claude-opus-5-5) and the P5 build unchanged, the same golden task set ran
both sides to completion (6/6 turns each).

Evidence: SELF - B report .task/model-ab/ab-opus.json (2793s) vs A baseline
.task/phase05-eval/after-p5.json; smoke through the live instance confirmed
the provider switch (after fixing the active-secret shadowing: overlayKey
prefers the secret table, so provider changes need /api/secrets rotation).

Impact: Selection report in the card - opus-5-5 eliminates invented numbers
(7 -> 0, the program's core metric), restores independent verification via
lightweight=deepseek-v4-flash; MiniMax stays as the fast/fallback tier.
Recommended default documented for the owner.
Record the model A/B make-test evidence
ci / test (pull_request) Successful in 2m19s
8f96b92071
weichen added 1 commit 2026-10-10 02:26:19 +00:00
Record the model A/B make-test evidence
ci / test (pull_request) Successful in 2m19s
8f96b92071
weichen merged commit 63a2485ec0 into main 2026-10-10 02:35:15 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Dev#296