weichen/Agent-System-Dev
1
0
Fork 0
Code Issues 1 Pull Requests 2 Actions 2 Packages Projects Releases Wiki Activity

Model A/B: opus-5-5 eliminates invented numbers (selection report) #296

Merged
weichen merged 5 commits from feature/agent-model-ab into main 2026-10-10 02:35:15 +00:00
Conversation 0 Commits 5 Files Changed 4
+1212
5 Commits
Author SHA1 Message Date
weichen 8f96b92071 Record the model A/B make-test evidence
ci / test (pull_request) Successful in 2m19s
Details
2026-10-10 02:26:17 +00:00
weichen 112d9e40c9 Complete the model A/B comparison and record the selection report 
Why: Ruling 5's final item. With the user-provided lmuai endpoint
(claude-opus-5-5) and the P5 build unchanged, the same golden task set ran
both sides to completion (6/6 turns each).

Evidence: SELF - B report .task/model-ab/ab-opus.json (2793s) vs A baseline
.task/phase05-eval/after-p5.json; smoke through the live instance confirmed
the provider switch (after fixing the active-secret shadowing: overlayKey
prefers the secret table, so provider changes need /api/secrets rotation).

Impact: Selection report in the card - opus-5-5 eliminates invented numbers
(7 -> 0, the program's core metric), restores independent verification via
lightweight=deepseek-v4-flash; MiniMax stays as the fast/fallback tier.
Recommended default documented for the owner.
 2026-10-10 02:24:56 +00:00
weichen 66c9edfe05 Record the second quota-blocked A/B attempt after the flash-tier probe 
Why: A small max_tokens probe passed (small pre-deduction), suggesting the
account might have been topped up; redeploying with chat=claude-sonnet-4-6
and lightweight=deepseek-v4-flash showed full-turn pre-deduction (CNY 0.62)
still exceeds the CNY 0.48 balance - 6/6 turns quota-failed again.

Evidence: SELF - .task/model-ab/ab-sonnet-quota-blocked-r2.json + evidence
row; settings confirmed chat=claude-sonnet-4-6, lightweight=deepseek-v4-flash
on the instance before the run.

Impact: Still blocked on user-only payment input. Estimated CNY 50-100 covers
multiple full comparison runs on dmxapi; alternatively any OpenAI-compatible
provider key via ~/.secrets/abmodel or .env.dev SVC_MODEL_*.
 2026-10-10 01:19:29 +00:00
weichen 4f97587c9a Record the model A/B pipeline validation and the quota block 
Why: Ruling 5 authorized the strong-model comparison with uncapped budget.
The A/B infrastructure is proven end to end (qa7 on remote main, seeded
claude-sonnet-4-6 via the user's existing dmxapi aggregator key, eval/live
ran the full golden set), but the account holds CNY 0.49 against a CNY 0.62
per-request pre-deduction - all six turns failed with insufficient quota,
captured honestly in the report.

Evidence: SELF - .task/model-ab/ab-sonnet-quota-blocked.json + evidence row;
smoke test of the dmxapi streaming endpoint succeeded before the run.

Impact: Task blocked on user input only (top up dmxapi or provide another
provider key). No code changes; worktree retained for immediate resumption.
 2026-10-10 01:04:47 +00:00
weichen 47be266062 Record the task card for feature-agent-model-ab 
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-agent-model-ab.md; R0=43f8e27b74e5e2213e2099ba04e5a69fe398586f.

Impact: task metadata only; the feature workspace starts at 43f8e27b74.

Refs: .task/feature-agent-model-ab.md
 2026-10-10 00:52:59 +00:00