Recall mis-execution fixes: negation-aware matching and fail-closed slots #717

Open
weichen wants to merge 9 commits from feature/recall-misexec-fixes into main
Owner

Fix the six mis-executions and three unclarified invalid-slot cases that the A3 recall baseline exposed, in the rule layer (no retrieval, no model changes - the baseline showed top-1 at 0.968 with all misses being rule-layer semantics).

Why: semantic negations (不要/别/不用/先不要 + action term) matched as positive intents - "不要重启任务" triggered a restart; a multi-intent tie-break preferred a bare greeting over a slot-bearing action; multi-candidate values, charset violations and symbols-only slot values started without clarification.

Evidence: one implementation unit (negation-aware per-hit check in compiledSpec.evaluate; specificity tie-break in adjudication; slot extraction multi-value/charset invalidity -> clarify) + the nested-pin cascade; the eval corpus mis_exec is now 0 with 37/37 cases passing and no regression in expired_confirmation, generation_consistency or overall top1 (verify-misexec2.json, race); full make test green (verify-misexec-combo2.json). Independent review lands in the task card before merge.

Impact: four component pins advance (native be556f4, intent 7f3ff6a, squad 95b387d, backend bd5c0f3) with matching nested bumps; clarify/fallback rates rise as dangerous inputs now clarify instead of executing. The retrieval/model-tuning decision stays open with a cleaner baseline.

Refs: evaluation/BASELINE.md; feature/catalog-recall-eval-a3

Fix the six mis-executions and three unclarified invalid-slot cases that the A3 recall baseline exposed, in the rule layer (no retrieval, no model changes - the baseline showed top-1 at 0.968 with all misses being rule-layer semantics). Why: semantic negations (不要/别/不用/先不要 + action term) matched as positive intents - "不要重启任务" triggered a restart; a multi-intent tie-break preferred a bare greeting over a slot-bearing action; multi-candidate values, charset violations and symbols-only slot values started without clarification. Evidence: one implementation unit (negation-aware per-hit check in compiledSpec.evaluate; specificity tie-break in adjudication; slot extraction multi-value/charset invalidity -> clarify) + the nested-pin cascade; the eval corpus mis_exec is now 0 with 37/37 cases passing and no regression in expired_confirmation, generation_consistency or overall top1 (verify-misexec2.json, race); full `make test` green (verify-misexec-combo2.json). Independent review lands in the task card before merge. Impact: four component pins advance (native be556f4, intent 7f3ff6a, squad 95b387d, backend bd5c0f3) with matching nested bumps; clarify/fallback rates rise as dangerous inputs now clarify instead of executing. The retrieval/model-tuning decision stays open with a cleaner baseline. Refs: evaluation/BASELINE.md; feature/catalog-recall-eval-a3
weichen added 5 commits 2026-10-23 04:24:10 +00:00
Why: establish the authorized task workspace and its initial baseline.

Evidence: preserved precheck receipt in .task/feature-recall-misexec-fixes.md; R0=bccb88f294038ce52a37fc5592a6daebd10e88b4.

Impact: task metadata only; the feature workspace starts at bccb88f294.

Refs: .task/feature-recall-misexec-fixes.md
Why: fix the design (negation-aware matching in compiledSpec.evaluate,
specificity tie-break in adjudication, slot multi-value/charset
invalidity) with the eval corpus as acceptance.
Evidence: A3 baseline analysis (evaluation/BASELINE.md).
Impact: task metadata only.
Refs: feature/catalog-recall-eval-a3 (baseline)
Why: land the A3-baseline-driven rule-layer fixes (negation-aware
matching, specificity tie-break, fail-closed slots) across the
composition.
Evidence: unit transport (misexec-git-run.json); eval mis_exec=0 with
37/37 pass (verify-misexec2.json); full combination re-run below.
Impact: four component gitlinks.
Refs: feature/recall-misexec-fixes
Why: the backend branch tip carries the corrected nested
intent/native/squad pins for the mis-execution fix composition.
Evidence: integration transport (misexec-git3-run.json); full
combination re-run below.
Impact: one component gitlink.
Refs: feature/recall-misexec-fixes
Record the mis-execution fix results in the task card
ci / test (pull_request) Successful in 11m7s
ci / test-pg (pull_request) Successful in 13m55s
ci / svc-realcombo (pull_request) Successful in 3m26s
8e3f9edb24
Why: audit trail for the rule-layer fixes and the pin cascade.
Evidence: verify-misexec2.json (mis_exec=0, 37/37 pass, race) and
verify-misexec-combo2.json exit=0.
Impact: task metadata only.
Refs: feature/recall-misexec-fixes
weichen added 2 commits 2026-10-23 04:44:19 +00:00
Record the mis-execution review approval in the task card
ci / test-pg (pull_request) Successful in 12m0s
ci / svc-realcombo (pull_request) Successful in 2m47s
ci / test (pull_request) Successful in 9m13s
b497f75e03
Why: independent review approved the rule-layer fixes (0 blockers,
0 majors).
Evidence: review receipt /tmp/agent-stack-b2/reviewm-run.json.
Impact: task metadata + review report.
Refs: PR #717
weichen added 1 commit 2026-10-23 04:56:45 +00:00
Fold-close the task card before merge
ci / svc-realcombo (pull_request) Successful in 2m56s
ci / test (pull_request) Successful in 9m48s
ci / test-pg (pull_request) Successful in 13m27s
27cd65e450
Why: close the task card inside its own PR so landing it needs no post-merge backfill or separate close PR.
Evidence: .task/feature-recall-misexec-fixes.md -> .task/archive/feature-recall-misexec-fixes.md; status 已完成(合入前闭合); PR #717 https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/717.
Impact: task metadata only; the card is archived within PR #717 and lands with its merge commit.
Refs: task.sh close --self
weichen added 1 commit 2026-10-23 05:10:52 +00:00
Merge authoritative main into the task branch
ci / test (pull_request) In progress
ci / test-pg (pull_request) In progress
ci / svc-realcombo (pull_request) Successful in 3m24s
226bf8d38c
Why: include the current integration baseline before merge.

Evidence: fetched origin/main=3dae91ec.

Impact: integrate upstream changes without rewriting task history.

Refs: task-sync
Some checks are pending
ci / test (pull_request) In progress
Required
Details
ci / test-pg (pull_request) In progress
Required
Details
ci / svc-realcombo (pull_request) Successful in 3m24s
Required
Details
Some required checks are missing.
This pull request is blocked because it's outdated.
You are not authorized to merge this pull request.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/recall-misexec-fixes:feature/recall-misexec-fixes
git checkout feature/recall-misexec-fixes
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Dev#717