Land functional follow-up A3: the sanitized capability-recall evaluation set with an offline metrics baseline, extending the existing intentctl eval mechanism.
Why: the decide-later question (retrieval? model tuning?) needs a repeatable baseline; the eval mechanism existed but had no corpus and no metrics.
Evidence: one implementation unit - five-category corpus (negation/reference, multi-intent, wrong-slot, expired confirmation, generation consistency; 30+ sanitized synthetic cases derived from the fixture specs), per-category metrics output (top-1 hit, mis-execution, clarification, fallback rates; rules-vs-generator agreement via the deterministic fake generator), committed baseline report (evaluation/BASELINE.md + metrics.json), structural coverage test. Combination: full make test green (verify-a3-combo3.json). Independent review lands in the task card before merge.
Impact: intent pin advances to 6aa4925 with matching nested pins (native d1f72aa, squad a5afe9c, backend 8dc91c1); intentctl eval output stays backward compatible. No retrieval implementation, no model/prompt changes - those are decided from this baseline separately. A4 remains out of scope.
Land functional follow-up A3: the sanitized capability-recall evaluation set with an offline metrics baseline, extending the existing intentctl eval mechanism.
Why: the decide-later question (retrieval? model tuning?) needs a repeatable baseline; the eval mechanism existed but had no corpus and no metrics.
Evidence: one implementation unit - five-category corpus (negation/reference, multi-intent, wrong-slot, expired confirmation, generation consistency; 30+ sanitized synthetic cases derived from the fixture specs), per-category metrics output (top-1 hit, mis-execution, clarification, fallback rates; rules-vs-generator agreement via the deterministic fake generator), committed baseline report (evaluation/BASELINE.md + metrics.json), structural coverage test. Combination: full `make test` green (verify-a3-combo3.json). Independent review lands in the task card before merge.
Impact: intent pin advances to 6aa4925 with matching nested pins (native d1f72aa, squad a5afe9c, backend 8dc91c1); intentctl eval output stays backward compatible. No retrieval implementation, no model/prompt changes - those are decided from this baseline separately. A4 remains out of scope.
Refs: docs/superpowers/plans/2026-10-21-agent-stack-audit-remediation-plan.md (A3)
Why: establish the authorized task workspace and its initial baseline.
Evidence: preserved precheck receipt in .task/feature-catalog-recall-eval-a3.md; R0=6ced938a126981ee2a4fab671a2da94f49cb56c5.
Impact: task metadata only; the feature workspace starts at 6ced938a12.
Refs: .task/feature-catalog-recall-eval-a3.md
Why: fix the evaluation scope (extend the existing intentctl eval
mechanism, five sanitized categories, metrics-only baseline, offline)
before implementation.
Evidence: scoping over cmd/intentctl, intent/eval_test.go and the
fixture corpus.
Impact: task metadata only.
Refs: docs/superpowers/plans/2026-10-21-agent-stack-audit-remediation-plan.md (A3)
Why: land the sanitized recall evaluation corpus, metrics output and
baseline report (A3) per the approved feature scope.
Evidence: integration transport (a3git3-run.json); unit tests green
(verify-a3.json exit=0); combination re-run below.
Impact: one component gitlink.
Refs: feature/catalog-recall-eval-a3
Why: land the A3 recall evaluation corpus and metrics baseline in
agent-intent (6aa4925) with matching nested pins in the native/squad/
backend containers.
Evidence: integration transport (a3git5-run.json); pins checks and
full make test re-run below.
Impact: three component gitlinks.
Refs: feature/catalog-recall-eval-a3
Why: the backend branch tip's nested agent-squad pin followed the
amended squad container commit (nested agent-native M5 tip).
Evidence: integration transport (a3git10-run.json); pins checks and
full make test re-run below.
Impact: one component gitlink.
Refs: feature/catalog-recall-eval-a3
Why: audit trail for the evaluation corpus/metrics unit, the nested
pin cascade and the green combination run.
Evidence: verify-a3.json and verify-a3-combo3.json exit=0.
Impact: task metadata only.
Refs: feature/catalog-recall-eval-a3
Why: close the task card inside its own PR so landing it needs no post-merge backfill or separate close PR.
Evidence: .task/feature-catalog-recall-eval-a3.md -> .task/archive/feature-catalog-recall-eval-a3.md; status 已完成（合入前闭合）; PR #713https://git-home.u1680832.nyat.app:30319/weichen/Agent-System-Dev/pulls/713.
Impact: task metadata only; the card is archived within PR #713 and lands with its merge commit.
Refs: task.sh close --self
Why: include the current integration baseline before merge.
Evidence: fetched origin/main=e62092a4.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
Why: include the current integration baseline before merge.
Evidence: fetched origin/main=fa79a8eb.
Impact: integrate upstream changes without rewriting task history.
Refs: task-sync
weichen
merged commit bccb88f294 into main2026-10-23 03:08:41 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Land functional follow-up A3: the sanitized capability-recall evaluation set with an offline metrics baseline, extending the existing intentctl eval mechanism.
Why: the decide-later question (retrieval? model tuning?) needs a repeatable baseline; the eval mechanism existed but had no corpus and no metrics.
Evidence: one implementation unit - five-category corpus (negation/reference, multi-intent, wrong-slot, expired confirmation, generation consistency; 30+ sanitized synthetic cases derived from the fixture specs), per-category metrics output (top-1 hit, mis-execution, clarification, fallback rates; rules-vs-generator agreement via the deterministic fake generator), committed baseline report (evaluation/BASELINE.md + metrics.json), structural coverage test. Combination: full
make testgreen (verify-a3-combo3.json). Independent review lands in the task card before merge.Impact: intent pin advances to 6aa4925 with matching nested pins (native d1f72aa, squad a5afe9c, backend 8dc91c1); intentctl eval output stays backward compatible. No retrieval implementation, no model/prompt changes - those are decided from this baseline separately. A4 remains out of scope.
Refs: docs/superpowers/plans/2026-10-21-agent-stack-audit-remediation-plan.md (A3)