weichen
613fb27ba0
Why: No task-level measurement of real agent behavior existed, so the failure chain documented in the diagnosis was invisible to CI. The harness drives a configured backend over its public HTTP surface (upload, message, poll, HITL auto-resolve), computes per-turn metrics (tool failures, full regenerations, API-discovery calls, original-material reads, plan usage, gate checks) and outcome checks (unverified facts vs material text, final claims vs artifact headings), and writes a JSON + Markdown report. Evidence: SELF - built behind the live build tag (CI skips); vet passes with -tags live; drove the qa instance (remote main 1ebd46e) through the four golden tasks end to end; report referenced on the phase01 task card. Impact: New eval/live package (tasks.json + runner). Materials stay outside the repo per eval/README; runs are opt-in via AGENT_EVAL_BASE. Refs: Dev task feature/agent-quality-phase01; documents proposal 2026-10-09-agent-quality-improvement-plan.md §4 P0.
27 lines
643 B
JSON
27 lines
643 B
JSON
[
|
|
{
|
|
"id": "minutes-basic",
|
|
"material": "meeting-rd-agent-asr.docx",
|
|
"turns": ["帮我总结一份会议纪要"]
|
|
},
|
|
{
|
|
"id": "plan-followup",
|
|
"material": "meeting-rd-agent-asr.docx",
|
|
"turns": [
|
|
"帮我总结一份会议纪要",
|
|
"根据这个纪要，制定一个方案",
|
|
"方案太简要了，给我一个详细的方案和计划"
|
|
]
|
|
},
|
|
{
|
|
"id": "qa-citation",
|
|
"material": "meeting-rd-agent-asr.docx",
|
|
"turns": ["这次会议最主要的事情是什么"]
|
|
},
|
|
{
|
|
"id": "minutes-pm-tool",
|
|
"material": "meeting-pm-tool-asr.docx",
|
|
"turns": ["生成会议纪要"]
|
|
}
|
|
]
|