diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index 6d175eee..e3a0c873 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -18,6 +18,7 @@ | owner | task | branch | worktree | mutex | status | |---|---|---|---|---|---| +| Cursor | jev-dev-assist-on-2026-09-24 | feature/jev-dev-assist-on-2026-09-24 | .worktrees/jev-dev-assist-on-2026-09-24 | env:dev+jevdecide | done | | Cursor | jev-dev-shadow-on-2026-09-24 | feature/jev-dev-shadow-on-2026-09-24 | .worktrees/jev-dev-shadow-on-2026-09-24 | env:dev+jevdecide | done | | Cursor | jev-product-closeout-2026-09-24 | feature/jev-product-closeout-2026-09-24 | .worktrees/jev-product-closeout-2026-09-24 | jevdecide+coordinator | done | | Cursor | jev-product-shadow-2026-09-24 | feature/jev-product-shadow-2026-09-24 | .worktrees/jev-product-shadow-2026-09-24 | jevdecide+coordinator | done | diff --git a/.doc/ops/short-term/2026-09-24-jev-dev-assist-enable.md b/.doc/ops/short-term/2026-09-24-jev-dev-assist-enable.md new file mode 100644 index 00000000..082cfc31 --- /dev/null +++ b/.doc/ops/short-term/2026-09-24-jev-dev-assist-enable.md @@ -0,0 +1,29 @@ +--- +status: done +write_scope: configs/.env.dev(local),env:dev,.doc +mutex: env:dev+jevdecide +base_dev_sha: 156c1130 +review_state: self-reviewed +integration_state: docs_pending +deployment: local_dev_assist_on +cleanup: pending +remote_sync: not_pushed +owner: Cursor +task_id: jev-dev-assist-on-2026-09-24 +--- + +# local_dev 启用 jev-service（assist）并多样本对话验收 + +## 目标 + +将 `JEV_DECIDE_MODE` 从 shadow 升为 **assist**，用 ≥20 条真实 `/api/v1/chat` 验证：规则漏检时能否加成打开实时检索，且业务问句不被误伤。 + +## 做法 + +1. `configs/.env.dev`：`JEV_DECIDE_MODE=assist`（URL/KEY 不变） +2. `env:dev` 租约下 force-recreate app → 日志 `jevdecide wired mode=assist` +3. 20 轮真实对话；证据 [assets](../../test-reports/assets/2026-09-24-jev-dev-assist-enable/)；报告 [test-report](../../test-reports/2026-09-24-jev-dev-assist-enable-test-report.md) + +## 回退 + +`JEV_DECIDE_MODE=shadow` 或 `off` 后 recreate app。 diff --git a/.doc/status/FEATURE-MATRIX.md b/.doc/status/FEATURE-MATRIX.md index 5c9f00fb..49329723 100644 --- a/.doc/status/FEATURE-MATRIX.md +++ b/.doc/status/FEATURE-MATRIX.md @@ -49,7 +49,7 @@ | record_id | 模块 | 声明范围 | 状态 | 发布关系 | 证据 | 剩余工作 / 关闭条件 | mutex | |------|----------|------|----------|------|---------------------|-------|-------| -| jevdecide-shadow | jev 产品决策 shadow | jevdecide 客户端 + coordinator realtime/ProfileTask shadow；compose 默认 off；**local_dev 已开 shadow**（`configs/.env.dev`，容器经 172.17.0.1:8767）；assist 未放量 | partial | deferred_enhancement | [产品报告](../test-reports/2026-09-24-jev-product-shadow-test-report.md)；[启用报告](../test-reports/2026-09-24-jev-dev-shadow-enable-test-report.md)；[启用短文](../ops/short-term/2026-09-24-jev-dev-shadow-enable.md) | 关闭条件：真实轮浸泡≥200 + 用户授权 assist；intent/risk 另立任务。本行不升 done | `jevdecide` | +| jevdecide-shadow | jev 产品决策 shadow/assist | jevdecide + coordinator realtime/ProfileTask；compose 默认 off；**local_dev 已开 assist**（`configs/.env.dev` → 172.17.0.1:8767）；决策层漏检加成已对话验收 | partial | deferred_enhancement | [产品报告](../test-reports/2026-09-24-jev-product-shadow-test-report.md)；[shadow 启用](../test-reports/2026-09-24-jev-dev-shadow-enable-test-report.md)；[assist 启用+对话](../test-reports/2026-09-24-jev-dev-assist-enable-test-report.md)；[assist 短文](../ops/short-term/2026-09-24-jev-dev-assist-enable.md) | 关闭条件：真实轮浸泡≥200 + 体验层检索空结果收敛；intent/risk 另立。本行不升 done | `jevdecide` | | chat-semantic-replace | 语义摘要 REPLACE | 以 shadow 叙事可选替换 served deterministic Narrative（fail-closed 灰度） | deferred | deferred_enhancement | [Wave5 X-2](../ops/short-term/2026-09-18-wave5-deadcode-decisions.md)；[pu6](../ops/long-term/2026-09-16-pu6-semantic-summary-contract.md)；[summary_flags](../../internal/ctxmgr/summary_flags.go) | 关闭条件：用户独立授权 + fail-closed 灰度设计；`SEMANTIC_SUMMARY_REPLACE` 当前密封不接线。本行不自动升 done | `chat` | | gov-agent-worktree | Agent worktree 治理自动化 | 主目录保护、任务 worktree、机器认领、候选集成、dev 部署与安全清理 | partial | non_blocker | [方案](../remediation/2026-09-21-agent-workflow-governance-plan.md)；[L1 验收](../test-reports/2026-09-21-agent-governance-test-report.md)；[W1-B macOS](../ops/short-term/2026-09-22-agentctl-macos.md)；[W4](../ops/short-term/2026-09-22-governance-automation.md)；residual `modular-decouple-residual-2026-09-23` | L1 与 macOS `with/docs apply/validate/integrate` 已合入。W4 landed：durable queue + 完整 docs projection apply（行/整文 CAS）+ deploy/recover 真实收据闭环 + Darwin cwd 可见性测试。仍 deferred：PI 源码镜像 compose、跨机隔离、automation/w4 实现收敛。本行不升 done | `agent-workflow-governance` | | intent-l1 | 意图识别 L1 | 18 类 Intent、三分支消解、写/只读分流、双标签 `DOC_BUSY`、intent 纯依赖边界 | done | non_blocker | [intent-recognition](../system-design/intent-recognition/)；[工作流作用域](../test-reports/2026-09-21-intent-workflow-scoping-test-report.md)；[审计](../audit/2026-09-13-conversation-intent-runtime-state.md)；[报告](../test-reports/2026-09-13-conversation-intent-runtime-state-test-report.md)；[双标签](../test-reports/assets/2026-09-15-conversation-intent-dual-tab/live.json)；[U1](../test-reports/2026-09-15-multi-intent-u1-contracts-test-report.md)–[U3](../test-reports/2026-09-15-multi-intent-u3-intentflow-pg-test-report.md) | **边界 before**：含 U9–U12 叙事混入 → **after**：U9–U12 迁至「多意图扩面」行。关闭：纯依赖边界回归即重开 | `intent` | diff --git a/.doc/test-reports/2026-09-24-jev-dev-assist-enable-test-report.md b/.doc/test-reports/2026-09-24-jev-dev-assist-enable-test-report.md new file mode 100644 index 00000000..f6ff12eb --- /dev/null +++ b/.doc/test-reports/2026-09-24-jev-dev-assist-enable-test-report.md @@ -0,0 +1,109 @@ +# local_dev jev-service assist 启用与对话加成 测试报告 + +## 一、 测试背景 (Background) + +shadow 已在 local_dev 观察一致率。用户授权启用 **assist**，并用多样本真实对话检验：规则词典漏检的时效问句能否被 jev 高置信加成打开实时检索，同时业务问句不被误开。 + +## 二、 测试目标 (Test Objectives) + +- 启动接线 `mode=assist` +- 20 条真实 chat：A 规则命中基线 / B 漏检候选 / C 应保持非实时 +- 决策层：`rule=false ∧ jev=true ∧ mode=assist` 计为加成信号 +- 体验层：回复是否出现可核验的实时信息（气温/汇率/AQI/金价等） +- C 组零错误加成 + +## 三、 测试方案与环境 (Test Plan & Environment) + +- OS：Linux；`shuimu-agent-app-dev`；jev `172.17.0.1:8767`；`JEV_DECIDE_MODE=assist` +- 层级：真实 Agent（pi-agent + 实时检索）端到端对话 +- 样本矩阵：A×4 + B×10 + C×6（见 assets `batch-results.json`） +- **全量 `go test ./...`：NOT REQUIRED**（运维开关 + 对话验收；产品单元/soak 已有既有报告） + +## 四、 预期结果 (Expected Results) + +| 组 | 预期 | +|----|------| +| A | rule=true，jev≈true，agree；行为与 shadow 时一致（assist 不压低规则） | +| B | 多数 rule=false、jev=true → 决策加成；部分回复含实时事实 | +| C | rule=false、jev=false，无 assist 加成 | + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +### 5.1 启动接线 + +- **输入**：`.env.dev` → assist；force-recreate app +- **实际输出**：`msg="jevdecide wired" mode=assist endpoint=http://172.17.0.1:8767` +- **判定**：PASS + +### 5.2 批量对话汇总（20/20 HTTP 200） + +证据：[`assets/2026-09-24-jev-dev-assist-enable/batch-summary.json`](./assets/2026-09-24-jev-dev-assist-enable/batch-summary.json) + +| 指标 | 结果 | 判定 | +|------|------|------| +| 决策层加成（B） | **10/10**（B1–B10 全部 `rule=false jev=true`） | PASS | +| A 组 agree | **4/4** | PASS | +| C 组错误加成 | **0/6** | PASS | +| UX 可见实时事实 | B3/B4/B6/B7/B8 明确；B9 部分；B1/B2/B5/B10 仍偏项目话术 | 决策 PASS；体验 **部分 PASS** | + +### 5.3 典型用例 + +#### A1 规则基线 + +- **输入**：`今天上海天气怎么样` +- **日志**：`mode=assist rule=true jev=true confidence=0.997951 agree=true` +- **判定**：PASS（assist 未破坏规则命中） + +#### B1 决策加成（体验弱） + +- **输入**：`外面冷不冷啊今天` +- **日志**：`mode=assist rule=false jev=true confidence=0.990937 agree=false` +- **回复**：偏项目 V0.5 话术（检索可能空/模型未用 snippet） +- **判定**：决策 PASS；体验弱 + +#### B3 决策+体验加成 + +- **输入**：`上海现在热不热` +- **日志**：`rule=false jev=true confidence=0.997512` +- **回复**：给出约 29°C、湿度等实时表述 +- **判定**：PASS（整链加成可见） + +#### B4 汇率漏检加成 + +- **输入**：`美元兑人民币现在多少了`（无「汇率」词典字） +- **日志**：`rule=false jev=true` +- **回复**：给出约 6.697 汇率 +- **判定**：PASS + +#### B6 英文天气 + +- **输入**：`What's the weather like in Beijing today` +- **日志**：`rule=false jev=true`（中文 marker 全漏） +- **回复**：北京气温 24.6°C 等 +- **判定**：PASS（对英文问句加成明显） + +#### C2 业务防误伤 + +- **输入**：`型检温度指标怎么定`（含「温度」但强业务词） +- **日志**：`rule=false jev=false confidence=0.994535 agree=true` +- **判定**：PASS（未错误加成） + +### 5.4 耗时 + +单轮 chat 约 6–29s（含 pi-agent / 检索）；jev decide 本身亚秒级，不主导端到端。 + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +NOT REQUIRED（运维启用 + 真实对话批次）。不将本报告记为产品全量回归 PASS。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +**结论**：assist 在决策层对规则漏检的时效问句有明确加成（B 组 10/10），且未误伤 C 组业务问句；A 组规则路径保持。端到端体验上约半数 B 样本能把实时事实写进回复，其余仍受检索空结果/对话模板牵引限制——**加成在「是否打开检索」上已成立，在「答复一定用上检索」上仍有缺口**。 + +**建议**： + +1. 保持 local_dev assist；继续统计 AssistHit / 检索空结果率 +2. 检索失败时对话提示可更明确，避免回落纯项目话术 +3. compose 默认仍 `off`；test/main 不放量直至浸泡与体验稳定 + +回退：`JEV_DECIDE_MODE=shadow|off` + recreate app。 diff --git a/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-results.json b/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-results.json new file mode 100644 index 00000000..dd3134dc --- /dev/null +++ b/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-results.json @@ -0,0 +1,402 @@ +[ + { + "id": "A1", + "bucket": "rule_true", + "input": "今天上海天气怎么样", + "rule_oracle": true, + "http": 200, + "err": null, + "duration_s": 11.74, + "reply_prefix": "好的，今天（9月24日）上海天气**小雨转多云**，气温约24～29℃，东风微风。 出门记得带把伞～ 如有其他需要随时说。", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:53:10.890Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=true jev=true confidence=0.997951 margin=0.99590182 agree=true", + "mode": "assist", + "rule": "true", + "jev": "true", + "confidence": "0.997951", + "margin": "0.99590182", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "time=2026-09-23T17:53:22.464Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.990003 agree=true" + }, + { + "id": "A2", + "bucket": "rule_true", + "input": "美元汇率多少", + "rule_oracle": true, + "http": 200, + "err": null, + "duration_s": 13.46, + "reply_prefix": "根据实时数据，当前（2026年9月23日）美元兑人民币汇率约为 **1 USD = 6.6977 CNY**（买入价/卖出价约 6.6975 / 6.6978）。 供参考，当前项目「医疗器械研发项目」处于 V0.1 摸底期，若 BOM 中涉及进口光学/电气元器件，汇率波动会直接影响成本边界，建议在预算评估时预留 1%～2% 的汇率缓冲空间。 如需我帮您按", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:53:22.966Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=true jev=true confidence=0.998322 margin=0.9966430399999999 agree=true", + "mode": "assist", + "rule": "true", + "jev": "true", + "confidence": "0.998322", + "margin": "0.9966430399999999", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "time=2026-09-23T17:53:36.239Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.973338 agree=true" + }, + { + "id": "A3", + "bucket": "rule_true", + "input": "今天有什么新闻", + "rule_oracle": true, + "http": 200, + "err": null, + "duration_s": 16.71, + "reply_prefix": "好的，为您梳理一下今天的要点新闻： **【国际热点】** 1. **川习会**：习近平主席9月23-25日访美，9月24日将访白宫与特朗普会面并出席国宴，双方就贸易及地区冲突议题展开磋商。 2. **霍尔木兹海峡局势**：一艘油轮21日遭袭、两名水手受伤；美军称近两周该海峡能源运输量达近六个月新高。 **【科技/商业】** 3. **AI竞争**：Ant", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:53:37.315Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=true jev=true confidence=0.996222 margin=0.99244306 agree=true", + "mode": "assist", + "rule": "true", + "jev": "true", + "confidence": "0.996222", + "margin": "0.99244306", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "time=2026-09-23T17:53:53.269Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.980692 agree=true" + }, + { + "id": "A4", + "bucket": "rule_true", + "input": "明天会下雨吗", + "rule_oracle": true, + "http": 200, + "err": null, + "duration_s": 15.79, + "reply_prefix": "您好！ 「明天」是否下雨取决于您所在的具体城市——检索到的实时预报显示，不同地区差异较大： - **大阪（大阪府）**：明天（9/25，周五）晴时多云，降雨概率 0%～20%，基本不下雨。 - **新加坡**：明天小雨，气温 29℃/25℃。 - **纽约**：明天有较高降雨概率。 - **香港、广州**：需查询具体地点预报。 请问您关心的是哪个城市或", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:53:53.765Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=true jev=true confidence=0.997117 margin=0.9942339200000001 agree=true", + "mode": "assist", + "rule": "true", + "jev": "true", + "confidence": "0.997117", + "margin": "0.9942339200000001", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "time=2026-09-23T17:54:09.370Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.986274 agree=true" + }, + { + "id": "B1", + "bucket": "assist_candidate", + "input": "外面冷不冷啊今天", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 13.81, + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册分类路径，并预估型检成本，便于与客户深入沟通； • 若先放一放 / 多做调研： 可先补齐预算、目标市场或使用场景等信息，降低后续返工风险。 💡 如果您接下来更希望先有一份可审阅的材料，我可以先整", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:54:11.130Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.990937 margin=0.9818732400000001 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.990937", + "margin": "0.9818732400000001", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:54:23.498Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.996736 agree=true" + }, + { + "id": "B2", + "bucket": "assist_candidate", + "input": "明天出门要不要带伞", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 13.98, + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册分类路径，并预估型检成本，便于与客户深入沟通； • 若先放一放 / 多做调研： 可先补齐预算、目标市场或使用场景等信息，降低后续返工风险。 💡 如果您接下来更希望先有一份可审阅的材料，我可以先整", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:54:25.319Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.996626 margin=0.99325288 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.996626", + "margin": "0.99325288", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:54:37.765Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.967817 agree=true" + }, + { + "id": "B3", + "bucket": "assist_candidate", + "input": "上海现在热不热", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 15.75, + "reply_prefix": "上海现在还挺热的~ 🌡️ 据实时数据显示，上海当前气温约 **29°C** 左右，天气多云，湿度较高（约80%），紫外线指数较强。夜间虽然会凉快一些，但整体仍维持夏季炎热的体感。 建议出行时做好防暑防晒，适当补水。 --- *注：以上天气信息来源于实时网络查询，供您参考。如您有其他问题需要协助，随时告诉我~*", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:54:39.587Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.997512 margin=0.99502314 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.997512", + "margin": "0.99502314", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:54:53.857Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.98938 agree=true" + }, + { + "id": "B4", + "bucket": "assist_candidate", + "input": "美元兑人民币现在多少了", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 11.06, + "reply_prefix": "好的，帮您查了一下—— **截至2026年9月22日，美元兑人民币汇率约 6.697。** 近期人民币有所走强，若您后续涉及进口元器件或海外采购预算，这算是个相对友好的窗口。 如果您接下来需要做BOM成本换算或预算边界梳理，我可以帮您一起核算。", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:54:54.433Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.998392 margin=0.99678406 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.998392", + "margin": "0.99678406", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:55:05.237Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.980923 agree=true" + }, + { + "id": "B5", + "bucket": "assist_candidate", + "input": "苹果股票今天表现如何", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 15.0, + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册分类路径，并预估型检成本，便于与客户深入沟通； • 若先放一放 / 多做调研： 可先补齐预算、目标市场或使用场景等信息，降低后续返工风险。 💡 如果您接下来更希望先有一份可审阅的材料，我可以先整", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:55:07.072Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.998564 margin=0.99712808 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.998564", + "margin": "0.99712808", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:55:20.555Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.97285 agree=true" + }, + { + "id": "B6", + "bucket": "assist_candidate", + "input": "What's the weather like in Beijing today", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 14.14, + "reply_prefix": "北京今日天气（2026年9月24日凌晨）： - 🌡️ 气温：**24.6°C** - 🌫️ 天气：烟霾 + 部分多云 - 💨 风向：东北风 - 🌧️ 降雨概率：低 - 🌬️ 空气质量：中等 如您接下来有项目推进或文档整理方面的需求，随时告诉我～", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:55:21.773Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.998316 margin=0.99663272 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.998316", + "margin": "0.99663272", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:55:35.016Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.984659 agree=true" + }, + { + "id": "B7", + "bucket": "assist_candidate", + "input": "黄金现在贵不贵", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 18.83, + "reply_prefix": "**结论：贵，且处于历史高位区间。** 根据实时数据（2026-09-23）： | 指标 | 价格 | |------|------| | 国际黄金现货 | **$4,333.25 / 盎司** | | 折合人民币 | 约 **¥934 / 克** | | 上海黄金现货 | 约 **¥936 / 克** | **参考背景：** - 年内峰值 $5,60", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:55:36.251Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.99545 margin=0.9908997799999999 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.99545", + "margin": "0.9908997799999999", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:55:54.154Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.964275 agree=true" + }, + { + "id": "B8", + "bucket": "assist_candidate", + "input": "北京空气质量怎么样最近", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 13.45, + "reply_prefix": "好的，帮您查了一下北京最近的空气质量： **截至 2026-09-23**，北京 AQI 约 **109**，处于**轻度污染**级别。PM2.5 浓度超出世卫组织指导值，对敏感人群可能有一定影响。 **小建议**：如果您或团队成员对空气质量比较敏感，建议减少长时间、高强度的户外活动，适当做好防护。 --- 这个时间点咱们项目组应该都在休息了吧？有什", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:55:54.702Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.997111 margin=0.9942215000000001 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.997111", + "margin": "0.9942215000000001", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:56:07.921Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.948139 agree=true" + }, + { + "id": "B9", + "bucket": "assist_candidate", + "input": "今晚适合户外跑步吗", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 17.21, + "reply_prefix": "好的，我帮您查了一下—— 当前时间是**凌晨1:56**，严格来说还没到\"今晚\"呢，是今晨 :) 如果您指的是**夜间跑步**的话，依据检索到的建议：夜跑可选，但建议注意以下几点： 1. **时间选择**：黄昏（17:00~19:00）人体运动状态最佳，氧气浓度高、感觉灵敏；若只能在深夜跑，建议选择照明充足的场所，避免运动损伤。 2. **空气质量**", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:56:08.454Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.992417 margin=0.98483312 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.992417", + "margin": "0.98483312", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:56:25.451Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.964401 agree=true" + }, + { + "id": "B10", + "bucket": "assist_candidate", + "input": "现在要穿羽绒服吗上海", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 17.08, + "reply_prefix": "您好，用户！我是您的项目助手，很高兴协助您推进医疗器械立项与研发协同。 针对「医疗器械研发项目」，我可以帮您： • 查询进展、成熟度与风险； • 沉淀客户需求与技术指标； • 起草沟通话术与方案要点； • 推进方案论证与报价核算。 如果您愿意，我们可以先从您当前最关心的一块开始——您想先看进展、补事实，还是直接聊方案？", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:56:26.005Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=true confidence=0.987759 margin=0.9755179199999999 agree=false", + "mode": "assist", + "rule": "false", + "jev": "true", + "confidence": "0.987759", + "margin": "0.9755179199999999", + "agree": "false", + "assist_additive_signal": true, + "log_profile": "time=2026-09-23T17:56:42.849Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.958358 agree=true" + }, + { + "id": "C1", + "bucket": "should_false", + "input": "项目进度怎么样", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 11.87, + "reply_prefix": "您好！帮您梳理一下当前项目进度： **项目状态概览** - **阶段**：V0.1（初始摸底期） - **成熟度**：65/100 - **工作流**：intake（信息采集/补充阶段） - **进展**：基准产品画像已梳理完成 **当前待办** | 槽位 | 状态 | 说明 | |------|------|------| | 设备类型（devic", + "realtime_note": "reply_suggests_live_or_search", + "log_realtime": "time=2026-09-23T17:56:44.121Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.987928 margin=0.97585502 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.987928", + "margin": "0.97585502", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "" + }, + { + "id": "C2", + "bucket": "should_false", + "input": "型检温度指标怎么定", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 29.19, + "reply_prefix": "您好！型检温度指标的确定通常遵循以下逻辑： ## 核心依据 1. **强标 GB 9706.1-2020** 〔引用键〕——有源医械的通用安全要求，规定了工作气候条件（通常 +10℃～+40℃）和贮存气候条件（通常 -40℃～+70℃）。 2. **GB/T 14710-2009《医用电器环境要求及试验方法》** 〔引用键〕——明确温度试验分组（Ⅰ/Ⅱ", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:56:56.232Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.994535 margin=0.98906928 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.994535", + "margin": "0.98906928", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "" + }, + { + "id": "C3", + "bucket": "should_false", + "input": "帮我推进 V0.5 方案", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 26.67, + "reply_prefix": "好的，我来帮您梳理一下推进V0.5方案的情况。 根据项目当前状态，V0.5方案推进需要关注以下几点： **当前阶段分析：** - 项目处于**V0.1阶段**（intake工作流） - 基准产品画像已梳理完成 ✓ - 成熟度：65/100 **推进到V0.5需要的关键信息：** | 槽位 | 状态 | 说明 | |------|------|----", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:57:25.749Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.994602 margin=0.98920304 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.994602", + "margin": "0.98920304", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "" + }, + { + "id": "C4", + "bucket": "should_false", + "input": "注册路径需要注意什么", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 20.98, + "reply_prefix": "您好！项目「医疗器械研发项目」当前处于【V0.1】阶段（成熟度 65/100）。 核心研判：已完成「医疗器械研发项目」的初始事实沉淀，当前具备进入方案与法规初筛的基准条件 ⚠️ 关键风险预警： • 若客户宣称中包含实质性疗效或治疗作用，可能触发器械分类升档（如二类转三类），研发与注册周期将增加 6-12 个月 • 时间与预算尚缺客户书面口径，直接对", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:57:52.630Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.998821 margin=0.9976410200000001 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.998821", + "margin": "0.9976410200000001", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "" + }, + { + "id": "C5", + "bucket": "should_false", + "input": "你好", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 6.21, + "reply_prefix": "您好，用户！我是您的项目助手，很高兴协助您推进医疗器械立项与研发协同。 针对「医疗器械研发项目」，我可以帮您： • 查询进展、成熟度与风险； • 沉淀客户需求与技术指标； • 起草沟通话术与方案要点； • 推进方案论证与报价核算。 如果您愿意，我们可以先从您当前最关心的一块开始——您想先看进展、补事实，还是直接聊方案？", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:58:13.380Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.998913 margin=0.9978259 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.998913", + "margin": "0.9978259", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "time=2026-09-23T17:58:19.348Z level=INFO msg=\"jevdecide profile shadow\" mode=assist rule_bucket=low jev_bucket=low score=0 confidence=0.99835 agree=true" + }, + { + "id": "C6", + "bucket": "should_false", + "input": "这份报价方案文档写得怎么样", + "rule_oracle": false, + "http": 200, + "err": null, + "duration_s": 13.07, + "reply_prefix": "用户，您好！针对「肌骨超声AI辅助诊断系统」的系统验证方案（SVP）与型式检验要求，我为您梳理以下核心难点项目与特别注意事项： ### 一、 核心测试难点项目 1. **声学输出安全与探头表面温升极限（GB 9706.237-2020）** • **难点分析**：肌骨超声常需配合高频线阵探头（10-18MHz）进行浅表滑膜微血流彩色多普勒（CDFI", + "realtime_note": "", + "log_realtime": "time=2026-09-23T17:58:20.485Z level=INFO msg=\"jevdecide realtime shadow\" mode=assist rule=false jev=false confidence=0.995574 margin=0.99114838 agree=true", + "mode": "assist", + "rule": "false", + "jev": "false", + "confidence": "0.995574", + "margin": "0.99114838", + "agree": "true", + "assist_additive_signal": false, + "log_profile": "" + } +] \ No newline at end of file diff --git a/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-summary.json b/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-summary.json new file mode 100644 index 00000000..2de658cc --- /dev/null +++ b/.doc/test-reports/assets/2026-09-24-jev-dev-assist-enable/batch-summary.json @@ -0,0 +1,104 @@ +{ + "n": 20, + "http_ok": 20, + "assist_additive_hits": 10, + "additive_ids": [ + "B1", + "B2", + "B3", + "B4", + "B5", + "B6", + "B7", + "B8", + "B9", + "B10" + ], + "additive_inputs": [ + { + "id": "B1", + "input": "外面冷不冷啊今天", + "confidence": "0.990937", + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册" + }, + { + "id": "B2", + "input": "明天出门要不要带伞", + "confidence": "0.996626", + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册" + }, + { + "id": "B3", + "input": "上海现在热不热", + "confidence": "0.997512", + "reply_prefix": "上海现在还挺热的~ 🌡️ 据实时数据显示，上海当前气温约 **29°C** 左右，天气多云，湿度较高（约80%），紫外线指数较强。夜间虽然会凉快一些，但整体仍" + }, + { + "id": "B4", + "input": "美元兑人民币现在多少了", + "confidence": "0.998392", + "reply_prefix": "好的，帮您查了一下—— **截至2026年9月22日，美元兑人民币汇率约 6.697。** 近期人民币有所走强，若您后续涉及进口元器件或海外采购预算，这算是" + }, + { + "id": "B5", + "input": "苹果股票今天表现如何", + "confidence": "0.998564", + "reply_prefix": "完全理解您想再掂量一下。针对「医疗器械研发项目」目前的节点，我帮您快速对照一下： • 若推进 V0.5（方案成型与法规对齐）： 我可帮您梳理初步架构与注册" + }, + { + "id": "B6", + "input": "What's the weather like in Beijing today", + "confidence": "0.998316", + "reply_prefix": "北京今日天气（2026年9月24日凌晨）： - 🌡️ 气温：**24.6°C** - 🌫️ 天气：烟霾 + 部分多云 - 💨 风向：东北风 - 🌧️ 降雨概率" + }, + { + "id": "B7", + "input": "黄金现在贵不贵", + "confidence": "0.99545", + "reply_prefix": "**结论：贵，且处于历史高位区间。** 根据实时数据（2026-09-23）： | 指标 | 价格 | |------|------| | 国际黄金现货 |" + }, + { + "id": "B8", + "input": "北京空气质量怎么样最近", + "confidence": "0.997111", + "reply_prefix": "好的，帮您查了一下北京最近的空气质量： **截至 2026-09-23**，北京 AQI 约 **109**，处于**轻度污染**级别。PM2.5 浓度超出世" + }, + { + "id": "B9", + "input": "今晚适合户外跑步吗", + "confidence": "0.992417", + "reply_prefix": "好的，我帮您查了一下—— 当前时间是**凌晨1:56**，严格来说还没到\"今晚\"呢，是今晨 :) 如果您指的是**夜间跑步**的话，依据检索到的建议：夜跑可" + }, + { + "id": "B10", + "input": "现在要穿羽绒服吗上海", + "confidence": "0.987759", + "reply_prefix": "您好，用户！我是您的项目助手，很高兴协助您推进医疗器械立项与研发协同。 针对「医疗器械研发项目」，我可以帮您： • 查询进展、成熟度与风险； • 沉淀客户需求" + } + ], + "agree": 10, + "disagree": 10, + "missing_log": [], + "should_false_wrong_assist": [], + "B_hit_rate": 1.0, + "failures": [], + "ux_visible_live_ids": [ + "B3", + "B4", + "B6", + "B7", + "B8" + ], + "ux_partial_search_ids": [ + "B9" + ], + "ux_weak_project_fallback_ids": [ + "B1", + "B2", + "B5", + "B10" + ], + "decision_layer_additive_rate": "10/10 B candidates", + "safety_false_assist_on_C": "0/6", + "baseline_A_agree": "4/4" +} \ No newline at end of file diff --git a/.env.example b/.env.example index 9e6cdc8a..b8006b72 100644 --- a/.env.example +++ b/.env.example @@ -113,8 +113,8 @@ INTENT_V2_DECISION=false INTENT_V2_DECISION_TIER=none # jev-service typed decide (product shadow/assist; default off) -# local_dev enable: set JEV_DECIDE_MODE=shadow in configs/.env.dev (gitignored) + rebuild app. -# From Docker app → host jev: use http://172.17.0.1:8767 (host.docker.internal may be unreachable). +# local_dev: set JEV_DECIDE_MODE=shadow|assist in configs/.env.dev (gitignored) + recreate/rebuild app. +# assist = rule || high-conf jev yes (additive recall only). From Docker → host: http://172.17.0.1:8767. JEV_DECIDE_MODE=off JEV_BASE_URL=http://127.0.0.1:8767 JEV_API_KEY=