From 29dfae6b40cc17f7e0a17fce343fcff07962bbe6 Mon Sep 17 00:00:00 2001 From: weichen Date: Mon, 21 Sep 2026 08:29:10 +0000 Subject: [PATCH] docs(test): record MinerU parsing implementation and live blocker Claim CURRENT mutex, archive short-term task, and document unit PASS plus live A0211 token-expired blocker. Co-authored-by: Cursor --- .doc/ops/CURRENT.md | 1 + .../2026-09-21-mineru-doc-parsing.md | 68 ++++++++++++++ ...26-09-21-mineru-doc-parsing-test-report.md | 90 +++++++++++++++++++ .../assets/mineru-doc-parsing/live-smoke.txt | 7 ++ 4 files changed, 166 insertions(+) create mode 100644 .doc/ops/short-term/2026-09-21-mineru-doc-parsing.md create mode 100644 .doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md create mode 100644 .doc/test-reports/assets/mineru-doc-parsing/live-smoke.txt diff --git a/.doc/ops/CURRENT.md b/.doc/ops/CURRENT.md index 54c2c6c4..371f58db 100644 --- a/.doc/ops/CURRENT.md +++ b/.doc/ops/CURRENT.md @@ -13,6 +13,7 @@ | owner | task | branch | worktree | mutex | status | |-------|------|--------|----------|-------|--------| +| Cursor | 2026-09-21-mineru-doc-parsing | feature/mineru-doc-parsing-2026-09-21 | .worktrees/mineru-doc-parsing-2026-09-21 | storage+bizdev+kb | in_progress | | Cursor | 2026-09-21-intent-workflow-scoping | fix/intent-workflow-scoping-2026-09-21 | - | intent+web+docgov | in_progress | | Codex | 2026-09-21-agent-workflow-governance-adoption | feature/agent-workflow-governance-plan | .worktrees/feature-agent-workflow-governance-plan | agent-workflow-governance | in_progress | | Codex | 2026-09-21-agent-workflow-governance-plan | feature/agent-workflow-governance-plan | .worktrees/feature-agent-workflow-governance-plan | agent-workflow-governance-plan | done | diff --git a/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md b/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md new file mode 100644 index 00000000..7c478f20 --- /dev/null +++ b/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md @@ -0,0 +1,68 @@ +--- +title: "MinerU 文档解析接入" +status: in_progress +created: 2026-09-21 +updated: 2026-09-21 +owner: Cursor +branch: feature/mineru-doc-parsing-2026-09-21 +worktree: .worktrees/mineru-doc-parsing-2026-09-21 +base_dev_sha: e0665ca446d7f3986484ddffa4112670906c0c7a +write_scope: + - deploy/postgres/goose/ + - internal/mineru/ + - internal/materialindex/ + - internal/bizdev/ + - internal/config/ + - internal/server/ + - web/src/ + - .env.example + - docker-compose.*.yml + - .doc/ops/ + - .doc/test-reports/ +head_sha: pending +deployment: pending +cleanup: pending +mutex: storage+bizdev+kb +--- + +# MinerU 文档解析接入 + +## 目标 + +接入 MinerU 精准解析 API，使上传 PDF/Office 等材料可被小队可靠理解；大文档走分块嵌入 + 重排检索注入。 + +## 范围 + +- `asset_parses` / `material_chunks` 迁移 +- `internal/mineru` 客户端与页锚点重组 +- `internal/materialindex` 异步解析 + 兜底等待 + RAG +- 上传触发、ResolveMaterials 混合注入、CAS embedder 接通 +- 配置透传；本机 `.env` 写入密钥（不入库） + +## 非目标 + +- 不合入/部署共享 KB 外部管线 +- 不改 production_ready；不操作 test/main 晋升 + +## 步骤 + +1. [x] DB 迁移 00061 +2. [x] mineru 客户端 +3. [x] materialindex 管线 +4. [x] 上传 hook + parse_state +5. [x] ResolveMaterials + CAS embedder +6. [x] 配置透传 +7. [x] 单测 / live 抽检（live 因 token 过期 FAIL，已记 blocker） +8. [x] 测试报告与 CURRENT 认领 + +## 验收标准 + +- [x] 相关 short 单测通过（mock） +- [ ] 压缩 PDF 全链路（blocked: MINERU_API_KEY A0211 expired） +- [x] 未配置 MinerU 时降级不回归 +- [x] 大文档 toc+rag 注入路径已实现 + +## 相关链接 + +- Test report: ../../test-reports/2026-09-21-mineru-doc-parsing-test-report.md +- Live evidence: ../../test-reports/assets/mineru-doc-parsing/live-smoke.txt diff --git a/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md b/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md new file mode 100644 index 00000000..7ae1af46 --- /dev/null +++ b/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md @@ -0,0 +1,90 @@ +# MinerU 文档解析接入 测试报告 + +## 一、 测试背景 (Background) + +用户上传 PDF 后，智能体小队编写时拿不到实质内容。根因是 `internal/squad/extract.go` 对 `.pdf` 走二进制刮串降级，压缩 PDF 提取为空后任务报 `extracted empty text`。仓库已有 Embedding/Reranker，但上传材料与向量检索断链，CAS embedder 装配为 `nil`。 + +## 二、 测试目标 (Test Objectives) + +- MinerU 客户端：batch 签名上传 → 轮询 → zip 提取 → 真实页锚点 Markdown +- 材料管线：上传异步解析 + 任务兜底等待；失败/未配置降级本地提取 +- 混合注入：≤60KB 全文；超限目录 + RAG（嵌入 + 重排） +- CAS embedder 接通真实 bge-m3 +- 配置透传与 `parse_state` API/前端徽标 + +## 三、 测试方案与环境 (Test Plan & Environment) + +- OS: Linux；Go short 单测；MinerU live 抽检 +- Worktree: `.worktrees/mineru-doc-parsing-2026-09-21` +- Branch: `feature/mineru-doc-parsing-2026-09-21` +- base_dev_sha: `e0665ca446d7f3986484ddffa4112670906c0c7a` +- Live 开关：`MINERU_LIVE=1` + 本机根 `.env` 的 `MINERU_API_KEY`（不入库） + +## 四、 预期结果 (Expected Results) + +| 用例 | 预期 | +|------|------| +| Mock ParseFile | 返回含真实页锚点的 Markdown | +| content_list 重组 | `` 1-based | +| ResolveMaterials 文本 | 行为与改造前一致 | +| Live 鉴权 | Token 有效时可提交任务 | +| 未配置 KEY | 降级本地提取，不 panic | + +## 五、 实际测试结果与判定 (Actual Results & Evaluation) + +### 5.1 单测（httptest mock） + +**测试输入**: `go test -short ./internal/mineru/ ./internal/materialindex/ ./internal/bizdev/ ./internal/config/` + +**实际输出**: + +```text +ok chat-agent-system/internal/mineru 3.008s +ok chat-agent-system/internal/materialindex 0.003s +ok chat-agent-system/internal/bizdev 0.127s +ok chat-agent-system/internal/config 0.003s +``` + +**评价判定**: PASS + +**性能与耗时**: ~3.1s（含 mock 轮询间隔） + +### 5.2 Live MinerU 鉴权抽检 + +**测试输入**: `MINERU_LIVE=1 go test -v ./internal/mineru/ -run TestLive_ParseTinyPDF` + +**实际输出**（完整日志见 [assets/mineru-doc-parsing/live-smoke.txt](./assets/mineru-doc-parsing/live-smoke.txt)）: + +```text +parse result err: mineru: http 401 Unauthorized: {"msgCode":"A0211","msg":"user token expired",...} +--- FAIL: TestLive_ParseTinyPDF +``` + +**评价判定**: FAIL（鉴权 blocker） + +**判定依据**: MinerU 返回 `A0211 user token expired`。实现链路已通到远端鉴权；**当前提供的 API Key 已过期**，需在 [MinerU API 管理页](https://mineru.net/apiManage/docs) 换发新 Token 后写入本机根 `.env` 的 `MINERU_API_KEY` 再复测。 + +**性能与耗时**: ~0.5s（鉴权快速失败） + +### 5.3 降级路径 + +未配置 / 鉴权失败时，`EnsureParsed` 回落 `squad.ExtractTextWithAnchors`；既有 `TestResolveMaterials_*` 仍 PASS。 + +**评价判定**: PASS + +## 六、 自动化测试套件汇总 (Test Suite Execution) + +相关包 short 回归：见 §5.1，通过率 100%。 + +全仓 `go test -short ./...` 建议在合入集成槽位时再跑；本任务未占共享集成槽、未 rebuild dev 容器（schema 仍为 60，迁移 `00061` 待部署后生效）。 + +## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) + +**代码交付**：MinerU 客户端、材料索引管线、上传异步触发、混合注入、配置透传、前端 `parse_state` 徽标已完成，单测通过。 + +**上线 blocker**： + +1. 刷新有效的 `MINERU_API_KEY`（当前 A0211 过期） +2. 占集成槽位：合入 `dev` → 跑迁移 `00061` → rebuild `agent-dev` → 用真实压缩 PDF 做全链路验收（上传 → `asset_parses.state=done` → 任务引用真实页码） + +**不改** `production_ready`；test/main 不晋升。 diff --git a/.doc/test-reports/assets/mineru-doc-parsing/live-smoke.txt b/.doc/test-reports/assets/mineru-doc-parsing/live-smoke.txt new file mode 100644 index 00000000..c6935550 --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/live-smoke.txt @@ -0,0 +1,7 @@ +=== RUN TestLive_ParseTinyPDF + live_test.go:34: parse result err: mineru: http 401 Unauthorized: {"traceId":"fec4ce085809","msgCode":"A0211","msg":"user token expired","data":null,"success":false,"total":0} + live_test.go:36: mineru auth failed (refresh MINERU_API_KEY): mineru: http 401 Unauthorized: {"traceId":"fec4ce085809","msgCode":"A0211","msg":"user token expired","data":null,"success":false,"total":0} +--- FAIL: TestLive_ParseTinyPDF (0.47s) +FAIL +FAIL chat-agent-system/internal/mineru 0.477s +FAIL