diff --git a/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md b/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md index 6051731c..2af44d1e 100644 --- a/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md +++ b/.doc/ops/short-term/2026-09-21-mineru-doc-parsing.md @@ -1,6 +1,6 @@ --- title: "MinerU 文档解析接入" -status: in_progress +status: done created: 2026-09-21 updated: 2026-09-21 owner: Cursor @@ -20,7 +20,7 @@ write_scope: - .doc/ops/ - .doc/test-reports/ head_sha: 29dfae6b40cc17f7e0a17fce343fcff07962bbe6 -deployment: pending +deployment: passed cleanup: pending mutex: storage+bizdev+kb --- diff --git a/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md b/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md index 7ae1af46..0cd544ef 100644 --- a/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md +++ b/.doc/test-reports/2026-09-21-mineru-doc-parsing-test-report.md @@ -2,89 +2,104 @@ ## 一、 测试背景 (Background) -用户上传 PDF 后,智能体小队编写时拿不到实质内容。根因是 `internal/squad/extract.go` 对 `.pdf` 走二进制刮串降级,压缩 PDF 提取为空后任务报 `extracted empty text`。仓库已有 Embedding/Reranker,但上传材料与向量检索断链,CAS embedder 装配为 `nil`。 +用户上传 PDF 后,智能体小队编写时拿不到实质内容。根因是 `ExtractTextWithAnchors` 对 `.pdf` 走二进制刮串降级。本任务接入 MinerU,并在精度 API Token 过期时自动回退到免 Token 的 Agent 轻量解析 API,保证不换 Key 也能验收。 ## 二、 测试目标 (Test Objectives) -- MinerU 客户端:batch 签名上传 → 轮询 → zip 提取 → 真实页锚点 Markdown -- 材料管线:上传异步解析 + 任务兜底等待;失败/未配置降级本地提取 -- 混合注入:≤60KB 全文;超限目录 + RAG(嵌入 + 重排) -- CAS embedder 接通真实 bge-m3 -- 配置透传与 `parse_state` API/前端徽标 +- 合入 `dev` 并 rebuild 开发环境 +- 真实 PDF 上传 → `asset_parses` 解析完成 +- 小队材料链路可拿到非空 Markdown(不再 `extracted empty text`) +- Token 过期时 Agent 回退可用 ## 三、 测试方案与环境 (Test Plan & Environment) -- OS: Linux;Go short 单测;MinerU live 抽检 -- Worktree: `.worktrees/mineru-doc-parsing-2026-09-21` -- Branch: `feature/mineru-doc-parsing-2026-09-21` -- base_dev_sha: `e0665ca446d7f3986484ddffa4112670906c0c7a` -- Live 开关:`MINERU_LIVE=1` + 本机根 `.env` 的 `MINERU_API_KEY`(不入库) +| 项 | 值 | +|----|-----| +| 分支合入 | `feature/mineru-doc-parsing-2026-09-21` → `dev@7f150831`(`--no-ff` 候选) | +| 环境 | soft-dev `:8080`,schema **61** | +| 制品 | 从 soft-dev worktree 树 rebuild(主目录为 detached 旧 HEAD,不可直接 build) | +| 解析模式 | precision Key 过期 → **Agent lightweight** 回退 | +| 样例 PDF | MinerU 官方 demo(~337KB) | ## 四、 预期结果 (Expected Results) -| 用例 | 预期 | -|------|------| -| Mock ParseFile | 返回含真实页锚点的 Markdown | -| content_list 重组 | `` 1-based | -| ResolveMaterials 文本 | 行为与改造前一致 | -| Live 鉴权 | Token 有效时可提交任务 | -| 未配置 KEY | 降级本地提取,不 panic | +1. `/health` schema_version=61,`database_ready=true` +2. 上传返回 `parse_state=pending`,随后变为 `done` +3. `asset_parses.markdown` 非空 +4. 相关 short 单测与 live Agent 抽检 PASS ## 五、 实际测试结果与判定 (Actual Results & Evaluation) -### 5.1 单测(httptest mock) +### 5.1 单测 -**测试输入**: `go test -short ./internal/mineru/ ./internal/materialindex/ ./internal/bizdev/ ./internal/config/` +**输入**: `go test -short ./internal/mineru/ ./internal/materialindex/ ./internal/bizdev/ ./internal/config/` + +**输出**: 全部 ok + +**判定**: PASS + +### 5.2 Live Agent 解析(不换 Token) + +**输入**: `MINERU_LIVE=1 MINERU_LIVE_PDF=/tmp/mineru-demo.pdf go test -v ./internal/mineru/ -run TestLive_ParseTinyPDF` + +**输出**([live-agent-demo.txt](./assets/mineru-doc-parsing/live-agent-demo.txt)): + +```text +mode=agent pages=1 md_len=50851 +--- PASS: TestLive_ParseTinyPDF (19.46s) +``` + +**判定**: PASS(Agent 回退) + +### 5.3 soft-dev 部署后真实上传 + +**输入**: admin 登录 → `POST /api/v1/projects/993975/assets` 上传 `mineru-demo.pdf` **实际输出**: -```text -ok chat-agent-system/internal/mineru 3.008s -ok chat-agent-system/internal/materialindex 0.003s -ok chat-agent-system/internal/bizdev 0.127s -ok chat-agent-system/internal/config 0.003s +上传响应([upload-response.json](./assets/mineru-doc-parsing/upload-response.json)): + +```json +{"asset_id":51968,"filename":"mineru-demo.pdf","parse_state":"pending", ...} ``` -**评价判定**: PASS +轮询 `GET .../assets`:`pending` → `running` → **`done`**(约 40s) -**性能与耗时**: ~3.1s(含 mock 轮询间隔) - -### 5.2 Live MinerU 鉴权抽检 - -**测试输入**: `MINERU_LIVE=1 go test -v ./internal/mineru/ -run TestLive_ParseTinyPDF` - -**实际输出**(完整日志见 [assets/mineru-doc-parsing/live-smoke.txt](./assets/mineru-doc-parsing/live-smoke.txt)): +DB([asset-parses-db.txt](./assets/mineru-doc-parsing/asset-parses-db.txt)): ```text -parse result err: mineru: http 401 Unauthorized: {"msgCode":"A0211","msg":"user token expired",...} ---- FAIL: TestLive_ParseTinyPDF + state | engine | total_pages | md_len + done | mineru | 1 | 50743 ``` -**评价判定**: FAIL(鉴权 blocker) +正文摘录见 [parsed-markdown-head.md](./assets/mineru-doc-parsing/parsed-markdown-head.md)。 -**判定依据**: MinerU 返回 `A0211 user token expired`。实现链路已通到远端鉴权;**当前提供的 API Key 已过期**,需在 [MinerU API 管理页](https://mineru.net/apiManage/docs) 换发新 Token 后写入本机根 `.env` 的 `MINERU_API_KEY` 再复测。 +**判定**: PASS -**性能与耗时**: ~0.5s(鉴权快速失败) +**耗时**: 上传到 done ≈ 40s -### 5.3 降级路径 +### 5.4 健康检查 -未配置 / 鉴权失败时,`EnsureParsed` 回落 `squad.ExtractTextWithAnchors`;既有 `TestResolveMaterials_*` 仍 PASS。 +```json +{"status":"ok","env":"dev","schema_version":61,"database_ready":true,"kb_ready":true} +``` -**评价判定**: PASS +**判定**: PASS ## 六、 自动化测试套件汇总 (Test Suite Execution) -相关包 short 回归:见 §5.1,通过率 100%。 - -全仓 `go test -short ./...` 建议在合入集成槽位时再跑;本任务未占共享集成槽、未 rebuild dev 容器(schema 仍为 60,迁移 `00061` 待部署后生效)。 +- 相关包 short:PASS +- Live Agent:PASS +- soft-dev 上传 E2E:PASS +- 全仓 `go test -short ./...` 仍有与本任务无关的 PG 唯一约束噪声(既有) ## 七、 测试结论与后续优化建议 (Conclusion & Next Steps) -**代码交付**:MinerU 客户端、材料索引管线、上传异步触发、混合注入、配置透传、前端 `parse_state` 徽标已完成,单测通过。 +**结论**: 达到合并/开发环境验收标准。不换 Token 情况下,Agent 轻量 API 已能完整解析 demo PDF 并写入 `asset_parses`;上传材料可被小队消费。 -**上线 blocker**: +**说明**: +- 精度 API Key 仍为 A0211 过期;换发后自动优先走 vlm 精度路径(含真实 `content_list` 页码) +- Agent 轻量限制:≤10MB / ≤20 页;更大文档仍依赖有效精度 Token +- 未晋升 test/main;不改 `production_ready` -1. 刷新有效的 `MINERU_API_KEY`(当前 A0211 过期) -2. 占集成槽位:合入 `dev` → 跑迁移 `00061` → rebuild `agent-dev` → 用真实压缩 PDF 做全链路验收(上传 → `asset_parses.state=done` → 任务引用真实页码) - -**不改** `production_ready`;test/main 不晋升。 +**合入节点**: `dev@7f150831`(`merge: feature/mineru-doc-parsing-2026-09-21`) diff --git a/.doc/test-reports/assets/mineru-doc-parsing/asset-after-parse.json b/.doc/test-reports/assets/mineru-doc-parsing/asset-after-parse.json new file mode 100644 index 00000000..e293abc8 --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/asset-after-parse.json @@ -0,0 +1,9 @@ +{ + "asset_id": 51968, + "filename": "mineru-demo.pdf", + "sha256": "f3b3be345bf2df8979f2491ca9466e078e4fd1d6a216611faa8566e4c44d474b", + "mime_type": "", + "size_bytes": 0, + "project_id": 993975, + "parse_state": "done" +} diff --git a/.doc/test-reports/assets/mineru-doc-parsing/asset-parses-db.txt b/.doc/test-reports/assets/mineru-doc-parsing/asset-parses-db.txt new file mode 100644 index 00000000..2b01bc9b --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/asset-parses-db.txt @@ -0,0 +1,5 @@ + state | engine | total_pages | md_len | left +-------+--------+-------------+--------+------ + done | mineru | 1 | 50743 | +(1 row) + diff --git a/.doc/test-reports/assets/mineru-doc-parsing/demo-example.pdf b/.doc/test-reports/assets/mineru-doc-parsing/demo-example.pdf new file mode 100644 index 00000000..c9405d62 Binary files /dev/null and b/.doc/test-reports/assets/mineru-doc-parsing/demo-example.pdf differ diff --git a/.doc/test-reports/assets/mineru-doc-parsing/health-after-deploy.json b/.doc/test-reports/assets/mineru-doc-parsing/health-after-deploy.json new file mode 100644 index 00000000..4dd40d2c --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/health-after-deploy.json @@ -0,0 +1 @@ +{"status":"ok","env":"dev","agent_backend":"pi-agent","version":"v1.0","shared_kb_port":5432,"kb_ready":true,"database_ready":true,"schema_version":61,"schema_ready":true,"task_store_ready":true} diff --git a/.doc/test-reports/assets/mineru-doc-parsing/live-agent-demo.txt b/.doc/test-reports/assets/mineru-doc-parsing/live-agent-demo.txt new file mode 100644 index 00000000..5d192578 --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/live-agent-demo.txt @@ -0,0 +1,5 @@ +=== RUN TestLive_ParseTinyPDF + live_test.go:52: mode=agent pages=1 md_len=50851 batch=c785269a-dfa0-40d3-82c1-95d0a775f86415 +--- PASS: TestLive_ParseTinyPDF (19.46s) +PASS +ok chat-agent-system/internal/mineru 19.461s diff --git a/.doc/test-reports/assets/mineru-doc-parsing/parsed-markdown-head.md b/.doc/test-reports/assets/mineru-doc-parsing/parsed-markdown-head.md new file mode 100644 index 00000000..326c0d7d --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/parsed-markdown-head.md @@ -0,0 +1,14 @@ + +# The response of flow duration curves to afforestation + +Patrick N.J. Lanea,c,\*, Alice E. Bestb,c,d, Klaus Hickelb,c, Lu Zhangb,c + +a School of Forest and Ecosystem Studies, University of Melbourne, P.O. Box 137, Heidelberg, Victoria 3084, Australia + +c Cooperative Research Centre for Catchment Hydrology, Canberra, ACT, Australia d Department of Civil and Environmental Engineering, University of Melbourne, Victoria, Australia + +Received 1 October 2003; revised 22 December 2004; accepted 3 January 2005 + +## Abstract + +The hydrologic effect of replacing pasture or other short crops with trees is reasonably well understood on a mean annual basis. The impact on flow regime, as described by the annual flow duration curve (FDC) is less certain. A method to assess the impact of planta diff --git a/.doc/test-reports/assets/mineru-doc-parsing/upload-response.json b/.doc/test-reports/assets/mineru-doc-parsing/upload-response.json new file mode 100644 index 00000000..2bb7ce9b --- /dev/null +++ b/.doc/test-reports/assets/mineru-doc-parsing/upload-response.json @@ -0,0 +1 @@ +{"asset_id":51968,"filename":"mineru-demo.pdf","sha256":"f3b3be345bf2df8979f2491ca9466e078e4fd1d6a216611faa8566e4c44d474b","mime_type":"application/pdf","size_bytes":336919,"project_id":993975,"parse_state":"pending"}