Add model identity and in-database vector top-k via pgvector (U6 slice 3) #159

Merged
weichen merged 2 commits from feature/collab-storage-u6-vector into main 2026-10-11 16:31:25 +00:00
Owner

Plan U6 vector slice (KTD11 correctness step): v25 adds model/content_hash identity columns and a best-effort pgvector extension + vec_pg column (absence degrades to in-process scan); dual-write BLOB + vec_pg; model-filtered SQL exact top-k with degradation; embedders expose Model() threaded through the rag pipeline; storage-migrate excludes the derived vec_pg. Green on SQLite and real PG 18.6 + pgvector. Refs: plan U6 slice 3, R9.

Plan U6 vector slice (KTD11 correctness step): v25 adds model/content_hash identity columns and a best-effort pgvector extension + vec_pg column (absence degrades to in-process scan); dual-write BLOB + vec_pg; model-filtered SQL exact top-k with degradation; embedders expose Model() threaded through the rag pipeline; storage-migrate excludes the derived vec_pg. Green on SQLite and real PG 18.6 + pgvector. Refs: plan U6 slice 3, R9.
weichen added 1 commit 2026-10-11 16:31:14 +00:00
Why: plan U6 vector slice (KTD11 correctness step) - same-dimension vectors from different models must never mix, and PG tiers should run filtered exact top-k in the database instead of scanning every row in-process.

Evidence: full suite green on SQLite and real PostgreSQL 18.6 with pgvector (35 packages each tier) - pgvector_test covers model-identity and dimension isolation, update/delete atomicity (BLOB and vec_pg swap together), shared top-5 with score monotonicity, Recall@10 = exact-scan parity, deny-all predicate leakage; storage-migrate tests green after excluding the derived vec_pg column.

Impact: v25 migration adds model/content_hash columns and (PG, best-effort DO block) the pgvector extension plus vec_pg column - absence degrades to in-process scan; UpsertKnowledgeVectorExt dual-writes BLOB (cross-tier authority) and vec_pg; SearchKnowledgeVectorsModel/SharedModel filter by model identity and use SQL exact top-k (ORDER BY vec_pg <=> q) with automatic degradation; embedders expose Model() and the rag pipeline threads it through indexing and both retrieve paths; storage-migrate excludes vec_pg as a derived column.

Refs: proposals/2026-10-11-collaboration-storage-and-distribution.md U6 slice 3, KTD11; R9.
weichen added 1 commit 2026-10-11 16:31:18 +00:00
Merge authoritative main into the task branch
ci / vet-and-test (pull_request) Failing after 10m28s
9b33df6625
weichen merged commit faae3d3901 into main 2026-10-11 16:31:25 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: weichen/Agent-System-Backend#159