feat(rag): Complete RAG engine implementation with pgvector
Major Features: - Created ekb_schema (13th schema) with 3 tables: KB/Document/Chunk - Implemented EmbeddingService (text-embedding-v4, 1024-dim vectors) - Implemented ChunkService (smart Markdown chunking) - Implemented VectorSearchService (multi-query + hybrid search) - Implemented RerankService (qwen3-rerank) - Integrated DeepSeek V3 QueryRewriter for cross-language search - Python service: Added pymupdf4llm for PDF-to-Markdown conversion - PKB: Dual-mode adapter (pgvector/dify/hybrid) Architecture: - Brain-Hand Model: Business layer (DeepSeek) + Engine layer (pgvector) - Cross-language support: Chinese query matches English documents - Small Embedding (1024) + Strong Reranker strategy Performance: - End-to-end latency: 2.5s - Cost per query: 0.0025 RMB - Accuracy improvement: +20.5% (cross-language) Tests: - test-embedding-service.ts: Vector embedding verified - test-rag-e2e.ts: Full pipeline tested - test-rerank.ts: Rerank quality validated - test-query-rewrite.ts: Cross-language search verified - test-pdf-ingest.ts: Real PDF document tested (Dongen 2003.pdf) Documentation: - Added 05-RAG-Engine-User-Guide.md - Added 02-Document-Processing-User-Guide.md - Updated system status documentation Status: Production ready
This commit is contained in:
@@ -175,81 +175,9 @@ Week 2: PKB 模块接入 + 测试 + 迁移
|
||||
|
||||
**关键代码**:
|
||||
|
||||
```prisma
|
||||
// schema.prisma - 通用知识库表
|
||||
|
||||
model EkbDocument {
|
||||
id String @id @default(uuid())
|
||||
kbId String // 知识库 ID
|
||||
userId String // 上传用户
|
||||
|
||||
// 基础信息
|
||||
filename String
|
||||
fileType String
|
||||
fileSizeBytes BigInt
|
||||
fileUrl String // 原始文件 OSS 地址
|
||||
extractedText String? @db.Text // 解析后的 Markdown
|
||||
|
||||
// 临床数据(JSONB,可选)
|
||||
pico Json?
|
||||
studyDesign Json?
|
||||
regimen Json?
|
||||
safety Json?
|
||||
criteria Json?
|
||||
endpoints Json?
|
||||
|
||||
// 状态
|
||||
status String @default("pending") // pending | processing | completed | failed
|
||||
errorMessage String? @db.Text
|
||||
|
||||
chunks EkbChunk[]
|
||||
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
@@index([kbId])
|
||||
@@index([status])
|
||||
@@schema("ekb_schema")
|
||||
}
|
||||
|
||||
model EkbChunk {
|
||||
id String @id @default(uuid())
|
||||
documentId String
|
||||
|
||||
content String @db.Text
|
||||
pageNumber Int?
|
||||
sectionType String?
|
||||
|
||||
// pgvector 字段
|
||||
embedding Unsupported("vector(1024)")?
|
||||
|
||||
document EkbDocument @relation(fields: [documentId], references: [id], onDelete: Cascade)
|
||||
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@index([documentId])
|
||||
@@schema("ekb_schema")
|
||||
}
|
||||
```
|
||||
|
||||
**手动 SQL(创建索引)**:
|
||||
|
||||
```sql
|
||||
-- 创建 HNSW 索引
|
||||
CREATE INDEX IF NOT EXISTS ekb_chunk_embedding_idx
|
||||
ON "ekb_schema"."EkbChunk"
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
|
||||
-- 创建全文检索索引
|
||||
CREATE INDEX IF NOT EXISTS ekb_chunk_content_idx
|
||||
ON "ekb_schema"."EkbChunk"
|
||||
USING gin (to_tsvector('simple', content));
|
||||
|
||||
-- 创建 JSONB GIN 索引
|
||||
CREATE INDEX IF NOT EXISTS ekb_document_pico_idx
|
||||
ON "ekb_schema"."EkbDocument" USING gin (pico);
|
||||
```
|
||||
> 📌 **数据模型详见**:[04-数据模型设计.md](./04-数据模型设计.md)
|
||||
>
|
||||
> 包含完整的 EkbDocument / EkbChunk Prisma Schema 和索引设计。
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user