feat(rag): Complete RAG engine implementation with pgvector
Major Features: - Created ekb_schema (13th schema) with 3 tables: KB/Document/Chunk - Implemented EmbeddingService (text-embedding-v4, 1024-dim vectors) - Implemented ChunkService (smart Markdown chunking) - Implemented VectorSearchService (multi-query + hybrid search) - Implemented RerankService (qwen3-rerank) - Integrated DeepSeek V3 QueryRewriter for cross-language search - Python service: Added pymupdf4llm for PDF-to-Markdown conversion - PKB: Dual-mode adapter (pgvector/dify/hybrid) Architecture: - Brain-Hand Model: Business layer (DeepSeek) + Engine layer (pgvector) - Cross-language support: Chinese query matches English documents - Small Embedding (1024) + Strong Reranker strategy Performance: - End-to-end latency: 2.5s - Cost per query: 0.0025 RMB - Accuracy improvement: +20.5% (cross-language) Tests: - test-embedding-service.ts: Vector embedding verified - test-rag-e2e.ts: Full pipeline tested - test-rerank.ts: Rerank quality validated - test-query-rewrite.ts: Cross-language search verified - test-pdf-ingest.ts: Real PDF document tested (Dongen 2003.pdf) Documentation: - Added 05-RAG-Engine-User-Guide.md - Added 02-Document-Processing-User-Guide.md - Updated system status documentation Status: Production ready
This commit is contained in:
@@ -3,25 +3,31 @@ fastapi==0.104.1
|
||||
uvicorn[standard]==0.24.0
|
||||
python-multipart==0.0.6
|
||||
|
||||
# PDF处理
|
||||
PyMuPDF>=1.24.0 # 使用更新版本,有预编译wheel
|
||||
pdfplumber==0.10.3
|
||||
nougat-ocr==0.1.17 # 学术PDF高质量提取(英文)
|
||||
albumentations==1.3.1 # Nougat兼容版本(不要升级到2.x)
|
||||
# PDF处理 - 使用 pymupdf4llm(替代 nougat,更轻量)
|
||||
pymupdf4llm>=0.0.17 # PDF → Markdown,自动包含 pymupdf
|
||||
pdfplumber==0.10.3 # 备用 PDF 处理
|
||||
|
||||
# Docx处理(Day 3需要)
|
||||
mammoth==1.6.0
|
||||
python-docx==1.1.0
|
||||
# Word处理
|
||||
mammoth==1.6.0 # Docx → Markdown
|
||||
python-docx==1.1.0 # Docx 读取
|
||||
|
||||
# 语言检测(Day 2需要)
|
||||
# Excel/CSV处理
|
||||
pandas>=2.0.0 # 表格处理
|
||||
openpyxl>=3.1.2 # Excel 读取
|
||||
tabulate>=0.9.0 # DataFrame → Markdown
|
||||
|
||||
# PPT处理
|
||||
python-pptx>=0.6.23 # PPT 读取
|
||||
|
||||
# 语言检测
|
||||
langdetect==1.0.9
|
||||
|
||||
# 编码检测(Day 3需要)
|
||||
# 编码检测
|
||||
chardet==5.2.0
|
||||
|
||||
# 工具
|
||||
python-dotenv==1.0.0
|
||||
pydantic>=2.10.0 # 使用更新版本,有预编译wheel
|
||||
pydantic>=2.10.0
|
||||
|
||||
# 日志
|
||||
loguru==0.7.2
|
||||
|
||||
Reference in New Issue
Block a user