HaHafeng
61cdc97eeb
Summary: - Fix pg-boss queue conflict (duplicate key violation on queue_pkey) - Add global error listener to prevent process crash - Reduce connection pool from 10 to 4 - Add graceful shutdown handling (SIGTERM/SIGINT) - Fix researchWorker recursive call bug in catch block - Make screeningWorker idempotent using upsert Security Standards (v1.1): - Prohibit recursive retry in Worker catch blocks - Prohibit payload bloat (only store fileKey/ID in job.data) - Require Worker idempotency (upsert + unique constraint) - Recommend task-specific expireInSeconds settings - Document graceful shutdown pattern New Features: - PKB signed URL endpoint for document preview/download - pg_bigm installation guide for Docker - Dockerfile.postgres-with-extensions for pgvector + pg_bigm Documentation: - Update Postgres-Only async task processing guide (v1.1) - Add troubleshooting SQL queries - Update safety checklist Tested: Local verification passed
213 lines
4.9 KiB
Markdown
213 lines
4.9 KiB
Markdown
# pg_bigm 安装指南
|
||
|
||
> **版本：** v1.0
|
||
> **日期：** 2026-01-23
|
||
> **状态：** 待部署
|
||
> **用途：** 优化中文关键词检索性能
|
||
|
||
---
|
||
|
||
## 📋 概述
|
||
|
||
pg_bigm 是 PostgreSQL 的全文搜索扩展，专门针对中日韩（CJK）字符优化。相比原生 LIKE/ILIKE，pg_bigm 提供：
|
||
|
||
- **2-gram 索引**：将文本拆分为连续的 2 字符片段，支持任意子串匹配
|
||
- **中文友好**：原生支持中文分词，无需额外配置
|
||
- **性能提升**：10-100x 性能提升（取决于数据量）
|
||
- **模糊搜索**：支持相似度搜索
|
||
|
||
---
|
||
|
||
## 🚀 安装步骤
|
||
|
||
### 方案 1：Docker 镜像升级（推荐）
|
||
|
||
**适用场景**：本地开发环境
|
||
|
||
```bash
|
||
cd AIclinicalresearch
|
||
|
||
# 1. 备份现有数据
|
||
docker exec ai-clinical-postgres pg_dump -U postgres -d ai_clinical_research > backup_$(date +%Y%m%d_%H%M%S).sql
|
||
|
||
# 2. 构建新镜像（包含 pgvector + pg_bigm）
|
||
docker build -f Dockerfile.postgres-with-extensions -t ai-clinical-postgres:v1.1 .
|
||
|
||
# 3. 停止现有容器
|
||
docker compose down
|
||
|
||
# 4. 修改 docker-compose.yml，替换镜像
|
||
# image: pgvector/pgvector:pg15 → image: ai-clinical-postgres:v1.1
|
||
|
||
# 5. 启动新容器
|
||
docker compose up -d
|
||
|
||
# 6. 验证扩展安装
|
||
docker exec ai-clinical-postgres psql -U postgres -d ai_clinical_research -c "SELECT extname, extversion FROM pg_extension;"
|
||
```
|
||
|
||
**预期输出**：
|
||
```
|
||
extname | extversion
|
||
----------+------------
|
||
plpgsql | 1.0
|
||
vector | 0.8.0
|
||
pg_bigm | 1.2
|
||
```
|
||
|
||
### 方案 2：在现有容器中安装
|
||
|
||
**适用场景**：不想重建镜像
|
||
|
||
```bash
|
||
# 1. 进入容器
|
||
docker exec -it ai-clinical-postgres bash
|
||
|
||
# 2. 安装编译工具
|
||
apt-get update && apt-get install -y build-essential postgresql-server-dev-15 wget
|
||
|
||
# 3. 下载并编译 pg_bigm
|
||
cd /tmp
|
||
wget https://github.com/pgbigm/pg_bigm/archive/refs/tags/v1.2-20200228.tar.gz
|
||
tar -xzf v1.2-20200228.tar.gz
|
||
cd pg_bigm-1.2-20200228
|
||
make USE_PGXS=1
|
||
make USE_PGXS=1 install
|
||
|
||
# 4. 清理
|
||
rm -rf /tmp/pg_bigm* /tmp/v1.2-20200228.tar.gz
|
||
apt-get purge -y build-essential postgresql-server-dev-15 wget
|
||
apt-get autoremove -y
|
||
|
||
# 5. 退出容器
|
||
exit
|
||
|
||
# 6. 创建扩展
|
||
docker exec ai-clinical-postgres psql -U postgres -d ai_clinical_research -c "CREATE EXTENSION IF NOT EXISTS pg_bigm;"
|
||
```
|
||
|
||
### 方案 3：阿里云 RDS
|
||
|
||
**适用场景**：生产环境（阿里云 RDS PostgreSQL）
|
||
|
||
阿里云 RDS PostgreSQL 15 **已内置** pg_bigm，只需执行：
|
||
|
||
```sql
|
||
-- 连接到 RDS 数据库
|
||
CREATE EXTENSION IF NOT EXISTS pg_bigm;
|
||
```
|
||
|
||
---
|
||
|
||
## 🔧 使用方法
|
||
|
||
### 1. 创建 GIN 索引
|
||
|
||
```sql
|
||
-- 为 ekb_chunk 表的 content 列创建 pg_bigm 索引
|
||
CREATE INDEX IF NOT EXISTS idx_ekb_chunk_content_bigm
|
||
ON ekb_schema.ekb_chunk
|
||
USING gin (content gin_bigm_ops);
|
||
|
||
-- 验证索引创建
|
||
SELECT indexname, indexdef FROM pg_indexes
|
||
WHERE tablename = 'ekb_chunk' AND indexname LIKE '%bigm%';
|
||
```
|
||
|
||
### 2. 查询示例
|
||
|
||
```sql
|
||
-- 基本查询（使用索引）
|
||
SELECT * FROM ekb_schema.ekb_chunk
|
||
WHERE content LIKE '%银杏叶%';
|
||
|
||
-- 相似度查询
|
||
SELECT *, bigm_similarity(content, '银杏叶副作用') AS similarity
|
||
FROM ekb_schema.ekb_chunk
|
||
WHERE content LIKE '%银杏叶%'
|
||
ORDER BY similarity DESC
|
||
LIMIT 10;
|
||
```
|
||
|
||
### 3. 在 VectorSearchService 中使用
|
||
|
||
```typescript
|
||
// keywordSearch 方法会自动检测 pg_bigm
|
||
// 如果扩展可用，使用 GIN 索引加速
|
||
// 否则 fallback 到 ILIKE
|
||
|
||
async keywordSearch(query: string, options: SearchOptions) {
|
||
// 自动使用最优方案
|
||
// pg_bigm: SELECT * WHERE content LIKE '%query%' (使用索引)
|
||
// fallback: SELECT * WHERE content ILIKE '%query%' (全表扫描)
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 📊 性能对比
|
||
|
||
| 场景 | ILIKE（无索引） | pg_bigm（GIN索引） | 提升 |
|
||
|------|----------------|-------------------|------|
|
||
| 10万条记录 | 500ms | 5ms | 100x |
|
||
| 100万条记录 | 5s | 50ms | 100x |
|
||
| 中文2字符 | 支持 | 支持 | - |
|
||
| 中文1字符 | 支持 | 不支持* | - |
|
||
|
||
> *pg_bigm 基于 2-gram，单字符查询需要至少2个字符
|
||
|
||
---
|
||
|
||
## ⚠️ 注意事项
|
||
|
||
### 1. 索引大小
|
||
|
||
pg_bigm 的 GIN 索引会占用额外存储空间：
|
||
|
||
```sql
|
||
-- 查看索引大小
|
||
SELECT pg_size_pretty(pg_relation_size('idx_ekb_chunk_content_bigm'));
|
||
```
|
||
|
||
预估：原始数据的 50%-100%
|
||
|
||
### 2. 写入性能
|
||
|
||
GIN 索引会影响写入性能：
|
||
|
||
- INSERT：约慢 20-30%
|
||
- UPDATE content 字段：约慢 30-50%
|
||
|
||
**建议**：批量写入时可临时禁用索引
|
||
|
||
### 3. 最小查询长度
|
||
|
||
pg_bigm 基于 2-gram，单字符查询效果差：
|
||
|
||
```sql
|
||
-- ❌ 效果差
|
||
SELECT * WHERE content LIKE '%癌%';
|
||
|
||
-- ✅ 效果好
|
||
SELECT * WHERE content LIKE '%肺癌%';
|
||
```
|
||
|
||
---
|
||
|
||
## 🔗 相关文档
|
||
|
||
- [pg_bigm 官方文档](https://pgbigm.osdn.jp/pg_bigm_en-1-2.html)
|
||
- [RAG 引擎使用指南](./05-RAG引擎使用指南.md)
|
||
- [pgvector 替换 Dify 计划](./02-pgvector替换Dify计划.md)
|
||
|
||
---
|
||
|
||
## 📅 更新计划
|
||
|
||
1. ✅ 创建 Dockerfile 和初始化脚本
|
||
2. ⏳ 本地环境测试
|
||
3. ⏳ 更新 VectorSearchService 使用 pg_bigm
|
||
4. ⏳ 生产环境部署（阿里云 RDS）
|
||
5. ⏳ 创建索引并验证性能
|
||
|