feat(admin): Add user management and upgrade to module permission system
Features - User Management (Phase 4.1): - Database: Add user_modules table for fine-grained module permissions - Database: Add 4 user permissions (view/create/edit/delete) to role_permissions - Backend: UserService (780 lines) - CRUD with tenant isolation - Backend: UserController + UserRoutes (648 lines) - 13 API endpoints - Backend: Batch import users from Excel - Frontend: UserListPage (412 lines) - list/filter/search/pagination - Frontend: UserFormPage (341 lines) - create/edit with module config - Frontend: UserDetailPage (393 lines) - details/tenant/module management - Frontend: 3 modal components (592 lines) - import/assign/configure - API: GET/POST/PUT/DELETE /api/admin/users/* endpoints Architecture Upgrade - Module Permission System: - Backend: Add getUserModules() method in auth.service - Backend: Login API returns modules array in user object - Frontend: AuthContext adds hasModule() method - Frontend: Navigation filters modules based on user.modules - Frontend: RouteGuard checks requiredModule instead of requiredVersion - Frontend: Remove deprecated version-based permission system - UX: Only show accessible modules in navigation (clean UI) - UX: Smart redirect after login (avoid 403 for regular users) Fixes: - Fix UTF-8 encoding corruption in ~100 docs files - Fix pageSize type conversion in userService (String to Number) - Fix authUser undefined error in TopNavigation - Fix login redirect logic with role-based access check - Update Git commit guidelines v1.2 with UTF-8 safety rules Database Changes: - CREATE TABLE user_modules (user_id, tenant_id, module_code, is_enabled) - ADD UNIQUE CONSTRAINT (user_id, tenant_id, module_code) - INSERT 4 permissions + role assignments - UPDATE PUBLIC tenant with 8 module subscriptions Technical: - Backend: 5 new files (~2400 lines) - Frontend: 10 new files (~2500 lines) - Docs: 1 development record + 2 status updates + 1 guideline update - Total: ~4900 lines of code Status: User management 100% complete, module permission system operational
This commit is contained in:
@@ -1,18 +1,18 @@
|
||||
# Redis缓存 vs Redis队列 - 详细说明
|
||||
|
||||
> **<EFBFBD><EFBFBD>﹝<EFBFBD><EFBFBD>𧋦嚗?* V1.0
|
||||
> **<EFBFBD>𥕦遣<EFBFBD>交<EFBFBD>嚗?* 2025-12-12
|
||||
> **<EFBFBD>格<EFBFBD>霂餉<EFBFBD><EFBFBD><EFBFBD>** 鈭批<E988AD>蝏讐<E89D8F><E8AE90><EFBFBD><EFBFBD><EFBFBD>睲犖<E79DB2>?
|
||||
> **<EFBFBD>桃<EFBFBD>嚗?* 瞉<><E79E89>Redis蝻枏<E89DBB><E69E8F>朙edis<69>笔<EFBFBD><E7AC94><EFBFBD>躹<EFBFBD>?
|
||||
> **文档版本:** V1.0
|
||||
> **创建日期:** 2025-12-12
|
||||
> **目标读者:** 产品经理、开发人员
|
||||
> **目的:** 澄清Redis缓存和Redis队列的区别
|
||||
|
||||
---
|
||||
|
||||
## 📋 目录
|
||||
|
||||
1. [核心概念](#1-核心概念)
|
||||
2. [敶枏<EFBFBD>蝟餌<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>嗆<EFBFBD><EFBFBD>(#2-敶枏<E695B6>蝟餌<E89D9F><E9A48C>笔<EFBFBD><E7AC94>嗆<EFBFBD>?
|
||||
2. [当前系统真实状态](#2-当前系统真实状态)
|
||||
3. [Redis缓存 vs Redis队列对比](#3-redis缓存-vs-redis队列对比)
|
||||
4. [<EFBFBD>芰鍂Redis蝻枏<EFBFBD><EFBFBD><EFBFBD>蔣<EFBFBD>䓞(#4-<2D>芰鍂redis蝻枏<E89DBB><E69E8F><EFBFBD>蔣<EFBFBD>?
|
||||
4. [只用Redis缓存的影响](#4-只用redis缓存的影响)
|
||||
5. [启用Redis队列的收益与风险](#5-启用redis队列的收益与风险)
|
||||
6. [推荐方案](#6-推荐方案)
|
||||
|
||||
@@ -20,60 +20,60 @@
|
||||
|
||||
## 1. 核心概念
|
||||
|
||||
### 1.1 隞<EFBFBD>銋<EFBFBD>糓Redis嚗?
|
||||
### 1.1 什么是Redis?
|
||||
|
||||
**Redis = 一个高性能的内存数据库**
|
||||
|
||||
可以理解为:
|
||||
- <EFBFBD>蘨 **頞<>之<EFBFBD><E4B98B><EFBFBD>鈭怠<E988AD>摮睃<E691AE><E79D83>函征<E587BD>?*
|
||||
- <EFBFBD><EFBFBD> **霂餃<E99C82><E9A483>笔漲<E7AC94><E6BCB2>翰**嚗?ms<6D><73><EFBFBD>
|
||||
- <EFBFBD>𠒣 **<EFBFBD>舀<EFBFBD><EFBFBD>唳旿<EFBFBD><EFBFBD><EFBFBD><EFBFBD>?*
|
||||
- <EFBFBD><EFBFBD> **憭帋葵<E5B88B>滚𦛚<E6BB9A>典虾隞亙<E99A9E>鈭?*
|
||||
- 🏪 **超大的共享内存存储空间**
|
||||
- 🚀 **读写速度极快**(1ms内)
|
||||
- 💾 **支持数据持久化**
|
||||
- 🌐 **多个服务器可以共享**
|
||||
|
||||
Redis<EFBFBD>祈澈<EFBFBD>臭<EFBFBD>銝?*撌亙<E6928C>蝞?*嚗屸<E59A97><E5B1B8>X<EFBFBD>敺<EFBFBD><E695BA>撌亙<E6928C>嚗<EFBFBD>㺭<EFBFBD>桃<EFBFBD><E6A183><EFBFBD><EFBFBD>嚗?
|
||||
- String嚗<EFBFBD><EFBFBD>蝚虫葡嚗争<EFBFBD>婙<EFBFBD>?<3F>其<EFBFBD>蝻枏<E89DBB>
|
||||
- List嚗<EFBFBD><EFBFBD>銵剁<EFBFBD><EFBFBD>婙<EFBFBD>?<3F>其<EFBFBD><E585B6>笔<EFBFBD>
|
||||
Redis本身是一个**工具箱**,里面有很多工具(数据结构):
|
||||
- String(字符串)—— 用于缓存
|
||||
- List(列表)—— 用于队列
|
||||
- Set(集合)
|
||||
- Hash嚗<EFBFBD><EFBFBD>撣諹”嚗?
|
||||
- Hash(哈希表)
|
||||
- Sorted Set(有序集合)
|
||||
|
||||
### 1.2 Redis缓存 vs Redis队列
|
||||
|
||||
餈嗘舅銝芣<EFBFBD>敹菜糓**雿輻鍂Redis<69><73><EFBFBD><EFBFBD>峕䲮撘?*嚗䔶<E59A97><E494B6>臭舅銝芯<E98A9D><E88AAF>𣬚<EFBFBD>鈭批<E988AD>嚗?
|
||||
这两个概念是**使用Redis的不同方式**,不是两个不同的产品!
|
||||
|
||||
```
|
||||
阿里云Redis实例(您购买的)
|
||||
<EFBFBD>?
|
||||
<EFBFBD>臭誑<EFBFBD>峕𧒄<EFBFBD>其<EFBFBD>嚗?
|
||||
<EFBFBD>鎿<EFBFBD> Redis蝻枏<EFBFBD>嚗<EFBFBD>鍂String<EFBFBD>唳旿蝏𤘪<EFBFBD>嚗?
|
||||
↓
|
||||
可以同时用于:
|
||||
├─ Redis缓存(用String数据结构)
|
||||
└─ Redis队列(用List数据结构 + BullMQ库)
|
||||
```
|
||||
|
||||
**蝐餅<EFBFBD><EFBFBD><EFBFBD>圾**嚗?
|
||||
**类比理解**:
|
||||
```
|
||||
Redis = 銝<EFBFBD><EFBFBD>见之璆潘<EFBFBD><EFBFBD>刻揚銋啁<EFBFBD>嚗?
|
||||
Redis = 一栋大楼(您购买的)
|
||||
|
||||
Redis缓存 = 大楼里的"快递柜"
|
||||
- 存东西、取东西
|
||||
- <EFBFBD>㕑<EFBFBD><EFBFBD><EFBFBD>𧒄<EFBFBD>?
|
||||
- <EFBFBD>券<EFBFBD>䈑<EFBFBD>敹恍<EFBFBD><EFBFBD>䰻霂?
|
||||
- 有过期时间
|
||||
- 用途:快速查询
|
||||
|
||||
Redis队列 = 大楼里的"传送带"
|
||||
- 任务排队
|
||||
- <EFBFBD>厰◇摨誩<EFBFBD><EFBFBD>?
|
||||
- 按顺序处理
|
||||
- 用途:异步任务
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 敶枏<EFBFBD>蝟餌<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>嗆<EFBFBD>?
|
||||
## 2. 当前系统真实状态
|
||||
|
||||
### 2.1 隞<EFBFBD><EFBFBD>璉<EFBFBD><EFBFBD>亦<EFBFBD><EFBFBD>?<3F>?
|
||||
### 2.1 代码检查结果 ✅
|
||||
|
||||
#### **<EFBFBD>𤑳緵1嚗鋳ullMQ撌脣<EFBFBD>鋆<EFBFBD><EFBFBD><EFBFBD>芯蝙<EFBFBD>?*
|
||||
#### **发现1:BullMQ已安装但未使用**
|
||||
```json
|
||||
// backend/package.json (蝚?6銵?
|
||||
"bullmq": "^5.65.0", // <EFBFBD>?撌脣<E6928C>鋆<EFBFBD><E98B86>隞<EFBFBD><E99A9E>銝剜𧊋雿輻鍂
|
||||
// backend/package.json (第36行)
|
||||
"bullmq": "^5.65.0", // ← 已安装但代码中未使用
|
||||
```
|
||||
|
||||
#### **发现2:当前使用自研的MemoryQueue**
|
||||
@@ -81,23 +81,23 @@ Redis队列 = 大楼里的"传送带"
|
||||
// backend/src/common/jobs/JobFactory.ts
|
||||
export class JobFactory {
|
||||
private static createQueue(): JobQueue {
|
||||
const queueType = process.env.QUEUE_TYPE || 'memory'; // <EFBFBD>?暺䁅恕memory
|
||||
const queueType = process.env.QUEUE_TYPE || 'memory'; // ← 默认memory
|
||||
|
||||
switch (queueType) {
|
||||
case 'memory':
|
||||
return this.createMemoryQueue(); // <EFBFBD>?敶枏<E695B6>雿輻鍂餈嗘葵
|
||||
return this.createMemoryQueue(); // ← 当前使用这个
|
||||
|
||||
case 'database':
|
||||
// TODO: 摰䂿緵DatabaseQueue // <EFBFBD>?餈䀹瓷摰䂿緵
|
||||
// TODO: 实现DatabaseQueue // ← 还没实现
|
||||
return this.createMemoryQueue();
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### **<EFBFBD>𤑳緵3嚗餷obQueue<EFBFBD>?銝芣<E98A9D>隞嗡葉鋡怠<E98BA1><E680A0>剁<EFBFBD><E58981>?0甈?*
|
||||
#### **发现3:jobQueue在8个文件中被引用,共50次**
|
||||
```typescript
|
||||
// 銝餉<EFBFBD>雿輻鍂雿滨蔭嚗?
|
||||
// 主要使用位置:
|
||||
1. ExtractionController.ts (TODO注释)
|
||||
2. DualModelExtractionService.ts (TODO注释)
|
||||
3. test-platform-api.ts (测试代码)
|
||||
@@ -106,17 +106,17 @@ export class JobFactory {
|
||||
|
||||
### 2.2 真实情况总结
|
||||
|
||||
| 蝏<EFBFBD>辣 | 摰㕑<E691B0><E39591>嗆<EFBFBD>?| 雿輻鍂<E8BCBB>嗆<EFBFBD>?| 霂湔<E99C82> |
|
||||
| 组件 | 安装状态 | 使用状态 | 说明 |
|
||||
|------|---------|---------|------|
|
||||
| **Redis蝻枏<EFBFBD>** | <EFBFBD>?<3F>芷<EFBFBD>蝵?| <20>?<3F>芯蝙<E88AAF>?| <20>桀<EFBFBD><E6A180>函<EFBFBD><E587BD>烘emoryCache |
|
||||
| **Redis<EFBFBD>笔<EFBFBD>** | <EFBFBD>?<3F>芷<EFBFBD>蝵?| <20>?<3F>芯蝙<E88AAF>?| BullMQ撌脣<E6928C>鋆<EFBFBD><E98B86><EFBFBD>芸鍳<E88AB8>?|
|
||||
| **<EFBFBD><EFBFBD><EFBFBD>蝻枏<EFBFBD>** | <EFBFBD>?撌脣<E6928C><E884A3>?| <20>?雿輻鍂銝?| HealthCheck<EFBFBD><EFBFBD>LM蝏𤘪<EFBFBD> |
|
||||
| **<EFBFBD><EFBFBD><EFBFBD><EFBFBD>笔<EFBFBD>** | <EFBFBD>?撌脣<E6928C><E884A3>?| <20>?雿輻鍂銝?| MemoryQueue |
|
||||
| **Redis缓存** | ❌ 未配置 | ❌ 未使用 | 目前用的是MemoryCache |
|
||||
| **Redis队列** | ❌ 未配置 | ❌ 未使用 | BullMQ已安装但未启用 |
|
||||
| **内存缓存** | ✅ 已实现 | ✅ 使用中 | HealthCheck、LLM结果 |
|
||||
| **内存队列** | ✅ 已实现 | ✅ 使用中 | MemoryQueue |
|
||||
|
||||
**蝏栞捏**嚗?
|
||||
- <EFBFBD>?<3F>函<EFBFBD>蝟餌<E89D9F>撌脩<E6928C>憸<EFBFBD><E686B8>鈭<EFBFBD><E988AD><EFBFBD>埈沲<E59F88><E6B2B2><EFBFBD>MemoryQueue嚗?
|
||||
- <EFBFBD>?撌脩<E6928C>摰㕑<E691B0>鈭<EFBFBD>ullMQ靘肽<EFBFBD>
|
||||
- <EFBFBD>?雿<><E99BBF><EFBFBD><EFBFBD><EFBFBD>瘝⊥<E7989D><E28AA5><EFBFBD>迤雿輻鍂Redis嚗<73><E59A97>霈箸糓蝻枏<E89DBB>餈䀹糓<E480B9>笔<EFBFBD>嚗?
|
||||
**结论**:
|
||||
- ✅ 您的系统已经预留了队列架构(MemoryQueue)
|
||||
- ✅ 已经安装了BullMQ依赖
|
||||
- ❌ 但实际还没有真正使用Redis(无论是缓存还是队列)
|
||||
|
||||
---
|
||||
|
||||
@@ -126,43 +126,43 @@ export class JobFactory {
|
||||
|
||||
| 维度 | Redis缓存 | Redis队列 |
|
||||
|------|----------|----------|
|
||||
| **<EFBFBD>券<EFBFBD>?* | 摮睃<E691AE>銝湔𧒄<E6B994>唳旿 | 撘<>郊隞餃𦛚蝞∠<E89D9E> |
|
||||
| **<EFBFBD>唳旿蝏𤘪<EFBFBD>** | String嚗<EFBFBD><EFBFBD>蝚虫葡嚗?| List嚗<74><E59A97>銵剁<E98AB5> |
|
||||
| **用途** | 存储临时数据 | 异步任务管理 |
|
||||
| **数据结构** | String(字符串) | List(列表) |
|
||||
| **操作** | GET / SET / DELETE | PUSH / POP / ACK |
|
||||
| **餈<EFBFBD><EFBFBD><EFBFBD>箏<EFBFBD>** | TTL<EFBFBD>芸𢆡餈<EFBFBD><EFBFBD> | 隞餃𦛚摰峕<E691B0><E5B395>𤾸<EFBFBD><F0A4BEB8>?|
|
||||
| **霂餃<EFBFBD>璅∪<EFBFBD>** | <EFBFBD>𤩺㦤霂餃<EFBFBD> | 憿箏<E686BF>憭<EFBFBD><E686AD>嚗㇅IFO嚗?|
|
||||
| **<EFBFBD>詨<EFBFBD><EFBFBD>箸艶** | LLM蝏𤘪<EFBFBD>蝻枏<EFBFBD><EFBFBD><EFBFBD>ession | <EFBFBD><EFBFBD>讃蝑偦<EFBFBD>劐遙<EFBFBD>~<EFBFBD><EFBFBD>鸌<EFBFBD>誩<EFBFBD><EFBFBD>?|
|
||||
| **摨?撌亙<E6928C>** | <EFBFBD>湔𦻖<EFBFBD>甬oredis | BullMQ嚗<EFBFBD>抅鈭箸oredis嚗?|
|
||||
| **过期机制** | TTL自动过期 | 任务完成后删除 |
|
||||
| **读写模式** | 随机读写 | 顺序处理(FIFO) |
|
||||
| **典型场景** | LLM结果缓存、Session | 文献筛选任务、批量处理 |
|
||||
| **库/工具** | 直接用ioredis | BullMQ(基于ioredis) |
|
||||
|
||||
### 3.2 在您系统中的实际应用
|
||||
|
||||
#### **Redis蝻枏<EFBFBD><EFBFBD><EFBFBD>鍂<EFBFBD>?*
|
||||
#### **Redis缓存的用途**
|
||||
|
||||
```typescript
|
||||
// 场景1:LLM结果缓存
|
||||
const cacheKey = `fulltext:${literatureId}:${model}`;
|
||||
await cache.set(cacheKey, llmResult, 3600); // 缓存1小时
|
||||
|
||||
// 銝𧢲活<EFBFBD>詨<EFBFBD>PDF<EFBFBD>齿<EFBFBD><EFBFBD>𡝗𧒄嚗?
|
||||
// 下次相同PDF再提取时:
|
||||
const cached = await cache.get(cacheKey);
|
||||
if (cached) {
|
||||
return cached; // <EFBFBD>?<3F>湔𦻖餈𥪜<E9A488>嚗諹<E59A97><E8ABB9><EFBFBD>PI韐寧鍂
|
||||
return cached; // ✅ 直接返回,节省API费用
|
||||
}
|
||||
```
|
||||
|
||||
**憟賢<EFBFBD>**嚗?
|
||||
- <EFBFBD>?<3F>踹<EFBFBD><E8B8B9>滚<EFBFBD>靚<EFBFBD>鍂DeepSeek API嚗<EFBFBD>?.43/蝭<EFBFBD><EFBFBD>
|
||||
- <EFBFBD>?<3F>滚<EFBFBD><E6BB9A>笔漲敹恬<E695B9>隞?<3F><><EFBFBD><EFBFBD>滚<EFBFBD>1ms嚗?
|
||||
- <EFBFBD>?憭𡁜<E686AD>靘见<E99D98>鈭怎<E988AD>摮?
|
||||
**好处**:
|
||||
- ✅ 避免重复调用DeepSeek API(¥0.43/篇)
|
||||
- ✅ 响应速度快(从1分钟降到1ms)
|
||||
- ✅ 多实例共享缓存
|
||||
|
||||
---
|
||||
|
||||
#### **Redis<EFBFBD>笔<EFBFBD><EFBFBD><EFBFBD>鍂<EFBFBD>?*
|
||||
#### **Redis队列的用途**
|
||||
|
||||
```typescript
|
||||
// <EFBFBD>箸艶2嚗𡁏<EFBFBD><EFBFBD>桃<EFBFBD><EFBFBD>劐遙<EFBFBD>∴<EFBFBD>199蝭<EFBFBD><EFBFBD><EFBFBD><EFBFBD>閬?0-60<36><30><EFBFBD>嚗?
|
||||
// 场景2:文献筛选任务(199篇,需要30-60分钟)
|
||||
export async function startScreening(projectId) {
|
||||
// 1. <EFBFBD>𥕦遣隞餃𦛚嚗<EFBFBD><EFBFBD><EFBFBD>餃<EFBFBD>霂瑟<EFBFBD>嚗?
|
||||
// 1. 创建任务(不阻塞请求)
|
||||
const task = await prisma.aslScreeningTask.create({...});
|
||||
|
||||
// 2. 推送到队列(异步处理)
|
||||
@@ -185,15 +185,15 @@ jobQueue.process('asl:screening', async (job) => {
|
||||
});
|
||||
```
|
||||
|
||||
**憟賢<EFBFBD>**嚗?
|
||||
- <EFBFBD>?隞餃𦛚<E9A483><F0A69B9A><EFBFBD><EFBFBD>吔<EFBFBD>摰硺<E691B0><E7A1BA>滚鍳銝滢腺憭梧<E686AD>
|
||||
- <EFBFBD>?<3F>舀<EFBFBD>隞餃𦛚隡睃<E99AA1>蝥?
|
||||
- <EFBFBD>?<3F>舀<EFBFBD>隞餃𦛚<E9A483>滩<EFBFBD>
|
||||
- <EFBFBD>?憭𡁜<E686AD>靘钅𡢿隞餃𦛚<E9A483><F0A69B9A><EFBFBD>
|
||||
**好处**:
|
||||
- ✅ 任务持久化(实例重启不丢失)
|
||||
- ✅ 支持任务优先级
|
||||
- ✅ 支持任务重试
|
||||
- ✅ 多实例间任务分配
|
||||
|
||||
---
|
||||
|
||||
### 3.3 <EFBFBD><EFBFBD><EFBFBD>臬<EFBFBD><EFBFBD>啣笆瘥?
|
||||
### 3.3 技术实现对比
|
||||
|
||||
```typescript
|
||||
// ==================== Redis缓存 ====================
|
||||
@@ -234,84 +234,84 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
});
|
||||
```
|
||||
|
||||
**<EFBFBD>喲睸<EFBFBD>?*嚗?
|
||||
- <EFBFBD>?銝方<E98A9D><E696B9><EFBFBD>餈墧𦻖<E5A2A7>啣<EFBFBD>銝<EFBFBD>銝析edis摰硺<EFBFBD>
|
||||
- <EFBFBD>?<3F>臭誑<E887AD>峕𧒄雿輻鍂嚗<E98D82><E59A97>銝滚僕<E6BB9A>堆<EFBFBD>
|
||||
- <EFBFBD>?BullMQ<EFBFBD><EFBFBD><EFBFBD>銋<EFBFBD>糓<EFBFBD>甬oredis摰䂿緵<EFBFBD>?
|
||||
**关键点**:
|
||||
- ✅ 两者都连接到同一个Redis实例
|
||||
- ✅ 可以同时使用(互不干扰)
|
||||
- ✅ BullMQ内部也是用ioredis实现的
|
||||
|
||||
---
|
||||
|
||||
## 4. <EFBFBD>芰鍂Redis蝻枏<EFBFBD><EFBFBD><EFBFBD>蔣<EFBFBD>?
|
||||
## 4. 只用Redis缓存的影响
|
||||
|
||||
### 4.1 <EFBFBD>臭誑皛∟雲<EFBFBD><EFBFBD><EFBFBD>瘙?<3F>?
|
||||
### 4.1 可以满足的需求 ✅
|
||||
|
||||
1. **LLM蝏𤘪<EFBFBD>蝻枏<EFBFBD>** <EFBFBD>?
|
||||
1. **LLM结果缓存** ✅
|
||||
- 避免重复API调用
|
||||
- 节省成本
|
||||
- 提升响应速度
|
||||
|
||||
2. **Session蝞∠<EFBFBD>** <EFBFBD>?
|
||||
2. **Session管理** ✅
|
||||
- 多实例共享Session
|
||||
- <EFBFBD>冽<EFBFBD><EFBFBD>嗆<EFBFBD><EFBFBD><EFBFBD>甇?
|
||||
- 用户状态同步
|
||||
|
||||
3. **<EFBFBD>亙熒璉<EFBFBD><EFBFBD>亦<EFBFBD>摮?* <20>?
|
||||
3. **健康检查缓存** ✅
|
||||
- 避免重复解析Excel
|
||||
|
||||
4. **<EFBFBD>剜<EFBFBD>隞餃𦛚** <20>?
|
||||
4. **短期任务** ✅
|
||||
- < 10秒的任务
|
||||
- <EFBFBD>湔𦻖<EFBFBD>沈TTP霂瑟<EFBFBD>銝剖<EFBFBD><EFBFBD>?
|
||||
- 直接在HTTP请求中处理
|
||||
|
||||
### 4.2 <EFBFBD>䭾<EFBFBD>皛∟雲<EFBFBD><EFBFBD><EFBFBD>瘙?<3F>?
|
||||
### 4.2 无法满足的需求 ❌
|
||||
|
||||
1. **<EFBFBD>踵𧒄<EFBFBD>港遙<EFBFBD>⊥<EFBFBD>銋<EFBFBD><EFBFBD>** <20>?
|
||||
1. **长时间任务持久化** ❌
|
||||
```
|
||||
<EFBFBD>箸艶嚗𡁶鍂<EFBFBD>瑟<EFBFBD>鈭?99蝭<39><E89DAD><EFBFBD>桃<EFBFBD><E6A183>㚁<EFBFBD><E39A81><EFBFBD>閬?0-60<36><30><EFBFBD>嚗?
|
||||
场景:用户提交199篇文献筛选(需要30-60分钟)
|
||||
|
||||
憒<EFBFBD><EFBFBD><EFBFBD>芰鍂Redis蝻枏<EFBFBD>嚗?
|
||||
- 隞餃𦛚<EFBFBD>嗆<EFBFBD><EFBFBD><EFBFBD><EFBFBD>典<EFBFBD>摮?<3F>?摰硺<E691B0><E7A1BA>滚鍳銝W仃
|
||||
- <EFBFBD>䭾<EFBFBD><EFBFBD>W<EFBFBD>銝剜鱏<EFBFBD><EFBFBD>遙<EFBFBD>?
|
||||
如果只用Redis缓存:
|
||||
- 任务状态存在内存 → 实例重启丢失
|
||||
- 无法恢复中断的任务
|
||||
- 用户看到任务消失
|
||||
```
|
||||
|
||||
2. **隞餃𦛚<EFBFBD>笔<EFBFBD>蝞∠<EFBFBD>** <EFBFBD>?
|
||||
2. **任务队列管理** ❌
|
||||
```
|
||||
<EFBFBD>箸艶嚗?0銝芰鍂<E88AB0>瑕<EFBFBD><E79195>嗆<EFBFBD>鈭斗鸌<E69697>譍遙<E8AD8D>?
|
||||
场景:10个用户同时提交批量任务
|
||||
|
||||
憒<EFBFBD><EFBFBD><EFBFBD>芰鍂Redis蝻枏<EFBFBD>嚗?
|
||||
- <EFBFBD>䭾<EFBFBD><EFBFBD>㘾<EFBFBD>嚗<EFBFBD><EFBFBD><EFBFBD>亙<EFBFBD><EFBFBD>滚𦛚嚗?
|
||||
如果只用Redis缓存:
|
||||
- 无法排队(先来先服务)
|
||||
- 无法限制并发
|
||||
- 服务器可能被打爆
|
||||
```
|
||||
|
||||
3. **隞餃𦛚<EFBFBD>滩<EFBFBD>** <EFBFBD>?
|
||||
3. **任务重试** ❌
|
||||
```
|
||||
<EFBFBD>箸艶嚗𡁏<EFBFBD>蝭<EFBFBD><EFBFBD><EFBFBD>桀<EFBFBD><EFBFBD><EFBFBD>仃韐?
|
||||
场景:某篇文献处理失败
|
||||
|
||||
憒<EFBFBD><EFBFBD><EFBFBD>芰鍂Redis蝻枏<EFBFBD>嚗?
|
||||
如果只用Redis缓存:
|
||||
- 无法自动重试
|
||||
- <EFBFBD><EFBFBD>閬<EFBFBD>鍂<EFBFBD>琿<EFBFBD><EFBFBD>唳<EFBFBD>鈭?
|
||||
- 需要用户重新提交
|
||||
```
|
||||
|
||||
4. **<EFBFBD><EFBFBD><EFBFBD>撘譍遙<EFBFBD>∪<EFBFBD><EFBFBD>?* <20>?
|
||||
4. **分布式任务分配** ❌
|
||||
```
|
||||
<EFBFBD>箸艶嚗锭AE<EFBFBD>?銝芸<E98A9D>靘?
|
||||
场景:SAE有3个实例
|
||||
|
||||
憒<EFBFBD><EFBFBD><EFBFBD>芰鍂Redis蝻枏<EFBFBD>嚗?
|
||||
如果只用Redis缓存:
|
||||
- 无法协调哪个实例处理哪个任务
|
||||
- 可能重复处理
|
||||
```
|
||||
|
||||
### 4.3 实际影响评估
|
||||
|
||||
#### **憒<EFBFBD><EFBFBD><EFBFBD>芸鍳<EFBFBD>沖edis蝻枏<EFBFBD>嚗?*
|
||||
#### **如果只启用Redis缓存:**
|
||||
|
||||
| 影响维度 | 评分 | 说明 |
|
||||
|---------|------|------|
|
||||
| **LLM<EFBFBD>鞉𧋦<EFBFBD>批<EFBFBD>** | <20>?隡条<E99AA1> | 蝻枏<E89DBB><E69E8F>賭葉嚗諹<E59A97><E8ABB9>?0-50%韐寧鍂 |
|
||||
| **憭𡁜<EFBFBD>靘𧢲𣈲<EFBFBD>?* | <20>?隡条<E99AA1> | 蝻枏<E89DBB><E69E8F>曹澈 |
|
||||
| **<EFBFBD>蹂遙<EFBFBD>∪虾<EFBFBD>䭾<EFBFBD>?* | <20>𩤃<EFBFBD> 銝<><E98A9D>?| 隞滨<E99A9E>靘肽<E99D98>MemoryQueue |
|
||||
| **隞餃𦛚蝞∠<EFBFBD><EFBFBD>賢<EFBFBD>** | <20>𩤃<EFBFBD> 銝<><E98A9D>?| <20>牐<EFBFBD><E78990><EFBFBD>漣<EFBFBD><E6BCA3><EFBFBD>霂?|
|
||||
| **蝟餌<EFBFBD><EFBFBD>舀<EFBFBD>撅閙<EFBFBD>?* | <20>椬 銝剔<E98A9D> | <20>訫<EFBFBD>靘见虾隞伐<E99A9E>憭𡁜<E686AD>靘𧢲<E99D98><F0A7A2B2>桅<EFBFBD> |
|
||||
| **LLM成本控制** | ✅ 优秀 | 缓存命中,节省30-50%费用 |
|
||||
| **多实例支持** | ✅ 优秀 | 缓存共享 |
|
||||
| **长任务可靠性** | ⚠️ 一般 | 仍然依赖MemoryQueue |
|
||||
| **任务管理能力** | ⚠️ 一般 | 无优先级、重试 |
|
||||
| **系统可扩展性** | 🟡 中等 | 单实例可以,多实例有问题 |
|
||||
|
||||
---
|
||||
|
||||
@@ -323,24 +323,24 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
```
|
||||
当前(MemoryQueue):
|
||||
- 用户提交批量任务
|
||||
- 摰硺<EFBFBD><EFBFBD>滚鍳 <20>?<3F>?隞餃𦛚銝W仃
|
||||
- 实例重启 → ❌ 任务丢失
|
||||
- 用户投诉率:5-10%
|
||||
|
||||
<EFBFBD>寧鍂Redis<EFBFBD>笔<EFBFBD>嚗?
|
||||
改用Redis队列:
|
||||
- 任务保存在Redis
|
||||
- 摰硺<EFBFBD><EFBFBD>滚鍳 <20>?<3F>?隞餃𦛚蝏抒賒
|
||||
- 实例重启 → ✅ 任务继续
|
||||
- 用户投诉率:< 1%
|
||||
```
|
||||
|
||||
#### **<EFBFBD>嗥<EFBFBD>2嚗帋遙<EFBFBD>⊿<EFBFBD>霂?*
|
||||
#### **收益2:任务重试**
|
||||
```
|
||||
当前(MemoryQueue):
|
||||
- LLM靚<EFBFBD>鍂憭梯揖 <20>?<3F>?隞餃𦛚<E9A483><F0A69B9A>扇銝箏仃韐?
|
||||
- <EFBFBD><EFBFBD>閬<EFBFBD>鍂<EFBFBD>琿<EFBFBD><EFBFBD>唳<EFBFBD>鈭?
|
||||
- LLM调用失败 → ❌ 任务标记为失败
|
||||
- 需要用户重新提交
|
||||
|
||||
<EFBFBD>寧鍂Redis<EFBFBD>笔<EFBFBD>嚗?
|
||||
- LLM靚<EFBFBD>鍂憭梯揖 <20>?<3F>?<3F>芸𢆡<E88AB8>滩<EFBFBD>3甈?
|
||||
- <EFBFBD><EFBFBD>㺭<EFBFBD><EFBFBD><EFBFBD>選<EFBFBD>2蝘鉝<EFBFBD>?蝘鉝<E89D98>?蝘𡜐<E89D98>
|
||||
改用Redis队列:
|
||||
- LLM调用失败 → ✅ 自动重试3次
|
||||
- 指数退避(2秒、4秒、8秒)
|
||||
```
|
||||
|
||||
#### **收益3:分布式任务分配**
|
||||
@@ -349,19 +349,19 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
- 每个实例独立处理任务
|
||||
- 无法协调
|
||||
|
||||
<EFBFBD>寧鍂Redis<EFBFBD>笔<EFBFBD>嚗?
|
||||
改用Redis队列:
|
||||
- 多个Worker抢占任务
|
||||
- 自动负载均衡
|
||||
- 某个Worker挂了,其他Worker接管
|
||||
```
|
||||
|
||||
#### **<EFBFBD>嗥<EFBFBD>4嚗帋遙<EFBFBD>∠<EFBFBD><EFBFBD>?*
|
||||
#### **收益4:任务监控**
|
||||
```
|
||||
当前(MemoryQueue):
|
||||
- 隞餃𦛚<EFBFBD>嗆<EFBFBD><EFBFBD>銁<EFBFBD><EFBFBD><EFBFBD>銝?
|
||||
- 任务状态在内存中
|
||||
- 无法查看历史任务
|
||||
|
||||
<EFBFBD>寧鍂Redis<EFBFBD>笔<EFBFBD>嚗?
|
||||
改用Redis队列:
|
||||
- 任务历史保存
|
||||
- 可以查看失败原因
|
||||
- 统计处理时长
|
||||
@@ -371,76 +371,76 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
|
||||
#### **风险1:增加复杂度** 🟡 中等
|
||||
```
|
||||
敶枏<EFBFBD>嚗?
|
||||
当前:
|
||||
- MemoryQueue:简单、易理解
|
||||
- 隞<EFBFBD><EFBFBD><EFBFBD>𧶏<EFBFBD>~200銵?
|
||||
- 代码量:~200行
|
||||
|
||||
<EFBFBD>嫣蛹BullMQ嚗?
|
||||
改为BullMQ:
|
||||
- 需要学习BullMQ API
|
||||
- 隞<EFBFBD><EFBFBD><EFBFBD>𧶏<EFBFBD>~500銵?
|
||||
- 代码量:~500行
|
||||
- 需要配置Worker
|
||||
```
|
||||
|
||||
**蝻栞圾**嚗?
|
||||
- <EFBFBD>?BullMQ<EFBFBD><EFBFBD>﹝摰<EFBFBD><EFBFBD>
|
||||
- <EFBFBD>?蝷曉躹瘣餉<E798A3>嚗𠃑itHub 15k+ stars嚗?
|
||||
- <EFBFBD>?<3F>睲賑撌脩<E6928C>摰㕑<E691B0>鈭<EFBFBD><E988AD>韏?
|
||||
**缓解**:
|
||||
- ✅ BullMQ文档完善
|
||||
- ✅ 社区活跃(GitHub 15k+ stars)
|
||||
- ✅ 我们已经安装了依赖
|
||||
|
||||
---
|
||||
|
||||
#### **憌𡡞埯2嚗帋<EFBFBD>韏餸edis蝔喳<EFBFBD><EFBFBD>?* <20>椬 銝剔<E98A9D>
|
||||
#### **风险2:依赖Redis稳定性** 🟡 中等
|
||||
```
|
||||
场景:Redis挂了
|
||||
- Redis缓存:可以降级到内存
|
||||
- Redis队列:无法降级(队列必须持久化)
|
||||
```
|
||||
|
||||
**蝻栞圾**嚗?
|
||||
- <EFBFBD>?<3F>刻揚銋啁<E98A8B><E59581>舫<EFBFBD><E888AB>舐鍂<E88890>㇌edis嚗?9.95%嚗?
|
||||
- <EFBFBD>?銝颱<E98A9D><E9A2B1>芸𢆡<E88AB8><F0A286A1>揢
|
||||
- <EFBFBD>?摰鮋<E691B0>憌𡡞埯敺<E59FAF><E695BA>
|
||||
**缓解**:
|
||||
- ✅ 您购买的是高可用版Redis(99.95%)
|
||||
- ✅ 主从自动切换
|
||||
- ✅ 实际风险很低
|
||||
|
||||
---
|
||||
|
||||
#### **憌𡡞埯3嚗朞<EFBFBD>霂閖𠗕摨血<EFBFBD><EFBFBD>?* <20>叚 雿?
|
||||
#### **风险3:调试难度增加** 🟢 低
|
||||
```
|
||||
敶枏<EFBFBD>嚗?
|
||||
当前:
|
||||
- console.log() 即可
|
||||
- 任务在内存中
|
||||
|
||||
<EFBFBD>嫣蛹BullMQ嚗?
|
||||
改为BullMQ:
|
||||
- 需要查看Redis数据
|
||||
- <EFBFBD><EFBFBD>閬<EFBFBD>鍂BullBoard嚗<EFBFBD>虾閫<EFBFBD><EFBFBD>撌亙<EFBFBD>嚗?
|
||||
- 需要用BullBoard(可视化工具)
|
||||
```
|
||||
|
||||
**蝻栞圾**嚗?
|
||||
- <EFBFBD>?BullBoard<EFBFBD>𣂷<EFBFBD>Web UI
|
||||
- <EFBFBD>?<3F>臭誑<E887AD>见<EFBFBD>隞餃𦛚<E9A483>嗆<EFBFBD><E59786><EFBFBD><EFBFBD><EFBFBD>霂閙活<E99699>?
|
||||
- <EFBFBD>?<3F>亙<EFBFBD><E4BA99>渲祕蝏?
|
||||
**缓解**:
|
||||
- ✅ BullBoard提供Web UI
|
||||
- ✅ 可以看到任务状态、重试次数
|
||||
- ✅ 日志更详细
|
||||
|
||||
---
|
||||
|
||||
#### **憌𡡞埯4嚗𡁜<EFBFBD>摮睃<EFBFBD><EFBFBD>?* <20>叚 雿?
|
||||
#### **风险4:内存占用** 🟢 低
|
||||
```
|
||||
担心:Redis队列会占用很多内存?
|
||||
|
||||
摰鮋<EFBFBD>嚗?
|
||||
实际:
|
||||
- 单个任务数据:~1KB
|
||||
- 100个并发任务:100KB
|
||||
- 1000个任务历史:1MB
|
||||
- 撖?56MB Redis敶勗<EFBFBD>敺<EFBFBD><EFBFBD>
|
||||
- 对256MB Redis影响很小
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5.3 风险矩阵
|
||||
|
||||
| 憌𡡞埯 | 銝仿<E98A9D><E4BBBF>?| 璁<><E79281> | 蝻栞圾<E6A09E>曉漲 | 撱箄悅 |
|
||||
| 风险 | 严重性 | 概率 | 缓解难度 | 建议 |
|
||||
|------|--------|------|----------|------|
|
||||
| **憓𧼮<EFBFBD>憭齿<EFBFBD>摨?* | <20>椬 銝?| <20>𣞁 擃?| <20>?<3F>?| <20>𣂷<EFBFBD><F0A382B7>寡悌<E5AFA1><E6828C>﹝ |
|
||||
| **Redis靘肽<EFBFBD>** | <EFBFBD>𣞁 擃?| <20>叚 雿?| <20>?<3F>?| 擃睃虾<E79D83>函<EFBFBD> |
|
||||
| **靚<EFBFBD><EFBFBD><EFBFBD>圈𠗕** | <20>叚 雿?| <20>椬 銝?| <20>?<3F>?| 雿輻鍂BullBoard |
|
||||
| **<EFBFBD><EFBFBD><EFBFBD><EFBFBD>删鍂** | <20>叚 雿?| <20>叚 雿?| <20>?<3F>?| <20>烐綉<E78390>喳虾 |
|
||||
| **增加复杂度** | 🟡 中 | 🔴 高 | ✅ 易 | 提供培训文档 |
|
||||
| **Redis依赖** | 🔴 高 | 🟢 低 | ✅ 易 | 高可用版 |
|
||||
| **调试困难** | 🟢 低 | 🟡 中 | ✅ 易 | 使用BullBoard |
|
||||
| **内存占用** | 🟢 低 | 🟢 低 | ✅ 易 | 监控即可 |
|
||||
|
||||
**结论:风险可控,收益大于风险**
|
||||
|
||||
@@ -453,19 +453,19 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
```
|
||||
阶段1(本周): 启用Redis缓存
|
||||
├─ 目标:解决LLM成本问题
|
||||
<EFBFBD>鎿<EFBFBD> 撌乩<E6928C><E4B9A9>𧶏<EFBFBD>2憭?
|
||||
<EFBFBD>鎿<EFBFBD> 憌𡡞埯嚗帋<E59A97>嚗<EFBFBD><E59A97><EFBFBD>滨漣<E6BBA8>寞<EFBFBD>嚗?
|
||||
<EFBFBD>婙<EFBFBD> <20>嗥<EFBFBD>嚗朞<E59A97><E69C9E>?0-50% API韐寧鍂
|
||||
├─ 工作量:2天
|
||||
├─ 风险:低(有降级方案)
|
||||
└─ 收益:节省30-50% API费用
|
||||
|
||||
阶段2(下周): 启用Redis队列
|
||||
<EFBFBD>鎿<EFBFBD> <20>格<EFBFBD>嚗朞圾<E69C9E>喲鵭隞餃𦛚<E9A483>舫<EFBFBD><E888AB>?
|
||||
<EFBFBD>鎿<EFBFBD> 撌乩<E6928C><E4B9A9>𧶏<EFBFBD>3憭?
|
||||
<EFBFBD>鎿<EFBFBD> 憌𡡞埯嚗帋葉嚗<E89189><E59A97><EFBFBD>舀綉嚗?
|
||||
<EFBFBD>婙<EFBFBD> <20>嗥<EFBFBD>嚗帋遙<E5B88B>∩腺憭梁<E686AD><E6A281>滢<EFBFBD>10<31>?
|
||||
├─ 目标:解决长任务可靠性
|
||||
├─ 工作量:3天
|
||||
├─ 风险:中(但可控)
|
||||
└─ 收益:任务丢失率降低10倍
|
||||
|
||||
<EFBFBD>嗆挾3嚗<EFBFBD>𧊋<EFBFBD>伐<EFBFBD>: 隡睃<E99AA1>銝𡒊<E98A9D><F0A1928A>?
|
||||
<EFBFBD>鎿<EFBFBD> BullBoard<EFBFBD>航<EFBFBD><EFBFBD>?
|
||||
<EFBFBD>鎿<EFBFBD> 隞餃𦛚隡睃<E99AA1>蝥?
|
||||
阶段3(未来): 优化与监控
|
||||
├─ BullBoard可视化
|
||||
├─ 任务优先级
|
||||
├─ 性能监控
|
||||
└─ 告警机制
|
||||
```
|
||||
@@ -476,21 +476,21 @@ const worker = new Worker('asl:screening', async (job) => {
|
||||
|
||||
**Day 1-2:实现RedisCacheAdapter**
|
||||
```bash
|
||||
# <EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>獢<EFBFBD><EFBFBD>04-Redis<69>寥<EFBFBD>惩<EFBFBD><E683A9>質恣<E8B3AA>?md
|
||||
# 参考文档:04-Redis改造实施计划.md
|
||||
# Phase 1-3
|
||||
|
||||
<EFBFBD>?摰㕑<E691B0>ioredis
|
||||
<EFBFBD>?摰䂿緵RedisCacheAdapter
|
||||
<EFBFBD>?瘛餃<E7989B><E9A483>滨漣蝑𣇉裦
|
||||
<EFBFBD>?<3F>砍𧑐瘚贝<E7989A>
|
||||
✅ 安装ioredis
|
||||
✅ 实现RedisCacheAdapter
|
||||
✅ 添加降级策略
|
||||
✅ 本地测试
|
||||
```
|
||||
|
||||
**撉峕𤣰<EFBFBD><EFBFBD><EFBFBD>**嚗?
|
||||
**验收标准**:
|
||||
```bash
|
||||
<EFBFBD>?HealthCheckService蝻枏<EFBFBD><EFBFBD>賭葉
|
||||
<EFBFBD>?LLM12FieldsService蝻枏<EFBFBD><EFBFBD>賭葉
|
||||
<EFBFBD>?摰硺<E691B0><E7A1BA>滚鍳蝻枏<E89DBB>銝滢腺憭?
|
||||
<EFBFBD>?Redis<EFBFBD><EFBFBD><EFBFBD>蝟餌<EFBFBD>隞滚虾<EFBFBD>剁<EFBFBD><EFBFBD>滨漣嚗?
|
||||
✅ HealthCheckService缓存命中
|
||||
✅ LLM12FieldsService缓存命中
|
||||
✅ 实例重启缓存不丢失
|
||||
✅ Redis挂了系统仍可用(降级)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -507,13 +507,13 @@ export class RedisQueue implements JobQueue {
|
||||
}
|
||||
```
|
||||
|
||||
**Day 2嚗朞<EFBFBD>蝘颱<EFBFBD><EFBFBD>∩誨<EFBFBD>?*
|
||||
**Day 2:迁移业务代码**
|
||||
```typescript
|
||||
// 修改:screeningService.ts
|
||||
// 从:
|
||||
processLiteraturesInBackground(task.id, ...);
|
||||
|
||||
// <EFBFBD>嫣蛹嚗?
|
||||
// 改为:
|
||||
await jobQueue.push('asl:screening', {
|
||||
taskId: task.id,
|
||||
projectId,
|
||||
@@ -523,109 +523,109 @@ await jobQueue.push('asl:screening', {
|
||||
|
||||
**Day 3:测试与上线**
|
||||
```bash
|
||||
<EFBFBD>?<3F>訫<EFBFBD>瘚贝<E7989A>
|
||||
<EFBFBD>?<3F><><EFBFBD>瘚贝<E7989A>
|
||||
<EFBFBD>?<3F>见<EFBFBD>瘚贝<E7989A>
|
||||
<EFBFBD>?<3F><><EFBFBD>璅⊥<E79285>瘚贝<E7989A>
|
||||
<EFBFBD>?<3F>啣漲<E595A3>穃<EFBFBD>
|
||||
✅ 单元测试
|
||||
✅ 集成测试
|
||||
✅ 压力测试
|
||||
✅ 故障模拟测试
|
||||
✅ 灰度发布
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6.3 <EFBFBD><EFBFBD>撠𤩺㺿<EFBFBD>冽䲮獢<EFBFBD><EFBFBD>憒<EFBFBD><EFBFBD>韏<EFBFBD><EFBFBD><EFBFBD>厰<EFBFBD>嚗?
|
||||
### 6.3 最小改动方案(如果资源有限)
|
||||
|
||||
**憒<EFBFBD><EFBFBD><EFBFBD>芣<EFBFBD>1<EFBFBD>冽𧒄<EFBFBD>湛<EFBFBD>撱箄悅嚗?*
|
||||
**如果只有1周时间,建议:**
|
||||
|
||||
```
|
||||
隡睃<EFBFBD>蝥?嚗<><E59A97>憿鳴<E686BF>嚗鑹edis蝻枏<EFBFBD>
|
||||
优先级1(必须):Redis缓存
|
||||
- 解决LLM成本问题
|
||||
- 撌乩<EFBFBD><EFBFBD>𧶏<EFBFBD>2憭?
|
||||
- 工作量:2天
|
||||
|
||||
隡睃<EFBFBD>蝥?嚗<>虾<EFBFBD>㚁<EFBFBD>嚗鑹edis<EFBFBD>笔<EFBFBD>
|
||||
优先级2(可选):Redis队列
|
||||
- 暂时保持MemoryQueue
|
||||
- 记录技术债务
|
||||
- 蝑厩鍂<EFBFBD>瑁<EFBFBD>璅∪<EFBFBD><EFBFBD>踹<EFBFBD><EFBFBD>齿㺿<EFBFBD>?
|
||||
- 等用户规模增长后再改造
|
||||
```
|
||||
|
||||
**<EFBFBD>斗鱏<EFBFBD><EFBFBD><EFBFBD>**嚗?
|
||||
**判断标准**:
|
||||
```
|
||||
憒<EFBFBD><EFBFBD>皛∟雲隞乩<EFBFBD><EFBFBD>∩辣嚗<EFBFBD>虾隞交<EFBFBD>蝻𣫮edis<EFBFBD>笔<EFBFBD>嚗?
|
||||
<EFBFBD>?<3F>冽<EFBFBD><E586BD>?< 20
|
||||
<EFBFBD>?SAE<EFBFBD>芣<EFBFBD>1銝芸<EFBFBD>靘?
|
||||
<EFBFBD>?<3F>寥<EFBFBD>隞餃𦛚銝漤<E98A9D>蝜<EFBFBD><E89D9C>瘥誩予 < 5銝迎<EFBFBD>
|
||||
<EFBFBD>?<3F>臭誑<E887AD>亙<EFBFBD><E4BA99>嗅<EFBFBD>隞餃𦛚銝W仃
|
||||
如果满足以下条件,可以暂缓Redis队列:
|
||||
✅ 用户数 < 20
|
||||
✅ SAE只有1个实例
|
||||
✅ 批量任务不频繁(每天 < 5个)
|
||||
✅ 可以接受偶尔任务丢失
|
||||
|
||||
否则,建议尽快启用Redis队列
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6.4 銝箔<EFBFBD>銋㇂ullMQ撌脩<EFBFBD>摰㕑<EFBFBD>雿<EFBFBD>𧊋雿輻鍂嚗?
|
||||
### 6.4 为什么BullMQ已经安装但未使用?
|
||||
|
||||
**<EFBFBD>冽<EFBFBD><EFBFBD>笔<EFBFBD>**嚗?
|
||||
1. **霈∪<EFBFBD>雿輻鍂雿<EFBFBD>𧊋摰墧鴌**嚗𡁜<E59A97><F0A1819C>烐𧒄霈∪<E99C88><E288AA>步ullMQ嚗<51><E59A97>摰㕑<E691B0>鈭<EFBFBD><E988AD>韏?
|
||||
2. **瘚贝<EFBFBD>餈<EFBFBD><EFBFBD><EFBFBD>芸鍳<EFBFBD>?*嚗𡁜虾<F0A1819C>賢銁瘚贝<E7989A><E8B49D>臬<EFBFBD>撉諹<E69289>餈?
|
||||
3. **靘肽<EFBFBD>隡𣳇<EFBFBD>?*嚗𡁜<E59A97>隞硋<E99A9E>靘肽<E99D98>鈭<EFBFBD>ullMQ
|
||||
**推测原因**:
|
||||
1. **计划使用但未实施**:开发时计划用BullMQ,先安装了依赖
|
||||
2. **测试过但未启用**:可能在测试环境验证过
|
||||
3. **依赖传递**:其他包依赖了BullMQ
|
||||
|
||||
**撱箄悅**嚗?
|
||||
- <EFBFBD>?靽萘<E99DBD>BullMQ靘肽<EFBFBD>嚗<EFBFBD>歇蝏誩<EFBFBD>鋆<EFBFBD><EFBFBD>嚗?
|
||||
- <EFBFBD>?<3F>券𧫴畾?<3F>嗥凒<E597A5>乩蝙<E4B9A9>剁<EFBFBD><E58981>𣳇<EFBFBD><F0A3B387>齿鰵摰㕑<E691B0>嚗?
|
||||
- <EFBFBD>?憒<><E68692><EFBFBD><EFBFBD>𧒄銝滨鍂嚗䔶<E59A97>銝滨鍂<E6BBA8>𣳇膄嚗<E88684><E59A97>敶勗<E695B6><E58B97>扯<EFBFBD>嚗?
|
||||
**建议**:
|
||||
- ✅ 保留BullMQ依赖(已经安装了)
|
||||
- ✅ 在阶段2时直接使用(无需重新安装)
|
||||
- ✅ 如果暂时不用,也不用删除(不影响性能)
|
||||
|
||||
---
|
||||
|
||||
## 7. 常见问题 FAQ
|
||||
|
||||
### Q1嚗鑹edis蝻枏<EFBFBD><EFBFBD>朙edis<EFBFBD>笔<EFBFBD><EFBFBD>賢<EFBFBD><EFBFBD>嗡蝙<EFBFBD>典<EFBFBD>嚗?
|
||||
**A1**嚗𡁜虾隞伐<EFBFBD>摰<EFBFBD>賑雿輻鍂<EFBFBD>䔶<EFBFBD>銝析edis摰硺<EFBFBD>嚗䔶<EFBFBD><EFBFBD>臭<EFBFBD><EFBFBD>𣬚<EFBFBD><EFBFBD>唳旿蝏𤘪<EFBFBD><EFBFBD>?
|
||||
### Q1:Redis缓存和Redis队列能同时使用吗?
|
||||
**A1**:可以!它们使用同一个Redis实例,但是不同的数据结构。
|
||||
|
||||
```
|
||||
Redis摰硺<EFBFBD>嚗?56MB嚗?
|
||||
Redis实例(256MB)
|
||||
├─ 缓存数据(String):占用 ~50MB
|
||||
└─ 队列数据(List):占用 ~10MB
|
||||
------------------------------------
|
||||
<EFBFBD>餉恣嚗鰺60MB / 256MB = 23% 雿輻鍂<EFBFBD>?
|
||||
总计:~60MB / 256MB = 23% 使用率
|
||||
```
|
||||
|
||||
### Q2嚗帋<EFBFBD><EFBFBD>沖edis<EFBFBD>笔<EFBFBD>嚗𣬚鍂<EFBFBD>唳旿摨㯄<EFBFBD><EFBFBD>堒虾隞亙<EFBFBD>嚗?
|
||||
**A2**嚗𡁜虾隞乩<EFBFBD>銝齿綫<EFBFBD>僐<EFBFBD>?
|
||||
### Q2:不用Redis队列,用数据库队列可以吗?
|
||||
**A2**:可以但不推荐。
|
||||
|
||||
| 方案 | 优势 | 劣势 |
|
||||
|------|------|------|
|
||||
| **Redis<EFBFBD>笔<EFBFBD>** | 敹恬<EFBFBD>< 1ms嚗剹<EFBFBD><EFBFBD><EFBFBD><EFBFBD>賢<EFBFBD><EFBFBD>?| <20><>閬<EFBFBD>edis |
|
||||
| **<EFBFBD>唳旿摨㯄<EFBFBD><EFBFBD>?* | 銝漤<E98A9D>閬<EFBFBD><E996AC>憭碶<E686AD>韏?| <20>g<EFBFBD>> 10ms嚗剹<E59A97><E589B9><EFBFBD><EFBFBD>賜<EFBFBD><E8B39C>?|
|
||||
| **Redis队列** | 快(< 1ms)、功能完善 | 需要Redis |
|
||||
| **数据库队列** | 不需要额外依赖 | 慢(> 10ms)、功能简陋 |
|
||||
|
||||
<EFBFBD>典歇蝏讛揚銋唬<EFBFBD>Redis嚗<EFBFBD>遣霈桃凒<EFBFBD>亦鍂Redis<EFBFBD>笔<EFBFBD><EFBFBD>?
|
||||
您已经购买了Redis,建议直接用Redis队列。
|
||||
|
||||
### Q3嚗𡁜<EFBFBD><EFBFBD>𨅯蘨<EFBFBD>舐鍂Redis蝻枏<EFBFBD>嚗𣬚頂蝏煺<EFBFBD><EFBFBD>厰䔮憸睃<EFBFBD>嚗?
|
||||
**A3**嚗𡁶<EFBFBD><EFBFBD>笔<EFBFBD>銝滢<EFBFBD><EFBFBD>匧之<EFBFBD>桅<EFBFBD>嚗䔶<EFBFBD>摮睃銁憌𡡞埯<EFBFBD>?
|
||||
### Q3:如果只启用Redis缓存,系统会有问题吗?
|
||||
**A3**:短期内不会有大问题,但存在风险。
|
||||
|
||||
```
|
||||
<EFBFBD>?<3F>臭誑甇<E8AA91>虜餈鞱<E9A488>嚗?
|
||||
- LLM<EFBFBD>鞉𧋦<EFBFBD>批<EFBFBD> <20>?
|
||||
- 憭𡁜<EFBFBD>靘讠<EFBFBD>摮睃<EFBFBD>鈭?<3F>?
|
||||
✅ 可以正常运行:
|
||||
- LLM成本控制 ✅
|
||||
- 多实例缓存共享 ✅
|
||||
|
||||
<EFBFBD>𩤃<EFBFBD> 瞏𨅯銁憌𡡞埯嚗?
|
||||
- <EFBFBD>蹂遙<EFBFBD>∪虾<EFBFBD>賭腺憭梧<EFBFBD>摰硺<EFBFBD><EFBFBD>滚鍳嚗?
|
||||
⚠️ 潜在风险:
|
||||
- 长任务可能丢失(实例重启)
|
||||
- 无法重试失败任务
|
||||
- 用户体验不佳
|
||||
```
|
||||
|
||||
### Q4嚗𡁜鍳<EFBFBD>沖edis<EFBFBD>笔<EFBFBD><EFBFBD>𠬍<EFBFBD><EFBFBD>賢炏<EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>衽emoryQueue嚗?
|
||||
**A4**嚗𡁜虾隞伐<EFBFBD>靽格㺿<EFBFBD>臬<EFBFBD><EFBFBD>㗛<EFBFBD><EFBFBD>喳虾<EFBFBD>?
|
||||
### Q4:启用Redis队列后,能否回退到MemoryQueue?
|
||||
**A4**:可以!修改环境变量即可。
|
||||
|
||||
```bash
|
||||
# 切换到Redis队列
|
||||
QUEUE_TYPE=redis
|
||||
|
||||
# <EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>啣<EFBFBD>摮㗛<EFBFBD><EFBFBD>?
|
||||
# 回退到内存队列
|
||||
QUEUE_TYPE=memory
|
||||
```
|
||||
|
||||
雿<EFBFBD>釣<EFBFBD>𧶏<EFBFBD>**Redis銝剔<EFBFBD><EFBFBD>芸<EFBFBD><EFBFBD>𣂷遙<EFBFBD>∩<EFBFBD>銝W仃**<2A>?
|
||||
但注意:**Redis中的未完成任务会丢失**。
|
||||
|
||||
### Q5:BullMQ难学吗?
|
||||
**A5**嚗帋<EFBFBD><EFBFBD>橘<EFBFBD><EFBFBD>詨<EFBFBD>API<EFBFBD>芣<EFBFBD>5銝芥<EFBFBD>?
|
||||
**A5**:不难,核心API只有5个。
|
||||
|
||||
```typescript
|
||||
// 1. 创建队列
|
||||
@@ -645,7 +645,7 @@ worker.on('failed', ...);
|
||||
const job = await queue.getJob(jobId);
|
||||
```
|
||||
|
||||
憸<EFBFBD>恣摮虫<EFBFBD><EFBFBD>園𡢿嚗?*<2A>𠰴予**
|
||||
预计学习时间:**半天**
|
||||
|
||||
---
|
||||
|
||||
@@ -653,52 +653,51 @@ const job = await queue.getJob(jobId);
|
||||
|
||||
### 8.1 核心要点
|
||||
|
||||
1. **Redis蝻枏<EFBFBD> <20>?Redis<EFBFBD>笔<EFBFBD>**
|
||||
1. **Redis缓存 ≠ Redis队列**
|
||||
- 都是使用同一个Redis实例
|
||||
- 只是使用方式不同
|
||||
|
||||
2. **您的系统现状**
|
||||
- BullMQ撌脣<EFBFBD>鋆<EFBFBD><EFBFBD><EFBFBD>芯蝙<EFBFBD>?
|
||||
- BullMQ已安装但未使用
|
||||
- 当前用MemoryQueue(自研内存队列)
|
||||
- 需要改造才能启用Redis队列
|
||||
|
||||
3. **推荐方案**
|
||||
- <EFBFBD>?<3F><>鍳<EFBFBD>沖edis蝻枏<E89DBB>嚗<EFBFBD>𧋦<EFBFBD>剁<EFBFBD>2憭抬<E686AD>
|
||||
- <EFBFBD>?<3F>滚鍳<E6BB9A>沖edis<69>笔<EFBFBD>嚗<EFBFBD><E59A97><EFBFBD>剁<EFBFBD>3憭抬<E686AD>
|
||||
- <EFBFBD>?皜鞱<E79A9C>撘誩<E69298><E8AAA9>踝<EFBFBD><E8B89D>滢<EFBFBD>憌𡡞埯
|
||||
- ✅ 先启用Redis缓存(本周,2天)
|
||||
- ✅ 再启用Redis队列(下周,3天)
|
||||
- ✅ 渐进式实施,降低风险
|
||||
|
||||
4. **<EFBFBD>芰鍂Redis蝻枏<EFBFBD><EFBFBD><EFBFBD>蔣<EFBFBD>?*
|
||||
- <EFBFBD>?<3F>臭誑閫<E8AA91><E996AB>LLM<4C>鞉𧋦<E99E89>桅<EFBFBD>
|
||||
- <EFBFBD>𩤃<EFBFBD> <20>蹂遙<E8B982>∪虾<E288AA>䭾<EFBFBD>找<EFBFBD><E689BE>厰<EFBFBD><E58EB0>?
|
||||
- <EFBFBD><EFBFBD> 撱箄悅嚗𡁏覔<F0A1818F>桃鍂<E6A183>瑁<EFBFBD>璅∪<E79285>摰𡁏糓<F0A1818F>血鍳<E8A180>券<EFBFBD><E588B8>?
|
||||
4. **只用Redis缓存的影响**
|
||||
- ✅ 可以解决LLM成本问题
|
||||
- ⚠️ 长任务可靠性仍有风险
|
||||
- 📊 建议:根据用户规模决定是否启用队列
|
||||
|
||||
5. **Redis<EFBFBD>笔<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?*
|
||||
5. **Redis队列的风险**
|
||||
- 🟡 增加复杂度(但可控)
|
||||
- <EFBFBD>叚 靘肽<E99D98>Redis蝔喳<E89D94><E596B3>改<EFBFBD>擃睃虾<E79D83>函<EFBFBD>99.95%嚗?
|
||||
- <EFBFBD>?<3F>嗥<EFBFBD> > 憌𡡞埯
|
||||
- 🟢 依赖Redis稳定性(高可用版99.95%)
|
||||
- ✅ 收益 > 风险
|
||||
|
||||
---
|
||||
|
||||
### 8.2 决策建议
|
||||
|
||||
**憒<EFBFBD><EFBFBD><EFBFBD>函<EFBFBD>蝟餌<EFBFBD>皛∟雲隞乩<EFBFBD><EFBFBD>∩辣銋衤<EFBFBD>嚗<EFBFBD>遣霈桀偷敹怠鍳<EFBFBD>沖edis<EFBFBD>笔<EFBFBD>嚗?*
|
||||
**如果您的系统满足以下条件之一,建议尽快启用Redis队列:**
|
||||
|
||||
```
|
||||
<EFBFBD>?<3F>冽<EFBFBD><E586BD>?> 20
|
||||
<EFBFBD>?SAE摰硺<EFBFBD><EFBFBD>?> 1
|
||||
<EFBFBD>?<3F>寥<EFBFBD>隞餃𦛚憸𤑳<E686B8>嚗<EFBFBD><E59A97>憭?> 5銝迎<E98A9D>
|
||||
<EFBFBD>?隞餃𦛚<E9A483>園鵭 > 10<EFBFBD><EFBFBD><EFBFBD>
|
||||
<EFBFBD>?<3F>冽<EFBFBD><E586BD>望<EFBFBD>其遙<E585B6>∩腺憭?
|
||||
✅ 用户数 > 20
|
||||
✅ SAE实例数 > 1
|
||||
✅ 批量任务频繁(每天 > 5个)
|
||||
✅ 任务时长 > 10分钟
|
||||
✅ 用户抱怨任务丢失
|
||||
```
|
||||
|
||||
**<EFBFBD>血<EFBFBD>嚗<EFBFBD>虾隞亙<EFBFBD><EFBFBD>舐鍂Redis蝻枏<EFBFBD>嚗屸<EFBFBD><EFBFBD>埈<EFBFBD>蝻瓐<EFBFBD>?*
|
||||
**否则,可以先启用Redis缓存,队列暂缓。**
|
||||
|
||||
---
|
||||
|
||||
**<EFBFBD><EFBFBD>﹝蝏湔擪<EFBFBD><EFBFBD><EFBFBD>** <20><><EFBFBD>臬𣪧<E887AC>?
|
||||
**文档维护者:** 技术团队
|
||||
**最后更新:** 2025-12-12
|
||||
**<EFBFBD>詨<EFBFBD><EFBFBD><EFBFBD>﹝嚗?* [Redis<EFBFBD>寥<EFBFBD>惩<EFBFBD><EFBFBD>質恣<EFBFBD>哋(./04-Redis<69>寥<EFBFBD>惩<EFBFBD><E683A9>質恣<E8B3AA>?md)
|
||||
|
||||
**相关文档:** [Redis改造实施计划](./04-Redis改造实施计划.md)
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user