feat(aia): Complete AIA V2.0 with universal streaming capabilities
Major Changes: - Add StreamingService with OpenAI Compatible format - Upgrade Chat component V2 with Ant Design X integration - Implement AIA module with 12 intelligent agents - Update API routes to unified /api/v1 prefix - Update system documentation Backend (~1300 lines): - common/streaming: OpenAI Compatible adapter - modules/aia: 12 agents, conversation service, streaming integration - Update route versions (RVW, PKB to v1) Frontend (~3500 lines): - modules/aia: AgentHub + ChatWorkspace (100% prototype restoration) - shared/Chat: AIStreamChat, ThinkingBlock, useAIStream Hook - Update API endpoints to v1 Documentation: - AIA module status guide - Universal capabilities catalog - System overview updates - All module documentation sync Tested: Stream response verified, authentication working Status: AIA V2.0 core completed (85%)
This commit is contained in:
@@ -1,25 +1,25 @@
|
||||
# CloseAI髮<49><E9ABAE>謖<EFBFBD>漉
|
||||
|
||||
> **文档版本:** v1.0
|
||||
> **创建日期:** 2025-11-09
|
||||
> **譁<EFBFBD>。」迚域悽<EFBFBD>?* v1.0
|
||||
> **蛻帛サコ譌・譛滂シ?* 2025-11-09
|
||||
> **逕ィ騾費シ<E8B2BB>** 騾夊ソ④loseAI莉」逅<EFBDA3>ケウ蜿ー隶ソ髣ョOpenAI GPT-5蜥靴laude-4.5
|
||||
> **适用场景:** AI智能文献双模型筛选、高质量文本生成
|
||||
> **騾ら畑蝨コ譎ッ<EFBFBD>?* AI譎コ閭ス譁<EFBDBD>鍵蜿梧ィ。蝙狗ュ幃峨<E5B3A8>ォ倩エィ驥乗枚譛ャ逕滓<E98095>
|
||||
|
||||
---
|
||||
|
||||
## 📋 CloseAI简介
|
||||
## <EFBFBD>搭 CloseAI邂莉?
|
||||
|
||||
### 什么是CloseAI?
|
||||
### 莉荵域弍CloseAI<EFBFBD>?
|
||||
|
||||
CloseAI是一个**API代理平台**,为中国用户提供稳定的OpenAI和Claude API访问服务。
|
||||
CloseAI譏ッ荳荳?*API莉」逅<EFBDA3>ケウ蜿ー**<2A>御クコ荳ュ蝗ス逕ィ謌キ謠蝉セ帷ィウ螳夂噪OpenAI蜥靴laude API隶ソ髣ョ譛榊苅縲?
|
||||
|
||||
**核心优势:**
|
||||
- ✅ 国内直连,无需科学上网
|
||||
- ✅ 一个API Key同时调用OpenAI和Claude
|
||||
- ✅ 兼容OpenAI SDK标准接口
|
||||
- ✅ 支持最新模型(GPT-5、Claude-4.5)
|
||||
**譬ク蠢<EFBFBD>シ伜漢<EFBFBD>?*
|
||||
- 笨?蝗ス蜀<EFBDBD>峩霑橸シ梧裏髴遘大ュヲ荳顔ス<E9A194>
|
||||
- 笨?荳荳ェAPI Key蜷梧慮隹<EFBFBD>畑OpenAI蜥靴laude
|
||||
- 笨?蜈シ螳ケOpenAI SDK譬<EFBFBD>㊥謗・蜿」
|
||||
- 笨?謾ッ謖∵怙譁ー讓。蝙具シ<E585B7>PT-5縲,laude-4.5<EFBFBD>?
|
||||
|
||||
**官网:** https://platform.openai-proxy.org
|
||||
**螳倡ス托シ?* https://platform.openai-proxy.org
|
||||
|
||||
---
|
||||
|
||||
@@ -38,15 +38,15 @@ CLOSEAI_OPENAI_BASE_URL=https://api.openai-proxy.org/v1
|
||||
CLOSEAI_CLAUDE_BASE_URL=https://api.openai-proxy.org/anthropic
|
||||
```
|
||||
|
||||
### 支持的模型
|
||||
### 謾ッ謖∫噪讓。蝙?
|
||||
|
||||
| 讓。蝙<EFBDA1> | Model ID | 隸エ譏<EFBDB4> | 騾ら畑蝨コ譎ッ |
|
||||
|------|---------|------|---------|
|
||||
| **GPT-5-Pro** | `gpt-5-pro` | 最新GPT-5 ⭐ | 文献精准筛选、复杂推理 |
|
||||
| GPT-4-Turbo | `gpt-4-turbo-preview` | GPT-4高性能版 | 质量要求高的任务 |
|
||||
| GPT-3.5-Turbo | `gpt-3.5-turbo` | 快速经济版 | 简单任务、成本优化 |
|
||||
| **Claude-4.5-Sonnet** | `claude-sonnet-4-5-20250929` | 最新Claude ⭐ | 第三方仲裁、结构化输出 |
|
||||
| Claude-3.5-Sonnet | `claude-3-5-sonnet-20241022` | Claude-3.5稳定版 | 高质量文本生成 |
|
||||
| **GPT-5-Pro** | `gpt-5-pro` | 譛譁ーGPT-5 箝?| 譁<>鍵邊セ蜃<EFBDBE>ュ幃峨∝、肴揩謗ィ逅?|
|
||||
| GPT-4-Turbo | `gpt-4-turbo-preview` | GPT-4鬮俶ァ閭ス迚?| 雍ィ驥剰ヲ∵アるォ倡噪莉サ蜉。 |
|
||||
| GPT-3.5-Turbo | `gpt-3.5-turbo` | 蠢ォ騾溽サ乗オ守沿 | 邂蜊穂ササ蜉。縲∵<E7B8B2>譛ャ莨伜<E88EA8>?|
|
||||
| **Claude-4.5-Sonnet** | `claude-sonnet-4-5-20250929` | 譛譁ーClaude 箝?| 隨ャ荳画婿莉イ陬√∫サ捺桷蛹冶セ灘<EFBDBE> |
|
||||
| Claude-3.5-Sonnet | `claude-3-5-sonnet-20241022` | Claude-3.5遞ウ螳夂<EFBFBD>?| 鬮倩エィ驥乗枚譛ャ逕滓<E98095>?|
|
||||
|
||||
---
|
||||
|
||||
@@ -58,9 +58,9 @@ CLOSEAI_CLAUDE_BASE_URL=https://api.openai-proxy.org/anthropic
|
||||
npm install openai
|
||||
```
|
||||
|
||||
### 2. 创建LLM服务类
|
||||
### 2. 蛻帛サコLLM譛榊苅邀?
|
||||
|
||||
**文件位置:** `backend/src/common/llm/closeai.service.ts`
|
||||
**譁<EFBFBD>サカ菴咲スョ<EFBFBD>?* `backend/src/common/llm/closeai.service.ts`
|
||||
|
||||
```typescript
|
||||
import OpenAI from 'openai';
|
||||
@@ -71,13 +71,13 @@ export class CloseAIService {
|
||||
private claudeClient: OpenAI;
|
||||
|
||||
constructor() {
|
||||
// OpenAI客户端(通过CloseAI)
|
||||
// OpenAI螳「謌キ遶ッ<EFBFBD>磯夊ソ④loseAI<EFBFBD>?
|
||||
this.openaiClient = new OpenAI({
|
||||
apiKey: config.closeaiApiKey,
|
||||
baseURL: config.closeaiOpenaiBaseUrl,
|
||||
});
|
||||
|
||||
// Claude客户端(通过CloseAI)
|
||||
// Claude螳「謌キ遶ッ<EFBFBD>磯夊ソ④loseAI<EFBFBD>?
|
||||
this.claudeClient = new OpenAI({
|
||||
apiKey: config.closeaiApiKey,
|
||||
baseURL: config.closeaiClaudeBaseUrl,
|
||||
@@ -135,7 +135,7 @@ export class CloseAIService {
|
||||
}
|
||||
|
||||
/**
|
||||
* 流式响应(GPT-5)
|
||||
* 豬∝シ丞桃蠎費シ<EFBFBD>PT-5<EFBFBD>?
|
||||
*/
|
||||
async *streamGPT5(prompt: string, systemPrompt?: string) {
|
||||
const messages: any[] = [];
|
||||
@@ -165,7 +165,7 @@ export class CloseAIService {
|
||||
|
||||
### 3. 扈滉クLLM譛榊苅<E6A68A>亥性4荳ェ讓。蝙具シ<E585B7>
|
||||
|
||||
**文件位置:** `backend/src/common/llm/llm.service.ts`
|
||||
**譁<EFBFBD>サカ菴咲スョ<EFBFBD>?* `backend/src/common/llm/llm.service.ts`
|
||||
|
||||
```typescript
|
||||
import OpenAI from 'openai';
|
||||
@@ -258,7 +258,7 @@ export class UnifiedLLMService {
|
||||
|
||||
### 蝨コ譎ッ1<EFBDAF>壼曙讓。蝙句ッケ豈皮ュ幃会シ域耳闕撰シ俄ュ<E4BF84>
|
||||
|
||||
**策略:** DeepSeek(快速初筛) + GPT-5(质量复核)
|
||||
**遲也払<EFBFBD>?* DeepSeek<EFBFBD>亥ソォ騾溷<EFBFBD>遲幢シ<EFBFBD> + GPT-5<>郁エィ驥丞、肴<EFBDA4>ク<EFBFBD><EFBDB8>
|
||||
|
||||
```typescript
|
||||
export class LiteratureScreeningService {
|
||||
@@ -269,23 +269,23 @@ export class LiteratureScreeningService {
|
||||
}
|
||||
|
||||
/**
|
||||
* 双模型文献筛选
|
||||
* 蜿梧ィ。蝙区枚迪ョ遲幃?
|
||||
*/
|
||||
async screenLiterature(title: string, abstract: string, picoConfig: any) {
|
||||
const prompt = `
|
||||
隸キ譬ケ謐ョ莉・荳輝ICO譬<EFBFBD>㊥<EFBFBD>悟愛譁ュ霑咏ッ<EFBFBD>枚迪ョ譏ッ蜷ヲ蠎碑ッ・郤ウ蜈・<EFBFBD><EFBFBD>
|
||||
|
||||
**PICO标准:**
|
||||
**PICO譬<EFBFBD>㊥<EFBFBD>?*
|
||||
- Population: ${picoConfig.population}
|
||||
- Intervention: ${picoConfig.intervention}
|
||||
- Comparison: ${picoConfig.comparison}
|
||||
- Outcome: ${picoConfig.outcome}
|
||||
|
||||
**文献信息:**
|
||||
标题:${title}
|
||||
摘要:${abstract}
|
||||
**譁<EFBFBD>鍵菫。諱ッ<EFBFBD>?*
|
||||
譬<EFBFBD>「假シ?{title}
|
||||
鞫倩ヲ<EFBFBD>シ?{abstract}
|
||||
|
||||
请输出JSON格式:
|
||||
隸キ霎灘<EFBFBD>JSON譬シ蠑擾シ?
|
||||
{
|
||||
"decision": "include/exclude/uncertain",
|
||||
"reason": "蛻、譁ュ逅<EFBDAD>罰",
|
||||
@@ -325,11 +325,11 @@ export class LiteratureScreeningService {
|
||||
|
||||
### 蝨コ譎ッ2<EFBDAF>壻ク画ィ。蝙句<E89D99>隸<EFBFBD>サイ陬<EFBDB2>
|
||||
|
||||
**策略:** 当两个模型冲突时,启用Claude作为第三方仲裁
|
||||
**遲也払<EFBFBD>?* 蠖謎ク、荳ェ讓。蝙句<E89D99>遯∵慮<E288B5>悟星逕ィClaude菴應クコ隨ャ荳画婿莉イ陬?
|
||||
|
||||
```typescript
|
||||
async screenWithArbitration(title: string, abstract: string, picoConfig: any) {
|
||||
// 第一轮:双模型筛选
|
||||
// 隨ャ荳霓ョ<EFBFBD>壼曙讓。蝙狗ュ幃?
|
||||
const initialScreen = await this.screenLiterature(title, abstract, picoConfig);
|
||||
|
||||
// 螯よ棡荳閾エ<E996BE>檎峩謗・霑泌屓
|
||||
@@ -343,7 +343,7 @@ async screenWithArbitration(title: string, abstract: string, picoConfig: any) {
|
||||
const claudeResult = await this.llm.chat('claude', prompt);
|
||||
const claudeDecision = JSON.parse(claudeResult.content);
|
||||
|
||||
// 三模型投票
|
||||
// 荳画ィ。蝙区兜逾?
|
||||
const decisions = [
|
||||
initialScreen.models[0].decision,
|
||||
initialScreen.models[1].decision,
|
||||
@@ -356,7 +356,7 @@ async screenWithArbitration(title: string, abstract: string, picoConfig: any) {
|
||||
uncertain: decisions.filter(d => d === 'uncertain').length,
|
||||
};
|
||||
|
||||
// 多数决
|
||||
// 螟壽焚蜀?
|
||||
const finalDecision = Object.keys(voteCount).reduce((a, b) =>
|
||||
voteCount[a] > voteCount[b] ? a : b
|
||||
);
|
||||
@@ -370,13 +370,13 @@ async screenWithArbitration(title: string, abstract: string, picoConfig: any) {
|
||||
}
|
||||
```
|
||||
|
||||
### 场景3:成本优化策略
|
||||
### 蝨コ譎ッ3<EFBFBD>壽<EFBFBD>譛ャ莨伜喧遲也<EFBFBD>?
|
||||
|
||||
**策略:** 只对不确定的结果使用GPT-5复核
|
||||
**遲也払<EFBFBD>?* 蜿ェ蟇ケ荳咲。ョ螳夂噪扈捺棡菴ソ逕ィGPT-5螟肴<EFBFBD>ク
|
||||
|
||||
```typescript
|
||||
async screenWithCostOptimization(title: string, abstract: string, picoConfig: any) {
|
||||
// 第一轮:用DeepSeek快速初筛(便宜)
|
||||
// 隨ャ荳霓ョ<EFBFBD>夂畑DeepSeek蠢ォ騾溷<EFBFBD>遲幢シ井セソ螳懶シ?
|
||||
const quickScreen = await this.llm.chat('deepseek', prompt);
|
||||
const quickDecision = JSON.parse(quickScreen.content);
|
||||
|
||||
@@ -405,39 +405,39 @@ async screenWithCostOptimization(title: string, abstract: string, picoConfig: an
|
||||
|
||||
---
|
||||
|
||||
## 📊 性能和成本对比
|
||||
## <EFBFBD>投 諤ァ閭ス蜥梧<E89CA5>譛ャ蟇ケ豈?
|
||||
|
||||
### 讓。蝙区ァ閭ス蟇ケ豈<EFBDB9>
|
||||
|
||||
| 謖<><E8AC96><EFBFBD> | DeepSeek-V3 | GPT-5-Pro | Claude-4.5 | Qwen-Max |
|
||||
|------|------------|-----------|-----------|----------|
|
||||
| **准确率** | 85% | **95%** ⭐ | 93% | 82% |
|
||||
| **速度** | **快** ⭐ | 中等 | 中等 | 快 |
|
||||
| **成本** | **¥0.001/1K** ⭐ | ¥0.10/1K | ¥0.021/1K | ¥0.004/1K |
|
||||
| **中文理解** | **优秀** ⭐ | 优秀 | 良好 | 优秀 |
|
||||
| **结构化输出** | 良好 | 优秀 | **优秀** ⭐ | 良好 |
|
||||
| **蜃<EFBFBD>。ョ邇?* | 85% | **95%** 箝?| 93% | 82% |
|
||||
| **騾溷コヲ** | **蠢?* 箝?| 荳ュ遲<EFBDAD> | 荳ュ遲<EFBDAD> | 蠢?|
|
||||
| **謌先悽** | **ツ・0.001/1K** 箝?| ツ・0.10/1K | ツ・0.021/1K | ツ・0.004/1K |
|
||||
| **荳ュ譁<EFBFBD>炊隗」** | **莨倡ァ** 箝?| 莨倡ァ | 濶ッ螂ス | 莨倡ァ |
|
||||
| **扈捺桷蛹冶セ灘<EFBFBD>?* | 濶ッ螂ス | 莨倡ァ | **莨倡ァ** 箝?| 濶ッ螂ス |
|
||||
|
||||
### 筛选1000篇文献的成本估算
|
||||
### 遲幃?000遽<30>枚迪ョ逧<EFBDAE><E980A7>譛ャ莨ー邂<EFBDB0>
|
||||
|
||||
**遲也払A<E68995>壼宵逕ィDeepSeek**
|
||||
- 成本:¥20-30
|
||||
- 謌先悽<EFBFBD>堋?0-30
|
||||
- 蜃<>。ョ邇<EFBDAE>シ<EFBFBD>85%
|
||||
- 适用:预算有限,可接受一定误差
|
||||
- 騾ら畑<EFBFBD>夐「<EFBFBD>ョ玲怏髯撰シ悟庄謗・蜿嶺ク螳夊ッッ蟾?
|
||||
|
||||
**策略B:DeepSeek + GPT-5 双模型**
|
||||
- 成本:¥150-200
|
||||
**遲也払B<EFBFBD>咼eepSeek + GPT-5 蜿梧ィ。蝙?*
|
||||
- 謌先悽<EFBFBD>堋?50-200
|
||||
- 蜃<>。ョ邇<EFBDAE>シ<EFBFBD>92%
|
||||
- 适用:质量要求高,预算充足 ⭐ 推荐
|
||||
- 騾ら畑<EFBFBD>夊エィ驥剰ヲ∵アるォ假シ碁「<EFBFBD>ョ怜<EFBFBD>雜?箝?謗ィ闕<EFBDA8>
|
||||
|
||||
**策略C:三模型共识(20%冲突启用Claude)**
|
||||
- 成本:¥180-220
|
||||
**遲也払C<EFBFBD>壻ク画ィ。蝙句<EFBFBD>隸<EFBFBD>シ?0%蜀イ遯∝星逕ィClaude<64>?*
|
||||
- 謌先悽<EFBFBD>堋?80-220
|
||||
- 蜃<>。ョ邇<EFBDAE>シ<EFBFBD>95%
|
||||
- 适用:最高质量要求
|
||||
- 騾ら畑<EFBFBD>壽怙鬮倩エィ驥剰ヲ∵ア?
|
||||
|
||||
**策略D:成本优化(80%用DeepSeek,20%用GPT-5)**
|
||||
- 成本:¥50-80
|
||||
**遲也払D<EFBFBD>壽<EFBFBD>譛ャ莨伜喧<EFBFBD><EFBFBD>80%逕ィDeepSeek<EFBFBD>?0%逕ィGPT-5<>?*
|
||||
- 謌先悽<EFBFBD>堋?0-80
|
||||
- 蜃<>。ョ邇<EFBDAE>シ<EFBFBD>90%
|
||||
- 适用:质量和成本平衡 ⭐ 性价比最高
|
||||
- 騾ら畑<EFBFBD>夊エィ驥丞柱謌先悽蟷ウ陦。 箝?諤ァ莉キ豈疲怙鬮?
|
||||
|
||||
---
|
||||
|
||||
@@ -446,12 +446,12 @@ async screenWithCostOptimization(title: string, abstract: string, picoConfig: an
|
||||
### 1. API Key螳牙<E89EB3>
|
||||
|
||||
```typescript
|
||||
// ❌ 错误:硬编码API Key
|
||||
// 笶?髞呵ッッ<EFBDAF>夂。ャ郛也<E9839B>、PI Key
|
||||
const client = new OpenAI({
|
||||
apiKey: 'sk-cu0iepbXYGGx2jc7BqP6ogtSWmP6fk918qV3RUdtGC3Edlpo',
|
||||
});
|
||||
|
||||
// ✅ 正确:从环境变量读取
|
||||
// 笨?豁」遑ョ<E98191>壻サ守識蠅<E8AD98>序驥剰ッサ蜿<EFBDBB>
|
||||
const client = new OpenAI({
|
||||
apiKey: process.env.CLOSEAI_API_KEY,
|
||||
});
|
||||
@@ -465,15 +465,15 @@ async chat(provider: LLMProvider, prompt: string) {
|
||||
const response = await this.llm.chat(provider, prompt);
|
||||
return response;
|
||||
} catch (error) {
|
||||
// CloseAI可能返回的错误
|
||||
// CloseAI蜿ッ閭ス霑泌屓逧<EFBFBD>漠隸?
|
||||
if (error.status === 429) {
|
||||
// 騾溽紫髯仙宛
|
||||
console.error('API隹<49>畑騾溽紫雜<E7B4AB>剞<EFBFBD>瑚ッキ遞榊錘驥崎ッ<E5B48E>');
|
||||
} else if (error.status === 401) {
|
||||
// 隶、隸∝、ア雍・
|
||||
console.error('API Key无效,请检查配置');
|
||||
console.error('API Key譌<EFBFBD>謨茨シ瑚ッキ譽譟・驟咲ス?);
|
||||
} else if (error.status === 500) {
|
||||
// 服务端错误
|
||||
// 譛榊苅遶ッ髞呵ッ?
|
||||
console.error('CloseAI譛榊苅蠑ょクク<EFBFBD>瑚ッキ遞榊錘驥崎ッ<EFBFBD>');
|
||||
}
|
||||
throw error;
|
||||
@@ -491,7 +491,7 @@ async chatWithRetry(provider: LLMProvider, prompt: string, maxRetries = 3) {
|
||||
} catch (error) {
|
||||
if (i === maxRetries - 1) throw error;
|
||||
|
||||
// 指数退避
|
||||
// 謖<EFBFBD>焚騾驕?
|
||||
const delay = Math.pow(2, i) * 1000;
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
}
|
||||
@@ -509,9 +509,9 @@ async chatWithRetry(provider: LLMProvider, prompt: string, maxRetries = 3) {
|
||||
|
||||
---
|
||||
|
||||
**更新日志:**
|
||||
**譖エ譁ー譌・蠢暦シ?*
|
||||
- 2025-11-09: 蛻帛サコ譁<EFBDBA>。」<EFBDA1>梧キサ蜉<EFBDBB>CloseAI髮<49><E9ABAE>謖<EFBFBD>漉
|
||||
- 支持GPT-5-Pro和Claude-4.5-Sonnet最新模型
|
||||
- 謾ッ謖;PT-5-Pro蜥靴laude-4.5-Sonnet譛譁ー讓。蝙?
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,51 +1,35 @@
|
||||
# LLM大模型网关
|
||||
|
||||
> **能力定位:** 通用能力层核心能力
|
||||
> **复用率:** 71% (5个模块依赖)
|
||||
> **优先级:** P0(最高)⭐
|
||||
> **状态:** ❌ 待实现
|
||||
|
||||
# LLM澶фā鍨嬬綉鍏?
|
||||
> **鑳藉姏瀹氫綅锛?* 閫氱敤鑳藉姏灞傛牳蹇冭兘鍔?
|
||||
> **澶嶇敤鐜囷細** 71% (5涓<35>ā鍧椾緷璧?
|
||||
> **浼樺厛绾э細** P0锛堟渶楂橈級猸?
|
||||
> **鐘舵€侊細** 鉂?寰呭疄鐜?
|
||||
---
|
||||
|
||||
## 馃搵 鑳藉姏姒傝堪
|
||||
|
||||
LLM大模型网关是平台AI能力的核心中枢,负责:
|
||||
- 统一管理所有LLM调用
|
||||
- 根据用户版本动态切换模型
|
||||
- 成本控制与限流
|
||||
- Token计数与计费
|
||||
|
||||
LLM澶фā鍨嬬綉鍏虫槸骞冲彴AI鑳藉姏鐨勬牳蹇冧腑鏋<EFBFBD>紝璐熻矗锛?- 缁熶竴绠$悊鎵€鏈塋LM璋冪敤
|
||||
- 鏍规嵁鐢ㄦ埛鐗堟湰鍔ㄦ€佸垏鎹㈡ā鍨?- 鎴愭湰鎺у埗涓庨檺娴?- Token璁℃暟涓庤<E6B693>璐?
|
||||
---
|
||||
|
||||
## 🎯 核心价值
|
||||
|
||||
### 1. 商业模式技术基础 ⭐
|
||||
```
|
||||
专业版 → DeepSeek-V3(便宜,¥1/百万tokens)
|
||||
高级版 → DeepSeek + Qwen3
|
||||
旗舰版 → DeepSeek + Qwen3 + Qwen-Long + Claude
|
||||
## 馃幆 鏍稿績浠峰€?
|
||||
### 1. 鍟嗕笟妯″紡鎶€鏈<E282AC>熀纭€ 猸?```
|
||||
涓撲笟鐗?鈫?DeepSeek-V3锛堜究瀹滐紝楼1/鐧句竾tokens锛?楂樼骇鐗?鈫?DeepSeek + Qwen3
|
||||
鏃楄埌鐗?鈫?DeepSeek + Qwen3 + Qwen-Long + Claude
|
||||
```
|
||||
|
||||
### 2. 鎴愭湰鎺у埗
|
||||
- 缁熶竴鐩戞帶鎵€鏈塋LM API璋冪敤
|
||||
- 瓒呭嚭閰嶉<E996B0>鑷<EFBFBD>姩闄愭祦
|
||||
- 按版本计费
|
||||
|
||||
- 鎸夌増鏈<E5A297><E98F88>璐?
|
||||
### 3. 缁熶竴鎺ュ彛
|
||||
- 屏蔽不同LLM API的差异
|
||||
- 统一的调用接口
|
||||
|
||||
- 灞忚斀涓嶅悓LLM API鐨勫樊寮?- 缁熶竴鐨勮皟鐢ㄦ帴鍙?
|
||||
---
|
||||
|
||||
## 馃搳 渚濊禆妯″潡
|
||||
|
||||
**5个模块依赖(71%复用率):**
|
||||
**5涓<EFBFBD>ā鍧椾緷璧栵紙71%澶嶇敤鐜囷級锛?*
|
||||
1. **AIA** - AI鏅鸿兘闂<E58598>瓟
|
||||
2. **ASL** - AI智能文献(双模型判断)
|
||||
3. **PKB** - 个人知识库(RAG问答)
|
||||
4. **DC** - 数据清洗(NER提取)
|
||||
5. **RVW** - 稿件审查(AI评估)
|
||||
|
||||
2. **ASL** - AI鏅鸿兘鏂囩尞锛堝弻妯″瀷鍒ゆ柇锛?3. **PKB** - 涓<>汉鐭ヨ瘑搴擄紙RAG闂<47>瓟锛?4. **DC** - 鏁版嵁娓呮礂锛圢ER鎻愬彇锛?5. **RVW** - 绋夸欢瀹℃煡锛圓I璇勪及锛?
|
||||
---
|
||||
|
||||
## 馃挕 鏍稿績鍔熻兘
|
||||
@@ -69,67 +53,48 @@ chat(params: {
|
||||
### 3. 閰嶉<E996B0>绠$悊
|
||||
```typescript
|
||||
checkQuota(userId: string): Promise<QuotaInfo>
|
||||
// 检查用户剩余配额
|
||||
```
|
||||
// 妫€鏌ョ敤鎴峰墿浣欓厤棰?```
|
||||
|
||||
### 4. Token璁℃暟
|
||||
```typescript
|
||||
countTokens(text: string): number
|
||||
// 使用tiktoken计算Token数
|
||||
```
|
||||
// 浣跨敤tiktoken璁$畻Token鏁?```
|
||||
|
||||
---
|
||||
|
||||
## 馃搨 鏂囨。缁撴瀯
|
||||
|
||||
```
|
||||
01-LLM大模型网关/
|
||||
├── [AI对接] LLM网关快速上下文.md # ✅ 已完成
|
||||
├── 03-CloseAI集成指南.md # ✅ 已完成 ⭐
|
||||
├── 00-需求分析/
|
||||
│ └── README.md
|
||||
01-LLM澶фā鍨嬬綉鍏?
|
||||
鈹溾攢鈹€ [AI瀵规帴] LLM缃戝叧蹇<EFBFBD>€熶笂涓嬫枃.md # 鉁?宸插畬鎴? 鈹溾攢鈹€ 03-CloseAI闆嗘垚鎸囧崡.md # 鉁?宸插畬鎴?猸? 鈹溾攢鈹€ 00-闇€姹傚垎鏋?
|
||||
鈹? 鈹斺攢鈹€ README.md
|
||||
鈹溾攢鈹€ 01-璁捐<E79281>鏂囨。/
|
||||
│ ├── 01-详细设计.md # ⏳ Week 5创建
|
||||
│ ├── 02-数据库设计.md # ⏳ Week 5创建
|
||||
│ ├── 03-API设计.md # ⏳ Week 5创建
|
||||
│ └── README.md
|
||||
└── README.md # ✅ 当前文档
|
||||
鈹? 鈹溾攢鈹€ 01-璇︾粏璁捐<E79281>.md # 鈴?Week 5鍒涘缓
|
||||
鈹? 鈹溾攢鈹€ 02-鏁版嵁搴撹<E690B4>璁?md # 鈴?Week 5鍒涘缓
|
||||
鈹? 鈹溾攢鈹€ 03-API璁捐<EFBFBD>.md # 鈴?Week 5鍒涘缓
|
||||
鈹? 鈹斺攢鈹€ README.md
|
||||
鈹斺攢鈹€ README.md # 鉁?褰撳墠鏂囨。
|
||||
```
|
||||
|
||||
### 快速入门文档 ⭐
|
||||
|
||||
| 文档 | 说明 | 状态 |
|
||||
### 蹇<EFBFBD>€熷叆闂ㄦ枃妗?猸?
|
||||
| 鏂囨。 | 璇存槑 | 鐘舵€?|
|
||||
|------|------|------|
|
||||
| **[AI对接] LLM网关快速上下文.md** | 快速了解LLM网关设计 | ✅ 已完成 |
|
||||
| **03-CloseAI集成指南.md** | CloseAI(GPT-5+Claude-4.5)集成文档 ⭐ | ✅ 已完成 |
|
||||
| **[AI瀵规帴] LLM缃戝叧蹇<EFBFBD>€熶笂涓嬫枃.md** | 蹇<>€熶簡瑙<E7B0A1>LM缃戝叧璁捐<E79281> | 鉁?宸插畬鎴?|
|
||||
| **03-CloseAI闆嗘垚鎸囧崡.md** | CloseAI锛圙PT-5+Claude-4.5锛夐泦鎴愭枃妗?猸?| 鉁?宸插畬鎴?|
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 开发计划调整
|
||||
|
||||
## 鈿狅笍 寮€鍙戣<E98D99>鍒掕皟鏁?
|
||||
### 鍘熻<E98D98>鍒掞細Week 2瀹屾垚LLM缃戝叧
|
||||
**调整:** LLM网关完整实现推迟到Week 5 ✅
|
||||
|
||||
**理由:**
|
||||
1. 现有LLM调用已经work(DeepSeek、Qwen)
|
||||
2. CloseAI集成配置已完成,可直接使用
|
||||
3. ASL开发不阻塞,先用简单调用
|
||||
4. Week 5有多个模块实践后,再抽取统一网关更合理
|
||||
|
||||
### 当前可用(Week 3 ASL开发)✅
|
||||
- ✅ DeepSeek API(直连)
|
||||
- ✅ GPT-5-Pro API(CloseAI代理)
|
||||
- ✅ Claude-4.5 API(CloseAI代理)
|
||||
- ✅ Qwen API(DashScope)
|
||||
- ✅ 4个模型的基础调用代码示例
|
||||
|
||||
### Week 5完善(LLM网关统一)
|
||||
- 统一调用接口
|
||||
- 版本分级(专业版/高级版/旗舰版)
|
||||
- 配额管理和限流
|
||||
- Token计数和计费
|
||||
- 使用记录和监控
|
||||
**璋冩暣锛?* LLM缃戝叧瀹屾暣瀹炵幇鎺ㄨ繜鍒癢eek 5 鉁?
|
||||
**鐞嗙敱锛?*
|
||||
1. 鐜版湁LLM璋冪敤宸茬粡work锛圖eepSeek銆丵wen锛?2. CloseAI闆嗘垚閰嶇疆宸插畬鎴愶紝鍙<E7B49D>洿鎺ヤ娇鐢?3. ASL寮€鍙戜笉闃诲<E99783>锛屽厛鐢ㄧ畝鍗曡皟鐢?4. Week 5鏈夊<E98F88>涓<EFBFBD>ā鍧楀疄璺靛悗锛屽啀鎶藉彇缁熶竴缃戝叧鏇村悎鐞?
|
||||
### 褰撳墠鍙<E5A2A0>敤锛圵eek 3 ASL寮€鍙戯級鉁?- 鉁?DeepSeek API锛堢洿杩烇級
|
||||
- 鉁?GPT-5-Pro API锛圕loseAI浠g悊锛?- 鉁?Claude-4.5 API锛圕loseAI浠g悊锛?- 鉁?Qwen API锛圖ashScope锛?- 鉁?4涓<34>ā鍨嬬殑鍩虹<E98DA9>璋冪敤浠g爜绀轰緥
|
||||
|
||||
### Week 5瀹屽杽锛圠LM缃戝叧缁熶竴锛?- 缁熶竴璋冪敤鎺ュ彛
|
||||
- 鐗堟湰鍒嗙骇锛堜笓涓氱増/楂樼骇鐗?鏃楄埌鐗堬級
|
||||
- 閰嶉<E996B0>绠$悊鍜岄檺娴?- Token璁℃暟鍜岃<E98D9C>璐?- 浣跨敤璁板綍鍜岀洃鎺?
|
||||
---
|
||||
|
||||
## 馃敆 鐩稿叧鏂囨。
|
||||
|
||||
@@ -1,29 +1,29 @@
|
||||
# 鏂囨。澶勭悊寮曟搸
|
||||
|
||||
> **能力定位:** 通用能力层
|
||||
> **复用率:** 86% (6个模块依赖)
|
||||
> **鑳藉姏瀹氫綅锛?* 閫氱敤鑳藉姏灞?
|
||||
> **澶嶇敤鐜囷細** 86% (6涓<EFBFBD>ā鍧椾緷璧?
|
||||
> **浼樺厛绾э細** P0
|
||||
> **状态:** ✅ 已实现(Python微服务)
|
||||
> **鐘舵€侊細** 鉁?宸插疄鐜帮紙Python寰<6E>湇鍔★級
|
||||
|
||||
---
|
||||
|
||||
## 馃搵 鑳藉姏姒傝堪
|
||||
|
||||
鏂囨。澶勭悊寮曟搸鏄<EFBFBD>钩鍙扮殑鏍稿績鍩虹<EFBFBD>鑳藉姏锛岃礋璐o細
|
||||
- 多格式文档文本提取(PDF、Docx、Txt、Excel)
|
||||
- 澶氭牸寮忔枃妗f枃鏈<EFBFBD>彁鍙栵紙PDF銆丏ocx銆乀xt銆丒xcel锛?
|
||||
- OCR澶勭悊
|
||||
- 琛ㄦ牸鎻愬彇
|
||||
- 语言检测
|
||||
- 璇<EFBFBD>█妫€娴?
|
||||
- 璐ㄩ噺璇勪及
|
||||
|
||||
---
|
||||
|
||||
## 馃搳 渚濊禆妯″潡
|
||||
|
||||
**6个模块依赖(86%复用率):**
|
||||
1. **ASL** - AI智能文献(文献PDF提取)
|
||||
**6涓<EFBFBD>ā鍧椾緷璧栵紙86%澶嶇敤鐜囷級锛?*
|
||||
1. **ASL** - AI鏅鸿兘鏂囩尞锛堟枃鐚甈DF鎻愬彇锛?
|
||||
2. **PKB** - 涓<>汉鐭ヨ瘑搴擄紙鐭ヨ瘑搴撴枃妗d笂浼狅級
|
||||
3. **DC** - 数据清洗(Excel/Docx数据导入)
|
||||
3. **DC** - 鏁版嵁娓呮礂锛圗xcel/Docx鏁版嵁瀵煎叆锛?
|
||||
4. **SSA** - 鏅鸿兘缁熻<E7BC81>鍒嗘瀽锛堟暟鎹<E69A9F><E98EB9>鍏ワ級
|
||||
5. **ST** - 缁熻<E7BC81>鍒嗘瀽宸ュ叿锛堟暟鎹<E69A9F><E98EB9>鍏ワ級
|
||||
6. **RVW** - 绋夸欢瀹℃煡锛堢ǹ浠舵枃妗f彁鍙栵級
|
||||
@@ -35,36 +35,36 @@
|
||||
### 1. PDF鎻愬彇
|
||||
- **Nougat**锛氳嫳鏂囧<E98F82>鏈<EFBFBD><E98F88>鏂囷紙楂樿川閲忥級
|
||||
- **PyMuPDF**锛氫腑鏂嘝DF + 鍏滃簳鏂规<E98F82>锛堝揩閫燂級
|
||||
- **语言检测**:自动识别中英文
|
||||
- **质量评估**:提取质量评分
|
||||
- **璇<EFBFBD>█妫€娴?*锛氳嚜鍔ㄨ瘑鍒<E79891>腑鑻辨枃
|
||||
- **璐ㄩ噺璇勪及**锛氭彁鍙栬川閲忚瘎鍒?
|
||||
|
||||
### 2. Docx鎻愬彇
|
||||
- **Mammoth**锛氳浆Markdown
|
||||
- **python-docx**锛氱粨鏋勫寲璇诲彇
|
||||
|
||||
### 3. Txt鎻愬彇
|
||||
- **多编码支持**:UTF-8、GBK等
|
||||
- **chardet**:自动检测编码
|
||||
- **澶氱紪鐮佹敮鎸?*锛歎TF-8銆丟BK绛?
|
||||
- **chardet**锛氳嚜鍔ㄦ<EFBFBD>娴嬬紪鐮?
|
||||
|
||||
### 4. Excel澶勭悊
|
||||
- **openpyxl**锛氳<E9949B>鍙朎xcel
|
||||
- **pandas**:数据处理
|
||||
- **pandas**锛氭暟鎹<EFBFBD><EFBFBD>鐞?
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 技术架构
|
||||
## 馃彈锔?鎶€鏈<E282AC>灦鏋?
|
||||
|
||||
**Python寰<6E>湇鍔★紙FastAPI锛夛細**
|
||||
```
|
||||
extraction_service/
|
||||
├── main.py (509行) - FastAPI主服务
|
||||
鈹溾攢鈹€ main.py (509琛? - FastAPI涓绘湇鍔?
|
||||
鈹溾攢鈹€ services/
|
||||
│ ├── pdf_extractor.py (242行) - PDF提取总协调
|
||||
│ ├── pdf_processor.py (280行) - PyMuPDF实现
|
||||
│ ├── language_detector.py (120行) - 语言检测
|
||||
│ ├── nougat_extractor.py (242行) - Nougat实现
|
||||
│ ├── docx_extractor.py (253行) - Docx提取
|
||||
│ └── txt_extractor.py (316行) - Txt提取(多编码)
|
||||
鈹? 鈹溾攢鈹€ pdf_extractor.py (242琛? - PDF鎻愬彇鎬诲崗璋?
|
||||
鈹? 鈹溾攢鈹€ pdf_processor.py (280琛? - PyMuPDF瀹炵幇
|
||||
鈹? 鈹溾攢鈹€ language_detector.py (120琛? - 璇<EFBFBD>█妫€娴?
|
||||
鈹? 鈹溾攢鈹€ nougat_extractor.py (242琛? - Nougat瀹炵幇
|
||||
鈹? 鈹溾攢鈹€ docx_extractor.py (253琛? - Docx鎻愬彇
|
||||
鈹? 鈹斺攢鈹€ txt_extractor.py (316琛? - Txt鎻愬彇锛堝<EFBFBD>缂栫爜锛?
|
||||
鈹斺攢鈹€ requirements.txt
|
||||
```
|
||||
|
||||
@@ -77,7 +77,7 @@ POST /api/extract/pdf - PDF文本提取
|
||||
POST /api/extract/docx - Docx鏂囨湰鎻愬彇
|
||||
POST /api/extract/txt - Txt鏂囨湰鎻愬彇
|
||||
POST /api/extract/excel - Excel琛ㄦ牸鎻愬彇
|
||||
GET /health - 健康检查
|
||||
GET /health - 鍋ュ悍妫€鏌?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -1,61 +1,61 @@
|
||||
# RAG寮曟搸
|
||||
|
||||
> **能力定位:** 通用能力层
|
||||
> **复用率:** 43% (3个模块依赖)
|
||||
> **鑳藉姏瀹氫綅锛?* 閫氱敤鑳藉姏灞?
|
||||
> **澶嶇敤鐜囷細** 43% (3涓<EFBFBD>ā鍧椾緷璧?
|
||||
> **浼樺厛绾э細** P1
|
||||
> **状态:** ✅ 已实现(基于Dify)
|
||||
> **鐘舵€侊細** 鉁?宸插疄鐜帮紙鍩轰簬Dify锛?
|
||||
|
||||
---
|
||||
|
||||
## 馃搵 鑳藉姏姒傝堪
|
||||
|
||||
RAG引擎负责:
|
||||
- 向量化存储(Embedding)
|
||||
- 语义检索(Semantic Search)
|
||||
- 检索增强生成(RAG)
|
||||
- Rerank重排序
|
||||
RAG寮曟搸璐熻矗锛?
|
||||
- 鍚戦噺鍖栧瓨鍌<EFBFBD>紙Embedding锛?
|
||||
- 璇<EFBFBD>箟妫€绱<EFBFBD>紙Semantic Search锛?
|
||||
- 妫€绱㈠<EFBFBD>寮虹敓鎴愶紙RAG锛?
|
||||
- Rerank閲嶆帓搴?
|
||||
|
||||
---
|
||||
|
||||
## 馃搳 渚濊禆妯″潡
|
||||
|
||||
**3个模块依赖(43%复用率):**
|
||||
**3涓<EFBFBD>ā鍧椾緷璧栵紙43%澶嶇敤鐜囷級锛?*
|
||||
1. **AIA** - AI鏅鸿兘闂<E58598>瓟锛園鐭ヨ瘑搴撻棶绛旓級
|
||||
2. **ASL** - AI鏅鸿兘鏂囩尞锛堟枃鐚<E69E83>唴瀹规<E780B9>绱<EFBFBD>級
|
||||
3. **PKB** - 个人知识库(RAG问答)
|
||||
3. **PKB** - 涓<EFBFBD>汉鐭ヨ瘑搴擄紙RAG闂<EFBFBD>瓟锛?
|
||||
|
||||
---
|
||||
|
||||
## 馃挕 鏍稿績鍔熻兘
|
||||
|
||||
### 1. 向量化存储
|
||||
### 1. 鍚戦噺鍖栧瓨鍌?
|
||||
- 鍩轰簬Dify骞冲彴
|
||||
- Qdrant向量数据库(Dify内置)
|
||||
- Qdrant鍚戦噺鏁版嵁搴擄紙Dify鍐呯疆锛?
|
||||
|
||||
### 2. 语义检索
|
||||
- Top-K检索
|
||||
- 相关度评分
|
||||
- 多知识库联合检索
|
||||
### 2. 璇<EFBFBD>箟妫€绱?
|
||||
- Top-K妫€绱?
|
||||
- 鐩稿叧搴﹁瘎鍒?
|
||||
- 澶氱煡璇嗗簱鑱斿悎妫€绱?
|
||||
|
||||
### 3. RAG闂<47>瓟
|
||||
- 检索 + 生成
|
||||
- 智能引用系统(100%准确溯源)
|
||||
- 妫€绱?+ 鐢熸垚
|
||||
- 鏅鸿兘寮曠敤绯荤粺锛?00%鍑嗙‘婧<E28098>簮锛?
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 技术架构
|
||||
## 馃彈锔?鎶€鏈<E282AC>灦鏋?
|
||||
|
||||
**基于Dify平台:**
|
||||
**鍩轰簬Dify骞冲彴锛?*
|
||||
```typescript
|
||||
// DifyClient灏佽<E7818F>
|
||||
interface RAGEngine {
|
||||
// 创建知识库
|
||||
// 鍒涘缓鐭ヨ瘑搴?
|
||||
createDataset(name: string): Promise<string>;
|
||||
|
||||
// 涓婁紶鏂囨。
|
||||
uploadDocument(datasetId: string, file: File): Promise<string>;
|
||||
|
||||
// 语义检索
|
||||
// 璇<EFBFBD>箟妫€绱?
|
||||
search(datasetId: string, query: string, topK?: number): Promise<SearchResult[]>;
|
||||
|
||||
// RAG闂<47>瓟
|
||||
@@ -68,19 +68,19 @@ interface RAGEngine {
|
||||
## 馃搱 浼樺寲鎴愭灉
|
||||
|
||||
**妫€绱㈠弬鏁颁紭鍖栵細**
|
||||
| 指标 | 优化前 | 优化后 | 提升 |
|
||||
| 鎸囨爣 | 浼樺寲鍓?| 浼樺寲鍚?| 鎻愬崌 |
|
||||
|------|--------|--------|------|
|
||||
| 检索数量 | 3 chunks | 15 chunks | 5倍 |
|
||||
| Chunk大小 | 500 tokens | 1500 tokens | 3倍 |
|
||||
| 总覆盖 | 1,500 tokens | 22,500 tokens | 15倍 |
|
||||
| 覆盖率 | ~5% | ~40-50% | 8-10倍 |
|
||||
| 妫€绱㈡暟閲?| 3 chunks | 15 chunks | 5鍊?|
|
||||
| Chunk澶у皬 | 500 tokens | 1500 tokens | 3鍊?|
|
||||
| 鎬昏<EFBFBD>鐩?| 1,500 tokens | 22,500 tokens | 15鍊?|
|
||||
| 瑕嗙洊鐜?| ~5% | ~40-50% | 8-10鍊?|
|
||||
|
||||
---
|
||||
|
||||
## 馃敆 鐩稿叧鏂囨。
|
||||
|
||||
- [閫氱敤鑳藉姏灞傛€昏<EFBFBD>](../README.md)
|
||||
- [Dify集成文档](../../00-系统总体设计/03-数据库架构说明.md)
|
||||
- [Dify闆嗘垚鏂囨。](../../00-绯荤粺鎬讳綋璁捐<EFBFBD>/03-鏁版嵁搴撴灦鏋勮<E98F8B>鏄?md)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
# æ•°æ<C2B0>®ETL引擎
|
||||
|
||||
> **能力定位:** 通用能力层
|
||||
> **复用率:** 29% (2个模块依赖)
|
||||
> **能力定ä½<EFBFBD>ï¼?* 通用能力å±?
|
||||
> **å¤<EFBFBD>用率:** 29% (2个模å<C2A1>—ä¾<C3A4>èµ?
|
||||
> **优先级:** P2
|
||||
> **状态:** ⏳ 待实现
|
||||
> **状æ€<EFBFBD>:** â<>?待实çŽ?
|
||||
|
||||
---
|
||||
|
||||
## 📋 能力概述
|
||||
|
||||
数据ETL引擎负责:
|
||||
æ•°æ<EFBFBD>®ETL引擎负责ï¼?
|
||||
- Excel多表JOIN
|
||||
- æ•°æ<C2B0>®æ¸…æ´—
|
||||
- æ•°æ<C2B0>®è½¬æ<C2AC>¢
|
||||
@@ -19,9 +19,9 @@
|
||||
|
||||
## 📊 ä¾<C3A4>赖模å<C2A1>—
|
||||
|
||||
**2个模块依赖(29%复用率):**
|
||||
**2个模å<EFBFBD>—ä¾<EFBFBD>赖(29%å¤<C3A5>用率)ï¼?*
|
||||
1. **DC** - æ•°æ<C2B0>®æ¸…æ´—æ•´ç<C2B4>†ï¼ˆæ ¸å¿ƒä¾<C3A4>赖)
|
||||
2. **SSA** - 智能统计分析(数据预处理)
|
||||
2. **SSA** - 智能统计分æž<EFBFBD>(数æ<EFBFBD>®é¢„处ç<EFBFBD>†ï¼?
|
||||
|
||||
---
|
||||
|
||||
@@ -33,21 +33,21 @@
|
||||
- GROUP BYè<59>šå<C5A1>ˆ
|
||||
|
||||
### 2. æ•°æ<C2B0>®æ¸…æ´—
|
||||
- 缺失值处理
|
||||
- 重复值处理
|
||||
- 异常值检测
|
||||
- 缺失值处ç<EFBFBD>?
|
||||
- é‡<EFBFBD>å¤<EFBFBD>值处ç<EFBFBD>?
|
||||
- 异常值检�
|
||||
|
||||
### 3. æ•°æ<C2B0>®è½¬æ<C2AC>¢
|
||||
- 类型转æ<C2AC>¢
|
||||
- 格式标准化
|
||||
- æ ¼å¼<EFBFBD>æ ‡å‡†åŒ?
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 技术方案
|
||||
## ðŸ<EFBFBD>—ï¸?技术方æ¡?
|
||||
|
||||
### 云端版(最优)
|
||||
```python
|
||||
# 基于Polars(性能极高)
|
||||
# 基于Polars(性能æž<EFBFBD>高ï¼?
|
||||
class ETLEngine:
|
||||
def read_excel(self, files: List[File]) -> List[DataFrame]
|
||||
def join(self, dfs: List[DataFrame], keys: List[str]) -> DataFrame
|
||||
@@ -55,7 +55,7 @@ class ETLEngine:
|
||||
def export(self, df: DataFrame, format: str) -> bytes
|
||||
```
|
||||
|
||||
### 单机版(兼容)
|
||||
### å<EFBFBD>•机版(兼容ï¼?
|
||||
```python
|
||||
# 基于SQLite(内å˜å<CB9C>‹å¥½ï¼‰
|
||||
# 分å<E280A0>—读å<C2BB>–,数æ<C2B0>®åº“引擎处ç<E2809E>†JOIN
|
||||
|
||||
@@ -1,25 +1,25 @@
|
||||
# 医å¦NLP引擎
|
||||
|
||||
> **能力定位:** 通用能力层
|
||||
> **复用率:** 14% (1个模块依赖)
|
||||
> **能力定ä½<EFBFBD>ï¼?* 通用能力å±?
|
||||
> **å¤<EFBFBD>用率:** 14% (1个模å<C2A1>—ä¾<C3A4>èµ?
|
||||
> **优先级:** P2
|
||||
> **状态:** ⏳ 待实现
|
||||
> **状æ€<EFBFBD>:** â<>?待实çŽ?
|
||||
|
||||
---
|
||||
|
||||
## 📋 能力概述
|
||||
|
||||
医学NLP引擎负责:
|
||||
- 医学实体识别(NER)
|
||||
- 医学术语标准化
|
||||
医å¦NLP引擎负责ï¼?
|
||||
- 医å¦å®žä½“识别(NERï¼?
|
||||
- åŒ»å¦æœ¯è¯æ ‡å‡†åŒ?
|
||||
- 疾病/è<>¯ç‰©è¯†åˆ«
|
||||
|
||||
---
|
||||
|
||||
## 📊 ä¾<C3A4>赖模å<C2A1>—
|
||||
|
||||
**1个模块依赖(14%复用率):**
|
||||
1. **DC** - 数据清洗整理(病例数据NER提取)
|
||||
**1个模å<EFBFBD>—ä¾<EFBFBD>赖(14%å¤<C3A5>用率)ï¼?*
|
||||
1. **DC** - æ•°æ<EFBFBD>®æ¸…æ´—æ•´ç<EFBFBD>†ï¼ˆç—…例数æ<EFBFBD>®NERæ<EFBFBD><EFBFBD>å<EFBFBD>–ï¼?
|
||||
|
||||
---
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
- 手术识别
|
||||
- TNM分期æ<C5B8><C3A6>å<EFBFBD>–
|
||||
|
||||
### 2. 术语标准化
|
||||
### 2. æœ¯è¯æ ‡å‡†åŒ?
|
||||
- ICDç¼–ç <C3A7>
|
||||
- ATCç¼–ç <C3A7>
|
||||
|
||||
@@ -41,15 +41,15 @@
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 技术方案
|
||||
## ðŸ<EFBFBD>—ï¸?技术方æ¡?
|
||||
|
||||
### 云端版(高准确率)
|
||||
### 云端版(高准确率�
|
||||
```python
|
||||
# 基于LLM API(Claude/GPT)
|
||||
# JSON Mode结构化输出
|
||||
# 基于LLM API(Claude/GPT�
|
||||
# JSON Mode结构化输�
|
||||
```
|
||||
|
||||
### 单机版(隐私优先)
|
||||
### å<EFBFBD>•机版(éš<EFBFBD>ç§<EFBFBD>优先ï¼?
|
||||
```python
|
||||
# 基于spaCy + åŒ»å¦æ¨¡åž‹
|
||||
# 100%本地è¿<C3A8>行
|
||||
|
||||
@@ -1,53 +1,53 @@
|
||||
# Postgres-Only 蠑よュ・莉サ蜉。螟<EFBDA1>炊謖<E7828A>漉
|
||||
|
||||
> **文档版本:** v1.0
|
||||
> **创建日期:** 2025-12-22
|
||||
> **譁<EFBFBD>。」迚域悽<EFBFBD>?* v1.0
|
||||
> **蛻帛サコ譌・譛滂シ?* 2025-12-22
|
||||
> **扈エ謚、閠<EFBDA4>シ<EFBFBD>** 蟷ウ蜿ー譫カ譫<EFBDB6>屬髦<E5B1AC>
|
||||
> **适用场景:** 长时间任务(>30秒)、大文件处理、后台Worker
|
||||
> **騾ら畑蝨コ譎ッ<EFBFBD>?* 髟ソ譌カ髣エ莉サ蜉。<E89C89><EFBDA1>>30遘抵シ峨∝、ァ譁<EFBDA7>サカ螟<EFBDB6>炊縲∝錘蜿ーWorker
|
||||
> **蜿り<E3828A>ョ樒鴫<E6A892><E9B4AB>** DC Tool C Excel隗」譫舌、SL譁<4C>鍵遲幃峨.C Tool B謨ー謐ョ謠仙叙
|
||||
|
||||
---
|
||||
|
||||
## <20>搭 讎りソー
|
||||
|
||||
本文档基于 **DC Tool C Excel解析功能** 的完整实践,总结 Postgres-Only 架构下异步任务处理的标准模式。
|
||||
譛ャ譁<EFBFBD>。」蝓コ莠?**DC Tool C Excel隗」譫仙粥閭ス** 逧<>ョ梧紛螳櫁キオ<EFBDB7>梧サ扈<EFBDBB> Postgres-Only 譫カ譫<EFBDB6>ク句シよュ・莉サ蜉。螟<EFBDA1>炊逧<E7828A><E980A7><EFBFBD>㊥讓。蠑上?
|
||||
|
||||
### 核心价值
|
||||
### 譬ク蠢<EFBFBD>サキ蛟?
|
||||
|
||||
1. ✅ **避免HTTP超时**:上传接口3秒返回,解析在后台完成(30-60秒)
|
||||
2. ✅ **用户体验优秀**:实时进度反馈,不需要傻等
|
||||
3. ✅ **符合云原生规范**:Platform-Only模式,pg-boss队列
|
||||
4. ✅ **性能优化**:clean data缓存,避免重复计算(-99%耗时)
|
||||
1. 笨?**驕ソ蜈工TTP雜<50>慮**<2A>壻ク贋シ<E8B48B>謗・蜿?遘定ソ泌屓<E6B38C>瑚ァ」譫仙惠蜷主床螳梧<E89EB3><E6A2A7><EFBFBD>30-60遘抵シ<EFBFBD>
|
||||
2. 笨?**逕ィ謌キ菴馴ェ御シ倡ァ**<2A>壼ョ樊慮霑帛コヲ蜿埼ヲ茨シ御ク埼怙隕∝そ遲?
|
||||
3. 笨?**隨ヲ蜷井コ大次逕溯ァ<E6BAAF><EFBDA7>?*<2A>啀latform-Only讓。蠑擾シ継g-boss髦溷<EFBFBD>
|
||||
4. 笨?**諤ァ閭ス莨伜喧**<2A>喞lean data郛灘ュ假シ碁∩蜈埼㍾螟崎ョ。邂暦シ<E69AA6>-99%閠玲慮<E78EB2>?
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 架构设计
|
||||
## <EFBFBD>女<EFBFBD>?譫カ譫<EFBDB6>ョセ隶。
|
||||
|
||||
### 荳牙アよ楔譫<E6A594>
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 前端层(React + React Query) │
|
||||
│ - 上传文件(立即返回 sessionId + jobId) │
|
||||
│ - 轮询状态(useQuery + refetchInterval,自动串行) │
|
||||
│ - 监听 status='ready',加载数据 │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
↓ HTTP
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 后端层(Fastify + Prisma) │
|
||||
│ - 快速上传到 OSS(2-3秒) │
|
||||
│ - 创建 Session(状态:processing) │
|
||||
│ - 推送任务到 pg-boss(立即返回) │
|
||||
│ - 提供状态查询 API │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
↓ pg-boss
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Worker层(pg-boss + Platform层) │
|
||||
│ - 从队列取任务(自动串行) │
|
||||
│ - 执行耗时操作(解析、清洗、统计) │
|
||||
│ - 保存结果(clean data 到 OSS) │
|
||||
│ - 更新 Session(填充元数据) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
笏娯楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
笏? 蜑咲ォッ螻ゑシ<E38291>eact + React Query<EFBFBD>? 笏?
|
||||
笏? - 荳贋シ<EFBFBD>譁<EFBFBD>サカ<EFBFBD>育ォ句叉霑泌<EFBFBD>?sessionId + jobId<EFBFBD>? 笏?
|
||||
笏? - 霓ョ隸「迥カ諤<EFBFBD>シ<EFBFBD>seQuery + refetchInterval<EFBFBD>瑚<EFBFBD>蜉ィ荳イ陦鯉シ<EFBFBD> 笏?
|
||||
笏? - 逶大成 status='ready'<EFBFBD>悟刈霓ス謨ー謐? 笏?
|
||||
笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
竊?HTTP
|
||||
笏娯楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
笏? 蜷守ォッ螻ゑシ<E38291>astify + Prisma<EFBFBD>? 笏?
|
||||
笏? - 蠢ォ騾滉ク贋シ<EFBFBD>蛻ー OSS<53>?-3遘抵シ<E68AB5> 笏?
|
||||
笏? - 蛻帛サコ Session<EFBFBD>育憾諤<EFBFBD>シ嗔rocessing<EFBFBD>? 笏?
|
||||
笏? - 謗ィ騾∽ササ蜉。蛻ー pg-boss<73>育ォ句叉霑泌屓<E6B38C><E5B193> 笏?
|
||||
笏? - 謠蝉セ帷憾諤∵衍隸?API 笏?
|
||||
笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
竊?pg-boss
|
||||
笏娯楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
笏? Worker螻ゑシ<EFBFBD>g-boss + Platform螻ゑシ<EFBFBD> 笏?
|
||||
笏? - 莉朱弌蛻怜叙莉サ蜉。<EFBFBD>郁<EFBFBD>蜉ィ荳イ陦鯉シ<EFBFBD> 笏?
|
||||
笏? - 謇ァ陦瑚玲慮謫堺ス懶シ郁ァ」譫舌∵ク<EFBFBD>エ励∫サ溯ョ。<EFBFBD><EFBFBD> 笏?
|
||||
笏? - 菫晏ュ倡サ捺棡<EFBFBD><EFBFBD>lean data 蛻?OSS<EFBFBD>? 笏?
|
||||
笏? - 譖エ譁ー Session<EFBFBD>亥。ォ蜈<EFBFBD><EFBFBD>謨ー謐ョ<EFBFBD>? 笏?
|
||||
笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||||
```
|
||||
|
||||
---
|
||||
@@ -63,15 +63,15 @@ model YourBusinessTable {
|
||||
userId String
|
||||
fileKey String // OSS蜴溷ァ区枚莉カ
|
||||
|
||||
// ✅ 性能优化:保存处理结果
|
||||
// 笨?諤ァ閭ス莨伜喧<E4BC9C>壻ソ晏ュ伜、<E4BC9C>炊扈捺<E68988>?
|
||||
cleanDataKey String? // 貂<>エ<EFBFBD>/螟<>炊蜷守噪謨ー謐ョ<E8AC90>磯∩蜈埼㍾螟崎ョ。邂暦シ<E69AA6>
|
||||
|
||||
// 数据元信息(异步填充)
|
||||
// 謨ー謐ョ蜈<EFBFBD>ソ。諱ッ<EFBFBD>亥シよュ・蝪ォ蜈<EFBFBD>シ?
|
||||
totalRows Int?
|
||||
totalCols Int?
|
||||
columns Json?
|
||||
|
||||
// 时间戳
|
||||
// 譌カ髣エ謌?
|
||||
createdAt DateTime
|
||||
updatedAt DateTime
|
||||
expiresAt DateTime
|
||||
@@ -80,13 +80,13 @@ model YourBusinessTable {
|
||||
}
|
||||
```
|
||||
|
||||
**关键点**:
|
||||
- ❌ 不要添加 `status`、`progress`、`errorMessage` 等任务管理字段
|
||||
- ✅ 这些字段由 pg-boss 的 `job` 表管理
|
||||
**蜈ウ髞ョ轤?*<2A>?
|
||||
- 笶?荳崎ヲ∵キサ蜉<EFBDBB> `status`縲~progress`縲~errorMessage` 遲我ササ蜉。邂。逅<EFBFBD>ュ玲ョ?
|
||||
- 笨?霑吩コ帛ュ玲ョオ逕?pg-boss 逧?`job` 陦ィ邂。逅?
|
||||
|
||||
---
|
||||
|
||||
### 步骤2:Service层 - 快速上传+推送任务
|
||||
### 豁・鬪、2<EFBFBD>售ervice螻?- 蠢ォ騾滉ク贋シ?謗ィ騾∽ササ蜉?
|
||||
|
||||
```typescript
|
||||
// backend/src/modules/your-module/services/YourService.ts
|
||||
@@ -97,13 +97,13 @@ import { prisma } from '@/config/database';
|
||||
|
||||
export class YourService {
|
||||
/**
|
||||
* 创建任务并推送到队列(Postgres-Only架构)
|
||||
* 蛻帛サコ莉サ蜉。蟷カ謗ィ騾∝芦髦溷<EFBFBD><EFBFBD><EFBFBD>ostgres-Only譫カ譫<EFBFBD>シ?
|
||||
*
|
||||
* ✅ Platform-Only 模式:
|
||||
* - 立即上传文件到 OSS
|
||||
* - 创建业务记录(元数据为null)
|
||||
* 笨?Platform-Only 讓。蠑擾シ?
|
||||
* - 遶句叉荳贋シ<EFBFBD>譁<EFBFBD>サカ蛻?OSS
|
||||
* - 蛻帛サコ荳壼苅隶ー蠖包シ亥<EFBFBD>謨ー謐ョ荳コnull<EFBFBD>?
|
||||
* - 謗ィ騾∽ササ蜉。蛻ー髦溷<E9ABA6>
|
||||
* - 立即返回(不阻塞请求)
|
||||
* - 遶句叉霑泌屓<EFBFBD>井ク埼仆蝪櫁ッキ豎ゑシ?
|
||||
*/
|
||||
async createTask(userId: string, fileName: string, fileBuffer: Buffer) {
|
||||
// 1. 鬪瑚ッ∵枚莉カ
|
||||
@@ -111,34 +111,34 @@ export class YourService {
|
||||
throw new Error('譁<>サカ螟ェ螟ァ');
|
||||
}
|
||||
|
||||
// 2. ⚡ 立即上传到 OSS(2-3秒)
|
||||
// 2. 笞?遶句叉荳贋シ<E8B48B>蛻?OSS<53>?-3遘抵シ<E68AB5>
|
||||
const fileKey = `path/${userId}/${Date.now()}-${fileName}`;
|
||||
await storage.upload(fileKey, fileBuffer);
|
||||
|
||||
// 3. ⚡ 创建业务记录(元数据为null,等Worker填充)
|
||||
// 3. 笞?蛻帛サコ荳壼苅隶ー蠖包シ亥<EFBDBC>謨ー謐ョ荳コnull<6C>檎ュ姥orker蝪ォ蜈<EFBDAB>シ?
|
||||
const record = await prisma.yourTable.create({
|
||||
data: {
|
||||
userId,
|
||||
fileName,
|
||||
fileKey,
|
||||
// ⚠️ 处理结果字段为 null
|
||||
// 笞<EFBFBD><EFBFBD><EFBFBD> 螟<>炊扈捺棡蟄玲ョオ荳?null
|
||||
totalRows: null,
|
||||
columns: null,
|
||||
expiresAt: new Date(Date.now() + 10 * 60 * 1000),
|
||||
},
|
||||
});
|
||||
|
||||
// 4. ⚡ 推送任务到 pg-boss(Platform-Only)
|
||||
// 4. 笞?謗ィ騾∽ササ蜉。蛻ー pg-boss<EFBFBD><EFBFBD>latform-Only<EFBFBD>?
|
||||
const job = await jobQueue.push('your_module_process', {
|
||||
recordId: record.id,
|
||||
fileKey,
|
||||
userId,
|
||||
});
|
||||
|
||||
// 5. ⚡ 立即返回(总耗时<3秒)
|
||||
// 5. 笞?遶句叉霑泌屓<E6B38C>域サ閠玲慮<3遘抵シ<E68AB5>
|
||||
return {
|
||||
...record,
|
||||
jobId: job.id, // ✅ 返回 jobId 供前端轮询
|
||||
jobId: job.id, // 笨?霑泌屓 jobId 萓帛燕遶ッ霓ョ隸?
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -146,7 +146,7 @@ export class YourService {
|
||||
|
||||
---
|
||||
|
||||
### 步骤3:Worker层 - 后台处理
|
||||
### 豁・鬪、3<EFBFBD>啗orker螻?- 蜷主床螟<E5BA8A>炊
|
||||
|
||||
```typescript
|
||||
// backend/src/modules/your-module/workers/yourWorker.ts
|
||||
@@ -163,7 +163,7 @@ interface YourJob {
|
||||
}
|
||||
|
||||
/**
|
||||
* 注册 Worker 到队列
|
||||
* 豕ィ蜀<EFBFBD> Worker 蛻ー髦溷<EFBFBD>?
|
||||
*/
|
||||
export function registerYourWorker() {
|
||||
logger.info('[YourWorker] Registering worker');
|
||||
@@ -175,14 +175,14 @@ export function registerYourWorker() {
|
||||
logger.info('[YourWorker] Processing job', { jobId: job.id, recordId });
|
||||
|
||||
try {
|
||||
// 1. 从 OSS 下载文件
|
||||
// 1. 莉?OSS 荳玖スス譁<EFBFBD>サカ
|
||||
const buffer = await storage.download(fileKey);
|
||||
|
||||
// 2. 謇ァ陦瑚玲慮謫堺ス懶シ郁ァ」譫舌∝、<E2889D>炊縲∬ョ。邂暦シ<E69AA6>
|
||||
const result = await yourLongTimeProcess(buffer);
|
||||
const { processedData, totalRows, columns } = result;
|
||||
|
||||
// 3. ✅ 保存处理结果到 OSS(避免重复计算)
|
||||
// 3. 笨?菫晏ュ伜、<E4BC9C>炊扈捺棡蛻?OSS<53>磯∩蜈埼㍾螟崎ョ。邂暦シ<E69AA6>
|
||||
const cleanDataKey = `${fileKey}_clean.json`;
|
||||
const cleanDataBuffer = Buffer.from(JSON.stringify(processedData), 'utf-8');
|
||||
await storage.upload(cleanDataKey, cleanDataBuffer);
|
||||
@@ -191,36 +191,36 @@ export function registerYourWorker() {
|
||||
size: `${(cleanDataBuffer.length / 1024).toFixed(2)} KB`
|
||||
});
|
||||
|
||||
// 4. 更新业务记录(填充元数据)
|
||||
// 4. 譖エ譁ー荳壼苅隶ー蠖包シ亥。ォ蜈<EFBFBD><EFBFBD>謨ー謐ョ<EFBFBD>?
|
||||
await prisma.yourTable.update({
|
||||
where: { id: recordId },
|
||||
data: {
|
||||
cleanDataKey, // ✅ 保存 clean data 位置
|
||||
cleanDataKey, // 笨?菫晏ュ<E6998F> clean data 菴咲スョ
|
||||
totalRows,
|
||||
columns,
|
||||
updatedAt: new Date(),
|
||||
},
|
||||
});
|
||||
|
||||
logger.info('[YourWorker] ✅ Job completed', { jobId: job.id });
|
||||
logger.info('[YourWorker] 笨?Job completed', { jobId: job.id });
|
||||
|
||||
return { success: true, recordId, totalRows };
|
||||
} catch (error: any) {
|
||||
logger.error('[YourWorker] ❌ Job failed', {
|
||||
logger.error('[YourWorker] 笶?Job failed', {
|
||||
jobId: job.id,
|
||||
error: error.message
|
||||
});
|
||||
throw error; // 让 pg-boss 处理重试
|
||||
throw error; // 隶?pg-boss 螟<EFBFBD>炊驥崎ッ<EFBFBD>
|
||||
}
|
||||
});
|
||||
|
||||
logger.info('[YourWorker] ✅ Worker registered: your_module_process');
|
||||
logger.info('[YourWorker] 笨?Worker registered: your_module_process');
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 步骤4:Controller层 - 状态查询API
|
||||
### 豁・鬪、4<EFBFBD>咾ontroller螻?- 迥カ諤∵衍隸「API
|
||||
|
||||
```typescript
|
||||
// backend/src/modules/your-module/controllers/YourController.ts
|
||||
@@ -229,10 +229,10 @@ import { jobQueue } from '@/common/jobs';
|
||||
|
||||
export class YourController {
|
||||
/**
|
||||
* 获取任务状态(Platform-Only模式)
|
||||
* 闔キ蜿紋ササ蜉。迥カ諤<EFBFBD>シ<EFBFBD>latform-Only讓。蠑擾シ?
|
||||
*
|
||||
* GET /api/v1/your-module/tasks/:id/status
|
||||
* Query: jobId (可选)
|
||||
* Query: jobId (蜿ッ騾?
|
||||
*/
|
||||
async getTaskStatus(request, reply) {
|
||||
const { id: recordId } = request.params;
|
||||
@@ -244,18 +244,18 @@ export class YourController {
|
||||
});
|
||||
|
||||
if (!record) {
|
||||
return reply.code(404).send({ success: false, error: '记录不存在' });
|
||||
return reply.code(404).send({ success: false, error: '隶ー蠖穂ク榊ュ伜<EFBFBD>? });
|
||||
}
|
||||
|
||||
// 2. 判断状态
|
||||
// - 如果 totalRows 不为 null,说明处理完成
|
||||
// - 否则查询 job 状态
|
||||
// 2. 蛻、譁ュ迥カ諤?
|
||||
// - 螯よ棡 totalRows 荳堺クコ null<EFBFBD>瑚ッエ譏主、<EFBFBD>炊螳梧<EFBFBD>?
|
||||
// - 蜷ヲ蛻呎衍隸「 job 迥カ諤?
|
||||
if (record.totalRows !== null) {
|
||||
return reply.send({
|
||||
success: true,
|
||||
data: {
|
||||
recordId,
|
||||
status: 'ready', // ✅ 处理完成
|
||||
status: 'ready', // 笨?螟<>炊螳梧<E89EB3>
|
||||
progress: 100,
|
||||
record,
|
||||
},
|
||||
@@ -274,7 +274,7 @@ export class YourController {
|
||||
});
|
||||
}
|
||||
|
||||
// 4. 从 pg-boss 查询 job 状态
|
||||
// 4. 莉?pg-boss 譟・隸「 job 迥カ諤?
|
||||
const job = await jobQueue.getJob(jobId);
|
||||
|
||||
const status = job?.status === 'completed' ? 'ready' :
|
||||
@@ -299,7 +299,7 @@ export class YourController {
|
||||
|
||||
---
|
||||
|
||||
### 步骤5:前端 - React Query 轮询
|
||||
### 豁・鬪、5<EFBFBD>壼燕遶?- React Query 霓ョ隸「
|
||||
|
||||
```typescript
|
||||
// frontend-v2/src/modules/your-module/hooks/useTaskStatus.ts
|
||||
@@ -308,12 +308,12 @@ import { useQuery } from '@tanstack/react-query';
|
||||
import * as api from '../api';
|
||||
|
||||
/**
|
||||
* 任务状态轮询 Hook
|
||||
* 莉サ蜉。迥カ諤∬スョ隸?Hook
|
||||
*
|
||||
* 特点:
|
||||
* 迚ケ轤ケ<EFBFBD>?
|
||||
* - 閾ェ蜉ィ荳イ陦瑚スョ隸「<E99AB8><EFBDA2>eact Query 蜀<>スョ髦イ蟷カ蜿托シ<E68998>
|
||||
* - 自动清理(组件卸载时停止)
|
||||
* - 条件停止(完成/失败时自动停止)
|
||||
* - 閾ェ蜉ィ貂<EFBFBD>炊<EFBFBD>育サ<EFBFBD>サカ蜊ク霓ス譌カ蛛懈ュ「<EFBFBD>?
|
||||
* - 譚。莉カ蛛懈ュ「<EFBFBD>亥ョ梧<EFBFBD>?螟ア雍・譌カ閾ェ蜉ィ蛛懈ュ「<EFBDAD><EFBDA2>
|
||||
*/
|
||||
export function useTaskStatus({
|
||||
recordId,
|
||||
@@ -327,15 +327,15 @@ export function useTaskStatus({
|
||||
refetchInterval: (query) => {
|
||||
const status = query.state.data?.data?.status;
|
||||
|
||||
// ✅ 完成或失败时停止轮询
|
||||
// 笨?螳梧<E89EB3>謌門、ア雍・譌カ蛛懈ュ「霓ョ隸「
|
||||
if (status === 'ready' || status === 'error') {
|
||||
return false;
|
||||
}
|
||||
|
||||
// ✅ 处理中时每2秒轮询(自动串行)
|
||||
// 笨?螟<>炊荳ュ譌カ豈?遘定スョ隸「<E99AB8>郁<EFBFBD>蜉ィ荳イ陦鯉シ?
|
||||
return 2000;
|
||||
},
|
||||
staleTime: 0, // 始终视为过时,确保轮询
|
||||
staleTime: 0, // 蟋狗サ郁ァ<EFBFBD>クコ霑<EFBFBD>慮<EFBFBD>檎。ョ菫晁スョ隸?
|
||||
retry: 1,
|
||||
});
|
||||
|
||||
@@ -356,7 +356,7 @@ export function useTaskStatus({
|
||||
|
||||
---
|
||||
|
||||
### 步骤6:前端组件 - 使用Hook
|
||||
### 豁・鬪、6<EFBFBD>壼燕遶ッ扈<EFBFBD>サ?- 菴ソ逕ィHook
|
||||
|
||||
```typescript
|
||||
// frontend-v2/src/modules/your-module/pages/YourPage.tsx
|
||||
@@ -369,17 +369,17 @@ const YourPage = () => {
|
||||
jobId: string;
|
||||
} | null>(null);
|
||||
|
||||
// ✅ 使用 React Query Hook 自动轮询
|
||||
// 笨?菴ソ逕ィ React Query Hook 閾ェ蜉ィ霓ョ隸「
|
||||
const { status, progress, isReady } = useTaskStatus({
|
||||
recordId: pollingInfo?.recordId || null,
|
||||
jobId: pollingInfo?.jobId || null,
|
||||
enabled: !!pollingInfo,
|
||||
});
|
||||
|
||||
// ✅ 监听状态变化
|
||||
// 笨?逶大成迥カ諤∝序蛹?
|
||||
useEffect(() => {
|
||||
if (isReady && pollingInfo) {
|
||||
console.log('✅ 处理完成,加载数据');
|
||||
console.log('笨?螟<EFBFBD>炊螳梧<EFBFBD><EFBFBD>悟刈霓ス謨ー謐?);
|
||||
|
||||
// 蛛懈ュ「霓ョ隸「
|
||||
setPollingInfo(null);
|
||||
@@ -394,13 +394,13 @@ const YourPage = () => {
|
||||
const result = await api.uploadFile(file);
|
||||
const { recordId, jobId } = result.data;
|
||||
|
||||
// ✅ 启动轮询(设置状态,React Query自动开始)
|
||||
// 笨?蜷ッ蜉ィ霓ョ隸「<E99AB8>郁ョセ鄂ョ迥カ諤<EFBDB6>シ軍eact Query閾ェ蜉ィ蠑蟋具シ<E585B7>
|
||||
setPollingInfo({ recordId, jobId });
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* 进度条 */}
|
||||
{/* 霑帛コヲ譚?*/}
|
||||
{pollingInfo && (
|
||||
<div className="progress-bar">
|
||||
<div style={{ width: `${progress}%` }} />
|
||||
@@ -421,16 +421,16 @@ const YourPage = () => {
|
||||
|
||||
### 1. 髦溷<E9ABA6>蜷咲ァー隗<EFBDB0>激
|
||||
|
||||
**错误**:
|
||||
**髞呵ッッ**<EFBFBD>?
|
||||
```typescript
|
||||
❌ 'asl:screening:batch' // 包含冒号,pg-boss不支持
|
||||
❌ 'dc.toolc.parse' // 包含点号,不推荐
|
||||
笶?'asl:screening:batch' // 蛹<EFBFBD>性蜀貞捷<EFBFBD>継g-boss荳肴髪謖?
|
||||
笶?'dc.toolc.parse' // 蛹<EFBFBD>性轤ケ蜿キ<EFBFBD>御ク肴耳闕<EFBFBD>
|
||||
```
|
||||
|
||||
**正确**:
|
||||
**豁」遑ョ**<EFBFBD>?
|
||||
```typescript
|
||||
✅ 'asl_screening_batch' // 下划线
|
||||
✅ 'dc_toolc_parse_excel' // 下划线
|
||||
笨?'asl_screening_batch' // 荳句<EFBFBD>郤?
|
||||
笨?'dc_toolc_parse_excel' // 荳句<EFBFBD>郤?
|
||||
```
|
||||
|
||||
---
|
||||
@@ -440,22 +440,22 @@ const YourPage = () => {
|
||||
```typescript
|
||||
// backend/src/index.ts
|
||||
|
||||
await jobQueue.start(); // ← 必须先启动队列
|
||||
await jobQueue.start(); // 竊?蠢<>。サ蜈亥星蜉ィ髦溷<E9ABA6>?
|
||||
|
||||
registerYourWorker(); // ← 再注册 Worker
|
||||
registerYourWorker(); // 竊?蜀肴ウィ蜀?Worker
|
||||
registerOtherWorker();
|
||||
|
||||
// ✅ 等待3秒,确保异步注册完成
|
||||
// 笨?遲牙セ<E78999>3遘抵シ檎。ョ菫晏シよュ・豕ィ蜀悟ョ梧<EFBDAE>
|
||||
await new Promise(resolve => setTimeout(resolve, 3000));
|
||||
|
||||
logger.info('✅ All workers registered');
|
||||
logger.info('笨?All workers registered');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. clean data 郛灘ュ俶惻蛻カ
|
||||
|
||||
**目的**:避免重复计算(性能提升99%)
|
||||
**逶ョ逧<EFBFBD>**<EFBFBD>夐∩蜈埼㍾螟崎ョ。邂暦シ域ァ閭ス謠仙合99%<EFBFBD>?
|
||||
|
||||
```typescript
|
||||
// Worker 菫晏ュ<E6998F> clean data
|
||||
@@ -465,17 +465,17 @@ await storage.upload(cleanDataKey, JSON.stringify(processedData));
|
||||
await prisma.update({
|
||||
where: { id },
|
||||
data: {
|
||||
cleanDataKey, // ← 记录位置
|
||||
cleanDataKey, // 竊?隶ー蠖穂ス咲スョ
|
||||
totalRows,
|
||||
columns,
|
||||
}
|
||||
});
|
||||
|
||||
// Service 读取数据(优先 clean data)
|
||||
// Service 隸サ蜿匁焚謐ョ<EFBFBD>井シ伜<EFBFBD>?clean data<EFBFBD>?
|
||||
async getFullData(recordId) {
|
||||
const record = await prisma.findUnique({ where: { id: recordId } });
|
||||
|
||||
// ✅ 优先读取 clean data(<1秒)
|
||||
// 笨?莨伜<E88EA8>隸サ蜿<EFBDBB> clean data<EFBFBD>?1遘抵シ<E68AB5>
|
||||
if (record.cleanDataKey) {
|
||||
const buffer = await storage.download(record.cleanDataKey);
|
||||
return JSON.parse(buffer.toString('utf-8'));
|
||||
@@ -486,19 +486,19 @@ async getFullData(recordId) {
|
||||
return parseFile(buffer);
|
||||
}
|
||||
|
||||
// ⚠️ 重要:操作后要同步更新 clean data
|
||||
// 笞<EFBFBD><EFBFBD><EFBFBD> 驥崎ヲ<E5B48E>シ壽桃菴懷錘隕∝酔豁・譖エ譁?clean data
|
||||
async saveProcessedData(recordId, newData) {
|
||||
const record = await getRecord(recordId);
|
||||
|
||||
// 覆盖原文件
|
||||
// 隕<EFBFBD>尠蜴滓枚莉?
|
||||
await storage.upload(record.fileKey, toExcel(newData));
|
||||
|
||||
// ✅ 同时更新 clean data
|
||||
// 笨?蜷梧慮譖エ譁ー clean data
|
||||
if (record.cleanDataKey) {
|
||||
await storage.upload(record.cleanDataKey, JSON.stringify(newData));
|
||||
}
|
||||
|
||||
// 更新元数据
|
||||
// 譖エ譁ー蜈<EFBFBD>焚謐?
|
||||
await prisma.update({ where: { id: recordId }, data: { ... } });
|
||||
}
|
||||
```
|
||||
@@ -507,15 +507,15 @@ async saveProcessedData(recordId, newData) {
|
||||
|
||||
### 4. React Query 霓ョ隸「<E99AB8>域耳闕撰シ<E692B0>
|
||||
|
||||
**优点**:
|
||||
- ✅ 自动串行(防并发风暴)
|
||||
- ✅ 自动去重(同一queryKey只有一个请求)
|
||||
- ✅ 自动清理(组件卸载时停止)
|
||||
- ✅ 条件停止(动态控制)
|
||||
**莨倡せ**<EFBFBD>?
|
||||
- 笨?閾ェ蜉ィ荳イ陦鯉シ磯亟蟷カ蜿鷹」取垓<E58F96>?
|
||||
- 笨?閾ェ蜉ィ蜴サ驥搾シ亥酔荳queryKey蜿ェ譛我ク荳ェ隸キ豎ゑシ<E38291>
|
||||
- 笨?閾ェ蜉ィ貂<EFBDA8>炊<EFBFBD>育サ<E882B2>サカ蜊ク霓ス譌カ蛛懈ュ「<EFBDAD>?
|
||||
- 笨?譚。莉カ蛛懈ュ「<EFBDAD>亥勘諤∵而蛻カ<E89BBB><EFBDB6>
|
||||
|
||||
**不要使用 setInterval**:
|
||||
**荳崎ヲ∽スソ逕ィ setInterval**<EFBFBD>?
|
||||
```typescript
|
||||
❌ const pollInterval = setInterval(() => {
|
||||
笶?const pollInterval = setInterval(() => {
|
||||
api.getStatus(); // 蜿ッ閭ス蟷カ蜿<EFBDB6>
|
||||
}, 2000);
|
||||
```
|
||||
@@ -524,16 +524,16 @@ async saveProcessedData(recordId, newData) {
|
||||
|
||||
## <20>投 諤ァ閭ス蟇ケ豈<EFBDB9>
|
||||
|
||||
### DC Tool C 实际数据(3339行×151列文件)
|
||||
### DC Tool C 螳樣刔謨ー謐ョ<EFBFBD>?339陦古?51蛻玲枚莉カ<E88E89><EFBDB6>
|
||||
|
||||
| 謖<><E8AC96><EFBFBD> | 蜷梧ュ・螟<EFBDA5>炊 | 蠑よュ・螟<EFBDA5>炊 | 謾ケ蝟<EFBDB9> |
|
||||
|------|---------|---------|------|
|
||||
| **上传耗时** | 47秒(阻塞) | 3秒(立即返回) | ✅ -94% |
|
||||
| **HTTP超时** | ❌ 经常超时 | ✅ 不会超时 | ✅ 100% |
|
||||
| **getPreviewData** | 43秒(重复解析) | 0.5秒(缓存) | ✅ -99% |
|
||||
| **getFullData** | 43秒(重复解析) | 0.5秒(缓存) | ✅ -99% |
|
||||
| **QuickAction操作** | 43秒 + Python | 0.5秒 + Python | ✅ -95% |
|
||||
| **并发请求** | 15+个 | 1个(串行) | ✅ -93% |
|
||||
| **荳贋シ<EFBFBD>閠玲慮** | 47遘抵シ磯仆蝪橸シ?| 3遘抵シ育ォ句叉霑泌屓<E6B38C>?| 笨?-94% |
|
||||
| **HTTP雜<EFBFBD>慮** | 笶?扈丞クク雜<EFBDB8>慮 | 笨?荳堺シ夊カ<E5A48A>慮 | 笨?100% |
|
||||
| **getPreviewData** | 43遘抵シ磯㍾螟崎ァ」譫撰シ?| 0.5遘抵シ育シ灘ュ假シ?| 笨?-99% |
|
||||
| **getFullData** | 43遘抵シ磯㍾螟崎ァ」譫撰シ?| 0.5遘抵シ育シ灘ュ假シ?| 笨?-99% |
|
||||
| **QuickAction謫堺ス<EFBFBD>** | 43遘?+ Python | 0.5遘?+ Python | 笨?-95% |
|
||||
| **蟷カ蜿題ッキ豎<EFBFBD>** | 15+荳?| 1荳ェ<EFBFBD>井クイ陦鯉シ?| 笨?-93% |
|
||||
|
||||
---
|
||||
|
||||
@@ -541,49 +541,50 @@ async saveProcessedData(recordId, newData) {
|
||||
|
||||
### Q1: Worker 豕ィ蜀御コ<E5BEA1>ス<EFBFBD>ク榊キ・菴懶シ<E687B6>
|
||||
|
||||
**检查**:
|
||||
- 队列名称是否包含冒号(`:`)?改为下划线(`_`)
|
||||
- 环境变量 `QUEUE_TYPE=pgboss` 是否设置?
|
||||
- Worker 注册是否在 `jobQueue.start()` 之后?
|
||||
**譽譟?*<2A>?
|
||||
- 髦溷<EFBFBD>蜷咲ァー譏ッ蜷ヲ蛹<EFBFBD>性蜀貞捷<EFBFBD><EFBFBD>:`<60>会シ滓隼荳コ荳句<E88DB3>郤ソ<E983A4><EFBDBF>_`<EFBFBD>?
|
||||
- 邇ッ蠅<EFBFBD>序驥<EFBFBD> `QUEUE_TYPE=pgboss` 譏ッ蜷ヲ隶セ鄂ョ<EFBFBD>?
|
||||
- Worker 豕ィ蜀梧弍蜷ヲ蝨?`jobQueue.start()` 荵句錘<EFBFBD>?
|
||||
|
||||
### Q2: 轮询风暴(多个并发请求)?
|
||||
### Q2: 霓ョ隸「鬟取垓<EFBFBD>亥、壻クェ蟷カ蜿題ッキ豎ゑシ会シ?
|
||||
|
||||
**解决**:使用 React Query,不要用 setInterval
|
||||
**隗」蜀ウ**<EFBFBD>壻スソ逕?React Query<EFBFBD>御ク崎ヲ∫畑 setInterval
|
||||
|
||||
### Q3: 蟇シ蜃コ謨ー謐ョ荳榊ッケ<EFBDAF>域弍蜴溷ァ区焚謐ョ<E8AC90>会シ<E4BC9A>
|
||||
|
||||
**蜴溷屏**<EFBFBD>啻saveProcessedData` 豐。譛画峩譁ー clean data
|
||||
**解决**:同时更新 fileKey 和 cleanDataKey
|
||||
**隗」蜀ウ**<2A>壼酔譌カ譖エ譁?fileKey 蜥?cleanDataKey
|
||||
|
||||
---
|
||||
|
||||
## 📚 参考实现
|
||||
## <EFBFBD>答 蜿り<E3828A>ョ樒<EFBDAE>?
|
||||
|
||||
| 讓。蝮<EFBDA1> | Worker | 蜑咲ォッHook | 譁<>。」 |
|
||||
|------|--------|---------|------|
|
||||
| **DC Tool C** | `parseExcelWorker.ts` | `useSessionStatus.ts` | 譛ャ謖<EFBDAC>漉蝓コ遑 |
|
||||
| **ASL 智能文献** | `screeningWorker.ts` | `useScreeningTask.ts` | [ASL模块状态](../03-业务模块/ASL-AI智能文献/00-模块当前状态与开发指南.md) |
|
||||
| **DC Tool B** | `extractionWorker.ts` | - | [DC模块状态](../03-业务模块/DC-数据清洗整理/00-模块当前状态与开发指南.md) |
|
||||
| **ASL 譎コ閭ス譁<EFBFBD>鍵** | `screeningWorker.ts` | `useScreeningTask.ts` | [ASL讓。蝮礼憾諤‐(../03-荳壼苅讓。蝮<EFBDA1>/ASL-AI譎コ閭ス譁<EFBDBD>鍵/00-讓。蝮怜ス灘燕迥カ諤∽ク主シ蜿第欠蜊?md) |
|
||||
| **DC Tool B** | `extractionWorker.ts` | - | [DC讓。蝮礼憾諤‐(../03-荳壼苅讓。蝮<EFBDA1>/DC-謨ー謐ョ貂<EFBDAE>エ玲紛逅<E7B49B>/00-讓。蝮怜ス灘燕迥カ諤∽ク主シ蜿第欠蜊?md) |
|
||||
|
||||
---
|
||||
|
||||
## ✅ 检查清单
|
||||
## 笨?譽譟・貂<EFBDA5><E8B282>?
|
||||
|
||||
在实施异步任务前,请确认:
|
||||
蝨ィ螳樊命蠑よュ・莉サ蜉。蜑搾シ瑚ッキ遑ョ隶、<EFBFBD>?
|
||||
|
||||
- [ ] 业务表只存业务信息(不包含 status 等字段)
|
||||
- [ ] 队列名称使用下划线(不含冒号)
|
||||
- [ ] 环境变量 `QUEUE_TYPE=pgboss` 已设置
|
||||
- [ ] Worker 在 `jobQueue.start()` 之后注册
|
||||
- [ ] 荳壼苅陦ィ蜿ェ蟄倅ク壼苅菫。諱ッ<E8ABB1>井ク榊桁蜷?status 遲牙ュ玲ョオ<EFBDAE><EFBDB5>
|
||||
- [ ] 髦溷<E9ABA6>蜷咲ァー菴ソ逕ィ荳句<E88DB3>郤ソ<E983A4>井ク榊性蜀貞捷<E8B29E>?
|
||||
- [ ] 邇ッ蠅<EFBDAF>序驥<E5BA8F> `QUEUE_TYPE=pgboss` 蟾イ隶セ鄂?
|
||||
- [ ] Worker 蝨?`jobQueue.start()` 荵句錘豕ィ蜀<EFBDA8>
|
||||
- [ ] 蜑咲ォッ菴ソ逕ィ React Query 霓ョ隸「
|
||||
- [ ] Service 莨伜<E88EA8>隸サ蜿<EFBDBB> clean data
|
||||
- [ ] saveProcessedData 蜷梧ュ・譖エ譁ー clean data
|
||||
|
||||
---
|
||||
|
||||
**维护者**: 平台架构团队
|
||||
**最后更新**: 2025-12-22
|
||||
**文档状态**: ✅ 已完成
|
||||
**扈エ謚、閠?*: 蟷ウ蜿ー譫カ譫<EFBDB6>屬髦<E5B1AC>
|
||||
**譛蜷取峩譁?*: 2025-12-22
|
||||
**譁<EFBFBD>。」迥カ諤?*: 笨?蟾イ螳梧<E89EB3>?
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,48 +1,48 @@
|
||||
# 通用能力层
|
||||
# 閫氱敤鑳藉姏灞?
|
||||
|
||||
> **层级定义:** 跨业务模块共享的核心技术能力
|
||||
> **核心原则:** 可复用、高内聚、独立部署
|
||||
> **灞傜骇瀹氫箟锛?* 璺ㄤ笟鍔℃ā鍧楀叡浜<E58FA1>殑鏍稿績鎶€鏈<E282AC>兘鍔?
|
||||
> **鏍稿績鍘熷垯锛?* 鍙<><E98D99>鐢ㄣ€侀珮鍐呰仛銆佺嫭绔嬮儴缃?
|
||||
|
||||
---
|
||||
|
||||
## 馃搵 鑳藉姏娓呭崟
|
||||
|
||||
| 能力 | 说明 | 复用率 | 优先级 | 状态 |
|
||||
| 鑳藉姏 | 璇存槑 | 澶嶇敤鐜?| 浼樺厛绾?| 鐘舵€?|
|
||||
|------|------|-------|--------|------|
|
||||
| **01-LLM大模型网关** | 统一管理LLM调用、成本控制、模型切换 | 71% (5/7) | P0 | ⏳ 待实现 |
|
||||
| **02-文档处理引擎** | PDF/Docx/Txt提取、OCR、表格提取 | 86% (6/7) | P0 | ✅ 已实现 |
|
||||
| **03-RAG引擎** | 向量检索、语义搜索、RAG问答 | 43% (3/7) | P1 | ✅ 已实现 |
|
||||
| **04-数据ETL引擎** | Excel JOIN、数据清洗、数据转换 | 29% (2/7) | P2 | ⏳ 待实现 |
|
||||
| **05-医学NLP引擎** | 医学实体识别、术语标准化 | 14% (1/7) | P2 | ⏳ 待实现 |
|
||||
| **01-LLM澶фā鍨嬬綉鍏?* | 缁熶竴绠$悊LLM璋冪敤銆佹垚鏈<E59E9A>帶鍒躲€佹ā鍨嬪垏鎹?| 71% (5/7) | P0 | 鈴?寰呭疄鐜?|
|
||||
| **02-鏂囨。澶勭悊寮曟搸** | PDF/Docx/Txt鎻愬彇銆丱CR銆佽〃鏍兼彁鍙?| 86% (6/7) | P0 | 鉁?宸插疄鐜?|
|
||||
| **03-RAG寮曟搸** | 鍚戦噺妫€绱€€佽<EFBFBD>涔夋悳绱€€丷AG闂<EFBFBD>瓟 | 43% (3/7) | P1 | 鉁?宸插疄鐜?|
|
||||
| **04-鏁版嵁ETL寮曟搸** | Excel JOIN銆佹暟鎹<EFBFBD>竻娲椼€佹暟鎹<EFBFBD>浆鎹?| 29% (2/7) | P2 | 鈴?寰呭疄鐜?|
|
||||
| **05-鍖诲<EFBFBD>NLP寮曟搸** | 鍖诲<E98D96>瀹炰綋璇嗗埆銆佹湳璇<E6B9B3>爣鍑嗗寲 | 14% (1/7) | P2 | 鈴?寰呭疄鐜?|
|
||||
|
||||
---
|
||||
|
||||
## 馃幆 璁捐<E79281>鍘熷垯
|
||||
|
||||
### 1. 可复用性
|
||||
### 1. 鍙<EFBFBD><EFBFBD>鐢ㄦ€?
|
||||
- 澶氫釜涓氬姟妯″潡鍏变韩
|
||||
- 避免重复开发
|
||||
- 閬垮厤閲嶅<EFBFBD>寮€鍙?
|
||||
|
||||
### 2. 鐙<>珛閮ㄧ讲
|
||||
- 鍙<>互鐙<E4BA92>珛涓哄井鏈嶅姟
|
||||
- 鏀<>寔鐙<E5AF94>珛鎵╁睍
|
||||
|
||||
### 3. 高内聚
|
||||
### 3. 楂樺唴鑱?
|
||||
- 姣忎釜鑳藉姏鑱岃矗鍗曚竴
|
||||
- 鎺ュ彛娓呮櫚
|
||||
|
||||
### 4. 棰嗗煙鐭ヨ瘑
|
||||
- 鍖呭惈涓氬姟棰嗗煙鐭ヨ瘑
|
||||
- 不是纯技术组件
|
||||
- 涓嶆槸绾<EFBFBD>妧鏈<EFBFBD>粍浠?
|
||||
|
||||
---
|
||||
|
||||
## 📊 复用率分析
|
||||
## 馃搳 澶嶇敤鐜囧垎鏋?
|
||||
|
||||
**LLM网关** - 71%复用率(最高优先级)
|
||||
- AIA(AI智能问答)
|
||||
- ASL(AI智能文献)
|
||||
- PKB(个人知识库)
|
||||
**LLM缃戝叧** - 71%澶嶇敤鐜囷紙鏈€楂樹紭鍏堢骇锛?
|
||||
- AIA锛圓I鏅鸿兘闂<EFBFBD>瓟锛?
|
||||
- ASL锛圓I鏅鸿兘鏂囩尞锛?
|
||||
- PKB锛堜釜浜虹煡璇嗗簱锛?
|
||||
- DC锛堟暟鎹<E69A9F>竻娲楋級
|
||||
- RVW锛堢ǹ浠跺<E6B5A0>鏌ワ級
|
||||
|
||||
@@ -54,15 +54,15 @@
|
||||
|
||||
---
|
||||
|
||||
## 📚 快速导航
|
||||
## 馃摎 蹇<>€熷<E282AC>鑸?
|
||||
|
||||
### 蹇<>€熶笂涓嬫枃
|
||||
- **[AI对接] 通用能力快速上下文.md** - 2-3分钟了解通用能力层
|
||||
- **[AI瀵规帴] 閫氱敤鑳藉姏蹇<E5A78F>€熶笂涓嬫枃.md** - 2-3鍒嗛挓浜嗚В閫氱敤鑳藉姏灞?
|
||||
|
||||
### 鏍稿績鑳藉姏
|
||||
1. [LLM大模型网关](./01-LLM大模型网关/README.md) - P0优先级 ⭐
|
||||
2. [文档处理引擎](./02-文档处理引擎/README.md) - 已实现
|
||||
3. [RAG引擎](./03-RAG引擎/README.md) - 已实现
|
||||
1. [LLM澶фā鍨嬬綉鍏砞(./01-LLM澶фā鍨嬬綉鍏?README.md) - P0浼樺厛绾?猸?
|
||||
2. [鏂囨。澶勭悊寮曟搸](./02-鏂囨。澶勭悊寮曟搸/README.md) - 宸插疄鐜?
|
||||
3. [RAG寮曟搸](./03-RAG寮曟搸/README.md) - 宸插疄鐜?
|
||||
4. [鏁版嵁ETL寮曟搸](./04-鏁版嵁ETL寮曟搸/README.md)
|
||||
5. [鍖诲<EFBFBD>NLP寮曟搸](./05-鍖诲<E98D96>NLP寮曟搸/README.md)
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
## 馃敆 鐩稿叧鏂囨。
|
||||
|
||||
- [绯荤粺鏋舵瀯鍒嗗眰璁捐<EFBFBD>](../00-绯荤粺鎬讳綋璁捐<E79281>/01-绯荤粺鏋舵瀯鍒嗗眰璁捐<E79281>.md)
|
||||
- [平台基础层](../01-平台基础层/README.md)
|
||||
- [骞冲彴鍩虹<EFBFBD>灞俔(../01-骞冲彴鍩虹<E98DA9>灞?README.md)
|
||||
- [涓氬姟妯″潡灞俔(../03-涓氬姟妯″潡/README.md)
|
||||
|
||||
---
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
# <20>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285><EFBFBD>臬<EFBFBD>箏𦛚皜<F0A69B9A><E79A9C>
|
||||
|
||||
> **文档版本:** v1.0
|
||||
> **创建日期:** 2025-12-22
|
||||
> **<EFBFBD><EFBFBD>﹝<EFBFBD><EFBFBD>𧋦嚗?* v1.0
|
||||
> **<EFBFBD>𥕦遣<EFBFBD>交<EFBFBD>嚗?* 2025-12-22
|
||||
> **蝏湔擪<E6B994><E693AA><EFBFBD>** 撟喳蝱<E596B3>嗆<EFBFBD><E59786>a<EFBFBD>
|
||||
> **文档目的:** 记录通用能力层待优化项,指导未来迭代
|
||||
> **<EFBFBD><EFBFBD>﹝<EFBFBD>桃<EFBFBD>嚗?* 霈啣<E99C88><E595A3>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285>隡睃<E99AA1>憿對<E686BF><E5B08D><EFBFBD>紡<EFBFBD>芣䔉餈凋誨
|
||||
|
||||
---
|
||||
|
||||
## <20><> 璁<>膩
|
||||
|
||||
本文档基于 **DC Tool C 异步架构实践**(2025-12-22),总结发现的可以抽象为通用能力的模式。
|
||||
<EFBFBD>祆<EFBFBD>獢<EFBFBD>抅鈭?**DC Tool C 撘<EFBFBD>郊<EFBFBD>嗆<EFBFBD>摰噼殿**嚗?025-12-22嚗㚁<E59A97><E39A81>餌<EFBFBD><E9A48C>𤑳緵<F0A491B3><E7B7B5>虾隞交𡂝鞊∩蛹<E288A9>𡁶鍂<F0A181B6>賢<EFBFBD><E8B3A2><EFBFBD>芋撘譌<E69298>?
|
||||
|
||||
**核心思想**:
|
||||
- ✅ **当前分层已经很好**(通用能力层提供完整的基础设施)
|
||||
- ✅ **业务模块正确使用了这些能力**
|
||||
- ⏭️ **未来可以进一步抽象**(锦上添花,非必需)
|
||||
**<EFBFBD>詨<EFBFBD><EFBFBD>脲<EFBFBD>**嚗?
|
||||
- <EFBFBD>?**敶枏<E695B6><E69E8F><EFBFBD><EFBFBD>撌脩<E6928C>敺<EFBFBD>末**嚗<><E59A97>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285>靘𥕦<E99D98><F0A595A6>渡<EFBFBD><E6B8A1>箇<EFBFBD>霈暹鴌嚗?
|
||||
- <EFBFBD>?**銝𡁜𦛚璅∪<E79285>甇<EFBFBD>&雿輻鍂鈭<E98D82><E988AD>鈭𥡝<E988AD><F0A5A19D>?*
|
||||
- <EFBFBD>哨<EFBFBD> **<EFBFBD>芣䔉<EFBFBD>臭誑餈𥕢<EFBFBD>甇交𡂝鞊?*嚗<>釺銝𦠜溶<F0A6A09C>梧<EFBFBD><E6A2A7>𧼮<EFBFBD><F0A7BCAE><EFBFBD>嚗?
|
||||
|
||||
---
|
||||
|
||||
@@ -22,18 +22,18 @@
|
||||
|
||||
### TD-COMMON-001: <20>滨垢<E6BBA8>𡁶鍂頧株砭Hook
|
||||
|
||||
**优先级**:⭐⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83> P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**憸<><E686B8><EFBFBD>嗥<EFBFBD>**嚗帋誨<E5B88B><E8AAA8><EFBFBD><EFBFBD>冽<EFBFBD>扳<EFBFBD><E689B3><EFBFBD><EFBFBD>璅∪<E79285><E288AA>渡<EFBFBD>銝<EFBFBD>
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗䫤dc/tool-c/hooks/useSessionStatus.ts`
|
||||
- ASL 摰䂿緵嚗䫤asl/hooks/useScreeningTask.ts`
|
||||
- 隞<><E99A9E><EFBFBD>滚<EFBFBD>摨佗<E691A8>70%
|
||||
|
||||
**重复的逻辑**:
|
||||
**<EFBFBD>滚<EFBFBD><EFBFBD><EFBFBD><EFBFBD>餉<EFBFBD>**嚗?
|
||||
```typescript
|
||||
// 瘥譍葵璅∪<E79285><E288AA>質<EFBFBD><E8B3AA>嗵掩隡潛<E99AA1>隞<EFBFBD><E99A9E>
|
||||
useQuery({
|
||||
@@ -51,7 +51,7 @@ useQuery({
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**抽象为通用Hook**:
|
||||
**<EFBFBD>質情銝粹<EFBFBD>𡁶鍂Hook**嚗?
|
||||
|
||||
```typescript
|
||||
// frontend-v2/src/common/hooks/useAsyncTaskPolling.ts嚗<73>鰵撱綽<E692B1>
|
||||
@@ -65,7 +65,7 @@ interface UseAsyncTaskPollingOptions<T> {
|
||||
/** <20>嗆<EFBFBD><E59786>䰻霂∕PI<50>賣㺭 */
|
||||
queryFn: (taskId: string) => Promise<T>;
|
||||
|
||||
/** 状态提取函数 */
|
||||
/** <EFBFBD>嗆<EFBFBD><EFBFBD><EFBFBD><EFBFBD>硋遆<EFBFBD>?*/
|
||||
getStatus: (data: T) => 'pending' | 'processing' | 'ready' | 'error' | string;
|
||||
|
||||
/** 餈𥕦漲<F0A595A6>𣂼<EFBFBD><F0A382BC>賣㺭嚗<E3BAAD>虾<EFBFBD>㚁<EFBFBD> */
|
||||
@@ -74,7 +74,7 @@ interface UseAsyncTaskPollingOptions<T> {
|
||||
/** <20>臬炏<E887AC>舐鍂 */
|
||||
enabled?: boolean;
|
||||
|
||||
/** 轮询间隔(毫秒),默认2000 */
|
||||
/** 頧株砭<EFBFBD>湧<EFBFBD>嚗<EFBFBD>神蝘𡜐<EFBFBD>嚗屸<EFBFBD>霈?000 */
|
||||
pollingInterval?: number;
|
||||
}
|
||||
|
||||
@@ -122,7 +122,7 @@ export function useAsyncTaskPolling<T>({
|
||||
}
|
||||
```
|
||||
|
||||
**使用示例**:
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
|
||||
```typescript
|
||||
// Tool C 雿輻鍂
|
||||
@@ -144,33 +144,33 @@ const { status, isReady } = useAsyncTaskPolling({
|
||||
|
||||
#### 敶勗<E695B6><E58B97><EFBFBD>凒
|
||||
|
||||
- Tool C: `useSessionStatus.ts` 可简化
|
||||
- ASL: `useScreeningTask.ts` 可简化
|
||||
- Tool B: 未来前端可直接使用
|
||||
- Tool C: `useSessionStatus.ts` <EFBFBD>舐<EFBFBD><EFBFBD>?
|
||||
- ASL: `useScreeningTask.ts` <EFBFBD>舐<EFBFBD><EFBFBD>?
|
||||
- Tool B: <EFBFBD>芣䔉<EFBFBD>滨垢<EFBFBD>舐凒<EFBFBD>乩蝙<EFBFBD>?
|
||||
- <20>嗡<EFBFBD>璅∪<E79285>: <20>湔𦻖憭滨鍂
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-002: Clean Data 蝻枏<E89DBB><E69E8F>滚𦛚
|
||||
|
||||
**优先级**:⭐⭐⭐⭐ P1(高)
|
||||
**工作量**:1天
|
||||
**预期收益**:性能提升99%,所有模块受益
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83>潃?P1嚗<31><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗𡁏<E59A97>扯<EFBFBD><E689AF>𣂼<EFBFBD>99%嚗峕<E59A97><E5B395>㗇芋<E39787>堒<EFBFBD><E5A092>?
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
- Tool C 实现:保存 `${fileKey}_clean.json`
|
||||
- ASL、Tool B:未实现,仍然每次重新解析
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗帋<EFBFBD>摮?`${fileKey}_clean.json`
|
||||
- ASL<EFBFBD><EFBFBD>ool B嚗𡁏𧊋摰䂿緵嚗䔶<EFBFBD><EFBFBD>嗆<EFBFBD>甈⊿<EFBFBD><EFBFBD>啗圾<EFBFBD>?
|
||||
|
||||
**重复计算问题**:
|
||||
- ASL 文献筛选:每次从 OSS 下载 PDF,重新解析(5-10秒/篇)
|
||||
- Tool B 数据提取:每次重新读取 Excel
|
||||
- Tool C 操作:已优化(clean data缓存)
|
||||
**<EFBFBD>滚<EFBFBD>霈∠<EFBFBD><EFBFBD>桅<EFBFBD>**嚗?
|
||||
- ASL <EFBFBD><EFBFBD>讃蝑偦<EFBFBD>㚁<EFBFBD>瘥𤩺活隞?OSS 銝贝蝸 PDF嚗屸<EFBFBD><EFBFBD>啗圾<EFBFBD>琜<EFBFBD>5-10蝘?蝭<><E89DAD>
|
||||
- Tool B <EFBFBD>唳旿<EFBFBD>𣂼<EFBFBD>嚗𡁏<EFBFBD>甈⊿<EFBFBD><EFBFBD>啗粉<EFBFBD>?Excel
|
||||
- Tool C <EFBFBD>滢<EFBFBD>嚗𡁜歇隡睃<EFBFBD>嚗Ếlean data蝻枏<EFBFBD>嚗?
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**抽象为通用服务**:
|
||||
**<EFBFBD>質情銝粹<EFBFBD>𡁶鍂<EFBFBD>滚𦛚**嚗?
|
||||
|
||||
```typescript
|
||||
// backend/src/common/services/DataCacheService.ts嚗<73>鰵撱綽<E692B1>
|
||||
@@ -180,12 +180,12 @@ import { prisma } from '../../config/database';
|
||||
import { logger } from '../logging';
|
||||
|
||||
/**
|
||||
* 数据缓存服务(通用)
|
||||
* <EFBFBD>唳旿蝻枏<EFBFBD><EFBFBD>滚𦛚嚗<EFBFBD><EFBFBD>𡁶鍂嚗?
|
||||
*
|
||||
* <20>券<EFBFBD>䈑<EFBFBD>
|
||||
* - Worker 解析后保存处理结果
|
||||
* - Service 优先读取缓存,避免重复计算
|
||||
* - 操作后同步更新缓存
|
||||
* - Worker 閫<EFBFBD><EFBFBD><EFBFBD>𦒘<EFBFBD>摮睃<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?
|
||||
* - Service 隡睃<EFBFBD>霂餃<EFBFBD>蝻枏<EFBFBD>嚗屸<EFBFBD><EFBFBD>漤<EFBFBD>憭滩恣蝞?
|
||||
* - <EFBFBD>滢<EFBFBD><EFBFBD>𤾸<EFBFBD>甇交凒<EFBFBD>啁<EFBFBD>摮?
|
||||
*/
|
||||
export class DataCacheService {
|
||||
/**
|
||||
@@ -193,8 +193,8 @@ export class DataCacheService {
|
||||
*
|
||||
* @param originalKey <20>笔<EFBFBD><E7AC94><EFBFBD>辣key
|
||||
* @param cleanData 皜<><E79A9C><EFBFBD>𡒊<EFBFBD><F0A1928A>唳旿
|
||||
* @param suffix 后缀(默认 '_clean.json')
|
||||
* @returns clean data 的 OSS key
|
||||
* @param suffix <EFBFBD>𡒊<EFBFBD>嚗<EFBFBD><EFBFBD>霈?'_clean.json'嚗?
|
||||
* @returns clean data <EFBFBD>?OSS key
|
||||
*/
|
||||
async saveCleanData(
|
||||
originalKey: string,
|
||||
@@ -223,7 +223,7 @@ export class DataCacheService {
|
||||
/**
|
||||
* 霂餃<E99C82>皜<EFBFBD><E79A9C><EFBFBD>𡒊<EFBFBD><F0A1928A>唳旿
|
||||
*
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
* @returns 皜<><E79A9C><EFBFBD>𡒊<EFBFBD><F0A1928A>唳旿
|
||||
*/
|
||||
async getCleanData(cleanDataKey: string): Promise<any> {
|
||||
@@ -242,8 +242,8 @@ export class DataCacheService {
|
||||
/**
|
||||
* <20>湔鰵皜<E9B0B5><E79A9C><EFBFBD>𡒊<EFBFBD><F0A1928A>唳旿
|
||||
*
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
* @param newData 新数据
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
* @param newData <EFBFBD>唳㺭<EFBFBD>?
|
||||
*/
|
||||
async updateCleanData(cleanDataKey: string, newData: any): Promise<void> {
|
||||
logger.info('[DataCacheService] Updating clean data', {
|
||||
@@ -260,7 +260,7 @@ export class DataCacheService {
|
||||
/**
|
||||
* <20>𣳇膄皜<E88684><E79A9C><EFBFBD>𡒊<EFBFBD><F0A1928A>唳旿
|
||||
*
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
*/
|
||||
async deleteCleanData(cleanDataKey: string): Promise<void> {
|
||||
try {
|
||||
@@ -278,46 +278,46 @@ export class DataCacheService {
|
||||
export const dataCacheService = new DataCacheService();
|
||||
```
|
||||
|
||||
**使用示例**:
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
|
||||
```typescript
|
||||
// Worker 中
|
||||
// Worker 銝?
|
||||
const cleanDataKey = await dataCacheService.saveCleanData(fileKey, cleanedData);
|
||||
await prisma.update({ where: { id }, data: { cleanDataKey } });
|
||||
|
||||
// Service 中
|
||||
// Service 銝?
|
||||
if (record.cleanDataKey) {
|
||||
return await dataCacheService.getCleanData(record.cleanDataKey);
|
||||
}
|
||||
|
||||
// 操作后更新
|
||||
// <EFBFBD>滢<EFBFBD><EFBFBD>擧凒<EFBFBD>?
|
||||
await dataCacheService.updateCleanData(record.cleanDataKey, newData);
|
||||
```
|
||||
|
||||
#### 敶勗<E695B6><E58B97><EFBFBD>凒
|
||||
|
||||
- Tool C: 简化现有代码
|
||||
- ASL: 文献解析结果缓存(提升99%)
|
||||
- Tool C: 蝞<EFBFBD><EFBFBD>𣇉緵<EFBFBD>劐誨<EFBFBD>?
|
||||
- ASL: <EFBFBD><EFBFBD>讃閫<EFBFBD><EFBFBD>蝏𤘪<EFBFBD>蝻枏<EFBFBD>嚗<EFBFBD><EFBFBD><EFBFBD>?9%嚗?
|
||||
- Tool B: <20>唳旿<E594B3>𣂼<EFBFBD>蝏𤘪<E89D8F>蝻枏<E89DBB>
|
||||
- 所有模块: 统一的缓存机制
|
||||
- <EFBFBD><EFBFBD><EFBFBD>㗇芋<EFBFBD>? 蝏煺<E89D8F><E785BA><EFBFBD><EFBFBD>摮䀹㦤<E480B9>?
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-003: 智能清洗算法通用化
|
||||
### TD-COMMON-003: <EFBFBD>箄<EFBFBD>皜<EFBFBD><EFBFBD>蝞埈<EFBFBD><EFBFBD>𡁶鍂<EFBFBD>?
|
||||
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**憸<><E686B8><EFBFBD>嗥<EFBFBD>**嚗帋誨<E5B88B><E8AAA8><EFBFBD><EFBFBD>剁<EFBFBD><E58981>踹<EFBFBD><E8B8B9>滚<EFBFBD>摰䂿緵
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
- Tool C 实现:`intelligentCleanData`(边界检测+安全阀)
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗䫤intelligentCleanData`嚗<EFBFBD>器<EFBFBD>峕<EFBFBD>瘚?摰匧<E691B0><E58CA7><EFBFBD>嚗?
|
||||
- <20>嗡<EFBFBD>璅∪<E79285>嚗𡁏𧊋摰䂿緵蝐颱撮<E9A2B1>蠘<EFBFBD>
|
||||
|
||||
**可以通用化的算法**:
|
||||
**<EFBFBD>臭誑<EFBFBD>𡁶鍂<EFBFBD>𣇉<EFBFBD>蝞埈<EFBFBD>**嚗?
|
||||
1. 撟賜<E6929F><E8B39C>埈<EFBFBD>瘚页<E7989A>颲寧<E9A2B2>璉<EFBFBD>瘚页<E7989A>
|
||||
2. 幽灵行过滤
|
||||
2. 撟賜<EFBFBD>銵諹<EFBFBD>皛?
|
||||
3. 摰匧<E691B0><E58CA7><EFBFBD>嚗<EFBFBD><E59A97>憭批<E686AD><E689B9>啜<EFBFBD><E5959C><EFBFBD><EFBFBD><EFBFBD>聢<EFBFBD>圈<EFBFBD><E59C88>塚<EFBFBD>
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
@@ -327,7 +327,7 @@ await dataCacheService.updateCleanData(record.cleanDataKey, newData);
|
||||
|
||||
export interface CleaningOptions {
|
||||
maxCols?: number; // <20><>憭批<E686AD><E689B9>堆<EFBFBD>暺䁅恕3000
|
||||
maxCells?: number; // 最大单元格数,默认500万
|
||||
maxCells?: number; // <EFBFBD><EFBFBD>憭批<EFBFBD><EFBFBD><EFBFBD>聢<EFBFBD>堆<EFBFBD>暺䁅恕500銝?
|
||||
removeEmptyRows?: boolean; // <20>臬炏<E887AC>𣳇膄蝛箄<E89D9B>嚗屸<E59A97>霈川rue
|
||||
removeEmptyCols?: boolean; // <20>臬炏<E887AC>𣳇膄蝛箏<E89D9B>嚗屸<E59A97>霈川rue
|
||||
}
|
||||
@@ -341,7 +341,7 @@ export function intelligentCleanData(
|
||||
}
|
||||
|
||||
export function isValidValue(value: any): boolean {
|
||||
// 统一的空值判断
|
||||
// 蝏煺<EFBFBD><EFBFBD><EFBFBD>征<EFBFBD>澆ế<EFBFBD>?
|
||||
// ...
|
||||
}
|
||||
```
|
||||
@@ -349,19 +349,19 @@ export function isValidValue(value: any): boolean {
|
||||
#### 敶勗<E695B6><E58B97><EFBFBD>凒
|
||||
|
||||
- Tool C: 憭滨鍂<E6BBA8>𡁶鍂摰䂿緵
|
||||
- 其他上传Excel的功能: 直接使用
|
||||
- <EFBFBD>嗡<EFBFBD>銝𠹺<EFBFBD>Excel<EFBFBD><EFBFBD><EFBFBD><EFBFBD>? <20>湔𦻖雿輻鍂
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-004: Worker瘜典<E7989C>颲<EFBFBD>𨭌撌亙<E6928C>
|
||||
|
||||
**优先级**:⭐ P3(低)
|
||||
**工作量**:0.3天
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97> P3嚗<33><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.3憭?
|
||||
**憸<><E686B8><EFBFBD>嗥<EFBFBD>**嚗𡁻<E59A97>雿竝orker瘜典<E7989C>隞<EFBFBD><E99A9E><EFBFBD>滚<EFBFBD>
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- 瘥譍葵璅∪<E79285><E288AA>質<EFBFBD><E8B3AA>见<EFBFBD> Worker 瘜典<E7989C>隞<EFBFBD><E99A9E>
|
||||
- <20>躰秤憭<E7A7A4><E686AD><EFBFBD>餉<EFBFBD><E9A489>滚<EFBFBD>
|
||||
|
||||
@@ -403,11 +403,11 @@ export function registerWorker(options: RegisterWorkerOptions) {
|
||||
}
|
||||
});
|
||||
|
||||
logger.info(`[WorkerHelper] ✅ Worker registered: ${queueName}`);
|
||||
logger.info(`[WorkerHelper] <EFBFBD>?Worker registered: ${queueName}`);
|
||||
}
|
||||
```
|
||||
|
||||
**使用示例**:
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
|
||||
```typescript
|
||||
// 蝞<><E89D9E>𣇉<EFBFBD>瘜典<E7989C>隞<EFBFBD><E99A9E>
|
||||
@@ -418,7 +418,7 @@ registerWorker({
|
||||
// 銝𡁜𦛚<F0A1819C>餉<EFBFBD>
|
||||
return result;
|
||||
},
|
||||
onStart: (job) => console.log(`开始处理: ${job.id}`),
|
||||
onStart: (job) => console.log(`撘<EFBFBD>憪见<EFBFBD><EFBFBD>? ${job.id}`),
|
||||
onComplete: (job, result) => console.log(`摰峕<EFBFBD>: ${result}`),
|
||||
onError: (job, error) => console.error(`憭梯揖: ${error}`),
|
||||
});
|
||||
@@ -428,36 +428,36 @@ registerWorker({
|
||||
|
||||
### TD-COMMON-005: <20>唳旿<E594B3><E697BF>𧋦蝞∠<E89D9E>蝟餌<E89D9F>
|
||||
|
||||
**优先级**:⭐⭐⭐⭐ P1(高)
|
||||
**工作量**:3天
|
||||
**预期收益**:支持链式操作、undo功能、导出历史版本
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83>潃?P1嚗<31><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗𡁏𣈲<F0A1818F><F0A388B2>曎撘𤩺<E69298>雿栶<E99BBF><E6A0B6>ndo<64>蠘<EFBFBD><E8A098><EFBFBD>紡<EFBFBD>箏<EFBFBD><E7AE8F>脩<EFBFBD><E884A9>?
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前限制**:
|
||||
- 操作不是累积的(每次基于原始数据)
|
||||
- 无法回退到某个操作前的状态
|
||||
- 无法导出中间版本的数据
|
||||
**敶枏<EFBFBD><EFBFBD>𣂼<EFBFBD>**嚗?
|
||||
- <EFBFBD>滢<EFBFBD>銝齿糓蝝舐妖<EFBFBD><EFBFBD><EFBFBD>瘥𤩺活<EFBFBD>箔<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>唳旿嚗?
|
||||
- <EFBFBD>䭾<EFBFBD><EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>唳<EFBFBD>銝芣<EFBFBD>雿𨅯<EFBFBD><EFBFBD><EFBFBD>𠶖<EFBFBD>?
|
||||
- <EFBFBD>䭾<EFBFBD>撖澆枂銝剝𡢿<EFBFBD><EFBFBD>𧋦<EFBFBD><EFBFBD>㺭<EFBFBD>?
|
||||
|
||||
**用户期望的工作流**:
|
||||
**<EFBFBD>冽<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>極雿𨀣<EFBFBD>**嚗?
|
||||
```
|
||||
銝𠹺<EFBFBD>嚗ǒ0: 100銵䕘<E98AB5>
|
||||
↓
|
||||
筛选(v1: 50行)← 可以回退到这里
|
||||
↓
|
||||
数值映射(v2: 50行,有映射)← 可以回退到这里
|
||||
↓
|
||||
Pivot(v3: 不同结构)
|
||||
↓
|
||||
导出:可以导出 v0、v1、v2、v3 任意版本
|
||||
<EFBFBD>?
|
||||
蝑偦<EFBFBD>㚁<EFBFBD>v1: 50銵䕘<EFBFBD><EFBFBD>?<3F>臭誑<E887AD>鮋<EFBFBD><E9AE8B><EFBFBD>啗<EFBFBD><E59597>?
|
||||
<EFBFBD>?
|
||||
<EFBFBD>啣<EFBFBD>潭<EFBFBD>撠<EFBFBD><EFBFBD>v2: 50銵䕘<E98AB5><E49598>㗇<EFBFBD>撠<EFBFBD><E692A0><EFBFBD>?<3F>臭誑<E887AD>鮋<EFBFBD><E9AE8B><EFBFBD>啗<EFBFBD><E59597>?
|
||||
<EFBFBD>?
|
||||
Pivot嚗ǒ3: 銝滚<EFBFBD>蝏𤘪<EFBFBD>嚗?
|
||||
<EFBFBD>?
|
||||
撖澆枂嚗𡁜虾隞亙紡<EFBFBD>?v0<76><30>1<EFBFBD><31>2<EFBFBD><32>3 隞餅<E99A9E><E9A485><EFBFBD>𧋦
|
||||
```
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**Prisma Schema设计**:
|
||||
**Prisma Schema霈曇恣**嚗?
|
||||
|
||||
```prisma
|
||||
// 版本管理表(通用)
|
||||
// <EFBFBD><EFBFBD>𧋦蝞∠<EFBFBD>銵剁<EFBFBD><EFBFBD>𡁶鍂嚗?
|
||||
model DataVersion {
|
||||
id String @id @default(uuid())
|
||||
|
||||
@@ -466,19 +466,19 @@ model DataVersion {
|
||||
entityId String // Session ID | Project ID | ...
|
||||
|
||||
// <20><>𧋦靽⊥<E99DBD>
|
||||
versionNumber Int // 0=原始, 1=第1次操作后, 2=第2次操作后...
|
||||
versionNumber Int // 0=<EFBFBD>笔<EFBFBD>, 1=蝚?甈⊥<E79488>雿𨅯<E99BBF>, 2=蝚?甈⊥<E79488>雿𨅯<E99BBF>...
|
||||
dataKey String // OSS銝剔<E98A9D><E58994>唳旿<E594B3><E697BF>辣key
|
||||
|
||||
// <20>滢<EFBFBD>霈啣<E99C88>
|
||||
operation String? // 'upload' | 'filter' | 'pivot' | 'recode' ...
|
||||
operationParams Json? // <20>滢<EFBFBD><E6BBA2><EFBFBD>㺭
|
||||
|
||||
// 元数据
|
||||
// <EFBFBD><EFBFBD>㺭<EFBFBD>?
|
||||
totalRows Int
|
||||
totalCols Int
|
||||
columns Json
|
||||
|
||||
// 时间戳
|
||||
// <EFBFBD>園𡢿<EFBFBD>?
|
||||
createdAt DateTime @default(now())
|
||||
createdBy String // <20>冽<EFBFBD>ID
|
||||
|
||||
@@ -488,22 +488,22 @@ model DataVersion {
|
||||
@@schema("platform_schema")
|
||||
}
|
||||
|
||||
// 业务表添加字段
|
||||
// 銝𡁜𦛚銵冽溶<EFBFBD>惩<EFBFBD>畾?
|
||||
model DcToolCSession {
|
||||
// ...<2E>唳<EFBFBD>摮埈挾
|
||||
|
||||
currentVersion Int @default(0) // 当前版本号
|
||||
currentVersion Int @default(0) // 敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
}
|
||||
```
|
||||
|
||||
**Service实现**:
|
||||
**Service摰䂿緵**嚗?
|
||||
|
||||
```typescript
|
||||
// backend/src/common/services/DataVersionService.ts嚗<73>鰵撱綽<E692B1>
|
||||
|
||||
export class DataVersionService {
|
||||
/**
|
||||
* 创建新版本
|
||||
* <EFBFBD>𥕦遣<EFBFBD>啁<EFBFBD><EFBFBD>?
|
||||
*/
|
||||
async createVersion(
|
||||
entityType: string,
|
||||
@@ -513,7 +513,7 @@ export class DataVersionService {
|
||||
operation?: string,
|
||||
params?: any
|
||||
): Promise<string> {
|
||||
// 保存数据到 OSS
|
||||
// 靽嘥<EFBFBD><EFBFBD>唳旿<EFBFBD>?OSS
|
||||
const dataKey = `versions/${entityType}/${entityId}/v${versionNumber}.json`;
|
||||
await storage.upload(dataKey, JSON.stringify(data));
|
||||
|
||||
@@ -553,21 +553,21 @@ export class DataVersionService {
|
||||
}
|
||||
});
|
||||
|
||||
if (!version) throw new Error('版本不存在');
|
||||
if (!version) throw new Error('<EFBFBD><EFBFBD>𧋦銝滚<EFBFBD><EFBFBD>?);
|
||||
|
||||
const buffer = await storage.download(version.dataKey);
|
||||
return JSON.parse(buffer.toString('utf-8'));
|
||||
}
|
||||
|
||||
/**
|
||||
* 回退到指定版本
|
||||
* <EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>唳<EFBFBD>摰𡁶<EFBFBD><EFBFBD>?
|
||||
*/
|
||||
async rollbackToVersion(
|
||||
entityType: string,
|
||||
entityId: string,
|
||||
versionNumber: number
|
||||
): Promise<void> {
|
||||
// 更新当前版本号
|
||||
// <EFBFBD>湔鰵敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
await this.updateCurrentVersion(entityType, entityId, versionNumber);
|
||||
}
|
||||
|
||||
@@ -583,16 +583,16 @@ export class DataVersionService {
|
||||
}
|
||||
```
|
||||
|
||||
**QuickAction 集成**:
|
||||
**QuickAction <EFBFBD><EFBFBD><EFBFBD>**嚗?
|
||||
|
||||
```typescript
|
||||
// QuickAction <20>扯<EFBFBD><E689AF>舘䌊<E88898>典<EFBFBD>撱箸鰵<E7AEB8><E9B0B5>𧋦
|
||||
const result = await python.execute(fullData, params);
|
||||
|
||||
// 读取当前版本号
|
||||
// 霂餃<EFBFBD>敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
const currentVersion = session.currentVersion || 0;
|
||||
|
||||
// 创建新版本
|
||||
// <EFBFBD>𥕦遣<EFBFBD>啁<EFBFBD><EFBFBD>?
|
||||
await dataVersionService.createVersion(
|
||||
'dc_toolc_session',
|
||||
sessionId,
|
||||
@@ -602,7 +602,7 @@ await dataVersionService.createVersion(
|
||||
params
|
||||
);
|
||||
|
||||
// 更新当前版本号
|
||||
// <EFBFBD>湔鰵敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
await prisma.update({
|
||||
where: { id: sessionId },
|
||||
data: { currentVersion: currentVersion + 1 }
|
||||
@@ -611,28 +611,28 @@ await prisma.update({
|
||||
|
||||
#### 敶勗<E695B6><E58B97><EFBFBD>凒
|
||||
|
||||
- Tool C: 支持链式操作、undo、导出历史
|
||||
- ASL: 文献筛选的多阶段结果管理
|
||||
- Tool B: 数据提取的版本管理
|
||||
- 所有模块: 统一的版本管理能力
|
||||
- Tool C: <EFBFBD>舀<EFBFBD><EFBFBD>曉<EFBFBD><EFBFBD>滢<EFBFBD><EFBFBD><EFBFBD>ndo<EFBFBD><EFBFBD>紡<EFBFBD>箏<EFBFBD><EFBFBD>?
|
||||
- ASL: <EFBFBD><EFBFBD>讃蝑偦<EFBFBD>厩<EFBFBD>憭𡁻𧫴畾萇<EFBFBD><EFBFBD>𦦵恣<EFBFBD>?
|
||||
- Tool B: <EFBFBD>唳旿<EFBFBD>𣂼<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>祉恣<EFBFBD>?
|
||||
- <EFBFBD><EFBFBD><EFBFBD>㗇芋<EFBFBD>? 蝏煺<E89D8F><E785BA><EFBFBD><EFBFBD><EFBFBD>祉恣<E7A589><E681A3><EFBFBD><EFBFBD>?
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-006: 幽灵列/行检测算法库
|
||||
### TD-COMMON-006: 撟賜<EFBFBD><EFBFBD>?銵峕<E98AB5>瘚讠<E7989A>瘜訫<E7989C>
|
||||
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**憸<><E686B8><EFBFBD>嗥<EFBFBD>**嚗鍃xcel憭<6C><E686AD>韐券<E99F90><E588B8>𣂼<EFBFBD>
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
- Tool C 实现:边界检测算法
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗朞器<EFBFBD>峕<EFBFBD>瘚讠<EFBFBD>瘜?
|
||||
- <20>嗡<EFBFBD>璅∪<E79285>嚗𡁏𧊋摰䂿緵
|
||||
|
||||
**Excel格式污染问题**:
|
||||
- 用户刷颜色到16384列 → 解析出16384列(实际只有151列有效)
|
||||
- 用户删除数据未清理 → 解析出大量空行
|
||||
**Excel<EFBFBD>澆<EFBFBD>瘙⊥<EFBFBD><EFBFBD>桅<EFBFBD>**嚗?
|
||||
- <EFBFBD>冽<EFBFBD><EFBFBD>琿<EFBFBD><EFBFBD>脣<EFBFBD>16384<EFBFBD>?<3F>?閫<><E996AB><EFBFBD>?6384<38>梹<EFBFBD>摰鮋<E691B0><E9AE8B>芣<EFBFBD>151<35>埈<EFBFBD><E59F88><EFBFBD><EFBFBD>
|
||||
- <EFBFBD>冽<EFBFBD><EFBFBD>𣳇膄<EFBFBD>唳旿<EFBFBD>芣<EFBFBD><EFBFBD>?<3F>?閫<><E996AB><EFBFBD>箏之<E7AE8F>讐征銵?
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
@@ -654,23 +654,23 @@ export function cleanExcelData(
|
||||
rawData: any[],
|
||||
options?: CleaningOptions
|
||||
): CleaningResult {
|
||||
// 实现通用的清洗算法
|
||||
// 摰䂿緵<EFBFBD>𡁶鍂<EFBFBD><EFBFBD><EFBFBD>瘣㛖<EFBFBD>瘜?
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-007: 前端进度条组件
|
||||
### TD-COMMON-007: <EFBFBD>滨垢餈𥕦漲<EFBFBD>∠<EFBFBD>隞?
|
||||
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.3天
|
||||
**预期收益**:UI统一,用户体验提升
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.3憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗䦧I蝏煺<E89D8F>嚗𣬚鍂<F0A3AC9A>瑚<EFBFBD>撉峕<E69289><E5B395>?
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>讛膩
|
||||
|
||||
**当前状态**:
|
||||
- Tool C 实现:内联进度条(蓝色,Header下方)
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗𡁜<EFBFBD><EFBFBD>磰<EFBFBD>摨行辺嚗<EFBFBD><EFBFBD><EFBFBD>莎<EFBFBD>Header銝𧢲䲮嚗?
|
||||
- <20>嗡<EFBFBD>璅∪<E79285>嚗𡁏𧊋摰䂿緵<E482BF>碶<EFBFBD>蝏煺<E89D8F>
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
@@ -714,63 +714,64 @@ export const AsyncProgressBar: React.FC<AsyncProgressBarProps> = ({
|
||||
|
||||
## <20><> 摰墧鴌霈∪<E99C88>
|
||||
|
||||
| 技术债务 | 优先级 | 工作量 | 建议时机 | 收益 |
|
||||
| <EFBFBD><EFBFBD><EFBFBD>臬<EFBFBD>箏𦛚 | 隡睃<E99AA1>蝥?| 撌乩<E6928C><E4B9A9>?| 撱箄悅<E7AE84>嗆㦤 | <20>嗥<EFBFBD> |
|
||||
|---------|--------|--------|---------|------|
|
||||
| TD-COMMON-001 前端轮询Hook | P2 | 0.5天 | 下次迭代 | 代码统一 |
|
||||
| TD-COMMON-002 Clean Data服务 | **P1** | 1天 | **下次迭代** | **性能提升99%** |
|
||||
| TD-COMMON-003 智能清洗算法 | P2 | 0.5天 | 需要时 | 质量提升 |
|
||||
| TD-COMMON-004 Worker注册辅助 | P3 | 0.3天 | 可选 | 代码简化 |
|
||||
| TD-COMMON-005 数据版本管理 | **P1** | 3天 | **下次迭代** | **链式操作** |
|
||||
| TD-COMMON-006 幽灵列/行检测 | P2 | 0.5天 | 需要时 | 质量提升 |
|
||||
| TD-COMMON-007 进度条组件 | P2 | 0.3天 | 需要时 | UI统一 |
|
||||
| TD-COMMON-001 <EFBFBD>滨垢頧株砭Hook | P2 | 0.5憭?| 銝𧢲活餈凋誨 | 隞<><E99A9E>蝏煺<E89D8F> |
|
||||
| TD-COMMON-002 Clean Data<EFBFBD>滚𦛚 | **P1** | 1憭?| **銝𧢲活餈凋誨** | **<EFBFBD>扯<EFBFBD><EFBFBD>𣂼<EFBFBD>99%** |
|
||||
| TD-COMMON-003 <EFBFBD>箄<EFBFBD>皜<EFBFBD><EFBFBD>蝞埈<EFBFBD> | P2 | 0.5憭?| <20><>閬<EFBFBD>𧒄 | 韐券<E99F90><E588B8>𣂼<EFBFBD> |
|
||||
| TD-COMMON-004 Worker瘜典<EFBFBD>颲<EFBFBD>𨭌 | P3 | 0.3憭?| <20>舫<EFBFBD>?| 隞<><E99A9E>蝞<EFBFBD><E89D9E>?|
|
||||
| TD-COMMON-005 <EFBFBD>唳旿<EFBFBD><EFBFBD>𧋦蝞∠<EFBFBD> | **P1** | 3憭?| **銝𧢲活餈凋誨** | **<EFBFBD>曉<EFBFBD><EFBFBD>滢<EFBFBD>** |
|
||||
| TD-COMMON-006 撟賜<EFBFBD><EFBFBD>?銵峕<E98AB5>瘚?| P2 | 0.5憭?| <20><>閬<EFBFBD>𧒄 | 韐券<E99F90><E588B8>𣂼<EFBFBD> |
|
||||
| TD-COMMON-007 餈𥕦漲<EFBFBD>∠<EFBFBD>隞?| P2 | 0.3憭?| <20><>閬<EFBFBD>𧒄 | UI蝏煺<EFBFBD> |
|
||||
|
||||
**推荐优先级**:
|
||||
1. ⭐⭐⭐⭐⭐ TD-COMMON-002(Clean Data服务)
|
||||
2. ⭐⭐⭐⭐⭐ TD-COMMON-005(数据版本管理)
|
||||
3. ⭐⭐⭐ TD-COMMON-001(前端轮询Hook)
|
||||
**<EFBFBD>刻<EFBFBD>隡睃<EFBFBD>蝥?*嚗?
|
||||
1. 潃鐥<EFBFBD>潃鐥<EFBFBD>潃?TD-COMMON-002嚗㇃lean Data<EFBFBD>滚𦛚嚗?
|
||||
2. 潃鐥<EFBFBD>潃鐥<EFBFBD>潃?TD-COMMON-005嚗<EFBFBD>㺭<EFBFBD>桃<EFBFBD><EFBFBD>祉恣<EFBFBD><EFBFBD><EFBFBD>
|
||||
3. 潃鐥<EFBFBD>潃?TD-COMMON-001嚗<EFBFBD><EFBFBD>蝡航蔭霂%ook嚗?
|
||||
|
||||
---
|
||||
|
||||
## <20><> 敶枏<E695B6><E69E8F>嗆<EFBFBD>霂<EFBFBD>遠
|
||||
|
||||
### ✅ 已经很好的部分
|
||||
### <EFBFBD>?撌脩<E6928C>敺<EFBFBD>末<EFBFBD><E69CAB><EFBFBD><EFBFBD>?
|
||||
|
||||
1. ✅ **通用能力层基础设施完整**
|
||||
- jobQueue(pg-boss队列)
|
||||
1. <EFBFBD>?**<2A>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD>抅蝖<E68A85>霈暹鴌摰峕㟲**
|
||||
- jobQueue嚗īg-boss<EFBFBD>笔<EFBFBD>嚗?
|
||||
- CheckpointService嚗<65>鱏<EFBFBD>寧賒隡𩤃<E99AA1>
|
||||
- 隞餃𦛚<E9A483><F0A69B9A><EFBFBD>撌亙<E6928C>
|
||||
- storage<67><65>ogger<65><72>ache
|
||||
|
||||
2. ✅ **业务模块正确使用通用能力**
|
||||
2. <EFBFBD>?**銝𡁜𦛚璅∪<E79285>甇<EFBFBD>&雿輻鍂<E8BCBB>𡁶鍂<F0A181B6>賢<EFBFBD>**
|
||||
- Tool C<><43>SL<53><4C>ool B 蝏煺<E89D8F>雿輻鍂 jobQueue
|
||||
- Platform-Only璅∪<E79285>
|
||||
- 銝滚銁銝𡁜𦛚銵其葉摮睃<E691AE>隞餃𦛚蝞∠<E89D9E>靽⊥<E99DBD>
|
||||
|
||||
3. ✅ **分层清晰,职责明确**
|
||||
3. <EFBFBD>?**<2A><><EFBFBD>皜<EFBFBD>苊嚗諹<E59A97>韐<EFBFBD><E99F90>蝖?*
|
||||
- <20>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285><EFBFBD>箇<EFBFBD>霈暹鴌
|
||||
- 业务模块:具体实现
|
||||
- 銝𡁜𦛚璅∪<EFBFBD>嚗𡁜<EFBFBD>雿枏<EFBFBD><EFBFBD>?
|
||||
|
||||
### ⏭️ 可以改进的部分(非必需)
|
||||
### <EFBFBD>哨<EFBFBD> <20>臭誑<E887AD>寡<EFBFBD><E5AFA1><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>𧼮<EFBFBD><F0A7BCAE><EFBFBD>嚗?
|
||||
|
||||
1. <20>哨<EFBFBD> <20>滨垢頧株砭璅∪<E79285><E288AA>臭誑<E887AD>渡<EFBFBD>銝<EFBFBD>
|
||||
2. <20>哨<EFBFBD> clean data蝻枏<E89DBB><E69E8F>臭誑<E887AD>湧<EFBFBD>𡁶鍂
|
||||
3. ⏭️ 数据版本管理待建立
|
||||
3. <EFBFBD>哨<EFBFBD> <20>唳旿<E594B3><E697BF>𧋦蝞∠<E89D9E>敺<EFBFBD>遣蝡?
|
||||
|
||||
**蝏栞捏**嚗𡁜<E59A97><F0A1819C>齿沲<E9BDBF><E6B2B2>歇蝏讐泵<E8AE90><E6B3B5><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>芣䔉隡睃<E99AA1><E79D83>舫釺銝𦠜溶<F0A6A09C>梧<EFBFBD>
|
||||
|
||||
---
|
||||
|
||||
## 📚 参考文档
|
||||
## <EFBFBD><EFBFBD> <20><><EFBFBD><EFBFBD><EFBFBD>獢?
|
||||
|
||||
- [Postgres-Only撘<79>郊隞餃𦛚憭<F0A69B9A><E686AD><EFBFBD><EFBFBD><EFBFBD>](./Postgres-Only撘<79>郊隞餃𦛚憭<F0A69B9A><E686AD><EFBFBD><EFBFBD><EFBFBD>.md) - 摰峕㟲摰噼殿
|
||||
- [云原生开发规范](../04-开发规范/08-云原生开发规范.md) - 开发规范
|
||||
- [DC Tool C状态](../03-业务模块/DC-数据清洗整理/00-工具C当前状态与开发指南.md) - Day 10实践
|
||||
- [鈭穃<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>𤏸<EFBFBD><EFBFBD><EFBFBD>(../04-撘<><E69298>𤏸<EFBFBD><F0A48FB8>?08-鈭穃<E988AD><E7A983>笔<EFBFBD><E7AC94>𤏸<EFBFBD><F0A48FB8>?md) - 撘<EFBFBD><EFBFBD>𤏸<EFBFBD><EFBFBD>?
|
||||
- [DC Tool C<EFBFBD>嗆<EFBFBD><EFBFBD>(../03-銝𡁜𦛚璅∪<E79285>/DC-<2D>唳旿皜<E697BF><E79A9C><EFBFBD>渡<EFBFBD>/00-撌亙<E6928C>C敶枏<E695B6><E69E8F>嗆<EFBFBD><E59786><EFBFBD>撘<EFBFBD><E69298>烐<EFBFBD><E78390>?md) - Day 10摰噼殿
|
||||
|
||||
---
|
||||
|
||||
**维护者**: 平台架构团队
|
||||
**最后更新**: 2025-12-22
|
||||
**文档状态**: ✅ 初始版本
|
||||
**蝏湔擪<EFBFBD>?*: 撟喳蝱<E596B3>嗆<EFBFBD><E59786>a<EFBFBD>
|
||||
**<EFBFBD><EFBFBD><EFBFBD>擧凒<EFBFBD>?*: 2025-12-22
|
||||
**<EFBFBD><EFBFBD>﹝<EFBFBD>嗆<EFBFBD>?*: <20>?<3F>嘥<EFBFBD><E598A5><EFBFBD>𧋦
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user