feat(admin): Add user management and upgrade to module permission system
Features - User Management (Phase 4.1): - Database: Add user_modules table for fine-grained module permissions - Database: Add 4 user permissions (view/create/edit/delete) to role_permissions - Backend: UserService (780 lines) - CRUD with tenant isolation - Backend: UserController + UserRoutes (648 lines) - 13 API endpoints - Backend: Batch import users from Excel - Frontend: UserListPage (412 lines) - list/filter/search/pagination - Frontend: UserFormPage (341 lines) - create/edit with module config - Frontend: UserDetailPage (393 lines) - details/tenant/module management - Frontend: 3 modal components (592 lines) - import/assign/configure - API: GET/POST/PUT/DELETE /api/admin/users/* endpoints Architecture Upgrade - Module Permission System: - Backend: Add getUserModules() method in auth.service - Backend: Login API returns modules array in user object - Frontend: AuthContext adds hasModule() method - Frontend: Navigation filters modules based on user.modules - Frontend: RouteGuard checks requiredModule instead of requiredVersion - Frontend: Remove deprecated version-based permission system - UX: Only show accessible modules in navigation (clean UI) - UX: Smart redirect after login (avoid 403 for regular users) Fixes: - Fix UTF-8 encoding corruption in ~100 docs files - Fix pageSize type conversion in userService (String to Number) - Fix authUser undefined error in TopNavigation - Fix login redirect logic with role-based access check - Update Git commit guidelines v1.2 with UTF-8 safety rules Database Changes: - CREATE TABLE user_modules (user_id, tenant_id, module_code, is_enabled) - ADD UNIQUE CONSTRAINT (user_id, tenant_id, module_code) - INSERT 4 permissions + role assignments - UPDATE PUBLIC tenant with 8 module subscriptions Technical: - Backend: 5 new files (~2400 lines) - Frontend: 10 new files (~2500 lines) - Docs: 1 development record + 2 status updates + 1 guideline update - Total: ~4900 lines of code Status: User management 100% complete, module permission system operational
This commit is contained in:
@@ -1,20 +1,20 @@
|
||||
# 通用能力层技术债务清单
|
||||
|
||||
> **<EFBFBD><EFBFBD>﹝<EFBFBD><EFBFBD>𧋦嚗?* v1.0
|
||||
> **<EFBFBD>𥕦遣<EFBFBD>交<EFBFBD>嚗?* 2025-12-22
|
||||
> **文档版本:** v1.0
|
||||
> **创建日期:** 2025-12-22
|
||||
> **维护者:** 平台架构团队
|
||||
> **<EFBFBD><EFBFBD>﹝<EFBFBD>桃<EFBFBD>嚗?* 霈啣<E99C88><E595A3>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285>隡睃<E99AA1>憿對<E686BF><E5B08D><EFBFBD>紡<EFBFBD>芣䔉餈凋誨
|
||||
> **文档目的:** 记录通用能力层待优化项,指导未来迭代
|
||||
|
||||
---
|
||||
|
||||
## 📋 概述
|
||||
|
||||
<EFBFBD>祆<EFBFBD>獢<EFBFBD>抅鈭?**DC Tool C 撘<EFBFBD>郊<EFBFBD>嗆<EFBFBD>摰噼殿**嚗?025-12-22嚗㚁<E59A97><E39A81>餌<EFBFBD><E9A48C>𤑳緵<F0A491B3><E7B7B5>虾隞交𡂝鞊∩蛹<E288A9>𡁶鍂<F0A181B6>賢<EFBFBD><E8B3A2><EFBFBD>芋撘譌<E69298>?
|
||||
本文档基于 **DC Tool C 异步架构实践**(2025-12-22),总结发现的可以抽象为通用能力的模式。
|
||||
|
||||
**<EFBFBD>詨<EFBFBD><EFBFBD>脲<EFBFBD>**嚗?
|
||||
- <EFBFBD>?**敶枏<E695B6><E69E8F><EFBFBD><EFBFBD>撌脩<E6928C>敺<EFBFBD>末**嚗<><E59A97>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD><E69285>靘𥕦<E99D98><F0A595A6>渡<EFBFBD><E6B8A1>箇<EFBFBD>霈暹鴌嚗?
|
||||
- <EFBFBD>?**銝𡁜𦛚璅∪<E79285>甇<EFBFBD>&雿輻鍂鈭<E98D82><E988AD>鈭𥡝<E988AD><F0A5A19D>?*
|
||||
- <EFBFBD>哨<EFBFBD> **<EFBFBD>芣䔉<EFBFBD>臭誑餈𥕢<EFBFBD>甇交𡂝鞊?*嚗<>釺銝𦠜溶<F0A6A09C>梧<EFBFBD><E6A2A7>𧼮<EFBFBD><F0A7BCAE><EFBFBD>嚗?
|
||||
**核心思想**:
|
||||
- ✅ **当前分层已经很好**(通用能力层提供完整的基础设施)
|
||||
- ✅ **业务模块正确使用了这些能力**
|
||||
- ⏭️ **未来可以进一步抽象**(锦上添花,非必需)
|
||||
|
||||
---
|
||||
|
||||
@@ -22,18 +22,18 @@
|
||||
|
||||
### TD-COMMON-001: 前端通用轮询Hook
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83> P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**优先级**:⭐⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**预期收益**:代码复用性提升,模块间统一
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
**当前状态**:
|
||||
- Tool C 实现:`dc/tool-c/hooks/useSessionStatus.ts`
|
||||
- ASL 实现:`asl/hooks/useScreeningTask.ts`
|
||||
- 代码重复度:70%
|
||||
|
||||
**<EFBFBD>滚<EFBFBD><EFBFBD><EFBFBD><EFBFBD>餉<EFBFBD>**嚗?
|
||||
**重复的逻辑**:
|
||||
```typescript
|
||||
// 每个模块都要写类似的代码
|
||||
useQuery({
|
||||
@@ -51,7 +51,7 @@ useQuery({
|
||||
|
||||
#### 解决方案
|
||||
|
||||
**<EFBFBD>質情銝粹<EFBFBD>𡁶鍂Hook**嚗?
|
||||
**抽象为通用Hook**:
|
||||
|
||||
```typescript
|
||||
// frontend-v2/src/common/hooks/useAsyncTaskPolling.ts(新建)
|
||||
@@ -65,7 +65,7 @@ interface UseAsyncTaskPollingOptions<T> {
|
||||
/** 状态查询API函数 */
|
||||
queryFn: (taskId: string) => Promise<T>;
|
||||
|
||||
/** <EFBFBD>嗆<EFBFBD><EFBFBD><EFBFBD><EFBFBD>硋遆<EFBFBD>?*/
|
||||
/** 状态提取函数 */
|
||||
getStatus: (data: T) => 'pending' | 'processing' | 'ready' | 'error' | string;
|
||||
|
||||
/** 进度提取函数(可选) */
|
||||
@@ -74,7 +74,7 @@ interface UseAsyncTaskPollingOptions<T> {
|
||||
/** 是否启用 */
|
||||
enabled?: boolean;
|
||||
|
||||
/** 頧株砭<EFBFBD>湧<EFBFBD>嚗<EFBFBD>神蝘𡜐<EFBFBD>嚗屸<EFBFBD>霈?000 */
|
||||
/** 轮询间隔(毫秒),默认2000 */
|
||||
pollingInterval?: number;
|
||||
}
|
||||
|
||||
@@ -122,7 +122,7 @@ export function useAsyncTaskPolling<T>({
|
||||
}
|
||||
```
|
||||
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
**使用示例**:
|
||||
|
||||
```typescript
|
||||
// Tool C 使用
|
||||
@@ -144,33 +144,33 @@ const { status, isReady } = useAsyncTaskPolling({
|
||||
|
||||
#### 影响范围
|
||||
|
||||
- Tool C: `useSessionStatus.ts` <EFBFBD>舐<EFBFBD><EFBFBD>?
|
||||
- ASL: `useScreeningTask.ts` <EFBFBD>舐<EFBFBD><EFBFBD>?
|
||||
- Tool B: <EFBFBD>芣䔉<EFBFBD>滨垢<EFBFBD>舐凒<EFBFBD>乩蝙<EFBFBD>?
|
||||
- Tool C: `useSessionStatus.ts` 可简化
|
||||
- ASL: `useScreeningTask.ts` 可简化
|
||||
- Tool B: 未来前端可直接使用
|
||||
- 其他模块: 直接复用
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-002: Clean Data 缓存服务
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83>潃?P1嚗<31><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗𡁏<E59A97>扯<EFBFBD><E689AF>𣂼<EFBFBD>99%嚗峕<E59A97><E5B395>㗇芋<E39787>堒<EFBFBD><E5A092>?
|
||||
**优先级**:⭐⭐⭐⭐ P1(高)
|
||||
**工作量**:1天
|
||||
**预期收益**:性能提升99%,所有模块受益
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗帋<EFBFBD>摮?`${fileKey}_clean.json`
|
||||
- ASL<EFBFBD><EFBFBD>ool B嚗𡁏𧊋摰䂿緵嚗䔶<EFBFBD><EFBFBD>嗆<EFBFBD>甈⊿<EFBFBD><EFBFBD>啗圾<EFBFBD>?
|
||||
**当前状态**:
|
||||
- Tool C 实现:保存 `${fileKey}_clean.json`
|
||||
- ASL、Tool B:未实现,仍然每次重新解析
|
||||
|
||||
**<EFBFBD>滚<EFBFBD>霈∠<EFBFBD><EFBFBD>桅<EFBFBD>**嚗?
|
||||
- ASL <EFBFBD><EFBFBD>讃蝑偦<EFBFBD>㚁<EFBFBD>瘥𤩺活隞?OSS 銝贝蝸 PDF嚗屸<EFBFBD><EFBFBD>啗圾<EFBFBD>琜<EFBFBD>5-10蝘?蝭<><E89DAD>
|
||||
- Tool B <EFBFBD>唳旿<EFBFBD>𣂼<EFBFBD>嚗𡁏<EFBFBD>甈⊿<EFBFBD><EFBFBD>啗粉<EFBFBD>?Excel
|
||||
- Tool C <EFBFBD>滢<EFBFBD>嚗𡁜歇隡睃<EFBFBD>嚗Ếlean data蝻枏<EFBFBD>嚗?
|
||||
**重复计算问题**:
|
||||
- ASL 文献筛选:每次从 OSS 下载 PDF,重新解析(5-10秒/篇)
|
||||
- Tool B 数据提取:每次重新读取 Excel
|
||||
- Tool C 操作:已优化(clean data缓存)
|
||||
|
||||
#### 解决方案
|
||||
|
||||
**<EFBFBD>質情銝粹<EFBFBD>𡁶鍂<EFBFBD>滚𦛚**嚗?
|
||||
**抽象为通用服务**:
|
||||
|
||||
```typescript
|
||||
// backend/src/common/services/DataCacheService.ts(新建)
|
||||
@@ -180,12 +180,12 @@ import { prisma } from '../../config/database';
|
||||
import { logger } from '../logging';
|
||||
|
||||
/**
|
||||
* <EFBFBD>唳旿蝻枏<EFBFBD><EFBFBD>滚𦛚嚗<EFBFBD><EFBFBD>𡁶鍂嚗?
|
||||
* 数据缓存服务(通用)
|
||||
*
|
||||
* 用途:
|
||||
* - Worker 閫<EFBFBD><EFBFBD><EFBFBD>𦒘<EFBFBD>摮睃<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?
|
||||
* - Service 隡睃<EFBFBD>霂餃<EFBFBD>蝻枏<EFBFBD>嚗屸<EFBFBD><EFBFBD>漤<EFBFBD>憭滩恣蝞?
|
||||
* - <EFBFBD>滢<EFBFBD><EFBFBD>𤾸<EFBFBD>甇交凒<EFBFBD>啁<EFBFBD>摮?
|
||||
* - Worker 解析后保存处理结果
|
||||
* - Service 优先读取缓存,避免重复计算
|
||||
* - 操作后同步更新缓存
|
||||
*/
|
||||
export class DataCacheService {
|
||||
/**
|
||||
@@ -193,8 +193,8 @@ export class DataCacheService {
|
||||
*
|
||||
* @param originalKey 原始文件key
|
||||
* @param cleanData 清洗后的数据
|
||||
* @param suffix <EFBFBD>𡒊<EFBFBD>嚗<EFBFBD><EFBFBD>霈?'_clean.json'嚗?
|
||||
* @returns clean data <EFBFBD>?OSS key
|
||||
* @param suffix 后缀(默认 '_clean.json')
|
||||
* @returns clean data 的 OSS key
|
||||
*/
|
||||
async saveCleanData(
|
||||
originalKey: string,
|
||||
@@ -223,7 +223,7 @@ export class DataCacheService {
|
||||
/**
|
||||
* 读取清洗后的数据
|
||||
*
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
* @returns 清洗后的数据
|
||||
*/
|
||||
async getCleanData(cleanDataKey: string): Promise<any> {
|
||||
@@ -242,8 +242,8 @@ export class DataCacheService {
|
||||
/**
|
||||
* 更新清洗后的数据
|
||||
*
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
* @param newData <EFBFBD>唳㺭<EFBFBD>?
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
* @param newData 新数据
|
||||
*/
|
||||
async updateCleanData(cleanDataKey: string, newData: any): Promise<void> {
|
||||
logger.info('[DataCacheService] Updating clean data', {
|
||||
@@ -260,7 +260,7 @@ export class DataCacheService {
|
||||
/**
|
||||
* 删除清洗后的数据
|
||||
*
|
||||
* @param cleanDataKey clean data <EFBFBD>?OSS key
|
||||
* @param cleanDataKey clean data 的 OSS key
|
||||
*/
|
||||
async deleteCleanData(cleanDataKey: string): Promise<void> {
|
||||
try {
|
||||
@@ -278,46 +278,46 @@ export class DataCacheService {
|
||||
export const dataCacheService = new DataCacheService();
|
||||
```
|
||||
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
**使用示例**:
|
||||
|
||||
```typescript
|
||||
// Worker 銝?
|
||||
// Worker 中
|
||||
const cleanDataKey = await dataCacheService.saveCleanData(fileKey, cleanedData);
|
||||
await prisma.update({ where: { id }, data: { cleanDataKey } });
|
||||
|
||||
// Service 銝?
|
||||
// Service 中
|
||||
if (record.cleanDataKey) {
|
||||
return await dataCacheService.getCleanData(record.cleanDataKey);
|
||||
}
|
||||
|
||||
// <EFBFBD>滢<EFBFBD><EFBFBD>擧凒<EFBFBD>?
|
||||
// 操作后更新
|
||||
await dataCacheService.updateCleanData(record.cleanDataKey, newData);
|
||||
```
|
||||
|
||||
#### 影响范围
|
||||
|
||||
- Tool C: 蝞<EFBFBD><EFBFBD>𣇉緵<EFBFBD>劐誨<EFBFBD>?
|
||||
- ASL: <EFBFBD><EFBFBD>讃閫<EFBFBD><EFBFBD>蝏𤘪<EFBFBD>蝻枏<EFBFBD>嚗<EFBFBD><EFBFBD><EFBFBD>?9%嚗?
|
||||
- Tool C: 简化现有代码
|
||||
- ASL: 文献解析结果缓存(提升99%)
|
||||
- Tool B: 数据提取结果缓存
|
||||
- <EFBFBD><EFBFBD><EFBFBD>㗇芋<EFBFBD>? 蝏煺<E89D8F><E785BA><EFBFBD><EFBFBD>摮䀹㦤<E480B9>?
|
||||
- 所有模块: 统一的缓存机制
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-003: <EFBFBD>箄<EFBFBD>皜<EFBFBD><EFBFBD>蝞埈<EFBFBD><EFBFBD>𡁶鍂<EFBFBD>?
|
||||
### TD-COMMON-003: 智能清洗算法通用化
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**预期收益**:代码复用,避免重复实现
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗䫤intelligentCleanData`嚗<EFBFBD>器<EFBFBD>峕<EFBFBD>瘚?摰匧<E691B0><E58CA7><EFBFBD>嚗?
|
||||
**当前状态**:
|
||||
- Tool C 实现:`intelligentCleanData`(边界检测+安全阀)
|
||||
- 其他模块:未实现类似功能
|
||||
|
||||
**<EFBFBD>臭誑<EFBFBD>𡁶鍂<EFBFBD>𣇉<EFBFBD>蝞埈<EFBFBD>**嚗?
|
||||
**可以通用化的算法**:
|
||||
1. 幽灵列检测(边界检测)
|
||||
2. 撟賜<EFBFBD>銵諹<EFBFBD>皛?
|
||||
2. 幽灵行过滤
|
||||
3. 安全阀(最大列数、单元格数限制)
|
||||
|
||||
#### 解决方案
|
||||
@@ -327,7 +327,7 @@ await dataCacheService.updateCleanData(record.cleanDataKey, newData);
|
||||
|
||||
export interface CleaningOptions {
|
||||
maxCols?: number; // 最大列数,默认3000
|
||||
maxCells?: number; // <EFBFBD><EFBFBD>憭批<EFBFBD><EFBFBD><EFBFBD>聢<EFBFBD>堆<EFBFBD>暺䁅恕500銝?
|
||||
maxCells?: number; // 最大单元格数,默认500万
|
||||
removeEmptyRows?: boolean; // 是否删除空行,默认true
|
||||
removeEmptyCols?: boolean; // 是否删除空列,默认true
|
||||
}
|
||||
@@ -341,7 +341,7 @@ export function intelligentCleanData(
|
||||
}
|
||||
|
||||
export function isValidValue(value: any): boolean {
|
||||
// 蝏煺<EFBFBD><EFBFBD><EFBFBD>征<EFBFBD>澆ế<EFBFBD>?
|
||||
// 统一的空值判断
|
||||
// ...
|
||||
}
|
||||
```
|
||||
@@ -349,19 +349,19 @@ export function isValidValue(value: any): boolean {
|
||||
#### 影响范围
|
||||
|
||||
- Tool C: 复用通用实现
|
||||
- <EFBFBD>嗡<EFBFBD>銝𠹺<EFBFBD>Excel<EFBFBD><EFBFBD><EFBFBD><EFBFBD>? <20>湔𦻖雿輻鍂
|
||||
- 其他上传Excel的功能: 直接使用
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-004: Worker注册辅助工具
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97> P3嚗<33><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.3憭?
|
||||
**优先级**:⭐ P3(低)
|
||||
**工作量**:0.3天
|
||||
**预期收益**:降低Worker注册代码重复
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
**当前状态**:
|
||||
- 每个模块都要手写 Worker 注册代码
|
||||
- 错误处理逻辑重复
|
||||
|
||||
@@ -403,11 +403,11 @@ export function registerWorker(options: RegisterWorkerOptions) {
|
||||
}
|
||||
});
|
||||
|
||||
logger.info(`[WorkerHelper] <EFBFBD>?Worker registered: ${queueName}`);
|
||||
logger.info(`[WorkerHelper] ✅ Worker registered: ${queueName}`);
|
||||
}
|
||||
```
|
||||
|
||||
**雿輻鍂蝷箔<EFBFBD>**嚗?
|
||||
**使用示例**:
|
||||
|
||||
```typescript
|
||||
// 简化的注册代码
|
||||
@@ -418,7 +418,7 @@ registerWorker({
|
||||
// 业务逻辑
|
||||
return result;
|
||||
},
|
||||
onStart: (job) => console.log(`撘<EFBFBD>憪见<EFBFBD><EFBFBD>? ${job.id}`),
|
||||
onStart: (job) => console.log(`开始处理: ${job.id}`),
|
||||
onComplete: (job, result) => console.log(`完成: ${result}`),
|
||||
onError: (job, error) => console.error(`失败: ${error}`),
|
||||
});
|
||||
@@ -428,36 +428,36 @@ registerWorker({
|
||||
|
||||
### TD-COMMON-005: 数据版本管理系统
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃鐥<E6BD83>潃?P1嚗<31><E59A97>嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗𡁏𣈲<F0A1818F><F0A388B2>曎撘𤩺<E69298>雿栶<E99BBF><E6A0B6>ndo<64>蠘<EFBFBD><E8A098><EFBFBD>紡<EFBFBD>箏<EFBFBD><E7AE8F>脩<EFBFBD><E884A9>?
|
||||
**优先级**:⭐⭐⭐⭐ P1(高)
|
||||
**工作量**:3天
|
||||
**预期收益**:支持链式操作、undo功能、导出历史版本
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>𣂼<EFBFBD>**嚗?
|
||||
- <EFBFBD>滢<EFBFBD>銝齿糓蝝舐妖<EFBFBD><EFBFBD><EFBFBD>瘥𤩺活<EFBFBD>箔<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>唳旿嚗?
|
||||
- <EFBFBD>䭾<EFBFBD><EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>唳<EFBFBD>銝芣<EFBFBD>雿𨅯<EFBFBD><EFBFBD><EFBFBD>𠶖<EFBFBD>?
|
||||
- <EFBFBD>䭾<EFBFBD>撖澆枂銝剝𡢿<EFBFBD><EFBFBD>𧋦<EFBFBD><EFBFBD>㺭<EFBFBD>?
|
||||
**当前限制**:
|
||||
- 操作不是累积的(每次基于原始数据)
|
||||
- 无法回退到某个操作前的状态
|
||||
- 无法导出中间版本的数据
|
||||
|
||||
**<EFBFBD>冽<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>極雿𨀣<EFBFBD>**嚗?
|
||||
**用户期望的工作流**:
|
||||
```
|
||||
上传(v0: 100行)
|
||||
<EFBFBD>?
|
||||
蝑偦<EFBFBD>㚁<EFBFBD>v1: 50銵䕘<EFBFBD><EFBFBD>?<3F>臭誑<E887AD>鮋<EFBFBD><E9AE8B><EFBFBD>啗<EFBFBD><E59597>?
|
||||
<EFBFBD>?
|
||||
<EFBFBD>啣<EFBFBD>潭<EFBFBD>撠<EFBFBD><EFBFBD>v2: 50銵䕘<E98AB5><E49598>㗇<EFBFBD>撠<EFBFBD><E692A0><EFBFBD>?<3F>臭誑<E887AD>鮋<EFBFBD><E9AE8B><EFBFBD>啗<EFBFBD><E59597>?
|
||||
<EFBFBD>?
|
||||
Pivot嚗ǒ3: 銝滚<EFBFBD>蝏𤘪<EFBFBD>嚗?
|
||||
<EFBFBD>?
|
||||
撖澆枂嚗𡁜虾隞亙紡<EFBFBD>?v0<76><30>1<EFBFBD><31>2<EFBFBD><32>3 隞餅<E99A9E><E9A485><EFBFBD>𧋦
|
||||
↓
|
||||
筛选(v1: 50行)← 可以回退到这里
|
||||
↓
|
||||
数值映射(v2: 50行,有映射)← 可以回退到这里
|
||||
↓
|
||||
Pivot(v3: 不同结构)
|
||||
↓
|
||||
导出:可以导出 v0、v1、v2、v3 任意版本
|
||||
```
|
||||
|
||||
#### 解决方案
|
||||
|
||||
**Prisma Schema霈曇恣**嚗?
|
||||
**Prisma Schema设计**:
|
||||
|
||||
```prisma
|
||||
// <EFBFBD><EFBFBD>𧋦蝞∠<EFBFBD>銵剁<EFBFBD><EFBFBD>𡁶鍂嚗?
|
||||
// 版本管理表(通用)
|
||||
model DataVersion {
|
||||
id String @id @default(uuid())
|
||||
|
||||
@@ -466,19 +466,19 @@ model DataVersion {
|
||||
entityId String // Session ID | Project ID | ...
|
||||
|
||||
// 版本信息
|
||||
versionNumber Int // 0=<EFBFBD>笔<EFBFBD>, 1=蝚?甈⊥<E79488>雿𨅯<E99BBF>, 2=蝚?甈⊥<E79488>雿𨅯<E99BBF>...
|
||||
versionNumber Int // 0=原始, 1=第1次操作后, 2=第2次操作后...
|
||||
dataKey String // OSS中的数据文件key
|
||||
|
||||
// 操作记录
|
||||
operation String? // 'upload' | 'filter' | 'pivot' | 'recode' ...
|
||||
operationParams Json? // 操作参数
|
||||
|
||||
// <EFBFBD><EFBFBD>㺭<EFBFBD>?
|
||||
// 元数据
|
||||
totalRows Int
|
||||
totalCols Int
|
||||
columns Json
|
||||
|
||||
// <EFBFBD>園𡢿<EFBFBD>?
|
||||
// 时间戳
|
||||
createdAt DateTime @default(now())
|
||||
createdBy String // 用户ID
|
||||
|
||||
@@ -488,22 +488,22 @@ model DataVersion {
|
||||
@@schema("platform_schema")
|
||||
}
|
||||
|
||||
// 銝𡁜𦛚銵冽溶<EFBFBD>惩<EFBFBD>畾?
|
||||
// 业务表添加字段
|
||||
model DcToolCSession {
|
||||
// ...现有字段
|
||||
|
||||
currentVersion Int @default(0) // 敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
currentVersion Int @default(0) // 当前版本号
|
||||
}
|
||||
```
|
||||
|
||||
**Service摰䂿緵**嚗?
|
||||
**Service实现**:
|
||||
|
||||
```typescript
|
||||
// backend/src/common/services/DataVersionService.ts(新建)
|
||||
|
||||
export class DataVersionService {
|
||||
/**
|
||||
* <EFBFBD>𥕦遣<EFBFBD>啁<EFBFBD><EFBFBD>?
|
||||
* 创建新版本
|
||||
*/
|
||||
async createVersion(
|
||||
entityType: string,
|
||||
@@ -513,7 +513,7 @@ export class DataVersionService {
|
||||
operation?: string,
|
||||
params?: any
|
||||
): Promise<string> {
|
||||
// 靽嘥<EFBFBD><EFBFBD>唳旿<EFBFBD>?OSS
|
||||
// 保存数据到 OSS
|
||||
const dataKey = `versions/${entityType}/${entityId}/v${versionNumber}.json`;
|
||||
await storage.upload(dataKey, JSON.stringify(data));
|
||||
|
||||
@@ -553,21 +553,21 @@ export class DataVersionService {
|
||||
}
|
||||
});
|
||||
|
||||
if (!version) throw new Error('<EFBFBD><EFBFBD>𧋦銝滚<EFBFBD><EFBFBD>?);
|
||||
if (!version) throw new Error('版本不存在');
|
||||
|
||||
const buffer = await storage.download(version.dataKey);
|
||||
return JSON.parse(buffer.toString('utf-8'));
|
||||
}
|
||||
|
||||
/**
|
||||
* <EFBFBD>鮋<EFBFBD><EFBFBD><EFBFBD>唳<EFBFBD>摰𡁶<EFBFBD><EFBFBD>?
|
||||
* 回退到指定版本
|
||||
*/
|
||||
async rollbackToVersion(
|
||||
entityType: string,
|
||||
entityId: string,
|
||||
versionNumber: number
|
||||
): Promise<void> {
|
||||
// <EFBFBD>湔鰵敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
// 更新当前版本号
|
||||
await this.updateCurrentVersion(entityType, entityId, versionNumber);
|
||||
}
|
||||
|
||||
@@ -583,16 +583,16 @@ export class DataVersionService {
|
||||
}
|
||||
```
|
||||
|
||||
**QuickAction <EFBFBD><EFBFBD><EFBFBD>**嚗?
|
||||
**QuickAction 集成**:
|
||||
|
||||
```typescript
|
||||
// QuickAction 执行后自动创建新版本
|
||||
const result = await python.execute(fullData, params);
|
||||
|
||||
// 霂餃<EFBFBD>敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
// 读取当前版本号
|
||||
const currentVersion = session.currentVersion || 0;
|
||||
|
||||
// <EFBFBD>𥕦遣<EFBFBD>啁<EFBFBD><EFBFBD>?
|
||||
// 创建新版本
|
||||
await dataVersionService.createVersion(
|
||||
'dc_toolc_session',
|
||||
sessionId,
|
||||
@@ -602,7 +602,7 @@ await dataVersionService.createVersion(
|
||||
params
|
||||
);
|
||||
|
||||
// <EFBFBD>湔鰵敶枏<EFBFBD><EFBFBD><EFBFBD>𧋦<EFBFBD>?
|
||||
// 更新当前版本号
|
||||
await prisma.update({
|
||||
where: { id: sessionId },
|
||||
data: { currentVersion: currentVersion + 1 }
|
||||
@@ -611,28 +611,28 @@ await prisma.update({
|
||||
|
||||
#### 影响范围
|
||||
|
||||
- Tool C: <EFBFBD>舀<EFBFBD><EFBFBD>曉<EFBFBD><EFBFBD>滢<EFBFBD><EFBFBD><EFBFBD>ndo<EFBFBD><EFBFBD>紡<EFBFBD>箏<EFBFBD><EFBFBD>?
|
||||
- ASL: <EFBFBD><EFBFBD>讃蝑偦<EFBFBD>厩<EFBFBD>憭𡁻𧫴畾萇<EFBFBD><EFBFBD>𦦵恣<EFBFBD>?
|
||||
- Tool B: <EFBFBD>唳旿<EFBFBD>𣂼<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>祉恣<EFBFBD>?
|
||||
- <EFBFBD><EFBFBD><EFBFBD>㗇芋<EFBFBD>? 蝏煺<E89D8F><E785BA><EFBFBD><EFBFBD><EFBFBD>祉恣<E7A589><E681A3><EFBFBD><EFBFBD>?
|
||||
- Tool C: 支持链式操作、undo、导出历史
|
||||
- ASL: 文献筛选的多阶段结果管理
|
||||
- Tool B: 数据提取的版本管理
|
||||
- 所有模块: 统一的版本管理能力
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-006: 撟賜<EFBFBD><EFBFBD>?銵峕<E98AB5>瘚讠<E7989A>瘜訫<E7989C>
|
||||
### TD-COMMON-006: 幽灵列/行检测算法库
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.5憭?
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.5天
|
||||
**预期收益**:Excel处理质量提升
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗朞器<EFBFBD>峕<EFBFBD>瘚讠<EFBFBD>瘜?
|
||||
**当前状态**:
|
||||
- Tool C 实现:边界检测算法
|
||||
- 其他模块:未实现
|
||||
|
||||
**Excel<EFBFBD>澆<EFBFBD>瘙⊥<EFBFBD><EFBFBD>桅<EFBFBD>**嚗?
|
||||
- <EFBFBD>冽<EFBFBD><EFBFBD>琿<EFBFBD><EFBFBD>脣<EFBFBD>16384<EFBFBD>?<3F>?閫<><E996AB><EFBFBD>?6384<38>梹<EFBFBD>摰鮋<E691B0><E9AE8B>芣<EFBFBD>151<35>埈<EFBFBD><E59F88><EFBFBD><EFBFBD>
|
||||
- <EFBFBD>冽<EFBFBD><EFBFBD>𣳇膄<EFBFBD>唳旿<EFBFBD>芣<EFBFBD><EFBFBD>?<3F>?閫<><E996AB><EFBFBD>箏之<E7AE8F>讐征銵?
|
||||
**Excel格式污染问题**:
|
||||
- 用户刷颜色到16384列 → 解析出16384列(实际只有151列有效)
|
||||
- 用户删除数据未清理 → 解析出大量空行
|
||||
|
||||
#### 解决方案
|
||||
|
||||
@@ -654,23 +654,23 @@ export function cleanExcelData(
|
||||
rawData: any[],
|
||||
options?: CleaningOptions
|
||||
): CleaningResult {
|
||||
// 摰䂿緵<EFBFBD>𡁶鍂<EFBFBD><EFBFBD><EFBFBD>瘣㛖<EFBFBD>瘜?
|
||||
// 实现通用的清洗算法
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### TD-COMMON-007: <EFBFBD>滨垢餈𥕦漲<EFBFBD>∠<EFBFBD>隞?
|
||||
### TD-COMMON-007: 前端进度条组件
|
||||
|
||||
**隡睃<EFBFBD>蝥?*嚗尠<E59A97>潃?P2嚗<32>葉嚗?
|
||||
**撌乩<EFBFBD><EFBFBD>?*嚗?.3憭?
|
||||
**憸<EFBFBD><EFBFBD><EFBFBD>嗥<EFBFBD>**嚗䦧I蝏煺<E89D8F>嚗𣬚鍂<F0A3AC9A>瑚<EFBFBD>撉峕<E69289><E5B395>?
|
||||
**优先级**:⭐⭐ P2(中)
|
||||
**工作量**:0.3天
|
||||
**预期收益**:UI统一,用户体验提升
|
||||
|
||||
#### 问题描述
|
||||
|
||||
**敶枏<EFBFBD><EFBFBD>嗆<EFBFBD>?*嚗?
|
||||
- Tool C 摰䂿緵嚗𡁜<EFBFBD><EFBFBD>磰<EFBFBD>摨行辺嚗<EFBFBD><EFBFBD><EFBFBD>莎<EFBFBD>Header銝𧢲䲮嚗?
|
||||
**当前状态**:
|
||||
- Tool C 实现:内联进度条(蓝色,Header下方)
|
||||
- 其他模块:未实现或不统一
|
||||
|
||||
#### 解决方案
|
||||
@@ -714,64 +714,63 @@ export const AsyncProgressBar: React.FC<AsyncProgressBarProps> = ({
|
||||
|
||||
## 📅 实施计划
|
||||
|
||||
| <EFBFBD><EFBFBD><EFBFBD>臬<EFBFBD>箏𦛚 | 隡睃<E99AA1>蝥?| 撌乩<E6928C><E4B9A9>?| 撱箄悅<E7AE84>嗆㦤 | <20>嗥<EFBFBD> |
|
||||
| 技术债务 | 优先级 | 工作量 | 建议时机 | 收益 |
|
||||
|---------|--------|--------|---------|------|
|
||||
| TD-COMMON-001 <EFBFBD>滨垢頧株砭Hook | P2 | 0.5憭?| 銝𧢲活餈凋誨 | 隞<><E99A9E>蝏煺<E89D8F> |
|
||||
| TD-COMMON-002 Clean Data<EFBFBD>滚𦛚 | **P1** | 1憭?| **銝𧢲活餈凋誨** | **<EFBFBD>扯<EFBFBD><EFBFBD>𣂼<EFBFBD>99%** |
|
||||
| TD-COMMON-003 <EFBFBD>箄<EFBFBD>皜<EFBFBD><EFBFBD>蝞埈<EFBFBD> | P2 | 0.5憭?| <20><>閬<EFBFBD>𧒄 | 韐券<E99F90><E588B8>𣂼<EFBFBD> |
|
||||
| TD-COMMON-004 Worker瘜典<EFBFBD>颲<EFBFBD>𨭌 | P3 | 0.3憭?| <20>舫<EFBFBD>?| 隞<><E99A9E>蝞<EFBFBD><E89D9E>?|
|
||||
| TD-COMMON-005 <EFBFBD>唳旿<EFBFBD><EFBFBD>𧋦蝞∠<EFBFBD> | **P1** | 3憭?| **銝𧢲活餈凋誨** | **<EFBFBD>曉<EFBFBD><EFBFBD>滢<EFBFBD>** |
|
||||
| TD-COMMON-006 撟賜<EFBFBD><EFBFBD>?銵峕<E98AB5>瘚?| P2 | 0.5憭?| <20><>閬<EFBFBD>𧒄 | 韐券<E99F90><E588B8>𣂼<EFBFBD> |
|
||||
| TD-COMMON-007 餈𥕦漲<EFBFBD>∠<EFBFBD>隞?| P2 | 0.3憭?| <20><>閬<EFBFBD>𧒄 | UI蝏煺<EFBFBD> |
|
||||
| TD-COMMON-001 前端轮询Hook | P2 | 0.5天 | 下次迭代 | 代码统一 |
|
||||
| TD-COMMON-002 Clean Data服务 | **P1** | 1天 | **下次迭代** | **性能提升99%** |
|
||||
| TD-COMMON-003 智能清洗算法 | P2 | 0.5天 | 需要时 | 质量提升 |
|
||||
| TD-COMMON-004 Worker注册辅助 | P3 | 0.3天 | 可选 | 代码简化 |
|
||||
| TD-COMMON-005 数据版本管理 | **P1** | 3天 | **下次迭代** | **链式操作** |
|
||||
| TD-COMMON-006 幽灵列/行检测 | P2 | 0.5天 | 需要时 | 质量提升 |
|
||||
| TD-COMMON-007 进度条组件 | P2 | 0.3天 | 需要时 | UI统一 |
|
||||
|
||||
**<EFBFBD>刻<EFBFBD>隡睃<EFBFBD>蝥?*嚗?
|
||||
1. 潃鐥<EFBFBD>潃鐥<EFBFBD>潃?TD-COMMON-002嚗㇃lean Data<EFBFBD>滚𦛚嚗?
|
||||
2. 潃鐥<EFBFBD>潃鐥<EFBFBD>潃?TD-COMMON-005嚗<EFBFBD>㺭<EFBFBD>桃<EFBFBD><EFBFBD>祉恣<EFBFBD><EFBFBD><EFBFBD>
|
||||
3. 潃鐥<EFBFBD>潃?TD-COMMON-001嚗<EFBFBD><EFBFBD>蝡航蔭霂%ook嚗?
|
||||
**推荐优先级**:
|
||||
1. ⭐⭐⭐⭐⭐ TD-COMMON-002(Clean Data服务)
|
||||
2. ⭐⭐⭐⭐⭐ TD-COMMON-005(数据版本管理)
|
||||
3. ⭐⭐⭐ TD-COMMON-001(前端轮询Hook)
|
||||
|
||||
---
|
||||
|
||||
## 📊 当前架构评价
|
||||
|
||||
### <EFBFBD>?撌脩<E6928C>敺<EFBFBD>末<EFBFBD><E69CAB><EFBFBD><EFBFBD>?
|
||||
### ✅ 已经很好的部分
|
||||
|
||||
1. <EFBFBD>?**<2A>𡁶鍂<F0A181B6>賢<EFBFBD>撅<EFBFBD>抅蝖<E68A85>霈暹鴌摰峕㟲**
|
||||
- jobQueue嚗īg-boss<EFBFBD>笔<EFBFBD>嚗?
|
||||
1. ✅ **通用能力层基础设施完整**
|
||||
- jobQueue(pg-boss队列)
|
||||
- CheckpointService(断点续传)
|
||||
- 任务拆分工具
|
||||
- storage、logger、cache
|
||||
|
||||
2. <EFBFBD>?**銝𡁜𦛚璅∪<E79285>甇<EFBFBD>&雿輻鍂<E8BCBB>𡁶鍂<F0A181B6>賢<EFBFBD>**
|
||||
2. ✅ **业务模块正确使用通用能力**
|
||||
- Tool C、ASL、Tool B 统一使用 jobQueue
|
||||
- Platform-Only模式
|
||||
- 不在业务表中存储任务管理信息
|
||||
|
||||
3. <EFBFBD>?**<2A><><EFBFBD>皜<EFBFBD>苊嚗諹<E59A97>韐<EFBFBD><E99F90>蝖?*
|
||||
3. ✅ **分层清晰,职责明确**
|
||||
- 通用能力层:基础设施
|
||||
- 銝𡁜𦛚璅∪<EFBFBD>嚗𡁜<EFBFBD>雿枏<EFBFBD><EFBFBD>?
|
||||
- 业务模块:具体实现
|
||||
|
||||
### <EFBFBD>哨<EFBFBD> <20>臭誑<E887AD>寡<EFBFBD><E5AFA1><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>𧼮<EFBFBD><F0A7BCAE><EFBFBD>嚗?
|
||||
### ⏭️ 可以改进的部分(非必需)
|
||||
|
||||
1. ⏭️ 前端轮询模式可以更统一
|
||||
2. ⏭️ clean data缓存可以更通用
|
||||
3. <EFBFBD>哨<EFBFBD> <20>唳旿<E594B3><E697BF>𧋦蝞∠<E89D9E>敺<EFBFBD>遣蝡?
|
||||
3. ⏭️ 数据版本管理待建立
|
||||
|
||||
**结论**:当前架构已经符合规范,未来优化是锦上添花!
|
||||
|
||||
---
|
||||
|
||||
## <EFBFBD><EFBFBD> <20><><EFBFBD><EFBFBD><EFBFBD>獢?
|
||||
## 📚 参考文档
|
||||
|
||||
- [Postgres-Only异步任务处理指南](./Postgres-Only异步任务处理指南.md) - 完整实践
|
||||
- [鈭穃<EFBFBD><EFBFBD>笔<EFBFBD><EFBFBD>𤏸<EFBFBD><EFBFBD><EFBFBD>(../04-撘<><E69298>𤏸<EFBFBD><F0A48FB8>?08-鈭穃<E988AD><E7A983>笔<EFBFBD><E7AC94>𤏸<EFBFBD><F0A48FB8>?md) - 撘<EFBFBD><EFBFBD>𤏸<EFBFBD><EFBFBD>?
|
||||
- [DC Tool C<EFBFBD>嗆<EFBFBD><EFBFBD>(../03-銝𡁜𦛚璅∪<E79285>/DC-<2D>唳旿皜<E697BF><E79A9C><EFBFBD>渡<EFBFBD>/00-撌亙<E6928C>C敶枏<E695B6><E69E8F>嗆<EFBFBD><E59786><EFBFBD>撘<EFBFBD><E69298>烐<EFBFBD><E78390>?md) - Day 10摰噼殿
|
||||
- [云原生开发规范](../04-开发规范/08-云原生开发规范.md) - 开发规范
|
||||
- [DC Tool C状态](../03-业务模块/DC-数据清洗整理/00-工具C当前状态与开发指南.md) - Day 10实践
|
||||
|
||||
---
|
||||
|
||||
**蝏湔擪<EFBFBD>?*: 撟喳蝱<E596B3>嗆<EFBFBD><E59786>a<EFBFBD>
|
||||
**<EFBFBD><EFBFBD><EFBFBD>擧凒<EFBFBD>?*: 2025-12-22
|
||||
**<EFBFBD><EFBFBD>﹝<EFBFBD>嗆<EFBFBD>?*: <20>?<3F>嘥<EFBFBD><E598A5><EFBFBD>𧋦
|
||||
|
||||
**维护者**: 平台架构团队
|
||||
**最后更新**: 2025-12-22
|
||||
**文档状态**: ✅ 初始版本
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user