feat(platform): Complete Postgres-Only architecture refactoring (Phase 1-7)
Major Changes: - Implement Platform-Only architecture pattern (unified task management) - Add PostgresCacheAdapter for unified caching (platform_schema.app_cache) - Add PgBossQueue for job queue management (platform_schema.job) - Implement CheckpointService using job.data (generic for all modules) - Add intelligent threshold-based dual-mode processing (THRESHOLD=50) - Add task splitting mechanism (auto chunk size recommendation) - Refactor ASL screening service with smart mode selection - Refactor DC extraction service with smart mode selection - Register workers for ASL and DC modules Technical Highlights: - All task management data stored in platform_schema.job.data (JSONB) - Business tables remain clean (no task management fields) - CheckpointService is generic (shared by all modules) - Zero code duplication (DRY principle) - Follows 3-layer architecture principle - Zero additional cost (no Redis needed, save 8400 CNY/year) Code Statistics: - New code: ~1750 lines - Modified code: ~500 lines - Test code: ~1800 lines - Documentation: ~3000 lines Testing: - Unit tests: 8/8 passed - Integration tests: 2/2 passed - Architecture validation: passed - Linter errors: 0 Files: - Platform layer: PostgresCacheAdapter, PgBossQueue, CheckpointService, utils - ASL module: screeningService, screeningWorker - DC module: ExtractionController, extractionWorker - Tests: 11 test files - Docs: Updated 4 key documents Status: Phase 1-7 completed, Phase 8-9 pending
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
# DC数据清洗整理模块 - 当前状态与开发指南
|
||||
|
||||
> **文档版本:** v3.1
|
||||
> **文档版本:** v3.2
|
||||
> **创建日期:** 2025-11-28
|
||||
> **维护者:** DC模块开发团队
|
||||
> **最后更新:** 2025-12-10 ✅ **Tool C UX重大改进完成!**
|
||||
> **重大里程碑:** Tool C MVP + 7个功能按钮 + NA处理 + Pivot优化 + UX重大改进(筛选/行号/滚动条/全量数据)
|
||||
> **最后更新:** 2025-12-13 🏆 **Postgres-Only 架构改造完成!**
|
||||
> **重大里程碑:** Tool C MVP完成 + Tool B Postgres-Only架构改造(智能双模式、任务拆分、断点续传)
|
||||
> **文档目的:** 反映模块真实状态,记录开发历程
|
||||
|
||||
---
|
||||
@@ -55,12 +55,18 @@
|
||||
DC数据清洗整理模块提供4个智能工具,帮助研究人员清洗、整理、提取医疗数据。
|
||||
|
||||
### 当前状态
|
||||
- **开发阶段**:✅ **Tool B MVP完成** + ✅ **Tool C MVP + NA处理优化 + Pivot优化完成**
|
||||
- **开发阶段**:✅ **Tool B MVP完成** + ✅ **Tool C MVP完成** + 🏆 **Postgres-Only 架构改造完成**
|
||||
- **已完成功能**:
|
||||
- ✅ Portal:智能数据清洗工作台(2025-12-02)
|
||||
- ✅ Tool B 后端:病历结构化机器人(2025-11-28重建完成)
|
||||
- ✅ Tool B 前端:5步工作流完整实现(2025-12-03)
|
||||
- ✅ Tool B API对接:6个端点全部集成(2025-12-03)
|
||||
- ✅ **🏆 Tool B Postgres-Only 架构改造**(2025-12-13,Phase 7):
|
||||
- ✅ 智能双模式处理(<50条直接,≥50条队列)
|
||||
- ✅ 任务拆分机制(1000条→20批)
|
||||
- ✅ 断点续传支持(支持长时间提取任务)
|
||||
- ✅ Platform层统一管理(job.data存储)
|
||||
- ✅ Worker注册(extractionWorker.ts)
|
||||
- ✅ **Tool C 完整实现**(2025-12-06 ~ 2025-12-10):
|
||||
- ✅ Python微服务(~1800行,Day 1 + NA处理优化 + 全量数据处理)
|
||||
- ✅ Node.js后端(~3500行,Day 2-3,Day 5-8增强 + 全量返回)
|
||||
@@ -75,6 +81,7 @@ DC数据清洗整理模块提供4个智能工具,帮助研究人员清洗、
|
||||
- **重大成就**:
|
||||
- 🎉 **前端通用能力层建设完成**
|
||||
- ✨ 基于 Ant Design X 的 Chat 组件库
|
||||
- 🏆 **Platform-Only 架构创新**(与 ASL 统一架构)
|
||||
- 🚀 可复用于 AIA、PKB、Tool C 等模块
|
||||
- ✅ **NA处理全面支持**:数值映射、分箱、条件生成列、筛选
|
||||
- ✅ **Pivot优化**:保留未选列+原始列顺序
|
||||
@@ -200,18 +207,54 @@ Excel处理: xlsx 库(内存模式)
|
||||
✅ 状态:代码已重建,100%功能恢复
|
||||
```
|
||||
|
||||
#### 基础设施(云原生)
|
||||
#### 基础设施(云原生 + 🏆 Postgres-Only 架构)
|
||||
```
|
||||
数据库: PostgreSQL 16 with Schema isolation
|
||||
Schema: dc_schema (独立隔离)
|
||||
存储: storage服务(OSS ↔ LocalFS)
|
||||
缓存: cache服务(Redis ↔ Memory)
|
||||
|
||||
🏆 Postgres-Only 架构(2025-12-13 改造完成):
|
||||
├── 统一缓存: platform_schema.app_cache (PostgresCacheAdapter)
|
||||
├── 统一队列: platform_schema.job (PgBossQueue, pg-boss)
|
||||
├── 任务管理: job.data 统一存储(不在业务表中)
|
||||
└── 断点续传: CheckpointService (操作 job.data,通用)
|
||||
|
||||
日志: logger服务(Winston结构化)
|
||||
任务队列: jobQueue服务(异步处理)
|
||||
|
||||
✅ 状态:100%复用平台基础设施
|
||||
✅ 架构:Platform-Only 模式,与 ASL 统一
|
||||
```
|
||||
|
||||
#### 🏆 Postgres-Only 架构亮点(Tool B)
|
||||
|
||||
**智能双模式处理:**
|
||||
```typescript
|
||||
const QUEUE_THRESHOLD = 50;
|
||||
|
||||
if (records.length >= 50) {
|
||||
// 队列模式:可靠性优先
|
||||
- 任务拆分(50条/批)
|
||||
- 断点续传(每10条保存)
|
||||
- 支持长时间提取(2-24小时)
|
||||
- 实例重启可恢复
|
||||
} else {
|
||||
// 直接模式:性能优先
|
||||
- 快速响应(<1分钟)
|
||||
- 无队列延迟
|
||||
}
|
||||
```
|
||||
|
||||
**核心组件:**
|
||||
- `ExtractionController.ts`:智能阈值判断,推送批次任务
|
||||
- `extractionWorker.ts`:批次处理,断点续传
|
||||
- `CheckpointService`:操作 job.data(与 ASL 共用)
|
||||
|
||||
**技术优势:**
|
||||
- ✅ 零额外成本(无需 Redis)
|
||||
- ✅ 统一架构(与 ASL 一致)
|
||||
- ✅ 高可靠性(自动重试、断点续传)
|
||||
- ✅ 易扩展(未来模块直接复用)
|
||||
|
||||
---
|
||||
|
||||
## 📂 真实代码结构
|
||||
|
||||
Reference in New Issue
Block a user