HaHafeng
4c6eaaecbf
Summary: - Implement async file upload processing (Platform-Only pattern) - Add parseExcelWorker with pg-boss queue - Implement React Query polling mechanism - Add clean data caching (avoid duplicate parsing) - Fix pivot single-value column tuple issue - Optimize performance by 99 percent Technical Details: 1. Async Architecture (Postgres-Only): - SessionService.createSession: Fast upload + push to queue (3s) - parseExcelWorker: Background parsing + save clean data (53s) - SessionController.getSessionStatus: Status query API for polling - React Query Hook: useSessionStatus (auto-serial polling) - Frontend progress bar with real-time feedback 2. Performance Optimization: - Clean data caching: Worker saves processed data to OSS - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent) - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent) - Intelligent cleaning: Boundary detection + ghost column/row removal - Safety valve: Max 3000 columns, 5M cells 3. Bug Fixes: - Fix pivot column name tuple issue for single value column - Fix queue name format (colon to underscore: asl:screening -> asl_screening) - Fix polling storm (15+ concurrent requests -> 1 serial request) - Fix QUEUE_TYPE environment variable (memory -> pgboss) - Fix logger import in PgBossQueue - Fix formatSession to return cleanDataKey - Fix saveProcessedData to update clean data synchronously 4. Database Changes: - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000) - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL 5. Documentation: - Create Postgres-Only async task processing guide (588 lines) - Update Tool C status document (Day 10 summary) - Update DC module status document - Update system overview document - Update cloud-native development guide Performance Improvements: - Upload + preview: 96s -> 53.5s (-44 percent) - Filter operation: 44s -> 2.5s (-94 percent) - Pivot operation: 45s -> 2.5s (-94 percent) - Concurrent requests: 15+ -> 1 (-93 percent) - Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent) Files Changed: - Backend: 15 files (Worker, Service, Controller, Schema, Config) - Frontend: 4 files (Hook, Component, API) - Docs: 4 files (Guide, Status, Overview, Spec) - Database: 4 column modifications - Total: ~1388 lines of new/modified code Status: Fully tested and verified, production ready
通用能力层
层级定义： 跨业务模块共享的核心技术能力
核心原则： 可复用、高内聚、独立部署
📋 能力清单
| 能力 | 说明 | 复用率 | 优先级 | 状态 |
|---|---|---|---|---|
| 01-LLM大模型网关 | 统一管理LLM调用、成本控制、模型切换 | 71% (5/7) | P0 | ⏳ 待实现 |
| 02-文档处理引擎 | PDF/Docx/Txt提取、OCR、表格提取 | 86% (6/7) | P0 | ✅ 已实现 |
| 03-RAG引擎 | 向量检索、语义搜索、RAG问答 | 43% (3/7) | P1 | ✅ 已实现 |
| 04-数据ETL引擎 | Excel JOIN、数据清洗、数据转换 | 29% (2/7) | P2 | ⏳ 待实现 |
| 05-医学NLP引擎 | 医学实体识别、术语标准化 | 14% (1/7) | P2 | ⏳ 待实现 |
🎯 设计原则
1. 可复用性
- 多个业务模块共享
- 避免重复开发
2. 独立部署
- 可以独立为微服务
- 支持独立扩展
3. 高内聚
- 每个能力职责单一
- 接口清晰
4. 领域知识
- 包含业务领域知识
- 不是纯技术组件
📊 复用率分析
LLM网关 - 71%复用率（最高优先级）
- AIA（AI智能问答）
- ASL（AI智能文献）
- PKB（个人知识库）
- DC（数据清洗）
- RVW（稿件审查）
文档处理引擎 - 86%复用率（已实现）
- ASL、PKB、DC、SSA、ST、RVW
RAG引擎 - 43%复用率（已实现）
- AIA、ASL、PKB
📚 快速导航
快速上下文
- [AI对接] 通用能力快速上下文.md - 2-3分钟了解通用能力层
核心能力
🔗 相关文档
最后更新： 2025-11-06
维护人： 技术架构师