HaHafeng
4c6eaaecbf
Summary: - Implement async file upload processing (Platform-Only pattern) - Add parseExcelWorker with pg-boss queue - Implement React Query polling mechanism - Add clean data caching (avoid duplicate parsing) - Fix pivot single-value column tuple issue - Optimize performance by 99 percent Technical Details: 1. Async Architecture (Postgres-Only): - SessionService.createSession: Fast upload + push to queue (3s) - parseExcelWorker: Background parsing + save clean data (53s) - SessionController.getSessionStatus: Status query API for polling - React Query Hook: useSessionStatus (auto-serial polling) - Frontend progress bar with real-time feedback 2. Performance Optimization: - Clean data caching: Worker saves processed data to OSS - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent) - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent) - Intelligent cleaning: Boundary detection + ghost column/row removal - Safety valve: Max 3000 columns, 5M cells 3. Bug Fixes: - Fix pivot column name tuple issue for single value column - Fix queue name format (colon to underscore: asl:screening -> asl_screening) - Fix polling storm (15+ concurrent requests -> 1 serial request) - Fix QUEUE_TYPE environment variable (memory -> pgboss) - Fix logger import in PgBossQueue - Fix formatSession to return cleanDataKey - Fix saveProcessedData to update clean data synchronously 4. Database Changes: - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000) - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL 5. Documentation: - Create Postgres-Only async task processing guide (588 lines) - Update Tool C status document (Day 10 summary) - Update DC module status document - Update system overview document - Update cloud-native development guide Performance Improvements: - Upload + preview: 96s -> 53.5s (-44 percent) - Filter operation: 44s -> 2.5s (-94 percent) - Pivot operation: 45s -> 2.5s (-94 percent) - Concurrent requests: 15+ -> 1 (-93 percent) - Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent) Files Changed: - Backend: 15 files (Worker, Service, Controller, Schema, Config) - Frontend: 4 files (Hook, Component, API) - Docs: 4 files (Guide, Status, Overview, Spec) - Database: 4 column modifications - Total: ~1388 lines of new/modified code Status: Fully tested and verified, production ready
113 lines
1.8 KiB
Markdown
113 lines
1.8 KiB
Markdown
# 🚀 快速开始 - 1分钟运行测试
|
||
|
||
## Windows用户
|
||
|
||
### 方法1：双击运行（最简单）
|
||
1. 双击 `run_tests.bat`
|
||
2. 等待测试完成
|
||
|
||
### 方法2：命令行
|
||
```cmd
|
||
cd AIclinicalresearch\tests
|
||
run_tests.bat
|
||
```
|
||
|
||
---
|
||
|
||
## Linux/Mac用户
|
||
|
||
```bash
|
||
cd AIclinicalresearch/tests
|
||
chmod +x run_tests.sh
|
||
./run_tests.sh
|
||
```
|
||
|
||
---
|
||
|
||
## ⚠️ 前提条件
|
||
|
||
**必须先启动Python服务！**
|
||
|
||
```bash
|
||
# 打开新终端
|
||
cd AIclinicalresearch/extraction_service
|
||
python main.py
|
||
```
|
||
|
||
看到这行表示启动成功：
|
||
```
|
||
INFO: Application startup complete.
|
||
INFO: Uvicorn running on http://0.0.0.0:8001
|
||
```
|
||
|
||
---
|
||
|
||
## 📊 预期结果
|
||
|
||
✅ **全部通过**：
|
||
```
|
||
总测试数: 18
|
||
✅ 通过: 18
|
||
❌ 失败: 0
|
||
通过率: 100.0%
|
||
|
||
🎉 所有测试通过！
|
||
```
|
||
|
||
⚠️ **部分失败**：
|
||
- 查看红色错误信息
|
||
- 检查失败的具体测试
|
||
- 查看Python服务日志
|
||
|
||
---
|
||
|
||
## 🎯 测试内容
|
||
|
||
- ✅ 6种简单填补方法（均值、中位数、众数、固定值、前向填充、后向填充）
|
||
- ✅ MICE多重插补（单列、多列）
|
||
- ✅ 边界情况（100%缺失、0%缺失、特殊字符）
|
||
- ✅ 各种数据类型（数值、分类、混合）
|
||
- ✅ 性能测试（1000行数据）
|
||
|
||
---
|
||
|
||
## 💡 提示
|
||
|
||
- **第一次运行**会自动安装依赖（pandas, numpy, requests）
|
||
- **测试时间**约 45-60 秒
|
||
- **测试数据**自动生成，无需手动准备
|
||
- **颜色输出**：绿色=通过，红色=失败，黄色=警告
|
||
|
||
---
|
||
|
||
## 🆘 遇到问题？
|
||
|
||
### 问题1：无法连接到服务
|
||
**解决**：确保Python服务在运行（`python main.py`）
|
||
|
||
### 问题2：依赖安装失败
|
||
**解决**：手动安装 `pip install pandas numpy requests`
|
||
|
||
### 问题3：测试失败
|
||
**解决**：查看错误信息，检查代码逻辑
|
||
|
||
---
|
||
|
||
**准备好了吗？启动服务，运行测试！** 🚀
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|