HaHafeng
4c6eaaecbf
Summary: - Implement async file upload processing (Platform-Only pattern) - Add parseExcelWorker with pg-boss queue - Implement React Query polling mechanism - Add clean data caching (avoid duplicate parsing) - Fix pivot single-value column tuple issue - Optimize performance by 99 percent Technical Details: 1. Async Architecture (Postgres-Only): - SessionService.createSession: Fast upload + push to queue (3s) - parseExcelWorker: Background parsing + save clean data (53s) - SessionController.getSessionStatus: Status query API for polling - React Query Hook: useSessionStatus (auto-serial polling) - Frontend progress bar with real-time feedback 2. Performance Optimization: - Clean data caching: Worker saves processed data to OSS - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent) - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent) - Intelligent cleaning: Boundary detection + ghost column/row removal - Safety valve: Max 3000 columns, 5M cells 3. Bug Fixes: - Fix pivot column name tuple issue for single value column - Fix queue name format (colon to underscore: asl:screening -> asl_screening) - Fix polling storm (15+ concurrent requests -> 1 serial request) - Fix QUEUE_TYPE environment variable (memory -> pgboss) - Fix logger import in PgBossQueue - Fix formatSession to return cleanDataKey - Fix saveProcessedData to update clean data synchronously 4. Database Changes: - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000) - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL 5. Documentation: - Create Postgres-Only async task processing guide (588 lines) - Update Tool C status document (Day 10 summary) - Update DC module status document - Update system overview document - Update cloud-native development guide Performance Improvements: - Upload + preview: 96s -> 53.5s (-44 percent) - Filter operation: 44s -> 2.5s (-94 percent) - Pivot operation: 45s -> 2.5s (-94 percent) - Concurrent requests: 15+ -> 1 (-93 percent) - Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent) Files Changed: - Backend: 15 files (Worker, Service, Controller, Schema, Config) - Frontend: 4 files (Hook, Component, API) - Docs: 4 files (Guide, Status, Overview, Spec) - Database: 4 column modifications - Total: ~1388 lines of new/modified code Status: Fully tested and verified, production ready
66 lines
1.6 KiB
PowerShell
66 lines
1.6 KiB
PowerShell
# DC模块代码恢复脚本运行器
|
||
Write-Host "==================================================================================" -ForegroundColor Cyan
|
||
Write-Host "DC模块代码恢复工具" -ForegroundColor Cyan
|
||
Write-Host "==================================================================================" -ForegroundColor Cyan
|
||
|
||
# 检查Python是否安装
|
||
try {
|
||
$pythonVersion = python --version 2>&1
|
||
Write-Host "✅ Python版本: $pythonVersion" -ForegroundColor Green
|
||
} catch {
|
||
Write-Host "❌ 错误: Python未安装或不在PATH中" -ForegroundColor Red
|
||
exit 1
|
||
}
|
||
|
||
# 运行恢复脚本
|
||
Write-Host "`n📦 开始运行恢复脚本..." -ForegroundColor Yellow
|
||
|
||
$output = python recover_dc_code.py 2>&1 | Out-String
|
||
Write-Host $output
|
||
|
||
# 保存输出到文件
|
||
$output | Out-File "recovery_output.log" -Encoding UTF8
|
||
Write-Host "`n💾 输出已保存到: recovery_output.log" -ForegroundColor Green
|
||
|
||
# 检查恢复结果
|
||
if (Test-Path "recovered_dc_code") {
|
||
Write-Host "`n✅ 恢复目录已创建!" -ForegroundColor Green
|
||
Write-Host "📁 恢复的文件:" -ForegroundColor Cyan
|
||
Get-ChildItem "recovered_dc_code" | Format-Table Name, Length, LastWriteTime
|
||
} else {
|
||
Write-Host "`n⚠️ 未创建恢复目录，可能未找到DC代码" -ForegroundColor Yellow
|
||
}
|
||
|
||
Write-Host "`n==================================================================================" -ForegroundColor Cyan
|
||
Write-Host "完成！请查看 recovered_dc_code 目录" -ForegroundColor Cyan
|
||
Write-Host "==================================================================================" -ForegroundColor Cyan
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|