Files
AIclinicalresearch/deploy-to-sae.ps1
HaHafeng 4c6eaaecbf feat(dc): Implement Postgres-Only async architecture and performance optimization
Summary:
- Implement async file upload processing (Platform-Only pattern)
- Add parseExcelWorker with pg-boss queue
- Implement React Query polling mechanism
- Add clean data caching (avoid duplicate parsing)
- Fix pivot single-value column tuple issue
- Optimize performance by 99 percent

Technical Details:

1. Async Architecture (Postgres-Only):
   - SessionService.createSession: Fast upload + push to queue (3s)
   - parseExcelWorker: Background parsing + save clean data (53s)
   - SessionController.getSessionStatus: Status query API for polling
   - React Query Hook: useSessionStatus (auto-serial polling)
   - Frontend progress bar with real-time feedback

2. Performance Optimization:
   - Clean data caching: Worker saves processed data to OSS
   - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent)
   - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent)
   - Intelligent cleaning: Boundary detection + ghost column/row removal
   - Safety valve: Max 3000 columns, 5M cells

3. Bug Fixes:
   - Fix pivot column name tuple issue for single value column
   - Fix queue name format (colon to underscore: asl:screening -> asl_screening)
   - Fix polling storm (15+ concurrent requests -> 1 serial request)
   - Fix QUEUE_TYPE environment variable (memory -> pgboss)
   - Fix logger import in PgBossQueue
   - Fix formatSession to return cleanDataKey
   - Fix saveProcessedData to update clean data synchronously

4. Database Changes:
   - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000)
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL

5. Documentation:
   - Create Postgres-Only async task processing guide (588 lines)
   - Update Tool C status document (Day 10 summary)
   - Update DC module status document
   - Update system overview document
   - Update cloud-native development guide

Performance Improvements:
- Upload + preview: 96s -> 53.5s (-44 percent)
- Filter operation: 44s -> 2.5s (-94 percent)
- Pivot operation: 45s -> 2.5s (-94 percent)
- Concurrent requests: 15+ -> 1 (-93 percent)
- Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent)

Files Changed:
- Backend: 15 files (Worker, Service, Controller, Schema, Config)
- Frontend: 4 files (Hook, Component, API)
- Docs: 4 files (Guide, Status, Overview, Spec)
- Database: 4 column modifications
- Total: ~1388 lines of new/modified code

Status: Fully tested and verified, production ready
2025-12-22 21:30:31 +08:00

146 lines
4.1 KiB
PowerShell
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# ===================================================================
# AI临床研究平台 - SAE部署自动化脚本
# ===================================================================
# 用途自动构建Docker镜像并推送到阿里云容器镜像服务
# 使用方法:.\deploy-to-sae.ps1 -Version "v1.0.0"
# 作者:技术架构师
# 日期2025-12-11
# ===================================================================
param(
[Parameter(Mandatory=$true)]
[string]$Version,
[Parameter(Mandatory=$false)]
[string]$Registry = "registry.cn-hangzhou.aliyuncs.com",
[Parameter(Mandatory=$false)]
[string]$Namespace = "aiclinical",
[Parameter(Mandatory=$false)]
[string]$Repository = "backend-dev"
)
# 颜色输出函数
function Write-ColorOutput($ForegroundColor) {
$fc = $host.UI.RawUI.ForegroundColor
$host.UI.RawUI.ForegroundColor = $ForegroundColor
if ($args) {
Write-Output $args
}
$host.UI.RawUI.ForegroundColor = $fc
}
# 标题
Write-ColorOutput Cyan "================================================"
Write-ColorOutput Cyan " AI临床研究平台 - SAE部署脚本"
Write-ColorOutput Cyan "================================================"
Write-Host ""
# 检查Docker是否运行
Write-ColorOutput Yellow "🔍 检查Docker状态..."
try {
docker ps > $null 2>&1
if ($LASTEXITCODE -ne 0) {
Write-ColorOutput Red "❌ Docker未运行请先启动Docker Desktop"
exit 1
}
Write-ColorOutput Green "✅ Docker运行正常"
} catch {
Write-ColorOutput Red "❌ Docker未安装或未运行"
exit 1
}
# 构建镜像
Write-Host ""
Write-ColorOutput Yellow "🔨 开始构建Docker镜像..."
Write-Host "版本: $Version"
Write-Host "目录: backend/"
Write-Host ""
Set-Location backend
$imageName = "aiclinical-backend"
$fullImageName = "${Registry}/${Namespace}/${Repository}:${Version}"
Write-ColorOutput Cyan "执行: docker build -t ${imageName}:${Version} ."
docker build -t "${imageName}:${Version}" .
if ($LASTEXITCODE -ne 0) {
Write-ColorOutput Red "❌ Docker镜像构建失败"
exit 1
}
Write-ColorOutput Green "✅ Docker镜像构建成功"
# 打标签
Write-Host ""
Write-ColorOutput Yellow "🏷️ 为镜像打标签..."
Write-ColorOutput Cyan "执行: docker tag ${imageName}:${Version} ${fullImageName}"
docker tag "${imageName}:${Version}" $fullImageName
if ($LASTEXITCODE -ne 0) {
Write-ColorOutput Red "❌ 打标签失败"
exit 1
}
Write-ColorOutput Green "✅ 标签创建成功"
# 推送镜像
Write-Host ""
Write-ColorOutput Yellow "📤 推送镜像到阿里云..."
Write-Host "目标仓库: $fullImageName"
Write-Host ""
Write-ColorOutput Cyan "提示:如果提示需要登录,请执行以下命令:"
Write-ColorOutput Cyan "docker login --username=<你的阿里云账号> ${Registry}"
Write-Host ""
Write-ColorOutput Cyan "执行: docker push ${fullImageName}"
docker push $fullImageName
if ($LASTEXITCODE -ne 0) {
Write-ColorOutput Red "❌ 镜像推送失败"
Write-ColorOutput Yellow "请检查:"
Write-Host " 1. 是否已登录阿里云容器镜像服务"
Write-Host " 2. 命名空间和仓库名称是否正确"
Write-Host " 3. 网络连接是否正常"
exit 1
}
Write-ColorOutput Green "✅ 镜像推送成功"
# 完成
Write-Host ""
Write-ColorOutput Cyan "================================================"
Write-ColorOutput Green "🎉 部署准备完成!"
Write-ColorOutput Cyan "================================================"
Write-Host ""
Write-Host "镜像信息:"
Write-ColorOutput Cyan " 完整地址: $fullImageName"
Write-ColorOutput Cyan " 版本号: $Version"
Write-Host ""
Write-Host "下一步操作:"
Write-Host " 1. 登录阿里云SAE控制台"
Write-Host " 2. 进入应用 '${Repository}'"
Write-Host " 3. 点击'部署应用'"
Write-Host " 4. 选择镜像版本: $Version"
Write-Host " 5. 点击'确定'开始部署"
Write-Host ""
Write-ColorOutput Cyan "部署文档: docs/05-部署文档/02-SAE部署完全指南(产品经理版).md"
Write-Host ""
Set-Location ..