Develop/AIclinicalresearch
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
109 Commits 1 Branch 0 Tags
dac3cecf78a976a1bdab5d223e55ba836969ef79
Commit Graph

15 Commits

Author SHA1 Message Date
HaHafeng
dac3cecf78 feat(iit): Complete IIT Manager Agent Day 1 - Environment initialization and WeChat integration 
Summary:
- Complete IIT Manager Agent MVP Day 1 (12.5% progress)
- Database: Create iit_schema with 5 tables (IitProject, IitPendingAction, IitTaskRun, IitUserMapping, IitAuditLog)
- Backend: Add module structure (577 lines) and types (223 lines)
- WeChat: Configure Enterprise WeChat app (CorpID, AgentID, Secret)
- WeChat: Obtain web authorization and JS-SDK authorization
- WeChat: Configure trusted domain (iit.xunzhengyixue.com)
- Frontend: Deploy v1.2 with WeChat domain verification file
- Frontend: Fix CRLF issue in docker-entrypoint.sh (CRLF -> LF)
- Testing: 11/11 database CRUD tests passed
- Testing: Access Token retrieval test passed
- Docs: Create module status and development guide
- Docs: Update MVP task list with Day 1 completion
- Docs: Rename deployment doc to SAE real-time status record
- Deployment: Update frontend internal IP to 172.17.173.80

Technical Details:
- Prisma: Multi-schema support (iit_schema)
- pg-boss: Job queue integration prepared
- Taro 4.x: Framework selected for WeChat Mini Program
- Shadow State: Architecture foundation laid
- Docker: Fix entrypoint script line endings for Linux container

Status: Day 1/14 complete, ready for Day 2 REDCap integration
 2026-01-01 14:32:58 +08:00
HaHafeng
4c5bb3d174 feat(iit): Initialize IIT Manager Agent MVP - Day 1 complete 
- Add iit_schema with 5 tables
- Create module structure and types (223 lines)
- WeChat integration verified (Access Token success)
- Update system docs to v2.4
- Add REDCap source folders to .gitignore
- Day 1/14 complete (11/11 tasks)
 2025-12-31 18:35:05 +08:00
HaHafeng
decff0bb1f docs(deploy): Complete full system deployment to Aliyun SAE 
Summary:
- Successfully deployed complete system to Aliyun SAE (2025-12-25)
- All services running: Python microservice + Node.js backend + Frontend Nginx + CLB
- Public access available at http://8.140.53.236/

Major Achievements:
1. Python microservice deployed (v1.0, internal IP: 172.17.173.66:8000)
2. Node.js backend deployed (v1.3, internal IP: 172.17.173.73:3001)
   - Fixed 4 critical issues: bash path, config directory, pino-pretty, ES Module
3. Frontend Nginx deployed (v1.0, internal IP: 172.17.173.72:80)
4. CLB load balancer configured (public IP: 8.140.53.236)

New Documentation (9 docs):
- 11-Node.js backend SAE deployment config checklist (21 env vars)
- 12-Node.js backend SAE deployment operation manual
- 13-Node.js backend image fix record (config directory)
- 14-Node.js backend pino-pretty fix
- 15-Node.js backend deployment success summary
- 16-Frontend Nginx deployment success summary
- 17-Complete deployment practical manual 2025 edition (1800 lines)
- 18-Deployment documentation usage guide
- 19-Daily update quick operation manual (670 lines)

Key Fixes:
- Environment variable name correction: EXTRACTION_SERVICE_URL (not PYTHON_SERVICE_URL)
- Dockerfile fix: added COPY config ./config
- Logger configuration: conditional pino-pretty for dev only
- Health check fix: ES Module compatibility (require -> import)

Updated Files:
- System status document updated with full deployment info
- Deployment progress overview updated with latest IPs
- All 3 Docker services' Dockerfiles and configs refined

Verification:
- All health checks passed
- Tool C 7 features working correctly
- Literature screening module functional
- Response time < 1 second

BREAKING CHANGE: Node.js backend internal IP changed from 172.17.173.71 to 172.17.173.73

Closes #deployment-milestone
 2025-12-25 21:24:37 +08:00
HaHafeng
691dc2bc98 docs(deploy): Update deployment documentation for Node.js backend 
Summary:
- Created Node.js backend Docker image build guide
- Updated deployment progress overview with backend status
- Updated system status documentation

Backend build achievements:
- Fixed 200+ TypeScript compilation errors (200+ to 0)
- Completed Prisma reverse sync (32 models from RDS)
- Manually added 30+ Prisma relation fields
- Successfully built Docker image (838MB)
- Pushed image to ACR (v1.0 + latest tags)

Documentation updates:
- Added 10-Node.js后端-Docker镜像构建手册.md
- Updated 00-部署进度总览.md with backend deployment status
- Updated 00-系统当前状态与开发指南.md with latest progress
- Fixed date format (2024 -> 2025)

Next steps:
- Deploy Node.js backend to SAE
- Configure environment variables
- Test end-to-end functionality

Status: Backend Docker image ready for SAE deployment
 2025-12-25 08:21:21 +08:00
HaHafeng
ef967d7d7c build(backend): Complete Node.js backend deployment preparation 
Major changes:
- Add Docker configuration (Dockerfile, .dockerignore)
- Fix 200+ TypeScript compilation errors
- Add Prisma schema relations for all models (30+ relations)
- Update tsconfig.json to relax non-critical checks
- Optimize Docker build with local dist strategy

Technical details:
- Exclude test files from TypeScript compilation
- Add manual relations for ASL, PKB, DC, AIA modules
- Use type assertions for JSON/Buffer compatibility
- Fix pg-boss, extractionWorker, and other legacy code issues

Build result:
- Docker image: 838MB (compressed ~186MB)
- Successfully pushed to ACR
- Zero TypeScript compilation errors

Related docs:
- Update deployment documentation
- Add Python microservice SAE deployment guide
 2025-12-24 22:12:00 +08:00
HaHafeng
b64896a307 feat(deploy): Complete PostgreSQL migration and Docker image build 
Summary:
- PostgreSQL database migration to RDS completed (90MB SQL, 11 schemas)
- Frontend Nginx Docker image built and pushed to ACR (v1.0, ~50MB)
- Python microservice Docker image built and pushed to ACR (v1.0, 1.12GB)
- Created 3 deployment documentation files

Docker Configuration Files:
- frontend-v2/Dockerfile: Multi-stage build with nginx:alpine
- frontend-v2/.dockerignore: Optimize build context
- frontend-v2/nginx.conf: SPA routing and API proxy
- frontend-v2/docker-entrypoint.sh: Dynamic env injection
- extraction_service/Dockerfile: Multi-stage build with Aliyun Debian mirror
- extraction_service/.dockerignore: Optimize build context
- extraction_service/requirements-prod.txt: Production dependencies (removed Nougat)

Deployment Documentation:
- docs/05-部署文档/00-部署进度总览.md: One-stop deployment status overview
- docs/05-部署文档/07-前端Nginx-SAE部署操作手册.md: Frontend deployment guide
- docs/05-部署文档/08-PostgreSQL数据库部署操作手册.md: Database deployment guide
- docs/00-系统总体设计/00-系统当前状态与开发指南.md: Updated with deployment status

Database Migration:
- RDS instance: pgm-2zex1m2y3r23hdn5 (2C4G, PostgreSQL 15.0)
- Database: ai_clinical_research
- Schemas: 11 business schemas migrated successfully
- Data: 3 users, 2 projects, 1204 literatures verified
- Backup: rds_init_20251224_154529.sql (90MB)

Docker Images:
- Frontend: crpi-cd5ij4pjt65mweeo.cn-beijing.personal.cr.aliyuncs.com/ai-clinical/ai-clinical_frontend-nginx:v1.0
- Python: crpi-cd5ij4pjt65mweeo.cn-beijing.personal.cr.aliyuncs.com/ai-clinical/python-extraction:v1.0

Key Achievements:
- Resolved Docker Hub network issues (using generic tags)
- Fixed 30 TypeScript compilation errors
- Removed Nougat OCR to reduce image size by 1.5GB
- Used Aliyun Debian mirror to resolve apt-get network issues
- Implemented multi-stage builds for optimization

Next Steps:
- Deploy Python microservice to SAE
- Build Node.js backend Docker image
- Deploy Node.js backend to SAE
- Deploy frontend Nginx to SAE
- End-to-end verification testing

Status: Docker images ready, SAE deployment pending
 2025-12-24 18:21:55 +08:00
HaHafeng
4c6eaaecbf feat(dc): Implement Postgres-Only async architecture and performance optimization 
Summary:
- Implement async file upload processing (Platform-Only pattern)
- Add parseExcelWorker with pg-boss queue
- Implement React Query polling mechanism
- Add clean data caching (avoid duplicate parsing)
- Fix pivot single-value column tuple issue
- Optimize performance by 99 percent

Technical Details:

1. Async Architecture (Postgres-Only):
   - SessionService.createSession: Fast upload + push to queue (3s)
   - parseExcelWorker: Background parsing + save clean data (53s)
   - SessionController.getSessionStatus: Status query API for polling
   - React Query Hook: useSessionStatus (auto-serial polling)
   - Frontend progress bar with real-time feedback

2. Performance Optimization:
   - Clean data caching: Worker saves processed data to OSS
   - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent)
   - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent)
   - Intelligent cleaning: Boundary detection + ghost column/row removal
   - Safety valve: Max 3000 columns, 5M cells

3. Bug Fixes:
   - Fix pivot column name tuple issue for single value column
   - Fix queue name format (colon to underscore: asl:screening -> asl_screening)
   - Fix polling storm (15+ concurrent requests -> 1 serial request)
   - Fix QUEUE_TYPE environment variable (memory -> pgboss)
   - Fix logger import in PgBossQueue
   - Fix formatSession to return cleanDataKey
   - Fix saveProcessedData to update clean data synchronously

4. Database Changes:
   - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000)
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL
   - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL

5. Documentation:
   - Create Postgres-Only async task processing guide (588 lines)
   - Update Tool C status document (Day 10 summary)
   - Update DC module status document
   - Update system overview document
   - Update cloud-native development guide

Performance Improvements:
- Upload + preview: 96s -> 53.5s (-44 percent)
- Filter operation: 44s -> 2.5s (-94 percent)
- Pivot operation: 45s -> 2.5s (-94 percent)
- Concurrent requests: 15+ -> 1 (-93 percent)
- Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent)

Files Changed:
- Backend: 15 files (Worker, Service, Controller, Schema, Config)
- Frontend: 4 files (Hook, Component, API)
- Docs: 4 files (Guide, Status, Overview, Spec)
- Database: 4 column modifications
- Total: ~1388 lines of new/modified code

Status: Fully tested and verified, production ready
 2025-12-22 21:30:31 +08:00
HaHafeng
9b81aef9a7 feat(dc): Add multi-metric transformation feature (direction 1+2) 
Summary:
- Implement intelligent multi-metric grouping detection algorithm
- Add direction 1: timepoint-as-row, metric-as-column (analysis format)
- Add direction 2: timepoint-as-column, metric-as-row (display format)
- Fix column name pattern detection (FMA___ issue)
- Maintain original Record ID order in output
- Add full-select/clear buttons in UI
- Integrate into TransformDialog with Radio selection
- Update 3 documentation files

Technical Details:
- Python: detect_metric_groups(), apply_multi_metric_to_long(), apply_multi_metric_to_matrix()
- Backend: 3 new methods in QuickActionService
- Frontend: MultiMetricPanel.tsx (531 lines)
- Total: ~1460 lines of new code

Status: Fully tested and verified, ready for production
 2025-12-21 15:06:15 +08:00
HaHafeng
19f9c5ea93 docs(deployment): Fix 8 critical deployment issues and enhance documentation 
Summary of fixes:
- Fix service discovery address (change .sae domain to internal IP)
- Unify timezone configuration (Asia/Shanghai for all services)
- Enhance ECS security group configuration (Redis/Weaviate port binding)
- Add image pull strategy best practices
- Add Python service memory management guidelines
- Update Dify API Key deployment strategy (avoid deadlock)
- Add SSH tunnel for RDS database access
- Add NAT gateway cost optimization explanation

Modified files (7 docs):
- 00-部署架构总览.md (enhanced with 7 sections)
- 03-Dify-ECS部署完全指南.md (security hardening)
- 04-Python微服务-SAE容器部署指南.md (timezone + service discovery)
- 05-Node.js后端-SAE容器部署指南.md (timezone configuration)
- PostgreSQL部署策略-摸底报告.md (timezone best practice)
- 07-关键配置补充说明.md (3 new sections)
- 08-部署检查清单.md (service address fix)

New files:
- 文档修正报告-20251214.md (comprehensive fix report)
- Review documents from technical team

Impact:
- Fixed 3 P0/P1 critical issues (100% connection failure risk)
- Fixed 3 P2 important issues (stability and maintainability)
- Added 2 P3 best practices (developer convenience)

Status: All deployment documents reviewed and corrected, ready for production deployment
 2025-12-14 13:25:28 +08:00
HaHafeng
fa72beea6c feat(platform): Complete Postgres-Only architecture refactoring (Phase 1-7) 
Major Changes:
- Implement Platform-Only architecture pattern (unified task management)
- Add PostgresCacheAdapter for unified caching (platform_schema.app_cache)
- Add PgBossQueue for job queue management (platform_schema.job)
- Implement CheckpointService using job.data (generic for all modules)
- Add intelligent threshold-based dual-mode processing (THRESHOLD=50)
- Add task splitting mechanism (auto chunk size recommendation)
- Refactor ASL screening service with smart mode selection
- Refactor DC extraction service with smart mode selection
- Register workers for ASL and DC modules

Technical Highlights:
- All task management data stored in platform_schema.job.data (JSONB)
- Business tables remain clean (no task management fields)
- CheckpointService is generic (shared by all modules)
- Zero code duplication (DRY principle)
- Follows 3-layer architecture principle
- Zero additional cost (no Redis needed, save 8400 CNY/year)

Code Statistics:
- New code: ~1750 lines
- Modified code: ~500 lines
- Test code: ~1800 lines
- Documentation: ~3000 lines

Testing:
- Unit tests: 8/8 passed
- Integration tests: 2/2 passed
- Architecture validation: passed
- Linter errors: 0

Files:
- Platform layer: PostgresCacheAdapter, PgBossQueue, CheckpointService, utils
- ASL module: screeningService, screeningWorker
- DC module: ExtractionController, extractionWorker
- Tests: 11 test files
- Docs: Updated 4 key documents

Status: Phase 1-7 completed, Phase 8-9 pending
 2025-12-13 16:10:04 +08:00
HaHafeng
74cf346453 feat(dc/tool-c): Add missing value imputation feature with 6 methods and MICE 
Major features:
1. Missing value imputation (6 simple methods + MICE):
   - Mean/Median/Mode/Constant imputation
   - Forward fill (ffill) and Backward fill (bfill) for time series
   - MICE multivariate imputation (in progress, shape issue to fix)

2. Auto precision detection:
   - Automatically match decimal places of original data
   - Prevent false precision (e.g. 13.57 instead of 13.566716417910449)

3. Categorical variable detection:
   - Auto-detect and skip categorical columns in MICE
   - Show warnings for unsuitable columns
   - Suggest mode imputation for categorical data

4. UI improvements:
   - Rename button: "Delete Missing" to "Missing Value Handling"
   - Remove standalone "Dedup" and "MICE" buttons
   - 3-tab dialog: Delete / Fill / Advanced Fill
   - Display column statistics and recommended methods
   - Extended warning messages (8 seconds for skipped columns)

5. Bug fixes:
   - Fix sessionService.updateSessionData -> saveProcessedData
   - Fix OperationResult interface (add message and stats)
   - Fix Toolbar button labels and removal

Modified files:
Python: operations/fillna.py (new, 556 lines), main.py (3 new endpoints)
Backend: QuickActionService.ts, QuickActionController.ts, routes/index.ts
Frontend: MissingValueDialog.tsx (new, 437 lines), Toolbar.tsx, index.tsx
Tests: test_fillna_operations.py (774 lines), test scripts and docs
Docs: 5 documentation files updated

Known issues:
- MICE imputation has DataFrame shape mismatch issue (under debugging)
- Workaround: Use 6 simple imputation methods first

Status: Development complete, MICE debugging in progress
Lines added: ~2000 lines across 3 tiers
 2025-12-10 13:06:00 +08:00
HaHafeng
75ceeb0653 hotfix(dc/tool-c): Fix compute formula validation and binning NaN serialization 
Critical fixes:
1. Compute column: Add Chinese comma support in formula validation
   - Problem: Formula with Chinese comma failed validation
   - Fix: Add Chinese comma character to allowed_chars regex
   - Example: Support formulas like 'col1（kg）+ col2，col3'

2. Binning operation: Fix NaN serialization error
   - Problem: 'Out of range float values are not JSON compliant: nan'
   - Fix: Enhanced NaN/inf handling in binning endpoint
   - Added np.inf/-np.inf replacement before JSON serialization
   - Added manual JSON serialization with NaN->null conversion

3. Enhanced all operation endpoints for consistency
   - Updated conditional, dropna endpoints with same NaN/inf handling
   - Ensures all operations return JSON-compliant data

Modified files:
- extraction_service/operations/compute.py: Add Chinese comma to regex
- extraction_service/main.py: Enhanced NaN handling in binning/conditional/dropna

Status: Hotfix complete, ready for testing
 2025-12-09 08:45:27 +08:00
HaHafeng
91cab452d1 fix(dc/tool-c): Fix special character handling and improve UX 
Major fixes:
- Fix pivot transformation with special characters in column names
- Fix compute column validation for Chinese punctuation
- Fix recode dialog to fetch unique values from full dataset via new API
- Add column mapping mechanism to handle special characters

Database migration:
- Add column_mapping field to dc_tool_c_sessions table
- Migration file: 20251208_add_column_mapping

UX improvements:
- Darken table grid lines for better visibility
- Reduce column width by 40% with tooltip support
- Insert new columns next to source columns
- Preserve original row order after operations
- Add notice about 50-row preview limit

Modified files:
- Backend: SessionService, SessionController, QuickActionService, routes
- Python: pivot.py, compute.py, recode.py, binning.py, conditional.py
- Frontend: DataGrid, RecodeDialog, index.tsx, ag-grid-custom.css
- Database: schema.prisma, migration SQL

Status: Code complete, database migrated, ready for testing
 2025-12-08 23:20:55 +08:00
HaHafeng
f729699510 feat(dc): Complete Tool C quick action buttons Phase 1-2 - 7 functions 
Summary:
- Implement 7 quick action functions (filter, recode, binning, conditional, dropna, compute, pivot)
- Refactor to pre-written Python functions architecture (stable and secure)
- Add 7 Python operations modules with full type hints
- Add 7 frontend Dialog components with user-friendly UI
- Fix NaN serialization issues and auto type conversion
- Update all related documentation

Technical Details:
- Python: operations/ module (filter.py, recode.py, binning.py, conditional.py, dropna.py, compute.py, pivot.py)
- Backend: QuickActionService.ts with 7 execute methods
- Frontend: 7 Dialog components with complete validation
- Toolbar: Enable 7 quick action buttons

Status: Phase 1-2 completed, basic testing passed, ready for further testing
 2025-12-08 17:38:08 +08:00
HaHafeng
2c7ed94161 feat(dc/tool-c): 完成前端基础框架（Day 4 MVP） 
核心功能：
- 新增Tool C主入口（index.tsx, 258行）：状态管理+布局
- 新增Header组件（91行）：顶栏+返回按钮+导出
- 新增Toolbar组件（104行）：7个快捷按钮+搜索框
- 新增DataGrid组件（111行）：AG Grid Community集成
- 新增Sidebar组件（149行）：右侧栏骨架版
- 新增API封装（toolC.ts, 218行）：8个API方法
- 新增类型定义（types/index.ts, 62行）

AG Grid集成：
- 安装ag-grid-community + ag-grid-react
- Excel风格表格渲染
- 列排序、过滤、调整宽度
- 缺失值高亮显示（红色斜体）
- 数值右对齐
- 自定义Emerald绿色主题（ag-grid-custom.css, 113行）
- 虚拟滚动支持大数据

路由配置：
- 更新dc/index.tsx：新增ToolCModule懒加载
- 更新Portal.tsx：Tool C状态改为ready
- 路径：/data-cleaning/tool-c

API封装（8个方法）：
- uploadFile（上传CSV/Excel）
- getSession（获取Session元数据）
- getPreviewData（获取预览数据）
- updateHeartbeat（延长10分钟）
- generateCode（生成代码，不执行）
- executeCode（执行代码）
- processMessage（生成+执行，一步到位）核心API
- getChatHistory（对话历史）

文档更新：
- 新增Day 4前端基础完成总结（213行）
- 更新工具C当前状态文档
- 更新TODO清单（Day 1-4标记完成）
- 更新系统总体设计文档

测试数据准备：
- cqol-demo.csv（21列x313行真实医疗数据）
- G鼓膜穿孔数据.xlsx（备用）

Day 5待完成：
- MessageItem组件（消息渲染）
- CodeBlock组件（Prism.js代码高亮）
- InputArea组件（输入框交互）
- InsightsPanel组件（数据洞察）
- 完善Sidebar（完整Chat交互）
- 端到端测试

影响范围：
- frontend-v2/src/modules/dc/pages/tool-c/*（新增11个文件）
- frontend-v2/src/modules/dc/api/toolC.ts（新增）
- frontend-v2/src/modules/dc/index.tsx（更新路由）
- frontend-v2/src/modules/dc/pages/Portal.tsx（启用Tool C）
- docs/03-业务模块/DC-数据清洗整理/*（文档更新）
- package.json（新增依赖）

Breaking Changes: 无

总代码行数：+1106行（前端基础框架）

Refs: #Tool-C-Day4
 2025-12-07 17:40:07 +08:00