3446909ff7
feat(ssa): Complete Phase I-IV intelligent dialogue and tool system development
...
Phase I - Session Blackboard + READ Layer:
- SessionBlackboardService with Postgres-Only cache
- DataProfileService for data overview generation
- PicoInferenceService for LLM-driven PICO extraction
- Frontend DataContextCard and VariableDictionaryPanel
- E2E tests: 31/31 passed
Phase II - Conversation Layer LLM + Intent Router:
- ConversationService with SSE streaming
- IntentRouterService (rule-first + LLM fallback, 6 intents)
- SystemPromptService with 6-segment dynamic assembly
- TokenTruncationService for context management
- ChatHandlerService as unified chat entry
- Frontend SSAChatPane and useSSAChat hook
- E2E tests: 38/38 passed
Phase III - Method Consultation + AskUser Standardization:
- ToolRegistryService with Repository Pattern
- MethodConsultService with DecisionTable + LLM enhancement
- AskUserService with global interrupt handling
- Frontend AskUserCard component
- E2E tests: 13/13 passed
Phase IV - Dialogue-Driven Analysis + QPER Integration:
- ToolOrchestratorService (plan/execute/report)
- analysis_plan SSE event for WorkflowPlan transmission
- Dual-channel confirmation (ask_user card + workspace button)
- PICO as optional hint for LLM parsing
- E2E tests: 25/25 passed
R Statistics Service:
- 5 new R tools: anova_one, baseline_table, fisher, linear_reg, wilcoxon
- Enhanced guardrails and block helpers
- Comprehensive test suite (run_all_tools_test.js)
Documentation:
- Updated system status document (v5.9)
- Updated SSA module status and development plan (v1.8)
Total E2E: 107/107 passed (Phase I: 31, Phase II: 38, Phase III: 13, Phase IV: 25)
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-02-22 18:53:39 +08:00
428a22adf2
feat(ssa): Complete Phase 2A frontend integration - multi-step workflow end-to-end
...
Phase 2A: WorkflowPlannerService, WorkflowExecutorService, Python data quality, 6 bug fixes, DescriptiveResultView, multi-step R code/Word export, MVP UI reuse. V11 UI: Gemini-style, multi-task, single-page scroll, Word export. Architecture: Block-based rendering consensus (4 block types). New R tools: chi_square, correlation, descriptive, logistic_binary, mann_whitney, t_test_paired. Docs: dev summary, block-based plan, status updates, task list v2.0.
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-02-20 23:09:27 +08:00
f9ed0c2528
feat(rvw): Complete V2.0 Week 3 - Statistical validation extension and UX improvements
...
Week 3 Development Summary:
- Implement negative sign normalization (6 Unicode variants)
- Enhance T-test validation with smart sample size extraction
- Enhance SE triangle and CI-P consistency validation with subrow support
- Add precise sub-cell highlighting for P-values in multi-line cells
- Add frontend issue type Chinese translations (6 new types)
- Add file format tips for PDF/DOC uploads
Technical improvements:
- Add _clean_statistical_text() in extractor.py
- Add _safe_float() wrapper in validator.py
- Add ForensicsReport.tsx component
- Update ISSUE_TYPE_LABELS translations
Documentation:
- Add 2026-02-18 development record
- Update RVW module status (v5.1)
- Update system status (v5.2)
Status: Week 3 complete, ready for Week 4 testing
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-02-18 18:26:16 +08:00
e785969e54
feat(rvw): Implement RVW V2.0 Data Forensics Module - Day 6 StatValidator
...
Summary:
- Implement L2 Statistical Validator (CI-P consistency, T-test reverse)
- Implement L2.5 Consistency Forensics (SE Triangle, SD>Mean check)
- Add error/warning severity classification with tolerance thresholds
- Support 5+ CI formats parsing (parentheses, brackets, 95% CI prefix)
- Complete Python forensics service (types, config, validator, extractor)
V2.0 Development Progress (Week 2 Day 6):
- Day 1-5: Python service setup, Word table extraction, L1 arithmetic validator
- Day 6: L2 StatValidator + L2.5 consistency forensics (promoted from V2.1)
Test Results:
- Unit tests: 4/4 passed (CI-P, SE Triangle, SD>Mean, T-test)
- Real document tests: 5/5 successful, 2 reasonable WARNINGs
Status: Day 6 completed, ready for Day 7 (Skills Framework)
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-02-17 22:15:27 +08:00
303dd78c54
feat(aia): Protocol Agent MVP complete with one-click generation and Word export
...
- Add one-click research protocol generation with streaming output
- Implement Word document export via Pandoc integration
- Add dynamic dual-panel layout with resizable split pane
- Implement collapsible content for StatePanel stages
- Add conversation history management with title auto-update
- Fix scroll behavior, markdown rendering, and UI layout issues
- Simplify conversation creation logic for reliability
2026-01-25 19:16:36 +08:00
40c2f8e148
feat(rag): Complete RAG engine implementation with pgvector
...
Major Features:
- Created ekb_schema (13th schema) with 3 tables: KB/Document/Chunk
- Implemented EmbeddingService (text-embedding-v4, 1024-dim vectors)
- Implemented ChunkService (smart Markdown chunking)
- Implemented VectorSearchService (multi-query + hybrid search)
- Implemented RerankService (qwen3-rerank)
- Integrated DeepSeek V3 QueryRewriter for cross-language search
- Python service: Added pymupdf4llm for PDF-to-Markdown conversion
- PKB: Dual-mode adapter (pgvector/dify/hybrid)
Architecture:
- Brain-Hand Model: Business layer (DeepSeek) + Engine layer (pgvector)
- Cross-language support: Chinese query matches English documents
- Small Embedding (1024) + Strong Reranker strategy
Performance:
- End-to-end latency: 2.5s
- Cost per query: 0.0025 RMB
- Accuracy improvement: +20.5% (cross-language)
Tests:
- test-embedding-service.ts: Vector embedding verified
- test-rag-e2e.ts: Full pipeline tested
- test-rerank.ts: Rerank quality validated
- test-query-rewrite.ts: Cross-language search verified
- test-pdf-ingest.ts: Real PDF document tested (Dongen 2003.pdf)
Documentation:
- Added 05-RAG-Engine-User-Guide.md
- Added 02-Document-Processing-User-Guide.md
- Updated system status documentation
Status: Production ready
2026-01-21 20:24:29 +08:00
9b81aef9a7
feat(dc): Add multi-metric transformation feature (direction 1+2)
...
Summary:
- Implement intelligent multi-metric grouping detection algorithm
- Add direction 1: timepoint-as-row, metric-as-column (analysis format)
- Add direction 2: timepoint-as-column, metric-as-row (display format)
- Fix column name pattern detection (FMA___ issue)
- Maintain original Record ID order in output
- Add full-select/clear buttons in UI
- Integrate into TransformDialog with Radio selection
- Update 3 documentation files
Technical Details:
- Python: detect_metric_groups(), apply_multi_metric_to_long(), apply_multi_metric_to_matrix()
- Backend: 3 new methods in QuickActionService
- Frontend: MultiMetricPanel.tsx (531 lines)
- Total: ~1460 lines of new code
Status: Fully tested and verified, ready for production
2025-12-21 15:06:15 +08:00
200eab5c2e
feat(dc-tool-c): Tool C UX重大改进 - 列头筛选/行号/滚动条/全量数据
...
新功能
- 列头筛选：Excel风格筛选功能（Community版本，中文本地化，显示唯一值及计数）
- 行号列：添加固定行号列（#列头，灰色背景，左侧固定）
- 全量数据加载：不再限制50行预览，Session加载全量数据
- 全量数据返回：所有快速操作（筛选/映射/分箱/条件/删NA/计算/Pivot）全量返回结果
Bug修复
- 滚动条终极修复：修改MainLayout为固定高度（h-screen + overflow-hidden），整个浏览器窗口无滚动条，只有AG Grid内部滚动
- 计算列全角字符修复：自动转换中文括号等全角字符为半角
- 计算列特殊字符列名修复：完善列别名机制，支持任意特殊字符列名
UI优化
- 删除'表格仅展示前50行'提示条，减少干扰
- 筛选对话框美化：白色背景，圆角，阴影
- 列头筛选图标优化：清晰可见，易于点击
文档更新
- 工具C_功能按钮开发计划_V1.0.md：添加V1.5版本记录
- 工具C_MVP开发_TODO清单.md：添加Day 8 UX优化内容
- 00-工具C当前状态与开发指南.md：更新进度为98%
- 00-模块当前状态与开发指南.md：更新DC模块状态
- 00-系统当前状态与开发指南.md：更新系统整体状态
影响范围
- Python微服务：无修改
- Node.js后端：5处代码修改（SessionService + QuickActionController + AICodeService）
- 前端：MainLayout + DataGrid + ag-grid-custom.css + index.tsx
- 完成度：Tool C整体完成度提升至98%
代码统计
- 修改文件：~15个文件
- 新增行数：~200行
- 修改行数：~150行
Co-authored-by: AI Assistant <assistant@example.com >
2025-12-10 18:02:42 +08:00
74cf346453
feat(dc/tool-c): Add missing value imputation feature with 6 methods and MICE
...
Major features:
1. Missing value imputation (6 simple methods + MICE):
- Mean/Median/Mode/Constant imputation
- Forward fill (ffill) and Backward fill (bfill) for time series
- MICE multivariate imputation (in progress, shape issue to fix)
2. Auto precision detection:
- Automatically match decimal places of original data
- Prevent false precision (e.g. 13.57 instead of 13.566716417910449)
3. Categorical variable detection:
- Auto-detect and skip categorical columns in MICE
- Show warnings for unsuitable columns
- Suggest mode imputation for categorical data
4. UI improvements:
- Rename button: "Delete Missing" to "Missing Value Handling"
- Remove standalone "Dedup" and "MICE" buttons
- 3-tab dialog: Delete / Fill / Advanced Fill
- Display column statistics and recommended methods
- Extended warning messages (8 seconds for skipped columns)
5. Bug fixes:
- Fix sessionService.updateSessionData -> saveProcessedData
- Fix OperationResult interface (add message and stats)
- Fix Toolbar button labels and removal
Modified files:
Python: operations/fillna.py (new, 556 lines), main.py (3 new endpoints)
Backend: QuickActionService.ts, QuickActionController.ts, routes/index.ts
Frontend: MissingValueDialog.tsx (new, 437 lines), Toolbar.tsx, index.tsx
Tests: test_fillna_operations.py (774 lines), test scripts and docs
Docs: 5 documentation files updated
Known issues:
- MICE imputation has DataFrame shape mismatch issue (under debugging)
- Workaround: Use 6 simple imputation methods first
Status: Development complete, MICE debugging in progress
Lines added: ~2000 lines across 3 tiers
2025-12-10 13:06:00 +08:00
f4f1d09837
feat(dc/tool-c): Add pivot column ordering and NA handling features
...
Major features:
1. Pivot transformation enhancements:
- Add option to keep unselected columns with 3 aggregation methods
- Maintain original column order after pivot (aligned with source file)
- Preserve pivot value order (first appearance order)
2. NA handling across 4 core functions:
- Recode: Support keep/map/drop for NA values
- Filter: Already supports is_null/not_null operators
- Binning: Support keep/label/assign for NA values (fix nan display)
- Conditional: Add is_null/not_null operators
3. UI improvements:
- Enable column header tooltips with custom header component
- Add closeable alert for 50-row preview
- Fix page scrollbar issues
Modified files:
Python: pivot.py, recode.py, binning.py, conditional.py, main.py
Backend: SessionController, QuickActionController, QuickActionService
Frontend: PivotDialog, RecodeDialog, BinningDialog, ConditionalDialog, DataGrid, index
Status: Ready for testing
2025-12-09 14:40:14 +08:00
75ceeb0653
hotfix(dc/tool-c): Fix compute formula validation and binning NaN serialization
...
Critical fixes:
1. Compute column: Add Chinese comma support in formula validation
- Problem: Formula with Chinese comma failed validation
- Fix: Add Chinese comma character to allowed_chars regex
- Example: Support formulas like 'col1（kg）+ col2，col3'
2. Binning operation: Fix NaN serialization error
- Problem: 'Out of range float values are not JSON compliant: nan'
- Fix: Enhanced NaN/inf handling in binning endpoint
- Added np.inf/-np.inf replacement before JSON serialization
- Added manual JSON serialization with NaN->null conversion
3. Enhanced all operation endpoints for consistency
- Updated conditional, dropna endpoints with same NaN/inf handling
- Ensures all operations return JSON-compliant data
Modified files:
- extraction_service/operations/compute.py: Add Chinese comma to regex
- extraction_service/main.py: Enhanced NaN handling in binning/conditional/dropna
Status: Hotfix complete, ready for testing
2025-12-09 08:45:27 +08:00
f729699510
feat(dc): Complete Tool C quick action buttons Phase 1-2 - 7 functions
...
Summary:
- Implement 7 quick action functions (filter, recode, binning, conditional, dropna, compute, pivot)
- Refactor to pre-written Python functions architecture (stable and secure)
- Add 7 Python operations modules with full type hints
- Add 7 frontend Dialog components with user-friendly UI
- Fix NaN serialization issues and auto type conversion
- Update all related documentation
Technical Details:
- Python: operations/ module (filter.py, recode.py, binning.py, conditional.py, dropna.py, compute.py, pivot.py)
- Backend: QuickActionService.ts with 7 execute methods
- Frontend: 7 Dialog components with complete validation
- Toolbar: Enable 7 quick action buttons
Status: Phase 1-2 completed, basic testing passed, ready for further testing
2025-12-08 17:38:08 +08:00
f01981bf78
feat(dc/tool-c): 完成AI代码生成服务（Day 3 MVP）
...
核心功能：
- 新增AICodeService（550行）：AI代码生成核心服务
- 新增AIController（257行）：4个API端点
- 新增dc_tool_c_ai_history表：存储对话历史
- 实现自我修正机制：最多3次智能重试
- 集成LLMFactory：复用通用能力层
- 10个Few-shot示例：覆盖Level 1-4场景
技术优化：
- 修复NaN序列化问题（Python端转None）
- 修复数据传递问题（从Session获取真实数据）
- 优化System Prompt（明确环境信息）
- 调整Few-shot示例（移除import语句）
测试结果：
- 通过率：9/11（81.8%） 达到MVP标准
- 成功场景：缺失值处理、编码、分箱、BMI、筛选、填补、统计、分类
- 待优化：数值清洗、智能去重（已记录技术债务TD-C-006）
API端点：
- POST /api/v1/dc/tool-c/ai/generate（生成代码）
- POST /api/v1/dc/tool-c/ai/execute（执行代码）
- POST /api/v1/dc/tool-c/ai/process（生成并执行，一步到位）
- GET /api/v1/dc/tool-c/ai/history/:sessionId（对话历史）
文档更新：
- 新增Day 3开发完成总结（770行）
- 新增复杂场景优化技术债务（TD-C-006）
- 更新工具C当前状态文档
- 更新技术债务清单
影响范围：
- backend/src/modules/dc/tool-c/*（新增2个文件，更新1个文件）
- backend/scripts/create-tool-c-ai-history-table.mjs（新增）
- backend/prisma/schema.prisma（新增DcToolCAiHistory模型）
- extraction_service/services/dc_executor.py（NaN序列化修复）
- docs/03-业务模块/DC-数据清洗整理/*（5份文档更新）
Breaking Changes: 无
总代码行数：+950行
Refs: #Tool-C-Day3
2025-12-07 16:21:32 +08:00
AI Clinical Dev Team
39eb62ee79
feat: add extraction_service (PDF/Docx/Txt) and update .gitignore to exclude venv
2025-11-16 15:32:44 +08:00
