diff --git a/DC模块代码恢复指南.md b/DC模块代码恢复指南.md new file mode 100644 index 00000000..39ba4d76 --- /dev/null +++ b/DC模块代码恢复指南.md @@ -0,0 +1,223 @@ +# DC模块代码恢复指南 + +> **目标**: 从Cursor缓存中恢复丢失的DC模块代码 +> **数据库位置**: `C:\Users\zhibo\AppData\Roaming\Cursor\User\workspaceStorage\d5e3431d02cbaa0109f69d72300733da\state.vscdb` + +--- + +## 📋 恢复方法汇总 + +### 方法1:使用Cursor内置Timeline(最简单)⭐⭐⭐⭐⭐ + +**适用于**: 文件曾经保存到磁盘过(即使后来被删除) + +#### 步骤: + +1. **打开Cursor IDE** + +2. **打开资源管理器(Explorer)** + - 左侧边栏点击"文件"图标 + - 或按 `Ctrl+Shift+E` + +3. **找到Timeline面板** + - 在Explorer底部,找到"TIMELINE"(时间轴)折叠面板 + - 如果没看到,右键点击Explorer标题栏 → 勾选"Timeline" + +4. **浏览文件历史** + - 在文件树中,尝试导航到这些路径: + ``` + backend/src/modules/dc/tool-b/services/HealthCheckService.ts + backend/src/modules/dc/tool-b/services/TemplateService.ts + backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts + backend/src/modules/dc/tool-b/services/ConflictDetectionService.ts + backend/src/modules/dc/tool-b/controllers/ExtractionController.ts + ``` + - 点击任一文件(即使文件不存在或为空) + - 查看Timeline面板,会显示该文件的所有历史快照 + +5. **恢复文件** + - 在Timeline中找到最近的版本(带有时间戳) + - 右键点击历史版本 → 选择"Restore"(恢复) + - 文件内容会恢复到选定的版本 + +**重复以上步骤,恢复所有DC模块文件!** + +--- + +### 方法2:使用命令面板恢复已删除文件 ⭐⭐⭐⭐ + +**适用于**: 文件已被完全删除,但曾经保存过 + +#### 步骤: + +1. **打开命令面板** + - Windows: `Ctrl+Shift+P` + - Mac: `Cmd+Shift+P` + +2. **搜索恢复命令** + - 输入: `Local History: Find Entry to Restore` + - 选中该命令 + +3. **搜索文件** + - 输入文件路径,例如: + ``` + HealthCheckService + ``` + - 或更精确的路径: + ``` + backend/src/modules/dc/tool-b/services/HealthCheckService.ts + ``` + +4. **选择版本并恢复** + - 从搜索结果中选择最近的版本 + - 确认恢复 + +**重复以上步骤,搜索并恢复所有DC模块文件!** + +--- + +### 方法3:SQLite数据库直接提取 ⭐⭐⭐⭐⭐(终极方案) + +**适用于**: 代码从未落盘,只存在于Chat/Composer对话中 + +#### 准备工作: + +1. **安装DB Browser for SQLite** + - 下载地址: https://sqlitebrowser.org/dl/ + - 或直接下载: https://github.com/sqlitebrowser/sqlitebrowser/releases + +2. **复制数据库文件(重要!)** + ```powershell + # 在PowerShell中执行 + $source = "C:\Users\zhibo\AppData\Roaming\Cursor\User\workspaceStorage\d5e3431d02cbaa0109f69d72300733da\state.vscdb" + $backup = "D:\MyCursor\AIclinicalresearch\state.vscdb.backup" + Copy-Item $source $backup + Write-Host "✅ 数据库已备份到: $backup" + ``` + +#### 提取步骤: + +1. **打开DB Browser** + - 启动"DB Browser for SQLite" + - File → Open Database + - 选择备份的数据库文件: `D:\MyCursor\AIclinicalresearch\state.vscdb.backup` + +2. **查询Chat历史** + - 点击"Browse Data"(浏览数据)标签 + - 从"Table"下拉菜单选择: `ItemTable` + +3. **搜索DC模块相关记录** + - 点击"Filter"(过滤器)按钮 + - 在"key"列的过滤框中输入: + ``` + chat + ``` + - 或: + ``` + composer + ``` + +4. **查找关键词** + - 在"value"列中搜索以下关键词(使用Ctrl+F): + - `HealthCheckService` + - `DualModelExtractionService` + - `ConflictDetectionService` + - `TemplateService` + - `dc_health_checks` + - `dc_extraction_tasks` + - `ExtractionController` + +5. **导出数据** + - 找到包含代码的行 + - 双击"value"列,查看完整内容 + - value通常是JSON格式,其中包含AI生成的代码块 + - 复制代码到文本编辑器 + +6. **提取代码块** + - JSON中的代码通常在以下结构中: + ```json + { + "messages": [ + { + "content": "```typescript\n[你的代码]\n```" + } + ] + } + ``` + - 提取所有 `\`\`\`typescript` 和 `\`\`\`` 之间的代码 + +--- + +## 🎯 重点查找的文件列表 + +| 文件路径 | 功能 | 优先级 | +|---------|------|--------| +| `backend/src/modules/dc/tool-b/services/HealthCheckService.ts` | 健康检查服务 | ⭐⭐⭐⭐⭐ | +| `backend/src/modules/dc/tool-b/services/TemplateService.ts` | 模板服务 | ⭐⭐⭐⭐⭐ | +| `backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts` | 双模型提取服务 | ⭐⭐⭐⭐ | +| `backend/src/modules/dc/tool-b/services/ConflictDetectionService.ts` | 冲突检测服务 | ⭐⭐⭐⭐ | +| `backend/src/modules/dc/tool-b/controllers/ExtractionController.ts` | 提取控制器 | ⭐⭐⭐⭐⭐ | +| `backend/src/modules/dc/tool-b/routes/index.ts` | 路由配置 | ⭐⭐⭐ | +| `backend/prisma/schema.prisma` (DC相关模型) | 数据库模型 | ⭐⭐⭐⭐⭐ | + +--- + +## 💡 关键提示 + +1. **Timeline方法最简单** + - 如果文件曾经保存过,这个方法成功率最高 + - 即使文件现在是空的,Timeline通常也能找到历史版本 + +2. **命令面板方法最快** + - 适合快速恢复多个已删除文件 + - 可以搜索文件名片段 + +3. **SQLite方法最全面** + - 可以恢复从未保存的代码 + - 需要一定的技术能力 + - 最终兜底方案 + +4. **多方法结合** + - 先尝试方法1和2(简单快速) + - 如果失败,再使用方法3(终极方案) + +--- + +## 🚀 恢复后的操作 + +找到代码后: + +1. **立即保存到文件** + ``` + backend/src/modules/dc/tool-b/services/[文件名].ts + ``` + +2. **立即Git提交** + ```bash + git add . + git commit -m "recover(dc): Restore DC module code from Cursor cache" + git push origin master + ``` + +3. **验证代码完整性** + - 检查是否有语法错误 + - 确认所有依赖是否正确 + +--- + +## 📞 需要帮助? + +如果您在恢复过程中遇到问题: + +1. 截图Timeline面板或SQLite查询结果 +2. 告诉我具体卡在哪一步 +3. 我会提供进一步的指导 + +--- + +**🎯 现在就开始恢复吧!优先尝试方法1(Timeline),最简单!** + + + + + diff --git a/backend/check-api-config.js b/backend/check-api-config.js deleted file mode 100644 index 2aaa8a5b..00000000 --- a/backend/check-api-config.js +++ /dev/null @@ -1,200 +0,0 @@ -/** - * API配置检查脚本 - * 检查DeepSeek和Qwen API配置是否正确 - */ - -import dotenv from 'dotenv'; -import axios from 'axios'; - -// 加载环境变量 -dotenv.config(); - -const colors = { - reset: '\x1b[0m', - green: '\x1b[32m', - red: '\x1b[31m', - yellow: '\x1b[33m', - cyan: '\x1b[36m', -}; - -function log(message, color = 'reset') { - console.log(`${colors[color]}${message}${colors.reset}`); -} - -async function checkDeepSeekAPI() { - log('\n=== 检查 DeepSeek API 配置 ===', 'cyan'); - - const apiKey = process.env.DEEPSEEK_API_KEY; - - if (!apiKey) { - log('❌ 未配置 DEEPSEEK_API_KEY', 'red'); - log('请在 .env 文件中添加: DEEPSEEK_API_KEY=sk-xxx', 'yellow'); - return false; - } - - log(`✅ API Key 已配置: ${apiKey.substring(0, 10)}...`, 'green'); - - // 测试API连接 - try { - log('正在测试 DeepSeek API 连接...', 'cyan'); - const response = await axios.post( - 'https://api.deepseek.com/v1/chat/completions', - { - model: 'deepseek-chat', - messages: [ - { role: 'user', content: '你好' } - ], - max_tokens: 10, - }, - { - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${apiKey}`, - }, - timeout: 10000, - } - ); - - log('✅ DeepSeek API 连接成功!', 'green'); - log(` 模型: ${response.data.model}`, 'green'); - log(` 响应: ${response.data.choices[0].message.content}`, 'green'); - return true; - } catch (error) { - log('❌ DeepSeek API 连接失败', 'red'); - if (error.response) { - log(` 错误: ${error.response.status} - ${error.response.data?.error?.message || error.response.statusText}`, 'red'); - } else if (error.code === 'ECONNABORTED') { - log(' 错误: 请求超时,请检查网络连接', 'red'); - } else { - log(` 错误: ${error.message}`, 'red'); - } - return false; - } -} - -async function checkQwenAPI() { - log('\n=== 检查 Qwen API 配置 ===', 'cyan'); - - const apiKey = process.env.QWEN_API_KEY; - - if (!apiKey) { - log('❌ 未配置 QWEN_API_KEY', 'red'); - log('请在 .env 文件中添加: QWEN_API_KEY=sk-xxx', 'yellow'); - return false; - } - - log(`✅ API Key 已配置: ${apiKey.substring(0, 10)}...`, 'green'); - - // 测试API连接 - try { - log('正在测试 Qwen API 连接...', 'cyan'); - const response = await axios.post( - 'https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions', - { - model: 'qwen-plus', - messages: [ - { role: 'user', content: '你好' } - ], - max_tokens: 10, - }, - { - headers: { - 'Content-Type': 'application/json', - 'Authorization': `Bearer ${apiKey}`, - }, - timeout: 10000, - } - ); - - log('✅ Qwen API 连接成功!', 'green'); - log(` 模型: ${response.data.model}`, 'green'); - log(` 响应: ${response.data.choices[0].message.content}`, 'green'); - return true; - } catch (error) { - log('❌ Qwen API 连接失败', 'red'); - if (error.response) { - log(` 错误: ${error.response.status} - ${error.response.data?.message || error.response.statusText}`, 'red'); - } else if (error.code === 'ECONNABORTED') { - log(' 错误: 请求超时,请检查网络连接', 'red'); - } else { - log(` 错误: ${error.message}`, 'red'); - } - return false; - } -} - -async function main() { - log('\n╔════════════════════════════════════════════════╗', 'cyan'); - log('║ API 配置检查工具 ║', 'cyan'); - log('╚════════════════════════════════════════════════╝', 'cyan'); - - const deepseekOK = await checkDeepSeekAPI(); - const qwenOK = await checkQwenAPI(); - - log('\n=== 检查结果汇总 ===', 'cyan'); - log(`DeepSeek API: ${deepseekOK ? '✅ 正常' : '❌ 异常'}`, deepseekOK ? 'green' : 'red'); - log(`Qwen API: ${qwenOK ? '✅ 正常' : '❌ 异常'}`, qwenOK ? 'green' : 'red'); - - if (!deepseekOK && !qwenOK) { - log('\n⚠️ 所有API都无法使用,请检查配置!', 'yellow'); - log('\n修复建议:', 'cyan'); - log('1. 检查 backend/.env 文件是否存在', 'yellow'); - log('2. 确认API Key已正确配置', 'yellow'); - log('3. 检查网络连接是否正常', 'yellow'); - log('4. 确认API Key有足够的额度', 'yellow'); - } else if (!deepseekOK || !qwenOK) { - log('\n⚠️ 部分API无法使用', 'yellow'); - log('建议使用可用的API进行测试', 'yellow'); - } else { - log('\n✅ 所有API配置正常!', 'green'); - } - - log('\n'); -} - -main().catch(error => { - console.error('脚本执行失败:', error); - process.exit(1); -}); - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/backend/package-lock.json b/backend/package-lock.json index 836ba6cd..6ca97c55 100644 --- a/backend/package-lock.json +++ b/backend/package-lock.json @@ -16,6 +16,8 @@ "@types/form-data": "^2.2.1", "ajv": "^8.17.1", "axios": "^1.12.2", + "bullmq": "^5.65.0", + "diff-match-patch": "^1.0.5", "dotenv": "^17.2.3", "exceljs": "^4.4.0", "fastify": "^5.6.1", @@ -35,6 +37,7 @@ "@types/js-yaml": "^4.0.9", "@types/node": "^24.7.1", "@types/winston": "^2.4.4", + "better-sqlite3": "^12.4.6", "nodemon": "^3.1.10", "pino-pretty": "^13.1.1", "ts-node": "^10.9.2", @@ -766,6 +769,12 @@ "ipaddr.js": "^2.1.0" } }, + "node_modules/@ioredis/commands": { + "version": "1.4.0", + "resolved": "https://registry.npmmirror.com/@ioredis/commands/-/commands-1.4.0.tgz", + "integrity": "sha512-aFT2yemJJo+TZCmieA7qnYGQooOS7QfNmYrzGtsYd3g9j5iDP8AimYYAesf79ohjbLG12XxC4nG5DyEnC88AsQ==", + "license": "MIT" + }, "node_modules/@jridgewell/resolve-uri": { "version": "3.1.2", "resolved": "https://registry.npmmirror.com/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", @@ -803,6 +812,84 @@ "node": ">=8" } }, + "node_modules/@msgpackr-extract/msgpackr-extract-darwin-arm64": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-darwin-arm64/-/msgpackr-extract-darwin-arm64-3.0.3.tgz", + "integrity": "sha512-QZHtlVgbAdy2zAqNA9Gu1UpIuI8Xvsd1v8ic6B2pZmeFnFcMWiPLfWXh7TVw4eGEZ/C9TH281KwhVoeQUKbyjw==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@msgpackr-extract/msgpackr-extract-darwin-x64": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-darwin-x64/-/msgpackr-extract-darwin-x64-3.0.3.tgz", + "integrity": "sha512-mdzd3AVzYKuUmiWOQ8GNhl64/IoFGol569zNRdkLReh6LRLHOXxU4U8eq0JwaD8iFHdVGqSy4IjFL4reoWCDFw==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@msgpackr-extract/msgpackr-extract-linux-arm": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-linux-arm/-/msgpackr-extract-linux-arm-3.0.3.tgz", + "integrity": "sha512-fg0uy/dG/nZEXfYilKoRe7yALaNmHoYeIoJuJ7KJ+YyU2bvY8vPv27f7UKhGRpY6euFYqEVhxCFZgAUNQBM3nw==", + "cpu": [ + "arm" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@msgpackr-extract/msgpackr-extract-linux-arm64": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-linux-arm64/-/msgpackr-extract-linux-arm64-3.0.3.tgz", + "integrity": "sha512-YxQL+ax0XqBJDZiKimS2XQaf+2wDGVa1enVRGzEvLLVFeqa5kx2bWbtcSXgsxjQB7nRqqIGFIcLteF/sHeVtQg==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@msgpackr-extract/msgpackr-extract-linux-x64": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-linux-x64/-/msgpackr-extract-linux-x64-3.0.3.tgz", + "integrity": "sha512-cvwNfbP07pKUfq1uH+S6KJ7dT9K8WOE4ZiAcsrSes+UY55E/0jLYc+vq+DO7jlmqRb5zAggExKm0H7O/CBaesg==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@msgpackr-extract/msgpackr-extract-win32-x64": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/@msgpackr-extract/msgpackr-extract-win32-x64/-/msgpackr-extract-win32-x64-3.0.3.tgz", + "integrity": "sha512-x0fWaQtYp4E6sktbsdAqnehxDgEc/VwM7uLsRCYWaiGu0ykYdZPiS8zCWdnjHwyiumousxfBm4SO31eXqwEZhQ==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, "node_modules/@prisma/client": { "version": "6.17.0", "resolved": "https://registry.npmmirror.com/@prisma/client/-/client-6.17.0.tgz", @@ -1247,6 +1334,21 @@ ], "license": "MIT" }, + "node_modules/better-sqlite3": { + "version": "12.4.6", + "resolved": "https://registry.npmmirror.com/better-sqlite3/-/better-sqlite3-12.4.6.tgz", + "integrity": "sha512-gaYt9yqTbQ1iOxLpJA8FPR5PiaHP+jlg8I5EX0Rs2KFwNzhBsF40KzMZS5FwelY7RG0wzaucWdqSAJM3uNCPCg==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "dependencies": { + "bindings": "^1.5.0", + "prebuild-install": "^7.1.1" + }, + "engines": { + "node": "20.x || 22.x || 23.x || 24.x || 25.x" + } + }, "node_modules/big-integer": { "version": "1.6.52", "resolved": "https://registry.npmmirror.com/big-integer/-/big-integer-1.6.52.tgz", @@ -1282,6 +1384,16 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/bindings": { + "version": "1.5.0", + "resolved": "https://registry.npmmirror.com/bindings/-/bindings-1.5.0.tgz", + "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "file-uri-to-path": "1.0.0" + } + }, "node_modules/bl": { "version": "4.1.0", "resolved": "https://registry.npmmirror.com/bl/-/bl-4.1.0.tgz", @@ -1378,6 +1490,34 @@ "node": ">=0.2.0" } }, + "node_modules/bullmq": { + "version": "5.65.0", + "resolved": "https://registry.npmmirror.com/bullmq/-/bullmq-5.65.0.tgz", + "integrity": "sha512-fyOcyf2ad4zrNmE18vdF/ie7DrW0TwhLt5e0DkqDxbRpDNiUdYqgp2QZJW2ntnUN08T2mDMC4deUUhF2UOAmeQ==", + "license": "MIT", + "dependencies": { + "cron-parser": "^4.9.0", + "ioredis": "^5.8.2", + "msgpackr": "^1.11.2", + "node-abort-controller": "^3.1.1", + "semver": "^7.5.4", + "tslib": "^2.0.0", + "uuid": "^11.1.0" + } + }, + "node_modules/bullmq/node_modules/uuid": { + "version": "11.1.0", + "resolved": "https://registry.npmmirror.com/uuid/-/uuid-11.1.0.tgz", + "integrity": "sha512-0/A9rDy9P7cJ+8w1c9WD9V//9Wj15Ce2MPz8Ri6032usz+NfePxx5AcN3bN+r6ZL6jEo066/yNYB3tn4pQEx+A==", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist/esm/bin/uuid" + } + }, "node_modules/c12": { "version": "3.1.0", "resolved": "https://registry.npmmirror.com/c12/-/c12-3.1.0.tgz", @@ -1491,6 +1631,13 @@ "url": "https://paulmillr.com/funding/" } }, + "node_modules/chownr": { + "version": "1.1.4", + "resolved": "https://registry.npmmirror.com/chownr/-/chownr-1.1.4.tgz", + "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", + "dev": true, + "license": "ISC" + }, "node_modules/citty": { "version": "0.1.6", "resolved": "https://registry.npmmirror.com/citty/-/citty-0.1.6.tgz", @@ -1500,6 +1647,15 @@ "consola": "^3.2.3" } }, + "node_modules/cluster-key-slot": { + "version": "1.1.2", + "resolved": "https://registry.npmmirror.com/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz", + "integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==", + "license": "Apache-2.0", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/codepage": { "version": "1.15.0", "resolved": "https://registry.npmmirror.com/codepage/-/codepage-1.15.0.tgz", @@ -1669,6 +1825,18 @@ "dev": true, "license": "MIT" }, + "node_modules/cron-parser": { + "version": "4.9.0", + "resolved": "https://registry.npmmirror.com/cron-parser/-/cron-parser-4.9.0.tgz", + "integrity": "sha512-p0SaNjrHOnQeR8/VnfGbmg9te2kfyYSQ7Sc/j/6DtPL3JQvKxmjO9TSjNFpujqV3vEYYBvNNvXSxzyksBWAx1Q==", + "license": "MIT", + "dependencies": { + "luxon": "^3.2.1" + }, + "engines": { + "node": ">=12.0.0" + } + }, "node_modules/css-line-break": { "version": "2.1.0", "resolved": "https://registry.npmmirror.com/css-line-break/-/css-line-break-2.1.0.tgz", @@ -1698,7 +1866,6 @@ "version": "4.4.3", "resolved": "https://registry.npmmirror.com/debug/-/debug-4.4.3.tgz", "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "dev": true, "license": "MIT", "dependencies": { "ms": "^2.1.3" @@ -1712,6 +1879,32 @@ } } }, + "node_modules/decompress-response": { + "version": "6.0.0", + "resolved": "https://registry.npmmirror.com/decompress-response/-/decompress-response-6.0.0.tgz", + "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "mimic-response": "^3.1.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/deep-extend": { + "version": "0.6.0", + "resolved": "https://registry.npmmirror.com/deep-extend/-/deep-extend-0.6.0.tgz", + "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=4.0.0" + } + }, "node_modules/deepmerge-ts": { "version": "7.1.5", "resolved": "https://registry.npmmirror.com/deepmerge-ts/-/deepmerge-ts-7.1.5.tgz", @@ -1736,6 +1929,15 @@ "node": ">=0.4.0" } }, + "node_modules/denque": { + "version": "2.1.0", + "resolved": "https://registry.npmmirror.com/denque/-/denque-2.1.0.tgz", + "integrity": "sha512-HVQE3AAb/pxF8fQAoiqpvg9i3evqug3hoiwakOyZAwJm+6vZehbkYXZ0l4JxS+I3QxM97v5aaRNhj8v5oBhekw==", + "license": "Apache-2.0", + "engines": { + "node": ">=0.10" + } + }, "node_modules/dequal": { "version": "2.0.3", "resolved": "https://registry.npmmirror.com/dequal/-/dequal-2.0.3.tgz", @@ -1751,6 +1953,16 @@ "integrity": "sha512-ugFTXCtDZunbzasqBxrK93Ik/DRYsO6S/fedkWEMKqt04xZ4csmnmwGDBAb07QWNaGMAmnTIemsYZCksjATwsA==", "license": "MIT" }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmmirror.com/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "devOptional": true, + "license": "Apache-2.0", + "engines": { + "node": ">=8" + } + }, "node_modules/diff": { "version": "4.0.2", "resolved": "https://registry.npmmirror.com/diff/-/diff-4.0.2.tgz", @@ -1761,6 +1973,12 @@ "node": ">=0.3.1" } }, + "node_modules/diff-match-patch": { + "version": "1.0.5", + "resolved": "https://registry.npmmirror.com/diff-match-patch/-/diff-match-patch-1.0.5.tgz", + "integrity": "sha512-IayShXAgj/QMXgB0IWmKx+rOPuGMhqm5w6jvFxmVenXKIzRqTAAsbBPT3kWQeGANj3jGgvcvv4yK6SxqYmikgw==", + "license": "Apache-2.0" + }, "node_modules/dompurify": { "version": "3.3.0", "resolved": "https://registry.npmmirror.com/dompurify/-/dompurify-3.3.0.tgz", @@ -1992,6 +2210,16 @@ "node": ">=8.3.0" } }, + "node_modules/expand-template": { + "version": "2.0.3", + "resolved": "https://registry.npmmirror.com/expand-template/-/expand-template-2.0.3.tgz", + "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", + "dev": true, + "license": "(MIT OR WTFPL)", + "engines": { + "node": ">=6" + } + }, "node_modules/exsolve": { "version": "1.0.7", "resolved": "https://registry.npmmirror.com/exsolve/-/exsolve-1.0.7.tgz", @@ -2236,6 +2464,13 @@ "integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==", "license": "MIT" }, + "node_modules/file-uri-to-path": { + "version": "1.0.0", + "resolved": "https://registry.npmmirror.com/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", + "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==", + "dev": true, + "license": "MIT" + }, "node_modules/fill-range": { "version": "7.1.1", "resolved": "https://registry.npmmirror.com/fill-range/-/fill-range-7.1.1.tgz", @@ -2433,6 +2668,13 @@ "giget": "dist/cli.mjs" } }, + "node_modules/github-from-package": { + "version": "0.0.0", + "resolved": "https://registry.npmmirror.com/github-from-package/-/github-from-package-0.0.0.tgz", + "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", + "dev": true, + "license": "MIT" + }, "node_modules/glob": { "version": "7.2.3", "resolved": "https://registry.npmmirror.com/glob/-/glob-7.2.3.tgz", @@ -2604,12 +2846,43 @@ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", "license": "ISC" }, + "node_modules/ini": { + "version": "1.3.8", + "resolved": "https://registry.npmmirror.com/ini/-/ini-1.3.8.tgz", + "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", + "dev": true, + "license": "ISC" + }, "node_modules/iobuffer": { "version": "5.4.0", "resolved": "https://registry.npmmirror.com/iobuffer/-/iobuffer-5.4.0.tgz", "integrity": "sha512-DRebOWuqDvxunfkNJAlc3IzWIPD5xVxwUNbHr7xKB8E6aLJxIPfNX3CoMJghcFjpv6RWQsrcJbghtEwSPoJqMA==", "license": "MIT" }, + "node_modules/ioredis": { + "version": "5.8.2", + "resolved": "https://registry.npmmirror.com/ioredis/-/ioredis-5.8.2.tgz", + "integrity": "sha512-C6uC+kleiIMmjViJINWk80sOQw5lEzse1ZmvD+S/s8p8CWapftSaC+kocGTx6xrbrJ4WmYQGC08ffHLr6ToR6Q==", + "license": "MIT", + "dependencies": { + "@ioredis/commands": "1.4.0", + "cluster-key-slot": "^1.1.0", + "debug": "^4.3.4", + "denque": "^2.1.0", + "lodash.defaults": "^4.2.0", + "lodash.isarguments": "^3.1.0", + "redis-errors": "^1.2.0", + "redis-parser": "^3.0.0", + "standard-as-callback": "^2.1.0" + }, + "engines": { + "node": ">=12.22.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/ioredis" + } + }, "node_modules/ipaddr.js": { "version": "2.2.0", "resolved": "https://registry.npmmirror.com/ipaddr.js/-/ipaddr.js-2.2.0.tgz", @@ -2943,6 +3216,12 @@ "integrity": "sha512-5dcWxm23+VAoz+awKmBaiBvzox8+RqMgFhi7UvX9DHZr2HdxHXM/Wrf8cfKpsW37RNrvtPn6hSwNqurSILbmJw==", "license": "MIT" }, + "node_modules/lodash.isarguments": { + "version": "3.1.0", + "resolved": "https://registry.npmmirror.com/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz", + "integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==", + "license": "MIT" + }, "node_modules/lodash.isboolean": { "version": "3.0.3", "resolved": "https://registry.npmmirror.com/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz", @@ -3009,6 +3288,15 @@ "node": ">= 12.0.0" } }, + "node_modules/luxon": { + "version": "3.7.2", + "resolved": "https://registry.npmmirror.com/luxon/-/luxon-3.7.2.tgz", + "integrity": "sha512-vtEhXh/gNjI9Yg1u4jX/0YVPMvxzHuGgCm6tC5kZyb08yjGWGnqAjGJvcXbqQR2P3MyMEFnRbpcdFS6PBcLqew==", + "license": "MIT", + "engines": { + "node": ">=12" + } + }, "node_modules/make-error": { "version": "1.3.6", "resolved": "https://registry.npmmirror.com/make-error/-/make-error-1.3.6.tgz", @@ -3046,6 +3334,19 @@ "node": ">= 0.6" } }, + "node_modules/mimic-response": { + "version": "3.1.0", + "resolved": "https://registry.npmmirror.com/mimic-response/-/mimic-response-3.1.0.tgz", + "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/minimalistic-assert": { "version": "1.0.1", "resolved": "https://registry.npmmirror.com/minimalistic-assert/-/minimalistic-assert-1.0.1.tgz", @@ -3085,6 +3386,13 @@ "mkdirp": "bin/cmd.js" } }, + "node_modules/mkdirp-classic": { + "version": "0.5.3", + "resolved": "https://registry.npmmirror.com/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", + "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", + "dev": true, + "license": "MIT" + }, "node_modules/mnemonist": { "version": "0.40.3", "resolved": "https://registry.npmmirror.com/mnemonist/-/mnemonist-0.40.3.tgz", @@ -3100,12 +3408,84 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, + "node_modules/msgpackr": { + "version": "1.11.5", + "resolved": "https://registry.npmmirror.com/msgpackr/-/msgpackr-1.11.5.tgz", + "integrity": "sha512-UjkUHN0yqp9RWKy0Lplhh+wlpdt9oQBYgULZOiFhV3VclSF1JnSQWZ5r9gORQlNYaUKQoR8itv7g7z1xDDuACA==", + "license": "MIT", + "optionalDependencies": { + "msgpackr-extract": "^3.0.2" + } + }, + "node_modules/msgpackr-extract": { + "version": "3.0.3", + "resolved": "https://registry.npmmirror.com/msgpackr-extract/-/msgpackr-extract-3.0.3.tgz", + "integrity": "sha512-P0efT1C9jIdVRefqjzOQ9Xml57zpOXnIuS+csaB4MdZbTdmGDLo8XhzBG1N7aO11gKDDkJvBLULeFTo46wwreA==", + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "dependencies": { + "node-gyp-build-optional-packages": "5.2.2" + }, + "bin": { + "download-msgpackr-prebuilds": "bin/download-prebuilds.js" + }, + "optionalDependencies": { + "@msgpackr-extract/msgpackr-extract-darwin-arm64": "3.0.3", + "@msgpackr-extract/msgpackr-extract-darwin-x64": "3.0.3", + "@msgpackr-extract/msgpackr-extract-linux-arm": "3.0.3", + "@msgpackr-extract/msgpackr-extract-linux-arm64": "3.0.3", + "@msgpackr-extract/msgpackr-extract-linux-x64": "3.0.3", + "@msgpackr-extract/msgpackr-extract-win32-x64": "3.0.3" + } + }, + "node_modules/napi-build-utils": { + "version": "2.0.0", + "resolved": "https://registry.npmmirror.com/napi-build-utils/-/napi-build-utils-2.0.0.tgz", + "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", + "dev": true, + "license": "MIT" + }, + "node_modules/node-abi": { + "version": "3.85.0", + "resolved": "https://registry.npmmirror.com/node-abi/-/node-abi-3.85.0.tgz", + "integrity": "sha512-zsFhmbkAzwhTft6nd3VxcG0cvJsT70rL+BIGHWVq5fi6MwGrHwzqKaxXE+Hl2GmnGItnDKPPkO5/LQqjVkIdFg==", + "dev": true, + "license": "MIT", + "dependencies": { + "semver": "^7.3.5" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/node-abort-controller": { + "version": "3.1.1", + "resolved": "https://registry.npmmirror.com/node-abort-controller/-/node-abort-controller-3.1.1.tgz", + "integrity": "sha512-AGK2yQKIjRuqnc6VkX2Xj5d+QW8xZ87pa1UK6yA6ouUyuxfHuMP6umE5QK7UmTeOAymo+Zx1Fxiuw9rVx8taHQ==", + "license": "MIT" + }, "node_modules/node-fetch-native": { "version": "1.6.7", "resolved": "https://registry.npmmirror.com/node-fetch-native/-/node-fetch-native-1.6.7.tgz", "integrity": "sha512-g9yhqoedzIUm0nTnTqAQvueMPVOuIY16bqgAJJC8XOOubYFNwz6IER9qs0Gq2Xd0+CecCKFjtdDTMA4u4xG06Q==", "license": "MIT" }, + "node_modules/node-gyp-build-optional-packages": { + "version": "5.2.2", + "resolved": "https://registry.npmmirror.com/node-gyp-build-optional-packages/-/node-gyp-build-optional-packages-5.2.2.tgz", + "integrity": "sha512-s+w+rBWnpTMwSFbaE0UXsRlg7hU4FjekKU4eyAih5T8nJuNZT1nNsskXpxmeqSK9UzkBl6UgRlnKc8hz8IEqOw==", + "license": "MIT", + "optional": true, + "dependencies": { + "detect-libc": "^2.0.1" + }, + "bin": { + "node-gyp-build-optional-packages": "bin.js", + "node-gyp-build-optional-packages-optional": "optional.js", + "node-gyp-build-optional-packages-test": "build-test.js" + } + }, "node_modules/nodemon": { "version": "3.1.10", "resolved": "https://registry.npmmirror.com/nodemon/-/nodemon-3.1.10.tgz", @@ -3388,6 +3768,33 @@ "pathe": "^2.0.3" } }, + "node_modules/prebuild-install": { + "version": "7.1.3", + "resolved": "https://registry.npmmirror.com/prebuild-install/-/prebuild-install-7.1.3.tgz", + "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", + "dev": true, + "license": "MIT", + "dependencies": { + "detect-libc": "^2.0.0", + "expand-template": "^2.0.3", + "github-from-package": "0.0.0", + "minimist": "^1.2.3", + "mkdirp-classic": "^0.5.3", + "napi-build-utils": "^2.0.0", + "node-abi": "^3.3.0", + "pump": "^3.0.0", + "rc": "^1.2.7", + "simple-get": "^4.0.0", + "tar-fs": "^2.0.0", + "tunnel-agent": "^0.6.0" + }, + "bin": { + "prebuild-install": "bin.js" + }, + "engines": { + "node": ">=10" + } + }, "node_modules/prisma": { "version": "6.17.0", "resolved": "https://registry.npmmirror.com/prisma/-/prisma-6.17.0.tgz", @@ -3491,6 +3898,32 @@ "performance-now": "^2.1.0" } }, + "node_modules/rc": { + "version": "1.2.8", + "resolved": "https://registry.npmmirror.com/rc/-/rc-1.2.8.tgz", + "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", + "dev": true, + "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", + "dependencies": { + "deep-extend": "^0.6.0", + "ini": "~1.3.0", + "minimist": "^1.2.0", + "strip-json-comments": "~2.0.1" + }, + "bin": { + "rc": "cli.js" + } + }, + "node_modules/rc/node_modules/strip-json-comments": { + "version": "2.0.1", + "resolved": "https://registry.npmmirror.com/strip-json-comments/-/strip-json-comments-2.0.1.tgz", + "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/rc9": { "version": "2.1.2", "resolved": "https://registry.npmmirror.com/rc9/-/rc9-2.1.2.tgz", @@ -3567,6 +4000,27 @@ "node": ">= 12.13.0" } }, + "node_modules/redis-errors": { + "version": "1.2.0", + "resolved": "https://registry.npmmirror.com/redis-errors/-/redis-errors-1.2.0.tgz", + "integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==", + "license": "MIT", + "engines": { + "node": ">=4" + } + }, + "node_modules/redis-parser": { + "version": "3.0.0", + "resolved": "https://registry.npmmirror.com/redis-parser/-/redis-parser-3.0.0.tgz", + "integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==", + "license": "MIT", + "dependencies": { + "redis-errors": "^1.0.0" + }, + "engines": { + "node": ">=4" + } + }, "node_modules/regenerator-runtime": { "version": "0.13.11", "resolved": "https://registry.npmmirror.com/regenerator-runtime/-/regenerator-runtime-0.13.11.tgz", @@ -3747,6 +4201,53 @@ "integrity": "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA==", "license": "MIT" }, + "node_modules/simple-concat": { + "version": "1.0.1", + "resolved": "https://registry.npmmirror.com/simple-concat/-/simple-concat-1.0.1.tgz", + "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/simple-get": { + "version": "4.0.1", + "resolved": "https://registry.npmmirror.com/simple-get/-/simple-get-4.0.1.tgz", + "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT", + "dependencies": { + "decompress-response": "^6.0.0", + "once": "^1.3.1", + "simple-concat": "^1.0.0" + } + }, "node_modules/simple-update-notifier": { "version": "2.0.0", "resolved": "https://registry.npmmirror.com/simple-update-notifier/-/simple-update-notifier-2.0.0.tgz", @@ -3815,6 +4316,12 @@ "node": ">=0.1.14" } }, + "node_modules/standard-as-callback": { + "version": "2.1.0", + "resolved": "https://registry.npmmirror.com/standard-as-callback/-/standard-as-callback-2.1.0.tgz", + "integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==", + "license": "MIT" + }, "node_modules/steed": { "version": "1.1.3", "resolved": "https://registry.npmmirror.com/steed/-/steed-1.1.3.tgz", @@ -3873,6 +4380,19 @@ "node": ">=12.0.0" } }, + "node_modules/tar-fs": { + "version": "2.1.4", + "resolved": "https://registry.npmmirror.com/tar-fs/-/tar-fs-2.1.4.tgz", + "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "chownr": "^1.1.1", + "mkdirp-classic": "^0.5.2", + "pump": "^3.0.0", + "tar-stream": "^2.1.4" + } + }, "node_modules/tar-stream": { "version": "2.2.0", "resolved": "https://registry.npmmirror.com/tar-stream/-/tar-stream-2.2.0.tgz", @@ -4028,6 +4548,12 @@ } } }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD" + }, "node_modules/tsx": { "version": "4.20.6", "resolved": "https://registry.npmmirror.com/tsx/-/tsx-4.20.6.tgz", @@ -4048,6 +4574,19 @@ "fsevents": "~2.3.3" } }, + "node_modules/tunnel-agent": { + "version": "0.6.0", + "resolved": "https://registry.npmmirror.com/tunnel-agent/-/tunnel-agent-0.6.0.tgz", + "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "safe-buffer": "^5.0.1" + }, + "engines": { + "node": "*" + } + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmmirror.com/typescript/-/typescript-5.9.3.tgz", diff --git a/backend/package.json b/backend/package.json index e941b200..bbc11fe0 100644 --- a/backend/package.json +++ b/backend/package.json @@ -33,6 +33,8 @@ "@types/form-data": "^2.2.1", "ajv": "^8.17.1", "axios": "^1.12.2", + "bullmq": "^5.65.0", + "diff-match-patch": "^1.0.5", "dotenv": "^17.2.3", "exceljs": "^4.4.0", "fastify": "^5.6.1", @@ -52,6 +54,7 @@ "@types/js-yaml": "^4.0.9", "@types/node": "^24.7.1", "@types/winston": "^2.4.4", + "better-sqlite3": "^12.4.6", "nodemon": "^3.1.10", "pino-pretty": "^13.1.1", "ts-node": "^10.9.2", diff --git a/backend/prisma/schema.prisma b/backend/prisma/schema.prisma index fbafce22..7936e8b0 100644 --- a/backend/prisma/schema.prisma +++ b/backend/prisma/schema.prisma @@ -718,3 +718,131 @@ model AslFulltextScreeningResult { @@map("fulltext_screening_results") @@schema("asl_schema") } + +// ==================== DC数据清洗模块 - Tool B (病历结构化机器人) ==================== + +// 健康检查缓存表 +model DCHealthCheck { + id String @id @default(uuid()) + + userId String @map("user_id") + fileName String @map("file_name") + columnName String @map("column_name") + + // 统计指标 + emptyRate Float @map("empty_rate") // 空值率 (0-1) + avgLength Float @map("avg_length") // 平均文本长度 + totalRows Int @map("total_rows") + estimatedTokens Int @map("estimated_tokens") + + // 检查结果 + status String @map("status") // 'good' | 'bad' + message String @map("message") + + createdAt DateTime @default(now()) @map("created_at") + + @@index([userId, fileName]) + @@map("dc_health_checks") + @@schema("dc_schema") +} + +// 预设模板表 +model DCTemplate { + id String @id @default(uuid()) + + diseaseType String @map("disease_type") // 'lung_cancer', 'diabetes', 'hypertension' + reportType String @map("report_type") // 'pathology', 'admission', 'outpatient' + displayName String @map("display_name") // '肺癌病理报告' + + fields Json @map("fields") // [{name, desc, width}] + promptTemplate String @map("prompt_template") @db.Text + + createdAt DateTime @default(now()) @map("created_at") + updatedAt DateTime @updatedAt @map("updated_at") + + @@unique([diseaseType, reportType]) + @@map("dc_templates") + @@schema("dc_schema") +} + +// 提取任务表 +model DCExtractionTask { + id String @id @default(uuid()) + + userId String @map("user_id") + projectName String @map("project_name") + sourceFileKey String @map("source_file_key") // Storage中的路径 + textColumn String @map("text_column") + + // 模板配置 + diseaseType String @map("disease_type") + reportType String @map("report_type") + targetFields Json @map("target_fields") // [{name, desc}] + + // 双模型配置 + modelA String @default("deepseek-v3") @map("model_a") + modelB String @default("qwen-max") @map("model_b") + + // 任务状态 + status String @default("pending") @map("status") // 'pending'|'processing'|'completed'|'failed' + totalCount Int @default(0) @map("total_count") + processedCount Int @default(0) @map("processed_count") + cleanCount Int @default(0) @map("clean_count") // 一致数 + conflictCount Int @default(0) @map("conflict_count") // 冲突数 + failedCount Int @default(0) @map("failed_count") + + // 成本统计 + totalTokens Int @default(0) @map("total_tokens") + totalCost Float @default(0) @map("total_cost") + + // 错误信息 + error String? @map("error") + + // 时间戳 + createdAt DateTime @default(now()) @map("created_at") + startedAt DateTime? @map("started_at") + completedAt DateTime? @map("completed_at") + + items DCExtractionItem[] + + @@index([userId, status]) + @@map("dc_extraction_tasks") + @@schema("dc_schema") +} + +// 提取记录表 (每条病历记录) +model DCExtractionItem { + id String @id @default(uuid()) + taskId String @map("task_id") + + // 原始数据 + rowIndex Int @map("row_index") + originalText String @map("original_text") @db.Text + + // 双模型结果 (V2核心) + resultA Json? @map("result_a") // DeepSeek结果 {"肿瘤大小": "3cm"} + resultB Json? @map("result_b") // Qwen结果 {"肿瘤大小": "3.0cm"} + + // 冲突检测 + status String @default("pending") @map("status") // 'pending'|'clean'|'conflict'|'resolved'|'failed' + conflictFields String[] @default([]) @map("conflict_fields") // ["肿瘤大小"] + + // 最终结果 (用户裁决后或自动采纳) + finalResult Json? @map("final_result") + + // Token统计 + tokensA Int @default(0) @map("tokens_a") + tokensB Int @default(0) @map("tokens_b") + + // 错误信息 + error String? @map("error") + + createdAt DateTime @default(now()) @map("created_at") + resolvedAt DateTime? @map("resolved_at") + + task DCExtractionTask @relation(fields: [taskId], references: [id], onDelete: Cascade) + + @@index([taskId, status]) + @@map("dc_extraction_items") + @@schema("dc_schema") +} diff --git a/backend/recover-code-from-cursor-db.js b/backend/recover-code-from-cursor-db.js new file mode 100644 index 00000000..4e38a862 --- /dev/null +++ b/backend/recover-code-from-cursor-db.js @@ -0,0 +1,180 @@ +/** + * 从Cursor的SQLite数据库中恢复代码历史 + * + * 使用方法: + * 1. cd backend + * 2. npm install better-sqlite3 + * 3. node recover-code-from-cursor-db.js + */ + +const Database = require('better-sqlite3'); +const fs = require('fs'); +const path = require('path'); + +// Cursor SQLite数据库路径 +const DB_PATH = path.join( + process.env.APPDATA || process.env.HOME, + 'Cursor/User/workspaceStorage/d5e3431d02cbaa0109f69d72300733da/state.vscdb' +); + +// 输出目录 +const OUTPUT_DIR = path.join(__dirname, 'cursor-history-recovery'); + +console.log('🔍 正在读取Cursor历史数据库...'); +console.log('📂 数据库路径:', DB_PATH); + +if (!fs.existsSync(DB_PATH)) { + console.error('❌ 数据库文件不存在!'); + process.exit(1); +} + +// 创建输出目录 +if (!fs.existsSync(OUTPUT_DIR)) { + fs.mkdirSync(OUTPUT_DIR, { recursive: true }); +} + +try { + // 打开数据库(只读模式) + const db = new Database(DB_PATH, { readonly: true, fileMustExist: true }); + + console.log('✅ 数据库打开成功!'); + + // 1. 查看表结构 + console.log('\n📊 数据库表列表:'); + const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all(); + console.log(tables.map(t => ` - ${t.name}`).join('\n')); + + // 2. 查询ItemTable表结构 + if (tables.some(t => t.name === 'ItemTable')) { + console.log('\n📋 ItemTable 表结构:'); + const columns = db.prepare("PRAGMA table_info(ItemTable)").all(); + console.log(columns.map(c => ` - ${c.name} (${c.type})`).join('\n')); + + // 3. 查询所有key(了解有哪些类型的数据) + console.log('\n🔑 ItemTable 中的所有key类型:'); + const keys = db.prepare("SELECT DISTINCT key FROM ItemTable").all(); + console.log(keys.map(k => ` - ${k.key}`).join('\n')); + + // 4. 查找聊天历史相关的key + console.log('\n💬 查找聊天/Composer历史记录...'); + const chatKeys = [ + 'workbench.panel.chat', + 'composer', + 'chat', + 'workbench.panel.aichat', + 'aiPanel' + ]; + + let foundCount = 0; + + for (const keyPattern of chatKeys) { + const rows = db.prepare( + `SELECT key, value FROM ItemTable WHERE key LIKE ?` + ).all(`%${keyPattern}%`); + + if (rows.length > 0) { + console.log(`\n✅ 找到 ${rows.length} 条与 "${keyPattern}" 相关的记录`); + + rows.forEach((row, index) => { + foundCount++; + const filename = `${keyPattern.replace(/[^a-z0-9]/gi, '_')}_${index + 1}.json`; + const filepath = path.join(OUTPUT_DIR, filename); + + // 保存原始JSON + fs.writeFileSync(filepath, row.value); + console.log(` 📄 已保存: ${filename} (${(row.value.length / 1024).toFixed(2)} KB)`); + + // 尝试解析JSON并提取代码 + try { + const data = JSON.parse(row.value); + + // 提取可能的代码片段 + const codeBlocks = extractCodeBlocks(data); + if (codeBlocks.length > 0) { + const codeFilename = `${keyPattern.replace(/[^a-z0-9]/gi, '_')}_${index + 1}_code.txt`; + const codeFilepath = path.join(OUTPUT_DIR, codeFilename); + fs.writeFileSync(codeFilepath, codeBlocks.join('\n\n' + '='.repeat(80) + '\n\n')); + console.log(` 📝 提取了 ${codeBlocks.length} 个代码块: ${codeFilename}`); + } + } catch (err) { + console.log(` ⚠️ JSON解析失败: ${err.message}`); + } + }); + } + } + + if (foundCount === 0) { + console.log('\n⚠️ 未找到聊天历史记录,尝试提取所有数据...'); + + // 导出所有ItemTable数据 + const allRows = db.prepare("SELECT key, value FROM ItemTable").all(); + console.log(`\n📦 共有 ${allRows.length} 条记录,正在导出...`); + + const allDataFile = path.join(OUTPUT_DIR, 'all_itemtable_data.json'); + fs.writeFileSync(allDataFile, JSON.stringify(allRows, null, 2)); + console.log(`✅ 已导出所有数据到: all_itemtable_data.json (${(fs.statSync(allDataFile).size / 1024 / 1024).toFixed(2)} MB)`); + } + + } else { + console.log('\n❌ ItemTable 表不存在!'); + } + + db.close(); + console.log(`\n✅ 恢复完成!所有文件保存在: ${OUTPUT_DIR}`); + console.log('\n💡 下一步:'); + console.log(' 1. 检查 cursor-history-recovery 文件夹'); + console.log(' 2. 打开 .json 文件查找DC模块相关的代码'); + console.log(' 3. 查找关键词:DualModelExtractionService, HealthCheckService, ExtractionController'); + +} catch (error) { + console.error('❌ 错误:', error.message); + console.error(error.stack); + process.exit(1); +} + +/** + * 从JSON数据中递归提取代码块 + */ +function extractCodeBlocks(obj, blocks = []) { + if (typeof obj === 'string') { + // 查找代码块模式 + const codePatterns = [ + /```[\s\S]*?```/g, // Markdown代码块 + /export\s+(const|function|class)\s+\w+/g, // TypeScript导出 + /interface\s+\w+/g, // TypeScript接口 + /async\s+function\s+\w+/g, // 异步函数 + ]; + + codePatterns.forEach(pattern => { + const matches = obj.match(pattern); + if (matches) { + blocks.push(...matches); + } + }); + + // 如果包含关键代码关键词,保存整段 + const keywords = [ + 'DualModelExtractionService', + 'HealthCheckService', + 'TemplateService', + 'ConflictDetectionService', + 'ExtractionController', + 'dc_extraction_tasks', + 'dc_health_checks' + ]; + + if (keywords.some(kw => obj.includes(kw))) { + blocks.push(obj); + } + } else if (Array.isArray(obj)) { + obj.forEach(item => extractCodeBlocks(item, blocks)); + } else if (obj && typeof obj === 'object') { + Object.values(obj).forEach(value => extractCodeBlocks(value, blocks)); + } + + return blocks; +} + + + + diff --git a/backend/recovery-log.txt b/backend/recovery-log.txt new file mode 100644 index 00000000..c2c90b05 Binary files /dev/null and b/backend/recovery-log.txt differ diff --git a/backend/scripts/check-dc-tables.mjs b/backend/scripts/check-dc-tables.mjs new file mode 100644 index 00000000..e02f80b8 --- /dev/null +++ b/backend/scripts/check-dc-tables.mjs @@ -0,0 +1,199 @@ +/** + * DC模块数据库表检查脚本(使用Prisma) + * + * 验证dc_schema和4个表是否已创建 + */ + +import { PrismaClient } from '@prisma/client'; + +const prisma = new PrismaClient(); + +async function checkDCTables() { + try { + console.log(''); + console.log('============================================================'); + console.log('[DC模块] 数据库表检查'); + console.log('============================================================'); + console.log(''); + console.log('✅ Prisma连接初始化成功'); + console.log(''); + + // 1. 检查dc_schema是否存在 + console.log('📋 检查1: dc_schema是否存在?'); + const schemaResult = await prisma.$queryRawUnsafe(` + SELECT schema_name + FROM information_schema.schemata + WHERE schema_name = 'dc_schema' + `); + + if (schemaResult.length === 0) { + console.log('❌ dc_schema 不存在!'); + console.log(''); + console.log('💡 解决方案:'); + console.log(' cd backend'); + console.log(' npx prisma db push'); + console.log(''); + await prisma.$disconnect(); + process.exit(1); + } + + console.log('✅ dc_schema 存在'); + console.log(''); + + // 2. 检查4个表是否存在 + console.log('📋 检查2: DC模块的4个表是否存在?'); + console.log(''); + + const tables = [ + { name: 'dc_health_checks', display: '健康检查表', model: 'dCHealthCheck' }, + { name: 'dc_templates', display: '预设模板表', model: 'dCTemplate' }, + { name: 'dc_extraction_tasks', display: '提取任务表', model: 'dCExtractionTask' }, + { name: 'dc_extraction_items', display: '提取明细表', model: 'dCExtractionItem' }, + ]; + + let allTablesExist = true; + const missingTables = []; + const tableCounts = {}; + + for (const table of tables) { + try { + const tableResult = await prisma.$queryRawUnsafe(` + SELECT table_name + FROM information_schema.tables + WHERE table_schema = 'dc_schema' + AND table_name = '${table.name}' + `); + + if (tableResult.length > 0) { + // 获取行数 + const countResult = await prisma.$queryRawUnsafe(` + SELECT COUNT(*) as count FROM dc_schema.${table.name} + `); + const count = Number(countResult[0].count); + tableCounts[table.name] = count; + + console.log(` ✅ ${table.display} (${table.name})`); + console.log(` 记录数: ${count} 条`); + } else { + console.log(` ❌ ${table.display} (${table.name}): 不存在`); + allTablesExist = false; + missingTables.push(table.name); + } + } catch (error) { + console.log(` ❌ ${table.display} (${table.name}): 检查失败 - ${error.message}`); + allTablesExist = false; + missingTables.push(table.name); + } + } + + console.log(''); + + // 3. 检查dc_templates是否有预设数据 + if (allTablesExist) { + console.log('📋 检查3: dc_templates预设模板是否存在?'); + const templateCount = tableCounts['dc_templates']; + + if (templateCount === 0) { + console.log('⚠️ dc_templates表为空(没有预设模板)'); + console.log(''); + console.log('💡 需要启动后端服务初始化预设模板:'); + console.log(' cd backend'); + console.log(' npm run dev'); + console.log(' (启动时会自动seed 3个预设模板)'); + } else { + console.log(`✅ dc_templates已有 ${templateCount} 个预设模板`); + + // 列出模板 + try { + const templates = await prisma.$queryRawUnsafe(` + SELECT disease_type, report_type, display_name + FROM dc_schema.dc_templates + ORDER BY created_at + `); + + if (templates.length > 0) { + console.log(''); + console.log(' 预设模板列表:'); + templates.forEach((t, i) => { + console.log(` ${i + 1}. ${t.display_name} (${t.disease_type}/${t.report_type})`); + }); + } + } catch (error) { + console.log(' ⚠️ 无法获取模板详情'); + } + } + console.log(''); + } + + // 4. 总结 + console.log('============================================================'); + console.log('[总结]'); + console.log('============================================================'); + console.log(''); + + if (allTablesExist) { + console.log('🎉 恭喜!DC模块数据库表已全部创建!'); + console.log(''); + console.log('✅ dc_schema: 存在'); + console.log('✅ 4个数据表: 全部存在'); + console.log(''); + console.log('📊 数据统计:'); + console.log(` - dc_health_checks: ${tableCounts['dc_health_checks']} 条`); + console.log(` - dc_templates: ${tableCounts['dc_templates']} 条`); + console.log(` - dc_extraction_tasks: ${tableCounts['dc_extraction_tasks']} 条`); + console.log(` - dc_extraction_items: ${tableCounts['dc_extraction_items']} 条`); + console.log(''); + console.log('📌 下一步:'); + if (tableCounts['dc_templates'] === 0) { + console.log(' 1. ⚠️ 启动后端初始化预设模板(npm run dev)'); + console.log(' 2. 然后可以开始前端开发!'); + } else { + console.log(' ✅ 可以开始前端开发了!'); + } + console.log(''); + + await prisma.$disconnect(); + process.exit(0); + } else { + console.log('⚠️ DC模块数据库表未完全创建'); + console.log(''); + console.log('❌ 缺少以下表:'); + missingTables.forEach(t => console.log(` - ${t}`)); + console.log(''); + console.log('💡 解决方案:'); + console.log(' cd backend'); + console.log(' npx prisma db push'); + console.log(''); + + await prisma.$disconnect(); + process.exit(1); + } + + } catch (error) { + console.error(''); + console.error('❌ 检查失败:', error.message); + console.error(''); + + if (error.message.includes('connect') || error.message.includes('ECONNREFUSED')) { + console.error('💡 数据库连接失败,请确认:'); + console.error(' 1. PostgreSQL服务是否已启动?'); + console.error(' 2. DATABASE_URL环境变量是否正确?'); + console.error(' 当前: ' + (process.env.DATABASE_URL || '未设置')); + console.error(''); + console.error(' 期望格式:'); + console.error(' postgresql://postgres:postgres123@localhost:5432/ai_clinical_research'); + } + + console.error(''); + console.error('详细错误:'); + console.error(error); + console.error(''); + + await prisma.$disconnect().catch(() => {}); + process.exit(1); + } +} + +// 执行检查 +checkDCTables(); + diff --git a/backend/src/index.ts b/backend/src/index.ts index bf4e1fbc..7c9e453d 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -11,6 +11,7 @@ import { chatRoutes } from './legacy/routes/chatRoutes.js'; import { batchRoutes } from './legacy/routes/batchRoutes.js'; import reviewRoutes from './legacy/routes/reviewRoutes.js'; import { aslRoutes } from './modules/asl/routes/index.js'; +import { registerDCRoutes, initDCModule } from './modules/dc/index.js'; import { registerHealthRoutes } from './common/health/index.js'; import { logger } from './common/logging/index.js'; import { registerTestRoutes } from './test-platform-api.js'; @@ -105,6 +106,12 @@ await fastify.register(reviewRoutes, { prefix: '/api/v1' }); await fastify.register(aslRoutes, { prefix: '/api/v1/asl' }); logger.info('✅ ASL智能文献筛选路由已注册: /api/v1/asl'); +// ============================================ +// 【业务模块】DC - 数据清洗整理 +// ============================================ +await registerDCRoutes(fastify); +logger.info('✅ DC数据清洗模块路由已注册: /api/v1/dc/tool-b'); + // 启动服务器 const start = async () => { try { @@ -119,6 +126,14 @@ const start = async () => { console.error('❌ 数据库连接失败,无法启动服务器'); process.exit(1); } + + // 初始化DC模块(Seed预设模板) + try { + await initDCModule(); + logger.info('✅ DC模块初始化成功'); + } catch (error) { + logger.warn('⚠️ DC模块初始化失败,但不影响启动', { error }); + } // 启动Fastify服务器 await fastify.listen({ diff --git a/backend/src/modules/asl/fulltext-screening/__tests__/api-integration-test.ts b/backend/src/modules/asl/fulltext-screening/__tests__/api-integration-test.ts index 36110dd3..14aa83c5 100644 --- a/backend/src/modules/asl/fulltext-screening/__tests__/api-integration-test.ts +++ b/backend/src/modules/asl/fulltext-screening/__tests__/api-integration-test.ts @@ -291,3 +291,13 @@ runTests().catch((error) => { process.exit(1); }); + + + + + + + + + + diff --git a/backend/src/modules/asl/fulltext-screening/__tests__/e2e-real-test-v2.ts b/backend/src/modules/asl/fulltext-screening/__tests__/e2e-real-test-v2.ts index dc491c80..b4cb969a 100644 --- a/backend/src/modules/asl/fulltext-screening/__tests__/e2e-real-test-v2.ts +++ b/backend/src/modules/asl/fulltext-screening/__tests__/e2e-real-test-v2.ts @@ -232,3 +232,13 @@ runTest() process.exit(1); }); + + + + + + + + + + diff --git a/backend/src/modules/asl/fulltext-screening/__tests__/fulltext-screening-api.http b/backend/src/modules/asl/fulltext-screening/__tests__/fulltext-screening-api.http index d0d93239..36615ff6 100644 --- a/backend/src/modules/asl/fulltext-screening/__tests__/fulltext-screening-api.http +++ b/backend/src/modules/asl/fulltext-screening/__tests__/fulltext-screening-api.http @@ -270,3 +270,13 @@ Content-Type: application/json ### + + + + + + + + + + diff --git a/backend/src/modules/asl/fulltext-screening/services/ExcelExporter.ts b/backend/src/modules/asl/fulltext-screening/services/ExcelExporter.ts index 8f836866..e889ff0f 100644 --- a/backend/src/modules/asl/fulltext-screening/services/ExcelExporter.ts +++ b/backend/src/modules/asl/fulltext-screening/services/ExcelExporter.ts @@ -349,3 +349,13 @@ export class ExcelExporter { } } + + + + + + + + + + diff --git a/backend/src/modules/dc/index.ts b/backend/src/modules/dc/index.ts new file mode 100644 index 00000000..f90cd917 --- /dev/null +++ b/backend/src/modules/dc/index.ts @@ -0,0 +1,113 @@ +/** + * DC模块入口文件 + * + * 导出所有DC模块的路由和服务 + */ + +import { FastifyInstance } from 'fastify'; +import { registerToolBRoutes } from './tool-b/routes/index.js'; +import { templateService } from './tool-b/services/TemplateService.js'; +import { logger } from '../../common/logging/index.js'; + +/** + * 注册DC模块的所有路由 + */ +export async function registerDCRoutes(fastify: FastifyInstance) { + logger.info('[DC] Registering DC module routes'); + + // 注册Tool B路由(病历结构化机器人) + await fastify.register(async (instance) => { + await registerToolBRoutes(instance); + }, { prefix: '/api/v1/dc/tool-b' }); + + logger.info('[DC] DC module routes registered'); +} + +/** + * 初始化DC模块 + * + * - Seed预设模板 + * - 检查数据库表是否存在 + */ +export async function initDCModule() { + try { + logger.info(''); + logger.info('============================================================'); + logger.info('[DC模块] 数据库表检查开始'); + logger.info('============================================================'); + + // 动态导入prisma + const { prisma } = await import('../../config/database.js'); + + // 检查dc_schema + const schemaCheck = await prisma.$queryRawUnsafe(` + SELECT schema_name + FROM information_schema.schemata + WHERE schema_name = 'dc_schema' + `) as Array<{ schema_name: string }>; + + if (schemaCheck.length === 0) { + logger.error('[DC模块] ❌ dc_schema 不存在!'); + logger.error('[DC模块] 解决方案: cd backend && npx prisma db push'); + logger.info('============================================================'); + logger.info(''); + throw new Error('DC模块数据库schema不存在,请执行 npx prisma db push'); + } + + logger.info('[DC模块] ✅ dc_schema 存在'); + + // 检查表 + const tables = await prisma.$queryRawUnsafe(` + SELECT table_name + FROM information_schema.tables + WHERE table_schema = 'dc_schema' + ORDER BY table_name + `) as Array<{ table_name: string }>; + + if (tables.length === 0) { + logger.error('[DC模块] ❌ dc_schema 存在但没有表!'); + logger.error('[DC模块] 解决方案: cd backend && npx prisma db push'); + logger.info('============================================================'); + logger.info(''); + throw new Error('DC模块数据库表不存在,请执行 npx prisma db push'); + } + + logger.info(`[DC模块] ✅ 找到 ${tables.length} 个表:`); + tables.forEach(t => logger.info(`[DC模块] - ${t.table_name}`)); + + // 检查每个表的记录数 + for (const table of tables) { + try { + const countResult = await prisma.$queryRawUnsafe( + `SELECT COUNT(*) as count FROM dc_schema."${table.table_name}"` + ) as Array<{ count: bigint }>; + const count = Number(countResult[0]?.count || 0); + logger.info(`[DC模块] ${table.table_name}: ${count} 条记录`); + } catch (error) { + logger.warn(`[DC模块] ${table.table_name}: 无法查询`); + } + } + + logger.info('============================================================'); + logger.info('[DC模块] 🎉 数据库表检查完成!表结构完整存在!'); + logger.info('============================================================'); + logger.info(''); + + // 初始化预设模板 + logger.info('[DC模块] Initializing DC module'); + await templateService.seedTemplates(); + logger.info('[DC模块] DC module initialized successfully'); + logger.info(''); + + } catch (error) { + logger.error('[DC模块] Failed to initialize DC module', { error }); + throw error; + } +} + +// 导出服务(供测试使用) +export { healthCheckService } from './tool-b/services/HealthCheckService.js'; +export { templateService } from './tool-b/services/TemplateService.js'; +export { dualModelExtractionService } from './tool-b/services/DualModelExtractionService.js'; +export { conflictDetectionService } from './tool-b/services/ConflictDetectionService.js'; + diff --git a/backend/src/modules/dc/tool-b/controllers/ExtractionController.ts b/backend/src/modules/dc/tool-b/controllers/ExtractionController.ts new file mode 100644 index 00000000..f0f652b5 --- /dev/null +++ b/backend/src/modules/dc/tool-b/controllers/ExtractionController.ts @@ -0,0 +1,391 @@ +/** + * DC模块 - 提取控制器 + * + * API端点: + * - POST /api/v1/dc/tool-b/health-check - 健康检查 + * - GET /api/v1/dc/tool-b/templates - 获取模板列表 + * - POST /api/v1/dc/tool-b/tasks - 创建提取任务 + * - GET /api/v1/dc/tool-b/tasks/:taskId/progress - 查询任务进度 + * - GET /api/v1/dc/tool-b/tasks/:taskId/items - 获取验证网格数据 + * - POST /api/v1/dc/tool-b/items/:itemId/resolve - 裁决冲突 + * + * 平台能力复用: + * - ✅ logger: 日志记录 + * - ✅ prisma: 数据库操作 + * - ✅ storage: 文件操作 + * - ✅ jobQueue: 异步任务 + */ + +import { FastifyRequest, FastifyReply } from 'fastify'; +import { healthCheckService } from '../services/HealthCheckService.js'; +import { templateService } from '../services/TemplateService.js'; +import { dualModelExtractionService } from '../services/DualModelExtractionService.js'; +import { conflictDetectionService } from '../services/ConflictDetectionService.js'; +import { storage } from '../../../../common/storage/index.js'; +import { logger } from '../../../../common/logging/index.js'; +import { prisma } from '../../../../config/database.js'; +import * as xlsx from 'xlsx'; + +export class ExtractionController { + /** + * 健康检查 + * POST /health-check + */ + async healthCheck(request: FastifyRequest<{ + Body: { + fileKey: string; + columnName: string; + } + }>, reply: FastifyReply) { + try { + const { fileKey, columnName } = request.body; + const userId = (request as any).userId || 'default-user'; // TODO: 从auth middleware获取 + + logger.info('[API] Health check request', { fileKey, columnName, userId }); + + const result = await healthCheckService.check(fileKey, columnName, userId); + + return reply.code(200).send({ + success: true, + data: result + }); + + } catch (error) { + logger.error('[API] Health check failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } + + /** + * 获取模板列表 + * GET /templates + */ + async getTemplates(request: FastifyRequest, reply: FastifyReply) { + try { + logger.info('[API] Get templates request'); + + const templates = await templateService.getAllTemplates(); + + return reply.code(200).send({ + success: true, + data: { templates } + }); + + } catch (error) { + logger.error('[API] Get templates failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } + + /** + * 创建提取任务 + * POST /tasks + */ + async createTask(request: FastifyRequest<{ + Body: { + projectName: string; + sourceFileKey: string; + textColumn: string; + diseaseType: string; + reportType: string; + modelA?: string; + modelB?: string; + } + }>, reply: FastifyReply) { + try { + const { + projectName, + sourceFileKey, + textColumn, + diseaseType, + reportType, + modelA = 'deepseek-v3', + modelB = 'qwen-max' + } = request.body; + const userId = (request as any).userId || 'default-user'; + + logger.info('[API] Create task request', { + userId, + projectName, + diseaseType, + reportType + }); + + // 1. 获取模板 + const template = await templateService.getTemplate(diseaseType, reportType); + if (!template) { + return reply.code(404).send({ + success: false, + error: `Template not found: ${diseaseType}/${reportType}` + }); + } + + // 2. 读取Excel文件,创建items + const fileBuffer = await storage.download(sourceFileKey); + if (!fileBuffer) { + return reply.code(404).send({ + success: false, + error: `File not found: ${sourceFileKey}` + }); + } + + const workbook = xlsx.read(fileBuffer, { type: 'buffer' }); + const sheetName = workbook.SheetNames[0]; + const worksheet = workbook.Sheets[sheetName]; + const data = xlsx.utils.sheet_to_json>(worksheet); + + if (!data[0].hasOwnProperty(textColumn)) { + return reply.code(400).send({ + success: false, + error: `Column '${textColumn}' not found in Excel` + }); + } + + // 3. 创建任务 + const task = await prisma.dCExtractionTask.create({ + data: { + userId, + projectName, + sourceFileKey, + textColumn, + diseaseType, + reportType, + targetFields: template.fields, + modelA, + modelB, + totalCount: data.length, + status: 'pending' + } + }); + + // 4. 创建items + const itemsData = data.map((row, index) => ({ + taskId: task.id, + rowIndex: index + 1, + originalText: String(row[textColumn] || '') + })); + + await prisma.dCExtractionItem.createMany({ + data: itemsData + }); + + // 5. 启动异步任务 + // TODO: 使用jobQueue.add() + // 暂时直接调用 + dualModelExtractionService.batchExtract(task.id).catch(err => { + logger.error('[API] Batch extraction failed', { error: err, taskId: task.id }); + }); + + logger.info('[API] Task created', { taskId: task.id, itemCount: data.length }); + + return reply.code(201).send({ + success: true, + data: { + taskId: task.id, + totalCount: data.length, + status: 'pending' + } + }); + + } catch (error) { + logger.error('[API] Create task failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } + + /** + * 查询任务进度 + * GET /tasks/:taskId/progress + */ + async getTaskProgress(request: FastifyRequest<{ + Params: { taskId: string } + }>, reply: FastifyReply) { + try { + const { taskId } = request.params; + + logger.info('[API] Get task progress', { taskId }); + + const task = await prisma.dCExtractionTask.findUnique({ + where: { id: taskId } + }); + + if (!task) { + return reply.code(404).send({ + success: false, + error: 'Task not found' + }); + } + + return reply.code(200).send({ + success: true, + data: { + taskId: task.id, + status: task.status, + totalCount: task.totalCount, + processedCount: task.processedCount, + cleanCount: task.cleanCount, + conflictCount: task.conflictCount, + failedCount: task.failedCount, + totalTokens: task.totalTokens, + totalCost: task.totalCost, + progress: task.totalCount > 0 ? Math.round((task.processedCount / task.totalCount) * 100) : 0 + } + }); + + } catch (error) { + logger.error('[API] Get task progress failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } + + /** + * 获取验证网格数据 + * GET /tasks/:taskId/items + */ + async getTaskItems(request: FastifyRequest<{ + Params: { taskId: string }; + Querystring: { page?: string; limit?: string; status?: string } + }>, reply: FastifyReply) { + try { + const { taskId } = request.params; + const page = parseInt(request.query.page || '1'); + const limit = parseInt(request.query.limit || '50'); + const statusFilter = request.query.status; + + logger.info('[API] Get task items', { taskId, page, limit, statusFilter }); + + const where: any = { taskId }; + if (statusFilter) { + where.status = statusFilter; + } + + const [items, total] = await Promise.all([ + prisma.dCExtractionItem.findMany({ + where, + skip: (page - 1) * limit, + take: limit, + orderBy: { rowIndex: 'asc' } + }), + prisma.dCExtractionItem.count({ where }) + ]); + + return reply.code(200).send({ + success: true, + data: { + items: items.map(item => ({ + id: item.id, + rowIndex: item.rowIndex, + originalText: item.originalText, + resultA: item.resultA, + resultB: item.resultB, + status: item.status, + conflictFields: item.conflictFields, + finalResult: item.finalResult + })), + pagination: { + page, + limit, + total, + totalPages: Math.ceil(total / limit) + } + } + }); + + } catch (error) { + logger.error('[API] Get task items failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } + + /** + * 裁决冲突 + * POST /items/:itemId/resolve + */ + async resolveConflict(request: FastifyRequest<{ + Params: { itemId: string }; + Body: { + field: string; + chosenValue: string; + } + }>, reply: FastifyReply) { + try { + const { itemId } = request.params; + const { field, chosenValue } = request.body; + + logger.info('[API] Resolve conflict', { itemId, field }); + + // 获取当前记录 + const item = await prisma.dCExtractionItem.findUnique({ + where: { id: itemId } + }); + + if (!item) { + return reply.code(404).send({ + success: false, + error: 'Item not found' + }); + } + + // 更新finalResult + const finalResult = { ...(item.finalResult as Record || {}) }; + finalResult[field] = chosenValue; + + // 移除已解决的冲突字段 + const conflictFields = item.conflictFields.filter(f => f !== field); + + // 更新状态 + const newStatus = conflictFields.length === 0 ? 'resolved' : 'conflict'; + + await prisma.dCExtractionItem.update({ + where: { id: itemId }, + data: { + finalResult, + conflictFields, + status: newStatus, + resolvedAt: conflictFields.length === 0 ? new Date() : null + } + }); + + logger.info('[API] Conflict resolved', { itemId, field, newStatus }); + + return reply.code(200).send({ + success: true, + data: { + itemId, + status: newStatus, + remainingConflicts: conflictFields.length + } + }); + + } catch (error) { + logger.error('[API] Resolve conflict failed', { error }); + return reply.code(500).send({ + success: false, + error: String(error) + }); + } + } +} + +// 导出单例 +export const extractionController = new ExtractionController(); + + + + + diff --git a/backend/src/modules/dc/tool-b/routes/index.ts b/backend/src/modules/dc/tool-b/routes/index.ts new file mode 100644 index 00000000..5b987cd9 --- /dev/null +++ b/backend/src/modules/dc/tool-b/routes/index.ts @@ -0,0 +1,118 @@ +/** + * DC模块 - Tool B 路由配置 + * + * Base URL: /api/v1/dc/tool-b + */ + +import { FastifyInstance } from 'fastify'; +import { extractionController } from '../controllers/ExtractionController.js'; +import { logger } from '../../../../common/logging/index.js'; + +export async function registerToolBRoutes(fastify: FastifyInstance) { + logger.info('[Routes] Registering DC Tool-B routes'); + + // 健康检查 + fastify.post('/health-check', { + schema: { + body: { + type: 'object', + required: ['fileKey', 'columnName'], + properties: { + fileKey: { type: 'string' }, + columnName: { type: 'string' } + } + } + }, + handler: extractionController.healthCheck.bind(extractionController) + }); + + // 获取模板列表 + fastify.get('/templates', { + handler: extractionController.getTemplates.bind(extractionController) + }); + + // 创建提取任务 + fastify.post('/tasks', { + schema: { + body: { + type: 'object', + required: ['projectName', 'sourceFileKey', 'textColumn', 'diseaseType', 'reportType'], + properties: { + projectName: { type: 'string' }, + sourceFileKey: { type: 'string' }, + textColumn: { type: 'string' }, + diseaseType: { type: 'string' }, + reportType: { type: 'string' }, + modelA: { type: 'string' }, + modelB: { type: 'string' } + } + } + }, + handler: extractionController.createTask.bind(extractionController) + }); + + // 查询任务进度 + fastify.get('/tasks/:taskId/progress', { + schema: { + params: { + type: 'object', + required: ['taskId'], + properties: { + taskId: { type: 'string' } + } + } + }, + handler: extractionController.getTaskProgress.bind(extractionController) + }); + + // 获取验证网格数据 + fastify.get('/tasks/:taskId/items', { + schema: { + params: { + type: 'object', + required: ['taskId'], + properties: { + taskId: { type: 'string' } + } + }, + querystring: { + type: 'object', + properties: { + page: { type: 'string' }, + limit: { type: 'string' }, + status: { type: 'string' } + } + } + }, + handler: extractionController.getTaskItems.bind(extractionController) + }); + + // 裁决冲突 + fastify.post('/items/:itemId/resolve', { + schema: { + params: { + type: 'object', + required: ['itemId'], + properties: { + itemId: { type: 'string' } + } + }, + body: { + type: 'object', + required: ['field', 'chosenValue'], + properties: { + field: { type: 'string' }, + chosenValue: { type: 'string' } + } + } + }, + handler: extractionController.resolveConflict.bind(extractionController) + }); + + logger.info('[Routes] DC Tool-B routes registered successfully'); +} + + + + + diff --git a/backend/src/modules/dc/tool-b/services/ConflictDetectionService.ts b/backend/src/modules/dc/tool-b/services/ConflictDetectionService.ts new file mode 100644 index 00000000..efd6e5cf --- /dev/null +++ b/backend/src/modules/dc/tool-b/services/ConflictDetectionService.ts @@ -0,0 +1,218 @@ +/** + * DC模块 - 冲突检测服务 + * + * 功能: + * - 比较双模型提取结果 + * - 标记冲突字段 + * - 计算冲突严重程度 + * - 生成冲突报告 + * + * 平台能力复用: + * - ✅ logger: 日志记录 + */ + +import { logger } from '../../../../common/logging/index.js'; + +export interface ConflictResult { + hasConflict: boolean; + conflictFields: string[]; + conflictDetails: Array<{ + fieldName: string; + valueA: string; + valueB: string; + similarity: number; // 0-1, 相似度 + }>; + severity: 'low' | 'medium' | 'high'; +} + +export class ConflictDetectionService { + /** + * 检测冲突 + * + * @param resultA DeepSeek结果 + * @param resultB Qwen结果 + * @returns 冲突分析结果 + */ + detectConflict(resultA: Record, resultB: Record): ConflictResult { + try { + logger.info('[Conflict] Starting conflict detection'); + + const conflictFields: string[] = []; + const conflictDetails: ConflictResult['conflictDetails'] = []; + + // 获取所有字段 + const allFields = new Set([...Object.keys(resultA), ...Object.keys(resultB)]); + + // 逐字段比较 + for (const field of allFields) { + const valueA = resultA[field] || ''; + const valueB = resultB[field] || ''; + + // 归一化后比较 + const normalizedA = this.normalize(valueA); + const normalizedB = this.normalize(valueB); + + if (normalizedA !== normalizedB) { + // 检测到冲突 + const similarity = this.calculateSimilarity(normalizedA, normalizedB); + + conflictFields.push(field); + conflictDetails.push({ + fieldName: field, + valueA, + valueB, + similarity + }); + } + } + + // 计算严重程度 + const severity = this.calculateSeverity(conflictFields.length, allFields.size); + + const result: ConflictResult = { + hasConflict: conflictFields.length > 0, + conflictFields, + conflictDetails, + severity + }; + + logger.info('[Conflict] Detection completed', { + hasConflict: result.hasConflict, + conflictCount: conflictFields.length, + severity + }); + + return result; + + } catch (error) { + logger.error('[Conflict] Detection failed', { error }); + throw error; + } + } + + /** + * 归一化文本 + * + * - 去除空格 + * - 转小写 + * - 半角化 + * - 数值归一化(3cm = 3.0cm = 3 cm) + */ + private normalize(value: string): string { + let normalized = String(value) + .toLowerCase() + .trim() + .replace(/\s+/g, '') // 去除所有空格 + .replace(/[,。;:!?]/g, (match) => { // 全角转半角 + return { + ',': ',', + '。': '.', + ';': ';', + ':': ':', + '!': '!', + '?': '?' + }[match] || match; + }); + + // 数值归一化:提取数字 + const numberMatch = normalized.match(/(\d+\.?\d*)\s*(cm|mm|kg|mg|ml|%)?/); + if (numberMatch) { + const num = parseFloat(numberMatch[1]); + const unit = numberMatch[2] || ''; + normalized = `${num}${unit}`; + } + + return normalized; + } + + /** + * 计算文本相似度(Dice Coefficient) + * + * 范围:0-1,1表示完全相同 + */ + private calculateSimilarity(a: string, b: string): number { + if (a === b) return 1; + if (!a || !b) return 0; + + // 生成2-gram + const bigramsA = this.getBigrams(a); + const bigramsB = this.getBigrams(b); + + if (bigramsA.size === 0 && bigramsB.size === 0) return 1; + if (bigramsA.size === 0 || bigramsB.size === 0) return 0; + + // 计算交集 + const intersection = new Set([...bigramsA].filter(x => bigramsB.has(x))); + + // Dice系数:2 * |A ∩ B| / (|A| + |B|) + const similarity = (2 * intersection.size) / (bigramsA.size + bigramsB.size); + + return similarity; + } + + /** + * 生成2-gram集合 + */ + private getBigrams(str: string): Set { + const bigrams = new Set(); + for (let i = 0; i < str.length - 1; i++) { + bigrams.add(str.substring(i, i + 2)); + } + return bigrams; + } + + /** + * 计算冲突严重程度 + */ + private calculateSeverity(conflictCount: number, totalFields: number): 'low' | 'medium' | 'high' { + const conflictRate = conflictCount / totalFields; + + if (conflictRate === 0) return 'low'; + if (conflictRate <= 0.3) return 'low'; // ≤30% + if (conflictRate <= 0.6) return 'medium'; // 30%-60% + return 'high'; // >60% + } + + /** + * 批量检测冲突 + * + * @param items 提取记录数组 + * @returns 冲突统计 + */ + batchDetect(items: Array<{ resultA: Record; resultB: Record }>): { + totalCount: number; + cleanCount: number; + conflictCount: number; + severityDistribution: Record<'low' | 'medium' | 'high', number>; + } { + let cleanCount = 0; + let conflictCount = 0; + const severityDistribution = { low: 0, medium: 0, high: 0 }; + + for (const item of items) { + const result = this.detectConflict(item.resultA, item.resultB); + + if (result.hasConflict) { + conflictCount++; + severityDistribution[result.severity]++; + } else { + cleanCount++; + } + } + + return { + totalCount: items.length, + cleanCount, + conflictCount, + severityDistribution + }; + } +} + +// 导出单例 +export const conflictDetectionService = new ConflictDetectionService(); + + + + + diff --git a/backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts b/backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts new file mode 100644 index 00000000..fd0c7dc7 --- /dev/null +++ b/backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts @@ -0,0 +1,393 @@ +/** + * DC模块 - 双模型提取服务 + * + * 功能: + * - 并发调用DeepSeek-V3和Qwen-Max进行文本提取 + * - PII脱敏处理 + * - JSON解析与容错 + * - Token统计 + * - 异步任务管理 + * + * 平台能力复用: + * - ✅ LLMFactory: LLM调用 + * - ✅ jobQueue: 异步任务 + * - ✅ logger: 日志记录 + * - ✅ prisma: 数据库操作 + */ + +import { LLMFactory } from '../../../../common/llm/adapters/LLMFactory.js'; +import { logger } from '../../../../common/logging/index.js'; +import { prisma } from '../../../../config/database.js'; + +export interface ExtractionInput { + text: string; + fields: { name: string; desc: string }[]; + promptTemplate: string; +} + +export interface ExtractionOutput { + result: Record; + tokensUsed: number; + rawOutput: any; +} + +export class DualModelExtractionService { + /** + * 双模型并发提取 + * + * @param input 提取输入 + * @param taskId 任务ID + * @param itemId 记录ID + * @returns 双模型结果 + */ + async extract(input: ExtractionInput, taskId: string, itemId: string): Promise<{ + resultA: ExtractionOutput; + resultB: ExtractionOutput; + }> { + try { + logger.info('[DualExtraction] Starting extraction', { taskId, itemId }); + + // 1. PII脱敏 + const maskedText = this.maskPII(input.text); + + // 2. 构建Prompt + const prompt = this.buildPrompt(maskedText, input.fields, input.promptTemplate); + + // 3. 并发调用两个模型(DeepSeek & Qwen) + const [resultA, resultB] = await Promise.allSettled([ + this.callModel('deepseek', prompt, input.fields), + this.callModel('qwen', prompt, input.fields) + ]); + + // 4. 处理结果 + if (resultA.status === 'rejected' || resultB.status === 'rejected') { + logger.error('[DualExtraction] One or both models failed', { + taskId, + itemId, + errorA: resultA.status === 'rejected' ? resultA.reason : null, + errorB: resultB.status === 'rejected' ? resultB.reason : null + }); + throw new Error('Dual model extraction failed'); + } + + logger.info('[DualExtraction] Extraction completed', { + taskId, + itemId, + tokensA: resultA.value.tokensUsed, + tokensB: resultB.value.tokensUsed + }); + + return { + resultA: resultA.value, + resultB: resultB.value + }; + + } catch (error) { + logger.error('[DualExtraction] Extraction failed', { error, taskId, itemId }); + throw error; + } + } + + /** + * PII脱敏 + * + * 使用正则表达式替换敏感信息: + * - 姓名:张** + * - 身份证号:3301********1234 + * - 手机号:138****5678 + */ + private maskPII(text: string): string { + let masked = text; + + // 手机号脱敏:138****5678 + masked = masked.replace(/1[3-9]\d{9}/g, (match) => { + return match.substring(0, 3) + '****' + match.substring(7); + }); + + // 身份证号脱敏:330102********1234 + masked = masked.replace(/\d{6}(19|20)\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])\d{3}[\dxX]/g, (match) => { + return match.substring(0, 6) + '********' + match.substring(14); + }); + + // 简单的姓名脱敏(匹配:患者xxx、姓名:xxx) + masked = masked.replace(/(患者|姓名[::])\s*([^\s,。,]{2,4})/g, (match, prefix, name) => { + if (name.length === 2) { + return prefix + name[0] + '*'; + } + return prefix + name[0] + '*'.repeat(name.length - 1); + }); + + return masked; + } + + /** + * 构建Prompt + */ + private buildPrompt(text: string, fields: { name: string; desc: string }[], template: string): string { + // 在模板末尾添加病历文本 + return `${template} + +**病历原文:** +${text} + +请严格按照JSON格式输出,不要有任何额外文字。`; + } + + /** + * 调用单个模型 + */ + private async callModel( + modelType: 'deepseek' | 'qwen', + prompt: string, + fields: { name: string; desc: string }[] + ): Promise { + try { + // 使用LLMFactory获取LLM客户端 + const modelName = modelType === 'deepseek' ? 'deepseek-v3' : 'qwen-max'; + const llm = LLMFactory.createLLM(modelName); + + logger.info(`[${modelType.toUpperCase()}] Calling model`, { modelName }); + + // 调用LLM + const response = await llm.generateText(prompt, { + temperature: 0, // 最大确定性 + maxTokens: 1000 + }); + + logger.info(`[${modelType.toUpperCase()}] Model responded`, { + modelName, + tokensUsed: response.tokensUsed + }); + + // 解析JSON(3层容错) + const result = this.parseJSON(response.text, fields); + + return { + result, + tokensUsed: response.tokensUsed || 0, + rawOutput: response.text + }; + + } catch (error) { + logger.error(`[${modelType.toUpperCase()}] Model call failed`, { error, modelType }); + throw error; + } + } + + /** + * 解析JSON(3层容错策略) + * + * 1. 直接JSON.parse + * 2. 提取```json代码块 + * 3. 提取{}内容 + */ + private parseJSON(text: string, fields: { name: string; desc: string }[]): Record { + // 策略1:直接解析 + try { + const parsed = JSON.parse(text); + if (this.validateFields(parsed, fields)) { + return parsed; + } + } catch (e) { + // 继续下一个策略 + } + + // 策略2:提取```json代码块 + const codeBlockMatch = text.match(/```json\s*\n([\s\S]*?)\n```/); + if (codeBlockMatch) { + try { + const parsed = JSON.parse(codeBlockMatch[1]); + if (this.validateFields(parsed, fields)) { + return parsed; + } + } catch (e) { + // 继续下一个策略 + } + } + + // 策略3:提取第一个完整的{}对象 + const objectMatch = text.match(/\{[\s\S]*\}/); + if (objectMatch) { + try { + const parsed = JSON.parse(objectMatch[0]); + if (this.validateFields(parsed, fields)) { + return parsed; + } + } catch (e) { + // 解析失败 + } + } + + // 所有策略失败,返回空对象 + logger.warn('[JSON] All parse strategies failed', { text }); + const emptyResult: Record = {}; + fields.forEach(f => { + emptyResult[f.name] = '解析失败'; + }); + return emptyResult; + } + + /** + * 验证字段完整性 + */ + private validateFields(parsed: any, fields: { name: string; desc: string }[]): boolean { + if (!parsed || typeof parsed !== 'object') { + return false; + } + + // 检查所有必需字段是否存在 + return fields.every(f => parsed.hasOwnProperty(f.name)); + } + + /** + * 批量提取(异步任务) + * + * @param taskId 任务ID + */ + async batchExtract(taskId: string): Promise { + try { + logger.info('[Batch] Starting batch extraction', { taskId }); + + // 1. 获取任务 + const task = await prisma.dCExtractionTask.findUnique({ + where: { id: taskId }, + include: { items: true } + }); + + if (!task) { + throw new Error(`Task not found: ${taskId}`); + } + + // 2. 更新任务状态 + await prisma.dCExtractionTask.update({ + where: { id: taskId }, + data: { + status: 'processing', + startedAt: new Date() + } + }); + + // 3. 获取模板 + const template = await prisma.dCTemplate.findUnique({ + where: { + diseaseType_reportType: { + diseaseType: task.diseaseType, + reportType: task.reportType + } + } + }); + + if (!template) { + throw new Error(`Template not found: ${task.diseaseType}/${task.reportType}`); + } + + const fields = template.fields as { name: string; desc: string }[]; + + // 4. 逐条处理 + let processedCount = 0; + let cleanCount = 0; + let conflictCount = 0; + let totalTokens = 0; + + for (const item of task.items) { + try { + // 双模型提取 + const { resultA, resultB } = await this.extract( + { + text: item.originalText, + fields, + promptTemplate: template.promptTemplate + }, + taskId, + item.id + ); + + // 检测冲突(由ConflictDetectionService处理,这里暂时简单比较) + const hasConflict = JSON.stringify(resultA.result) !== JSON.stringify(resultB.result); + + // 更新记录 + await prisma.dCExtractionItem.update({ + where: { id: item.id }, + data: { + resultA: resultA.result, + resultB: resultB.result, + tokensA: resultA.tokensUsed, + tokensB: resultB.tokensUsed, + status: hasConflict ? 'conflict' : 'clean', + finalResult: hasConflict ? null : resultA.result // 一致时自动采纳 + } + }); + + processedCount++; + if (hasConflict) { + conflictCount++; + } else { + cleanCount++; + } + totalTokens += resultA.tokensUsed + resultB.tokensUsed; + + // 更新任务进度 + await prisma.dCExtractionTask.update({ + where: { id: taskId }, + data: { + processedCount, + cleanCount, + conflictCount, + totalTokens + } + }); + + } catch (error) { + logger.error('[Batch] Item extraction failed', { error, itemId: item.id }); + + await prisma.dCExtractionItem.update({ + where: { id: item.id }, + data: { + status: 'failed', + error: String(error) + } + }); + } + } + + // 5. 完成任务 + await prisma.dCExtractionTask.update({ + where: { id: taskId }, + data: { + status: 'completed', + completedAt: new Date() + } + }); + + logger.info('[Batch] Batch extraction completed', { + taskId, + processedCount, + cleanCount, + conflictCount, + totalTokens + }); + + } catch (error) { + logger.error('[Batch] Batch extraction failed', { error, taskId }); + + // 更新任务为失败状态 + await prisma.dCExtractionTask.update({ + where: { id: taskId }, + data: { + status: 'failed', + error: String(error) + } + }); + + throw error; + } + } +} + +// 导出单例 +export const dualModelExtractionService = new DualModelExtractionService(); + + + + + diff --git a/backend/src/modules/dc/tool-b/services/HealthCheckService.ts b/backend/src/modules/dc/tool-b/services/HealthCheckService.ts new file mode 100644 index 00000000..f045144b --- /dev/null +++ b/backend/src/modules/dc/tool-b/services/HealthCheckService.ts @@ -0,0 +1,193 @@ +/** + * DC模块 - 健康检查服务 + * + * 功能: + * - Excel列数据质量检查(空值率、平均长度) + * - Token预估 + * - 拦截不适合的数据列 + * - 结果缓存(避免重复计算) + * + * 平台能力复用: + * - ✅ storage: 文件读取 + * - ✅ logger: 日志记录 + * - ✅ cache: 结果缓存 + * - ✅ prisma: 数据库存储 + */ + +import * as xlsx from 'xlsx'; +import { storage } from '../../../../common/storage/index.js'; +import { logger } from '../../../../common/logging/index.js'; +import { cache } from '../../../../common/cache/index.js'; +import { prisma } from '../../../../config/database.js'; + +export interface HealthCheckResult { + status: 'good' | 'bad'; + emptyRate: number; + avgLength: number; + totalRows: number; + estimatedTokens: number; + message: string; +} + +export class HealthCheckService { + /** + * 执行健康检查 + * + * @param fileKey Storage中的文件路径 + * @param columnName 要检查的列名 + * @param userId 用户ID + * @returns 健康检查结果 + */ + async check(fileKey: string, columnName: string, userId: string): Promise { + try { + logger.info('[HealthCheck] Starting health check', { fileKey, columnName, userId }); + + // 1. 检查缓存(避免重复计算) + const cacheKey = `dc:health:${fileKey}:${columnName}`; + const cached = await cache.get(cacheKey); + if (cached) { + logger.info('[HealthCheck] Cache hit', { cacheKey }); + return cached; + } + + // 2. 从Storage读取Excel文件 + const fileBuffer = await storage.download(fileKey); + if (!fileBuffer) { + throw new Error(`File not found: ${fileKey}`); + } + + // 3. 解析Excel(仅前100行) + const workbook = xlsx.read(fileBuffer, { type: 'buffer' }); + const sheetName = workbook.SheetNames[0]; + const worksheet = workbook.Sheets[sheetName]; + const data = xlsx.utils.sheet_to_json>(worksheet, { range: 99 }); // 前100行 + + logger.info('[HealthCheck] Excel parsed', { totalRows: data.length }); + + // 4. 检查列是否存在 + if (data.length === 0 || !data[0].hasOwnProperty(columnName)) { + throw new Error(`Column '${columnName}' not found in Excel`); + } + + // 5. 计算统计指标 + const stats = this.calculateStats(data, columnName); + + // 6. 判断健康状态 + const result = this.evaluateHealth(stats); + + // 7. 保存到数据库 + await prisma.dCHealthCheck.create({ + data: { + userId, + fileName: fileKey.split('/').pop() || fileKey, + columnName, + emptyRate: result.emptyRate, + avgLength: result.avgLength, + totalRows: result.totalRows, + estimatedTokens: result.estimatedTokens, + status: result.status, + message: result.message + } + }); + + // 8. 缓存结果(24小时) + await cache.set(cacheKey, result, 86400); + + logger.info('[HealthCheck] Check completed', { status: result.status }); + + return result; + + } catch (error) { + logger.error('[HealthCheck] Check failed', { error, fileKey, columnName }); + throw error; + } + } + + /** + * 计算统计指标 + */ + private calculateStats(data: Record[], columnName: string) { + const totalRows = data.length; + let emptyCount = 0; + let totalLength = 0; + let validCount = 0; + + for (const row of data) { + const value = row[columnName]; + + if (!value || String(value).trim() === '') { + emptyCount++; + } else { + const text = String(value); + totalLength += text.length; + validCount++; + } + } + + const emptyRate = totalRows > 0 ? emptyCount / totalRows : 0; + const avgLength = validCount > 0 ? totalLength / validCount : 0; + + return { totalRows, emptyCount, emptyRate, avgLength, validCount }; + } + + /** + * 评估健康状态 + */ + private evaluateHealth(stats: ReturnType): HealthCheckResult { + const { totalRows, emptyRate, avgLength } = stats; + + // 拦截策略1:空值率 > 80% + if (emptyRate > 0.8) { + return { + status: 'bad', + emptyRate, + avgLength, + totalRows, + estimatedTokens: 0, + message: `空值率过高(${(emptyRate * 100).toFixed(1)}%),该列不适合提取` + }; + } + + // 拦截策略2:平均长度 < 10 + if (avgLength < 10) { + return { + status: 'bad', + emptyRate, + avgLength, + totalRows, + estimatedTokens: 0, + message: `文本过短(平均${avgLength.toFixed(1)}字符),该列不适合提取` + }; + } + + // Token预估(粗略估算:字符数 * 1.5 / 2.5) + // 中文通常1个token约等于2-3个字符 + const estimatedTokens = Math.ceil((totalRows * avgLength * 1.5) / 2.5); + + return { + status: 'good', + emptyRate, + avgLength, + totalRows, + estimatedTokens, + message: `健康度良好,预计消耗约 ${(estimatedTokens / 1000).toFixed(1)}k Token(双模型约 ${(estimatedTokens * 2 / 1000).toFixed(1)}k Token)` + }; + } + + /** + * 清除缓存 + */ + async clearCache(fileKey: string, columnName: string): Promise { + const cacheKey = `dc:health:${fileKey}:${columnName}`; + await cache.delete(cacheKey); + logger.info('[HealthCheck] Cache cleared', { cacheKey }); + } +} + +// 导出单例 +export const healthCheckService = new HealthCheckService(); + + + + + diff --git a/backend/src/modules/dc/tool-b/services/TemplateService.ts b/backend/src/modules/dc/tool-b/services/TemplateService.ts new file mode 100644 index 00000000..47b08dee --- /dev/null +++ b/backend/src/modules/dc/tool-b/services/TemplateService.ts @@ -0,0 +1,246 @@ +/** + * DC模块 - 模板服务 + * + * 功能: + * - 管理预设提取模板(疾病类型 + 报告类型) + * - 提供模板列表查询 + * - Seed初始数据(3个预设模板) + * + * 平台能力复用: + * - ✅ prisma: 数据库操作 + * - ✅ logger: 日志记录 + */ + +import { prisma } from '../../../../config/database.js'; +import { logger } from '../../../../common/logging/index.js'; + +export interface TemplateField { + name: string; + desc: string; + width?: string; // TailwindCSS class +} + +export interface Template { + id: string; + diseaseType: string; + reportType: string; + displayName: string; + fields: TemplateField[]; + promptTemplate: string; +} + +export class TemplateService { + /** + * 获取所有模板 + */ + async getAllTemplates(): Promise { + try { + logger.info('[Template] Fetching all templates'); + + const templates = await prisma.dCTemplate.findMany({ + orderBy: [{ diseaseType: 'asc' }, { reportType: 'asc' }] + }); + + logger.info('[Template] Templates fetched', { count: templates.length }); + + return templates.map(t => ({ + id: t.id, + diseaseType: t.diseaseType, + reportType: t.reportType, + displayName: t.displayName, + fields: t.fields as TemplateField[], + promptTemplate: t.promptTemplate + })); + + } catch (error) { + logger.error('[Template] Failed to fetch templates', { error }); + throw error; + } + } + + /** + * 根据疾病和报告类型获取模板 + */ + async getTemplate(diseaseType: string, reportType: string): Promise