Develop/AIclinicalresearch
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dc): Complete Tool B MVP with full API integration and bug fixes

Browse Source 
Phase 5: Export Feature
- Add Excel export API endpoint (GET /tasks/:id/export)
- Fix Content-Disposition header encoding for Chinese filenames
- Fix export field order to match template definition
- Export finalResult or resultA as fallback

API Integration Fixes (Phase 1-5):
- Fix API response parsing (return result.data consistently)
- Fix field name mismatch (fileKey -> sourceFileKey)
- Fix Excel parsing bug (range:99 -> slice(0,100))
- Add file upload with Excel parsing (columns, totalRows)
- Add detailed error logging for debugging

LLM Integration Fixes:
- Fix LLM call method: LLMFactory.createLLM -> getAdapter
- Fix adapter interface: generateText -> chat([messages])
- Fix response fields: text -> content, tokensUsed -> usage.totalTokens
- Fix model names: qwen-max -> qwen3-72b

React Infinite Loop Fixes:
- Step2: Remove updateState from useEffect deps
- Step3: Add useRef to prevent Strict Mode double execution
- Step3: Clear interval on API failure (max 3 retries)
- Step4: Add useRef to prevent infinite data loading
- Add cleanup functions to all useEffect hooks

Frontend Enhancements:
- Add comprehensive error handling with user-friendly messages
- Remove debug console.logs (production ready)
- Fix TypeScript type definitions (TaskProgress, ExtractionItem)
- Improve Step4Verify data transformation logic

Backend Enhancements:
- Add detailed logging at each step for debugging
- Add parameter validation in controllers
- Improve error messages with stack traces (dev mode)
- Add export field ordering by template definition

Documentation Updates:
- Update module status: Tool B MVP completed
- Create MVP completion summary (06-开发记录)
- Create technical debt document (07-技术债务)
- Update API documentation with test status
- Update database documentation with verified status
- Update system overview with DC module status
- Document 4 known issues (Excel preprocessing, progress display, etc.)

Testing Results:
- File upload: 9 rows parsed successfully
- Health check: Column validation working
- Dual model extraction: DeepSeek-V3 + Qwen-Max both working
- Processing time: ~49s for 9 records (~5s per record)
- Token usage: ~10k tokens total (~1.1k per record)
- Conflict detection: 1 clean, 8 conflicts (88.9% conflict rate)
- Excel export: Working with proper encoding

Files Changed:
Backend (~500 lines):
- ExtractionController.ts: Add upload endpoint, improve logging
- DualModelExtractionService.ts: Fix LLM call methods, add detailed logs
- HealthCheckService.ts: Fix Excel range parsing
- routes/index.ts: Add upload route

Frontend (~200 lines):
- toolB.ts: Fix API response parsing, add error handling
- Step1Upload.tsx: Integrate upload and health check APIs
- Step2Schema.tsx: Fix infinite loop, load templates from API
- Step3Processing.tsx: Fix infinite loop, integrate progress polling
- Step4Verify.tsx: Fix infinite loop, transform backend data correctly
- Step5Result.tsx: Integrate export API
- index.tsx: Add file metadata to state

Scripts:
- check-task-progress.mjs: Database inspection utility

Docs (~8 files):
- 00-模块当前状态与开发指南.md: Update to v2.0
- API设计文档.md: Mark all endpoints as tested
- 数据库设计文档.md: Update verification status
- DC模块Tool-B开发计划.md: Add MVP completion notice
- DC模块Tool-B开发任务清单.md: Update progress to 100%
- Tool-B-MVP完成总结.md: New completion summary
- Tool-B技术债务清单.md: New technical debt document
- 00-系统当前状态与开发指南.md: Update DC module status

Status: Tool B MVP complete and production ready
This commit is contained in:
HaHafeng
2025-12-03 15:07:39 +08:00
parent 5f1e7af92c
commit 8a17369138
39 changed files with 1756 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files

213
backend/src/modules/dc/tool-b/controllers/ExtractionController.ts
View File

@@ -27,6 +27,70 @@ import { prisma } from '../../../../config/database.js';
import * as xlsx from 'xlsx';
export class ExtractionController {
/**
* 文件上传
* POST /upload
*/
async uploadFile(request: FastifyRequest, reply: FastifyReply) {
try {
const data = await request.file();
if (!data) {
return reply.code(400).send({
success: false,
error: 'No file uploaded'
});
}
const userId = (request as any).userId || 'default-user';
const buffer = await data.toBuffer();
const originalFilename = data.filename;
const timestamp = Date.now();
const fileKey = `dc/tool-b/${userId}/${timestamp}_${originalFilename}`;
logger.info('[API] File upload request', {
filename: originalFilename,
size: buffer.length,
userId
});
// 解析Excel文件获取列名和行数
const workbook = xlsx.read(buffer, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const worksheet = workbook.Sheets[sheetName];
const jsonData = xlsx.utils.sheet_to_json<Record<string, any>>(worksheet);
// 获取列名（从第一行数据的keys）
const columns = jsonData.length > 0 ? Object.keys(jsonData[0]) : [];
const totalRows = jsonData.length;
logger.info('[API] Excel parsed', { columns, totalRows });
// 上传到storage
const url = await storage.upload(fileKey, buffer);
logger.info('[API] File uploaded successfully', { fileKey, url });
return reply.code(200).send({
success: true,
data: {
fileKey,
url,
filename: originalFilename,
size: buffer.length,
totalRows,
columns
}
});
} catch (error) {
logger.error('[API] File upload failed', { error });
return reply.code(500).send({
success: false,
error: String(error)
});
}
}
/**
* 健康检查
* POST /health-check
@@ -43,18 +107,36 @@ export class ExtractionController {
logger.info('[API] Health check request', { fileKey, columnName, userId });
// 参数验证
if (!fileKey || !columnName) {
logger.error('[API] Missing required parameters', { fileKey, columnName });
return reply.code(400).send({
success: false,
error: 'Missing required parameters: fileKey or columnName'
});
}
const result = await healthCheckService.check(fileKey, columnName, userId);
logger.info('[API] Health check success', { status: result.status });
return reply.code(200).send({
success: true,
data: result
});
} catch (error) {
logger.error('[API] Health check failed', { error });
} catch (error: any) {
logger.error('[API] Health check failed', {
error: error.message,
stack: error.stack,
fileKey: request.body?.fileKey,
columnName: request.body?.columnName
});
return reply.code(500).send({
success: false,
error: String(error)
error: error.message || String(error),
details: process.env.NODE_ENV === 'development' ? error.stack : undefined
});
}
}
@@ -99,6 +181,8 @@ export class ExtractionController {
}
}>, reply: FastifyReply) {
try {
logger.info('[API] ===== CREATE TASK START =====');
const {
projectName,
sourceFileKey,
@@ -113,34 +197,48 @@ export class ExtractionController {
logger.info('[API] Create task request', {
userId,
projectName,
sourceFileKey,
textColumn,
diseaseType,
reportType
});
// 1. 获取模板
logger.info('[API] Step 1: Getting template', { diseaseType, reportType });
const template = await templateService.getTemplate(diseaseType, reportType);
if (!template) {
logger.error('[API] Template not found', { diseaseType, reportType });
return reply.code(404).send({
success: false,
error: `Template not found: ${diseaseType}/${reportType}`
});
}
logger.info('[API] Template found', { templateId: template.id });
// 2. 读取Excel文件，创建items
logger.info('[API] Step 2: Downloading Excel file', { sourceFileKey });
const fileBuffer = await storage.download(sourceFileKey);
if (!fileBuffer) {
logger.error('[API] File not found in storage', { sourceFileKey });
return reply.code(404).send({
success: false,
error: `File not found: ${sourceFileKey}`
});
}
logger.info('[API] File downloaded', { size: fileBuffer.length });
logger.info('[API] Step 3: Parsing Excel file');
const workbook = xlsx.read(fileBuffer, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const worksheet = workbook.Sheets[sheetName];
const data = xlsx.utils.sheet_to_json<Record<string, any>>(worksheet);
logger.info('[API] Excel parsed', { rowCount: data.length });
if (!data[0].hasOwnProperty(textColumn)) {
logger.error('[API] Column not found', {
textColumn,
availableColumns: Object.keys(data[0])
});
return reply.code(400).send({
success: false,
error: `Column '${textColumn}' not found in Excel`
@@ -148,6 +246,7 @@ export class ExtractionController {
}
// 3. 创建任务
logger.info('[API] Step 4: Creating task in database');
const task = await prisma.dCExtractionTask.create({
data: {
userId,
@@ -156,15 +255,17 @@ export class ExtractionController {
textColumn,
diseaseType,
reportType,
targetFields: template.fields,
targetFields: template.fields as any, // Prisma Json类型
modelA,
modelB,
totalCount: data.length,
status: 'pending'
}
});
logger.info('[API] Task created in database', { taskId: task.id });
// 4. 创建items
logger.info('[API] Step 5: Creating extraction items', { count: data.length });
const itemsData = data.map((row, index) => ({
taskId: task.id,
rowIndex: index + 1,
@@ -174,13 +275,24 @@ export class ExtractionController {
await prisma.dCExtractionItem.createMany({
data: itemsData
});
logger.info('[API] Items created', { count: itemsData.length });
// 5. 启动异步任务
// TODO: 使用jobQueue.add()
// 暂时直接调用
dualModelExtractionService.batchExtract(task.id).catch(err => {
logger.error('[API] Batch extraction failed', { error: err, taskId: task.id });
});
logger.info('[API] Starting batch extraction (async)', { taskId: task.id });
dualModelExtractionService.batchExtract(task.id)
.then(() => {
logger.info('[API] Batch extraction completed successfully', { taskId: task.id });
})
.catch(err => {
logger.error('[API] Batch extraction failed', {
error: err.message,
stack: err.stack,
taskId: task.id
});
});
logger.info('[API] Task created', { taskId: task.id, itemCount: data.length });
@@ -380,6 +492,93 @@ export class ExtractionController {
});
}
}
/**
* 导出结果
* GET /tasks/:taskId/export
*/
async exportResults(request: FastifyRequest<{
Params: { taskId: string };
}>, reply: FastifyReply) {
try {
const { taskId } = request.params;
logger.info('[API] Export results request', { taskId });
// 获取任务和所有items
const task = await prisma.dCExtractionTask.findUnique({
where: { id: taskId },
include: {
items: {
orderBy: { rowIndex: 'asc' }
}
}
});
if (!task) {
return reply.code(404).send({
success: false,
error: 'Task not found'
});
}
// 创建Excel工作簿
const workbook = xlsx.utils.book_new();
// 🔑 获取字段顺序（从targetFields）
const targetFields = task.targetFields as { name: string; desc: string }[];
const fieldNames = targetFields.map(f => f.name);
// 构建数据行，按模板字段顺序
const rows = task.items.map(item => {
// 优先使用finalResult，如果为空则使用resultA
const finalResult = item.finalResult as Record<string, string> | null;
const resultA = item.resultA as Record<string, string> | null;
const extractedData = finalResult || resultA || {};
// 🔑 按字段顺序构建行对象
const row: Record<string, any> = {
'行号': item.rowIndex,
'原文': item.originalText,
'状态': item.status === 'resolved' ? '已解决' : item.status === 'clean' ? '一致' : '待裁决'
};
// 按模板定义的顺序添加字段
fieldNames.forEach(fieldName => {
row[fieldName] = extractedData[fieldName] || '未提及';
});
return row;
});
// 创建工作表
const worksheet = xlsx.utils.json_to_sheet(rows);
xlsx.utils.book_append_sheet(workbook, worksheet, '提取结果');
// 生成Excel Buffer
const excelBuffer = xlsx.write(workbook, { type: 'buffer', bookType: 'xlsx' });
logger.info('[API] Export results success', { taskId, rowCount: rows.length });
// 返回文件
// 🔑 对文件名进行URL编码以支持中文
const filename = `${task.projectName}_结果.xlsx`;
const encodedFilename = encodeURIComponent(filename);
return reply
.code(200)
.header('Content-Type', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
.header('Content-Disposition', `attachment; filename*=UTF-8''${encodedFilename}`)
.send(excelBuffer);
} catch (error) {
logger.error('[API] Export results failed', { error });
return reply.code(500).send({
success: false,
error: String(error)
});
}
}
}
// 导出单例

19
backend/src/modules/dc/tool-b/routes/index.ts
View File

@@ -11,6 +11,11 @@ import { logger } from '../../../../common/logging/index.js';
export async function registerToolBRoutes(fastify: FastifyInstance) {
logger.info('[Routes] Registering DC Tool-B routes');
// 文件上传
fastify.post('/upload', {
handler: extractionController.uploadFile.bind(extractionController)
});
// 健康检查
fastify.post('/health-check', {
schema: {
@@ -109,6 +114,20 @@ export async function registerToolBRoutes(fastify: FastifyInstance) {
handler: extractionController.resolveConflict.bind(extractionController)
});
// 导出结果
fastify.get('/tasks/:taskId/export', {
schema: {
params: {
type: 'object',
required: ['taskId'],
properties: {
taskId: { type: 'string' }
}
}
},
handler: extractionController.exportResults.bind(extractionController)
});
logger.info('[Routes] DC Tool-B routes registered successfully');
}

65
backend/src/modules/dc/tool-b/services/DualModelExtractionService.ts
View File

@@ -142,34 +142,56 @@ ${text}
fields: { name: string; desc: string }[]
): Promise<ExtractionOutput> {
try {
// 使用LLMFactory获取LLM客户端
const modelName = modelType === 'deepseek' ? 'deepseek-v3' : 'qwen-max';
const llm = LLMFactory.createLLM(modelName);
// 🔑 使用LLMFactory获取适配器（正确的方法）
const modelName = modelType === 'deepseek' ? 'deepseek-v3' : 'qwen3-72b';
logger.info(`[${modelType.toUpperCase()}] Calling model`, { modelName });
logger.info(`[${modelType.toUpperCase()}] Getting adapter`, { modelName });
const adapter = LLMFactory.getAdapter(modelName as any);
logger.info(`[${modelType.toUpperCase()}] Adapter created successfully`);
// 调用LLM
const response = await llm.generateText(prompt, {
logger.info(`[${modelType.toUpperCase()}] Calling model with prompt`, {
modelName,
promptLength: prompt.length,
promptPreview: prompt.substring(0, 100) + '...'
});
// 🔑 调用LLM（使用chat方法，符合ILLMAdapter接口）
const startTime = Date.now();
const response = await adapter.chat([
{ role: 'user', content: prompt }
], {
temperature: 0, // 最大确定性
maxTokens: 1000
});
const elapsedTime = Date.now() - startTime;
logger.info(`[${modelType.toUpperCase()}] Model responded`, {
logger.info(`[${modelType.toUpperCase()}] Model responded successfully`, {
modelName,
tokensUsed: response.tokensUsed
tokensUsed: response.usage?.totalTokens,
elapsedMs: elapsedTime,
contentLength: response.content.length,
contentPreview: response.content.substring(0, 200)
});
// 解析JSON（3层容错）
const result = this.parseJSON(response.text, fields);
logger.info(`[${modelType.toUpperCase()}] Parsing JSON response`);
const result = this.parseJSON(response.content, fields);
logger.info(`[${modelType.toUpperCase()}] JSON parsed successfully`, {
fieldCount: Object.keys(result).length
});
return {
result,
tokensUsed: response.tokensUsed || 0,
rawOutput: response.text
tokensUsed: response.usage?.totalTokens || 0,
rawOutput: response.content
};
} catch (error) {
logger.error(`[${modelType.toUpperCase()}] Model call failed`, { error, modelType });
} catch (error: any) {
logger.error(`[${modelType.toUpperCase()}] Model call failed`, {
error: error.message,
stack: error.stack,
modelType
});
throw error;
}
}
@@ -246,18 +268,27 @@ ${text}
*/
async batchExtract(taskId: string): Promise<void> {
try {
logger.info('[Batch] Starting batch extraction', { taskId });
logger.info('[Batch] ===== Starting batch extraction =====', { taskId });
// 1. 获取任务
logger.info('[Batch] Step 1: Fetching task from database', { taskId });
const task = await prisma.dCExtractionTask.findUnique({
where: { id: taskId },
include: { items: true }
});
if (!task) {
logger.error('[Batch] Task not found in database', { taskId });
throw new Error(`Task not found: ${taskId}`);
}
logger.info('[Batch] Task fetched successfully', {
taskId,
itemCount: task.items.length,
diseaseType: task.diseaseType,
reportType: task.reportType
});
// 2. 更新任务状态
await prisma.dCExtractionTask.update({
where: { id: taskId },
@@ -309,12 +340,12 @@ ${text}
await prisma.dCExtractionItem.update({
where: { id: item.id },
data: {
resultA: resultA.result,
resultB: resultB.result,
resultA: resultA.result as any,
resultB: resultB.result as any,
tokensA: resultA.tokensUsed,
tokensB: resultB.tokensUsed,
status: hasConflict ? 'conflict' : 'clean',
finalResult: hasConflict ? null : resultA.result // 一致时自动采纳
finalResult: (hasConflict ? null : resultA.result) as any // 一致时自动采纳
}
});

83
backend/src/modules/dc/tool-b/services/HealthCheckService.ts
View File

@@ -51,22 +51,73 @@ export class HealthCheckService {
}
// 2. 从Storage读取Excel文件
const fileBuffer = await storage.download(fileKey);
if (!fileBuffer) {
throw new Error(`File not found: ${fileKey}`);
logger.info('[HealthCheck] Downloading file from storage', { fileKey });
let fileBuffer: Buffer;
try {
fileBuffer = await storage.download(fileKey);
if (!fileBuffer) {
throw new Error(`File not found in storage: ${fileKey}`);
}
logger.info('[HealthCheck] File downloaded successfully', {
fileKey,
size: fileBuffer.length
});
} catch (storageError: any) {
logger.error('[HealthCheck] Storage download failed', {
fileKey,
error: storageError.message,
stack: storageError.stack
});
throw new Error(`Failed to download file from storage: ${storageError.message}`);
}
// 3. 解析Excel（仅前100行）
const workbook = xlsx.read(fileBuffer, { type: 'buffer' });
const sheetName = workbook.SheetNames[0];
const worksheet = workbook.Sheets[sheetName];
const data = xlsx.utils.sheet_to_json<Record<string, any>>(worksheet, { range: 99 }); // 前100行
// 3. 解析Excel（取前100行用于采样）
logger.info('[HealthCheck] Parsing Excel file');
let workbook: xlsx.WorkBook;
let data: Record<string, any>[];
logger.info('[HealthCheck] Excel parsed', { totalRows: data.length });
try {
workbook = xlsx.read(fileBuffer, { type: 'buffer' });
if (!workbook.SheetNames || workbook.SheetNames.length === 0) {
throw new Error('Excel文件中没有工作表');
}
const sheetName = workbook.SheetNames[0];
const worksheet = workbook.Sheets[sheetName];
// 读取所有数据
const allData = xlsx.utils.sheet_to_json<Record<string, any>>(worksheet);
// 取前100行作为采样（如果不足100行则取全部）
data = allData.slice(0, 100);
logger.info('[HealthCheck] Excel parsed successfully', {
sheetName,
totalRows: allData.length,
sampleRows: data.length
});
} catch (xlsxError: any) {
logger.error('[HealthCheck] Excel parsing failed', {
error: xlsxError.message,
stack: xlsxError.stack
});
throw new Error(`Excel解析失败: ${xlsxError.message}`);
}
// 4. 检查列是否存在
if (data.length === 0 || !data[0].hasOwnProperty(columnName)) {
throw new Error(`Column '${columnName}' not found in Excel`);
if (data.length === 0) {
throw new Error('Excel文件无有效数据');
}
const availableColumns = Object.keys(data[0]);
logger.info('[HealthCheck] Available columns', { availableColumns });
if (!data[0].hasOwnProperty(columnName)) {
throw new Error(
`列 "${columnName}" 不存在。可用列：${availableColumns.join(', ')}`
);
}
// 5. 计算统计指标
@@ -97,8 +148,14 @@ export class HealthCheckService {
return result;
} catch (error) {
logger.error('[HealthCheck] Check failed', { error, fileKey, columnName });
} catch (error: any) {
logger.error('[HealthCheck] Check failed', {
error: error.message,
stack: error.stack,
fileKey,
columnName,
userId
});
throw error;
}
}