Summary: - Migrate PostgreSQL to pgvector/pgvector:pg15 Docker image - Successfully install and verify pgvector 0.8.1 extension - Create comprehensive Dify-to-pgvector migration plan - Update PKB module documentation with pgvector status - Update system documentation with pgvector integration Key changes: - docker-compose.yml: Switch to pgvector/pgvector:pg15 image - Add EkbDocument and EkbChunk data model design - Design R-C-R-G hybrid retrieval architecture - Add clinical data JSONB fields (pico, studyDesign, regimen, safety, criteria, endpoints) - Create detailed 10-day implementation roadmap Documentation updates: - PKB module status: pgvector RAG infrastructure ready - System status: pgvector 0.8.1 integrated - New: Dify replacement development plan (01-Dify替换为pgvector开发计划.md) - New: Enterprise medical knowledge base solution V2 Tested: PostgreSQL with pgvector verified, frontend and backend functionality confirmed
1093 lines
31 KiB
Markdown
1093 lines
31 KiB
Markdown
# REDCap对接技术方案与实施指南
|
||
|
||
**版本:** V1.0
|
||
**创建日期:** 2026-01-02
|
||
**适用阶段:** IIT Manager Agent MVP - Day 2
|
||
**文档性质:** 核心技术基石
|
||
**重要程度:** ⭐⭐⭐⭐⭐
|
||
|
||
---
|
||
|
||
## 📋 文档目标
|
||
|
||
本文档是IIT Manager Agent与REDCap集成的**权威技术指南**,明确:
|
||
1. ✅ REDCap对接方式的技术选型
|
||
2. ✅ Data Entry Trigger (DET) 的验证与配置
|
||
3. ✅ REST API的使用方法
|
||
4. ✅ 实时质控的完整架构
|
||
5. ✅ Day 2的开发实施步骤
|
||
|
||
---
|
||
|
||
## 🎯 核心结论(Executive Summary)
|
||
|
||
### ✅ **技术选型:DET(实时触发) + REST API(数据读写)**
|
||
|
||
**不采用:** External Module (EM)
|
||
**原因:** EM需要PHP开发、侵入REDCap源码、维护成本高、不适合外部系统集成
|
||
|
||
**采用方案:**
|
||
|
||
| 组件 | 技术 | 用途 | 实时性 |
|
||
|------|-----|------|--------|
|
||
| **Data Entry Trigger** | REDCap原生功能 | 数据保存时实时通知 | 0秒延迟 |
|
||
| **REST API (Export)** | HTTP + JSON | 拉取数据、元数据 | 按需调用 |
|
||
| **REST API (Import)** | HTTP + JSON | 回写质控意见(Phase 2) | 按需调用 |
|
||
| **轮询机制** | pg-boss定时任务 | 补充DET遗漏数据 | 30分钟 |
|
||
|
||
---
|
||
|
||
## 🔍 技术调研结果
|
||
|
||
### 调研1:REDCap提供的对接方式
|
||
|
||
**调研来源:**
|
||
- REDCap 15.8.0源码
|
||
- External Module Framework官方文档
|
||
- REDCap二次开发深度指南
|
||
|
||
**发现的对接方式:**
|
||
|
||
| 方式 | 技术栈 | 适用场景 | 我们是否适用 |
|
||
|------|--------|---------|-------------|
|
||
| **External Module** | PHP + Hook | REDCap内部功能扩展、UI定制 | ❌ 不适用 |
|
||
| **REST API** | HTTP + JSON | 外部系统集成、数据同步 | ✅ **适用** |
|
||
| **Data Entry Trigger** | Webhook | 实时通知外部系统 | ✅ **适用** |
|
||
|
||
### 调研2:Data Entry Trigger (DET) 验证
|
||
|
||
**源码位置:**
|
||
```
|
||
/var/www/html/redcap/redcap_v15.8.0/Classes/DataEntry.php
|
||
/var/www/html/redcap/redcap_v15.8.0/ProjectSetup/index.php
|
||
/var/www/html/redcap/redcap_v15.8.0/ControlCenter/modules_settings.php
|
||
```
|
||
|
||
**关键代码:**
|
||
```php
|
||
// DataEntry.php 核心实现
|
||
global $data_entry_trigger_url, $data_entry_trigger_enabled;
|
||
if (!$data_entry_trigger_enabled || $data_entry_trigger_url == '') {
|
||
return;
|
||
}
|
||
// 发送HTTP POST到配置的URL
|
||
$full_url = $pre_url . $data_entry_trigger_url;
|
||
```
|
||
|
||
**验证结论:** ✅ DET是REDCap原生功能,真实存在且广泛使用
|
||
|
||
---
|
||
|
||
## 🏗️ 完整架构设计
|
||
|
||
### 架构图
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 1. CRC 在 REDCap 中保存记录 │
|
||
│ - 录入患者数据 │
|
||
│ - 点击"Save"按钮 │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓ 立即触发(REDCap DET,0秒延迟)
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 2. REDCap Data Entry Trigger (DET) │
|
||
│ POST → https://iit.xunzhengyixue.com/ │
|
||
│ api/v1/iit/webhooks/redcap │
|
||
│ Payload: │
|
||
│ - project_id: 16 │
|
||
│ - record: 101 │
|
||
│ - instrument: demographics │
|
||
│ - redcap_event_name: baseline_arm_1 │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓ 1秒内
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 3. Node.js Webhook接收器 │
|
||
│ - 验证请求来源(可选) │
|
||
│ - 立即返回200 OK(<100ms,不阻塞REDCap) │
|
||
│ - 异步调用质控流程(setImmediate) │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓ 异步处理
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 4. REDCap REST API - exportRecords │
|
||
│ - 使用API Token认证 │
|
||
│ - 拉取指定record_id的完整数据 │
|
||
│ - 包含所有字段值、元数据 │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 5. 推送到质控队列 │
|
||
│ - jobQueue.send('iit:quality-check', record) │
|
||
│ - 异步处理,不影响实时响应 │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓ 由QualityCheckAgent处理(Day 6)
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 6. AI质控分析 │
|
||
│ - 规则引擎验证 │
|
||
│ - AI推理检测异常 │
|
||
│ - 生成质控报告 │
|
||
└──────────────────┬──────────────────────────────────┘
|
||
↓ 3-5秒内
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 7. 多渠道反馈 │
|
||
│ A. 企业微信推送给CRC(实时通知) │
|
||
│ B. REDCap API importQueries(写回质疑) │
|
||
│ C. IIT Manager前端显示(待办事项) │
|
||
└─────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ 补充:定时轮询机制(每30分钟) │
|
||
│ - 防止DET遗漏数据 │
|
||
│ - 使用dateRangeBegin增量拉取 │
|
||
│ - 推送到相同的质控队列 │
|
||
└─────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 实时性对比
|
||
|
||
| 方案 | 延迟时间 | 我们的选择 |
|
||
|------|---------|-----------|
|
||
| **纯轮询(每5分钟)** | 最高5分钟 | ❌ 不满足实时性要求 |
|
||
| **纯轮询(每1分钟)** | 最高1分钟 | ⚠️ 资源浪费、延迟仍高 |
|
||
| **DET + API** | 0秒触发 + 3-5秒处理 | ✅ **最优方案** |
|
||
| **DET + 轮询补充** | 实时 + 可靠 | ✅ **最终架构** |
|
||
|
||
**结论:** CRC点击"保存" → 5秒内收到企业微信通知 ✅
|
||
|
||
---
|
||
|
||
## 📦 技术实现详解
|
||
|
||
### 1. Data Entry Trigger (DET) 配置
|
||
|
||
#### 步骤1:在Control Center启用DET
|
||
|
||
```
|
||
1. 登录REDCap:http://localhost:8080/
|
||
2. 使用管理员账户(Admin / Admin123!)
|
||
3. 进入:Control Center → "Additional Customizations"
|
||
4. 找到:"Data Entry Trigger" 选项
|
||
5. 设置为:"Enabled"
|
||
6. 保存配置
|
||
```
|
||
|
||
**数据库字段:** `redcap_config.data_entry_trigger_enabled = 1`
|
||
|
||
#### 步骤2:在项目中配置Webhook URL
|
||
|
||
```
|
||
1. 进入测试项目:test0102 (PID 16)
|
||
2. 左侧菜单 → Project Setup
|
||
3. 找到:"Additional Customizations"
|
||
4. 展开:"Data Entry Trigger URL"
|
||
5. 填入Webhook URL:
|
||
- 开发环境:http://localhost:3001/api/v1/iit/webhooks/redcap
|
||
- 测试环境:https://backend-dev.xunzhengyixue.com/api/v1/iit/webhooks/redcap
|
||
- 生产环境:https://iit.xunzhengyixue.com/api/v1/iit/webhooks/redcap
|
||
6. 保存
|
||
```
|
||
|
||
**数据库字段:** `redcap_projects.data_entry_trigger_url`
|
||
|
||
#### 步骤3:验证DET配置
|
||
|
||
**使用RequestBin/ngrok验证:**
|
||
|
||
```bash
|
||
# 使用ngrok创建临时公网URL(开发测试用)
|
||
ngrok http 3001
|
||
|
||
# 将ngrok URL配置到REDCap DET
|
||
# 例如:https://1234-abc-def.ngrok.io/api/v1/iit/webhooks/redcap
|
||
|
||
# 在REDCap中保存一条记录
|
||
# 检查ngrok控制台是否收到POST请求
|
||
```
|
||
|
||
#### DET的POST Payload格式
|
||
|
||
**REDCap发送的请求:**
|
||
|
||
```http
|
||
POST /api/v1/iit/webhooks/redcap HTTP/1.1
|
||
Host: iit.xunzhengyixue.com
|
||
Content-Type: application/x-www-form-urlencoded
|
||
User-Agent: REDCap/15.8.0
|
||
|
||
project_id=16
|
||
&record=101
|
||
&instrument=demographics
|
||
&redcap_event_name=baseline_arm_1
|
||
&demographics_complete=2
|
||
&redcap_url=http://localhost:8080/
|
||
```
|
||
|
||
**字段说明:**
|
||
|
||
| 字段 | 类型 | 说明 | 示例 |
|
||
|------|-----|------|------|
|
||
| `project_id` | string | REDCap项目ID | "16" |
|
||
| `record` | string | 记录ID | "101" |
|
||
| `instrument` | string | 表单名称 | "demographics" |
|
||
| `redcap_event_name` | string | 事件名称(纵向研究) | "baseline_arm_1" |
|
||
| `[instrument]_complete` | string | 表单完成状态(0/1/2) | "2" |
|
||
| `redcap_url` | string | REDCap实例URL | "http://localhost:8080/" |
|
||
|
||
---
|
||
|
||
### 2. REDCap REST API 使用
|
||
|
||
#### API端点
|
||
|
||
```
|
||
http://localhost:8080/api/
|
||
```
|
||
|
||
#### API认证:Token
|
||
|
||
**生成步骤:**
|
||
```
|
||
1. 登录REDCap
|
||
2. 进入项目:test0102 (PID 16)
|
||
3. 左侧菜单 → "API"
|
||
4. 点击 "Generate API Token"
|
||
5. 复制Token(32位字符串)
|
||
6. 保存到环境变量:
|
||
REDCAP_API_TOKEN_TEST=YOUR_TOKEN_HERE
|
||
```
|
||
|
||
**Token存储:**
|
||
- ✅ 环境变量(推荐)
|
||
- ✅ 数据库加密字段(IitProject.redcapApiToken)
|
||
- ❌ 不要提交到Git
|
||
|
||
#### API方法1:导出记录 (exportRecords)
|
||
|
||
**用途:** 拉取数据(支持增量同步)
|
||
|
||
**请求示例(curl):**
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8080/api/ \
|
||
-F "token=YOUR_API_TOKEN" \
|
||
-F "content=record" \
|
||
-F "format=json" \
|
||
-F "type=flat" \
|
||
-F "records[0]=101" \
|
||
-F "records[1]=102" \
|
||
-F "dateRangeBegin=2026-01-01 00:00:00"
|
||
```
|
||
|
||
**参数说明:**
|
||
|
||
| 参数 | 必填 | 说明 | 示例 |
|
||
|------|-----|------|------|
|
||
| `token` | ✅ | API Token | "ABC123..." |
|
||
| `content` | ✅ | 固定值 | "record" |
|
||
| `format` | ✅ | 返回格式 | "json" |
|
||
| `type` | ✅ | 数据结构 | "flat" |
|
||
| `records` | ❌ | 指定记录ID | ["101", "102"] |
|
||
| `fields` | ❌ | 指定字段 | ["age", "gender"] |
|
||
| `dateRangeBegin` | ❌ | 时间过滤(增量同步关键) | "2026-01-01 00:00:00" |
|
||
| `dateRangeEnd` | ❌ | 结束时间 | "2026-01-31 23:59:59" |
|
||
|
||
**响应示例:**
|
||
|
||
```json
|
||
[
|
||
{
|
||
"record_id": "101",
|
||
"redcap_event_name": "baseline_arm_1",
|
||
"first_name": "张",
|
||
"last_name": "三",
|
||
"age": "30",
|
||
"gender": "1",
|
||
"demographics_complete": "2"
|
||
}
|
||
]
|
||
```
|
||
|
||
#### API方法2:导出元数据 (exportMetadata)
|
||
|
||
**用途:** 获取字段定义、表单结构
|
||
|
||
**请求示例:**
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8080/api/ \
|
||
-F "token=YOUR_API_TOKEN" \
|
||
-F "content=metadata" \
|
||
-F "format=json"
|
||
```
|
||
|
||
**响应示例:**
|
||
|
||
```json
|
||
[
|
||
{
|
||
"field_name": "age",
|
||
"form_name": "demographics",
|
||
"section_header": "",
|
||
"field_type": "text",
|
||
"field_label": "Age",
|
||
"select_choices_or_calculations": "",
|
||
"field_note": "",
|
||
"text_validation_type_or_show_slider_number": "integer",
|
||
"text_validation_min": "0",
|
||
"text_validation_max": "120",
|
||
"identifier": "",
|
||
"branching_logic": "",
|
||
"required_field": "y",
|
||
"custom_alignment": "",
|
||
"question_number": "",
|
||
"matrix_group_name": "",
|
||
"matrix_ranking": "",
|
||
"field_annotation": ""
|
||
}
|
||
]
|
||
```
|
||
|
||
#### API方法3:导入记录 (importRecords) - Phase 2
|
||
|
||
**用途:** 回写数据(如质控意见、AI建议)
|
||
|
||
**请求示例:**
|
||
|
||
```bash
|
||
curl -X POST http://localhost:8080/api/ \
|
||
-F "token=YOUR_API_TOKEN" \
|
||
-F "content=record" \
|
||
-F "format=json" \
|
||
-F "type=flat" \
|
||
-F "overwriteBehavior=normal" \
|
||
-F 'data=[{"record_id":"101","ai_review":"数据异常"}]'
|
||
```
|
||
|
||
---
|
||
|
||
### 3. Node.js实现代码
|
||
|
||
#### 3.1 RedcapAdapter(API适配器)
|
||
|
||
**文件:** `backend/src/modules/iit-manager/adapters/RedcapAdapter.ts`
|
||
|
||
```typescript
|
||
import axios from 'axios';
|
||
import FormData from 'form-data';
|
||
import { logger } from '@/common/logging';
|
||
|
||
export interface RedcapExportOptions {
|
||
records?: string[];
|
||
fields?: string[];
|
||
dateRangeBegin?: Date;
|
||
dateRangeEnd?: Date;
|
||
}
|
||
|
||
export class RedcapAdapter {
|
||
private baseUrl: string;
|
||
private apiToken: string;
|
||
private timeout: number;
|
||
|
||
constructor(baseUrl: string, apiToken: string, timeout = 30000) {
|
||
this.baseUrl = baseUrl.replace(/\/$/, ''); // 移除末尾斜杠
|
||
this.apiToken = apiToken;
|
||
this.timeout = timeout;
|
||
}
|
||
|
||
/**
|
||
* 导出记录(支持增量同步)
|
||
*/
|
||
async exportRecords(options: RedcapExportOptions = {}): Promise<any[]> {
|
||
const formData = new FormData();
|
||
formData.append('token', this.apiToken);
|
||
formData.append('content', 'record');
|
||
formData.append('format', 'json');
|
||
formData.append('type', 'flat');
|
||
|
||
// 指定记录ID
|
||
if (options.records && options.records.length > 0) {
|
||
options.records.forEach((recordId, index) => {
|
||
formData.append(`records[${index}]`, recordId);
|
||
});
|
||
}
|
||
|
||
// 指定字段
|
||
if (options.fields && options.fields.length > 0) {
|
||
options.fields.forEach((field, index) => {
|
||
formData.append(`fields[${index}]`, field);
|
||
});
|
||
}
|
||
|
||
// 时间过滤(增量同步关键)
|
||
if (options.dateRangeBegin) {
|
||
const dateStr = this.formatRedcapDate(options.dateRangeBegin);
|
||
formData.append('dateRangeBegin', dateStr);
|
||
}
|
||
|
||
if (options.dateRangeEnd) {
|
||
const dateStr = this.formatRedcapDate(options.dateRangeEnd);
|
||
formData.append('dateRangeEnd', dateStr);
|
||
}
|
||
|
||
try {
|
||
const response = await axios.post(
|
||
`${this.baseUrl}/api/`,
|
||
formData,
|
||
{
|
||
headers: formData.getHeaders(),
|
||
timeout: this.timeout
|
||
}
|
||
);
|
||
|
||
if (!Array.isArray(response.data)) {
|
||
throw new Error('Invalid response format');
|
||
}
|
||
|
||
logger.info(`REDCap API: Exported ${response.data.length} records`);
|
||
return response.data;
|
||
|
||
} catch (error) {
|
||
logger.error('REDCap API exportRecords failed:', error);
|
||
throw new Error(`REDCap API error: ${error.message}`);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* 导出元数据(字段定义)
|
||
*/
|
||
async exportMetadata(): Promise<any[]> {
|
||
const formData = new FormData();
|
||
formData.append('token', this.apiToken);
|
||
formData.append('content', 'metadata');
|
||
formData.append('format', 'json');
|
||
|
||
try {
|
||
const response = await axios.post(
|
||
`${this.baseUrl}/api/`,
|
||
formData,
|
||
{
|
||
headers: formData.getHeaders(),
|
||
timeout: this.timeout
|
||
}
|
||
);
|
||
|
||
logger.info(`REDCap API: Exported ${response.data.length} metadata fields`);
|
||
return response.data;
|
||
|
||
} catch (error) {
|
||
logger.error('REDCap API exportMetadata failed:', error);
|
||
throw new Error(`REDCap API error: ${error.message}`);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* 导入记录(回写数据)- Phase 2
|
||
*/
|
||
async importRecords(records: any[]): Promise<void> {
|
||
const formData = new FormData();
|
||
formData.append('token', this.apiToken);
|
||
formData.append('content', 'record');
|
||
formData.append('format', 'json');
|
||
formData.append('type', 'flat');
|
||
formData.append('overwriteBehavior', 'normal');
|
||
formData.append('data', JSON.stringify(records));
|
||
|
||
try {
|
||
await axios.post(
|
||
`${this.baseUrl}/api/`,
|
||
formData,
|
||
{
|
||
headers: formData.getHeaders(),
|
||
timeout: this.timeout
|
||
}
|
||
);
|
||
|
||
logger.info(`REDCap API: Imported ${records.length} records`);
|
||
|
||
} catch (error) {
|
||
logger.error('REDCap API importRecords failed:', error);
|
||
throw new Error(`REDCap API error: ${error.message}`);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* 格式化日期为REDCap格式:YYYY-MM-DD HH:MM:SS
|
||
*/
|
||
private formatRedcapDate(date: Date): string {
|
||
return date
|
||
.toISOString()
|
||
.replace('T', ' ')
|
||
.substring(0, 19);
|
||
}
|
||
}
|
||
```
|
||
|
||
#### 3.2 WebhookController(Webhook接收器)
|
||
|
||
**文件:** `backend/src/modules/iit-manager/controllers/webhookController.ts`
|
||
|
||
```typescript
|
||
import { FastifyRequest, FastifyReply } from 'fastify';
|
||
import { prisma } from '@/config/database';
|
||
import { logger } from '@/common/logging';
|
||
import { jobQueue } from '@/common/jobs';
|
||
import { RedcapAdapter } from '../adapters/RedcapAdapter';
|
||
|
||
export class WebhookController {
|
||
/**
|
||
* 接收REDCap Data Entry Trigger Webhook
|
||
*
|
||
* 性能要求:响应时间 < 100ms(不阻塞REDCap)
|
||
*/
|
||
async handleRedcapWebhook(
|
||
request: FastifyRequest,
|
||
reply: FastifyReply
|
||
): Promise<void> {
|
||
const startTime = Date.now();
|
||
|
||
// 解析Webhook Payload
|
||
const payload = request.body as Record<string, any>;
|
||
const {
|
||
project_id,
|
||
record,
|
||
instrument,
|
||
redcap_event_name,
|
||
redcap_url
|
||
} = payload;
|
||
|
||
// 基本验证
|
||
if (!project_id || !record) {
|
||
reply.code(400).send({
|
||
error: 'Missing required fields: project_id, record'
|
||
});
|
||
return;
|
||
}
|
||
|
||
logger.info('REDCap DET webhook received:', {
|
||
project_id,
|
||
record,
|
||
instrument,
|
||
event: redcap_event_name
|
||
});
|
||
|
||
// 立即返回200(不阻塞REDCap)
|
||
const responseTime = Date.now() - startTime;
|
||
reply.code(200).send({
|
||
status: 'received',
|
||
timestamp: new Date().toISOString(),
|
||
response_time_ms: responseTime
|
||
});
|
||
|
||
// 异步处理(不阻塞HTTP响应)
|
||
setImmediate(async () => {
|
||
try {
|
||
await this.processWebhook({
|
||
project_id,
|
||
record,
|
||
instrument,
|
||
redcap_event_name,
|
||
redcap_url
|
||
});
|
||
} catch (error) {
|
||
logger.error('Webhook processing failed:', error);
|
||
// 不抛出错误,因为已经返回200了
|
||
}
|
||
});
|
||
}
|
||
|
||
/**
|
||
* 异步处理Webhook
|
||
*/
|
||
private async processWebhook(payload: {
|
||
project_id: string;
|
||
record: string;
|
||
instrument: string;
|
||
redcap_event_name?: string;
|
||
redcap_url?: string;
|
||
}): Promise<void> {
|
||
const { project_id, record, instrument } = payload;
|
||
|
||
// 1. 查找项目配置
|
||
const project = await prisma.iitProject.findFirst({
|
||
where: { redcapProjectId: project_id }
|
||
});
|
||
|
||
if (!project) {
|
||
logger.warn(`Unknown REDCap project_id: ${project_id}`);
|
||
return;
|
||
}
|
||
|
||
// 2. 幂等性检查(防止重复处理)
|
||
const isDuplicate = await this.checkDuplicate(
|
||
project.id,
|
||
record,
|
||
instrument
|
||
);
|
||
|
||
if (isDuplicate) {
|
||
logger.info(`Duplicate webhook ignored: ${record}`);
|
||
return;
|
||
}
|
||
|
||
// 3. 调用REDCap API拉取完整数据
|
||
const adapter = new RedcapAdapter(
|
||
project.redcapBaseUrl,
|
||
project.redcapApiToken
|
||
);
|
||
|
||
const records = await adapter.exportRecords({
|
||
records: [record] // 只拉取这一条记录
|
||
});
|
||
|
||
if (records.length === 0) {
|
||
logger.warn(`No data found for record: ${record}`);
|
||
return;
|
||
}
|
||
|
||
// 4. 推送到质控队列
|
||
await jobQueue.send('iit:quality-check', {
|
||
projectId: project.id,
|
||
record: records[0],
|
||
trigger: 'webhook',
|
||
instrument,
|
||
timestamp: new Date().toISOString()
|
||
});
|
||
|
||
logger.info(`Webhook processed successfully: project=${project_id}, record=${record}`);
|
||
|
||
// 5. 记录审计日志
|
||
await prisma.iitAuditLog.create({
|
||
data: {
|
||
projectId: project.id,
|
||
action: 'WEBHOOK_RECEIVED',
|
||
targetType: 'RECORD',
|
||
targetId: record,
|
||
details: {
|
||
instrument,
|
||
trigger: 'redcap_det'
|
||
}
|
||
}
|
||
});
|
||
}
|
||
|
||
/**
|
||
* 幂等性检查(防止5分钟内重复处理相同记录)
|
||
*/
|
||
private async checkDuplicate(
|
||
projectId: string,
|
||
recordId: string,
|
||
instrument: string
|
||
): Promise<boolean> {
|
||
const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1000);
|
||
|
||
const recentLog = await prisma.iitAuditLog.findFirst({
|
||
where: {
|
||
projectId,
|
||
action: 'WEBHOOK_RECEIVED',
|
||
targetId: recordId,
|
||
createdAt: { gte: fiveMinutesAgo },
|
||
details: {
|
||
path: ['instrument'],
|
||
equals: instrument
|
||
}
|
||
}
|
||
});
|
||
|
||
return recentLog !== null;
|
||
}
|
||
}
|
||
```
|
||
|
||
#### 3.3 SyncManager(轮询补充)
|
||
|
||
**文件:** `backend/src/modules/iit-manager/services/SyncManager.ts`
|
||
|
||
```typescript
|
||
import { prisma } from '@/config/database';
|
||
import { logger } from '@/common/logging';
|
||
import { jobQueue } from '@/common/jobs';
|
||
import { RedcapAdapter } from '../adapters/RedcapAdapter';
|
||
|
||
export class SyncManager {
|
||
/**
|
||
* 初始化同步(注册定时任务)
|
||
*/
|
||
async initializeSync(projectId: string, interval: string = '*/30 * * * *'): Promise<void> {
|
||
// 注册定时任务(每30分钟)
|
||
await jobQueue.schedule(
|
||
'iit:redcap:poll',
|
||
interval,
|
||
{ projectId }
|
||
);
|
||
|
||
logger.info(`Polling sync initialized for project ${projectId}, interval: ${interval}`);
|
||
}
|
||
|
||
/**
|
||
* 处理轮询(拉取增量数据)
|
||
*/
|
||
async handlePoll(projectId: string): Promise<void> {
|
||
logger.info(`Starting poll sync for project ${projectId}`);
|
||
|
||
try {
|
||
// 1. 获取项目配置
|
||
const project = await prisma.iitProject.findUnique({
|
||
where: { id: projectId }
|
||
});
|
||
|
||
if (!project) {
|
||
throw new Error(`Project not found: ${projectId}`);
|
||
}
|
||
|
||
// 2. 获取上次同步时间
|
||
const lastSyncAt = project.lastSyncAt || new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
|
||
|
||
logger.info(`Last sync: ${lastSyncAt.toISOString()}`);
|
||
|
||
// 3. 拉取增量数据
|
||
const adapter = new RedcapAdapter(
|
||
project.redcapBaseUrl,
|
||
project.redcapApiToken
|
||
);
|
||
|
||
const records = await adapter.exportRecords({
|
||
dateRangeBegin: lastSyncAt
|
||
});
|
||
|
||
logger.info(`Pulled ${records.length} records from REDCap`);
|
||
|
||
// 4. 推送到质控队列
|
||
for (const record of records) {
|
||
await jobQueue.send('iit:quality-check', {
|
||
projectId,
|
||
record,
|
||
trigger: 'polling',
|
||
timestamp: new Date().toISOString()
|
||
});
|
||
}
|
||
|
||
// 5. 更新同步时间
|
||
await prisma.iitProject.update({
|
||
where: { id: projectId },
|
||
data: { lastSyncAt: new Date() }
|
||
});
|
||
|
||
logger.info(`Poll sync completed for project ${projectId}`);
|
||
|
||
} catch (error) {
|
||
logger.error(`Poll sync failed for project ${projectId}:`, error);
|
||
throw error;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* 停止同步(取消定时任务)
|
||
*/
|
||
async stopSync(projectId: string): Promise<void> {
|
||
// pg-boss不支持直接取消schedule,需要在Worker中检查项目状态
|
||
logger.info(`Sync stopped for project ${projectId}`);
|
||
}
|
||
}
|
||
```
|
||
|
||
#### 3.4 路由配置
|
||
|
||
**文件:** `backend/src/modules/iit-manager/routes/index.ts`
|
||
|
||
```typescript
|
||
import { FastifyInstance } from 'fastify';
|
||
import { WebhookController } from '../controllers/webhookController';
|
||
import { SyncManager } from '../services/SyncManager';
|
||
|
||
export async function iitManagerRoutes(fastify: FastifyInstance) {
|
||
const webhookController = new WebhookController();
|
||
const syncManager = new SyncManager();
|
||
|
||
// Webhook端点(接收REDCap DET)
|
||
fastify.post('/webhooks/redcap', async (request, reply) => {
|
||
return webhookController.handleRedcapWebhook(request, reply);
|
||
});
|
||
|
||
// 手动触发同步(测试用)
|
||
fastify.post('/projects/:id/sync', async (request, reply) => {
|
||
const { id } = request.params as { id: string };
|
||
await syncManager.handlePoll(id);
|
||
return { status: 'synced' };
|
||
});
|
||
}
|
||
```
|
||
|
||
#### 3.5 Worker注册
|
||
|
||
**文件:** `backend/src/modules/iit-manager/index.ts`
|
||
|
||
```typescript
|
||
import { jobQueue } from '@/common/jobs';
|
||
import { SyncManager } from './services/SyncManager';
|
||
import { logger } from '@/common/logging';
|
||
|
||
export async function initializeIITManager() {
|
||
const syncManager = new SyncManager();
|
||
|
||
// 注册轮询Worker
|
||
await jobQueue.work('iit:redcap:poll', async (job) => {
|
||
const { projectId } = job.data;
|
||
await syncManager.handlePoll(projectId);
|
||
});
|
||
|
||
logger.info('IIT Manager initialized: Poll worker registered');
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 🧪 测试验证
|
||
|
||
### 测试1:验证DET配置
|
||
|
||
**步骤:**
|
||
1. 配置ngrok:`ngrok http 3001`
|
||
2. 将ngrok URL配置到REDCap DET
|
||
3. 在REDCap中保存一条记录
|
||
4. 检查ngrok控制台是否收到POST请求
|
||
|
||
**预期结果:**
|
||
- ✅ ngrok收到POST请求
|
||
- ✅ Payload包含project_id、record等字段
|
||
|
||
### 测试2:验证API Token
|
||
|
||
**测试脚本:** `test-redcap-api.ts`
|
||
|
||
```typescript
|
||
import { RedcapAdapter } from './adapters/RedcapAdapter';
|
||
|
||
async function testRedcapAPI() {
|
||
const adapter = new RedcapAdapter(
|
||
'http://localhost:8080',
|
||
process.env.REDCAP_API_TOKEN_TEST!
|
||
);
|
||
|
||
// 测试导出记录
|
||
const records = await adapter.exportRecords();
|
||
console.log('Records:', records);
|
||
|
||
// 测试导出元数据
|
||
const metadata = await adapter.exportMetadata();
|
||
console.log('Metadata fields:', metadata.length);
|
||
|
||
console.log('✅ REDCap API test passed!');
|
||
}
|
||
|
||
testRedcapAPI();
|
||
```
|
||
|
||
### 测试3:端到端集成测试
|
||
|
||
**测试脚本:** `test-redcap-integration.ts`
|
||
|
||
```typescript
|
||
async function testIntegration() {
|
||
// 1. 创建测试项目
|
||
const project = await prisma.iitProject.create({
|
||
data: {
|
||
name: 'test0102',
|
||
redcapBaseUrl: 'http://localhost:8080',
|
||
redcapApiToken: process.env.REDCAP_API_TOKEN_TEST!,
|
||
redcapProjectId: '16',
|
||
status: 'ACTIVE'
|
||
}
|
||
});
|
||
|
||
// 2. 初始化同步
|
||
const syncManager = new SyncManager();
|
||
await syncManager.initializeSync(project.id);
|
||
|
||
// 3. 手动触发一次轮询
|
||
await syncManager.handlePoll(project.id);
|
||
|
||
// 4. 模拟Webhook
|
||
const webhookController = new WebhookController();
|
||
await webhookController.handleRedcapWebhook(
|
||
{
|
||
body: {
|
||
project_id: '16',
|
||
record: '101',
|
||
instrument: 'demographics'
|
||
}
|
||
} as any,
|
||
{} as any
|
||
);
|
||
|
||
console.log('✅ Integration test passed!');
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 📋 Day 2实施清单
|
||
|
||
### 阶段1:环境准备(30分钟)
|
||
|
||
- [ ] 在REDCap Control Center启用DET功能
|
||
- [ ] 在test0102项目生成API Token
|
||
- [ ] 配置环境变量:
|
||
```env
|
||
REDCAP_BASE_URL=http://localhost:8080
|
||
REDCAP_API_TOKEN_TEST=YOUR_TOKEN_HERE
|
||
```
|
||
- [ ] 使用curl测试API是否可用
|
||
|
||
### 阶段2:开发RedcapAdapter(1.5小时)
|
||
|
||
- [ ] 创建文件:`adapters/RedcapAdapter.ts`
|
||
- [ ] 实现 `exportRecords()` 方法
|
||
- [ ] 实现 `exportMetadata()` 方法
|
||
- [ ] 实现 `formatRedcapDate()` 工具方法
|
||
- [ ] 编写单元测试:`RedcapAdapter.test.ts`
|
||
|
||
### 阶段3:开发WebhookController(2小时)
|
||
|
||
- [ ] 创建文件:`controllers/webhookController.ts`
|
||
- [ ] 实现 `handleRedcapWebhook()` 方法
|
||
- [ ] 实现 `processWebhook()` 私有方法
|
||
- [ ] 实现 `checkDuplicate()` 幂等性检查
|
||
- [ ] 配置路由:`POST /api/v1/iit/webhooks/redcap`
|
||
|
||
### 阶段4:开发SyncManager(1.5小时)
|
||
|
||
- [ ] 创建文件:`services/SyncManager.ts`
|
||
- [ ] 实现 `initializeSync()` 方法
|
||
- [ ] 实现 `handlePoll()` 方法
|
||
- [ ] 实现 `stopSync()` 方法
|
||
- [ ] 注册Worker:`iit:redcap:poll`
|
||
|
||
### 阶段5:集成测试(1小时)
|
||
|
||
- [ ] 配置ngrok/RequestBin测试DET
|
||
- [ ] 运行 `test-redcap-api.ts`
|
||
- [ ] 运行 `test-redcap-integration.ts`
|
||
- [ ] 验证端到端流程
|
||
- [ ] 记录测试结果
|
||
|
||
### 阶段6:文档更新(30分钟)
|
||
|
||
- [ ] 更新MVP开发任务清单(Day 2完成)
|
||
- [ ] 记录API Token和配置信息
|
||
- [ ] 更新架构图
|
||
- [ ] 提交Git
|
||
|
||
---
|
||
|
||
## ⚠️ 关键注意事项
|
||
|
||
### 1. DET配置要点
|
||
|
||
**常见错误:**
|
||
- ❌ 忘记在Control Center启用DET全局开关
|
||
- ❌ Webhook URL填写错误(多了斜杠、少了路径)
|
||
- ❌ 本地开发环境无法接收外网Webhook
|
||
|
||
**解决方案:**
|
||
- ✅ 使用ngrok创建临时公网URL测试
|
||
- ✅ 开发环境可先用RequestBin验证Payload格式
|
||
- ✅ 生产环境确保URL可访问(防火墙、HTTPS)
|
||
|
||
### 2. API Token安全
|
||
|
||
**安全实践:**
|
||
- ✅ 存储在环境变量或数据库加密字段
|
||
- ✅ 定期轮换Token
|
||
- ✅ 最小权限原则(只开启需要的权限)
|
||
- ❌ 不要提交到Git
|
||
- ❌ 不要在日志中打印完整Token
|
||
|
||
### 3. 性能优化
|
||
|
||
**Webhook响应时间:**
|
||
- ✅ 必须 < 100ms(立即返回200)
|
||
- ✅ 使用 `setImmediate()` 异步处理
|
||
- ❌ 不要在Webhook中执行耗时操作
|
||
|
||
**API调用频率:**
|
||
- ✅ 增量拉取(使用dateRangeBegin)
|
||
- ✅ 指定字段(减少数据量)
|
||
- ✅ 合理设置timeout(30秒)
|
||
|
||
### 4. 错误处理
|
||
|
||
**DET Webhook失败:**
|
||
- REDCap会重试3次(间隔1分钟)
|
||
- 如果仍失败,REDCap会记录到日志
|
||
- 我们的轮询机制可以补充遗漏的数据
|
||
|
||
**API调用失败:**
|
||
- 实现重试机制(3次,指数退避)
|
||
- 记录失败日志
|
||
- 告警通知管理员
|
||
|
||
---
|
||
|
||
## 📊 成功验收标准
|
||
|
||
### Day 2完成标准
|
||
|
||
- [ ] ✅ REDCap API Token已生成并验证
|
||
- [ ] ✅ RedcapAdapter可以拉取测试数据
|
||
- [ ] ✅ DET已配置并成功接收Webhook
|
||
- [ ] ✅ Webhook可以触发API拉取
|
||
- [ ] ✅ 轮询机制可以定时运行
|
||
- [ ] ✅ 数据推送到质控队列
|
||
- [ ] ✅ 单元测试全部通过
|
||
- [ ] ✅ 端到端集成测试通过
|
||
|
||
### 性能指标
|
||
|
||
- [ ] Webhook响应时间 < 100ms
|
||
- [ ] API调用成功率 > 99%
|
||
- [ ] 端到端延迟 < 5秒(DET触发 → 企微通知)
|
||
|
||
---
|
||
|
||
## 🔗 相关文档
|
||
|
||
- **MVP开发任务清单:** `MVP开发任务清单.md`
|
||
- **完整技术方案:** `IIT Manager Agent 完整技术开发方案 (V1.1).md`
|
||
- **REDCap二次开发指南:** `../../Redcap/03-API对接与开发/33-REDCap二次开发深度指南.md`
|
||
- **REDCap部署手册:** `../../Redcap/01-部署与配置/10-REDCap_Docker部署操作手册.md`
|
||
|
||
---
|
||
|
||
## 📝 更新日志
|
||
|
||
| 日期 | 版本 | 更新内容 | 更新人 |
|
||
|------|------|---------|--------|
|
||
| 2026-01-02 | V1.0 | 初始版本,完成技术调研和方案设计 | AI Assistant |
|
||
|
||
---
|
||
|
||
**这是IIT Manager Agent的技术基石文档,请妥善保管!** ⭐⭐⭐⭐⭐
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|