HaHafeng
1b53ab9d52
Major Changes: - Add StreamingService with OpenAI Compatible format - Upgrade Chat component V2 with Ant Design X integration - Implement AIA module with 12 intelligent agents - Update API routes to unified /api/v1 prefix - Update system documentation Backend (~1300 lines): - common/streaming: OpenAI Compatible adapter - modules/aia: 12 agents, conversation service, streaming integration - Update route versions (RVW, PKB to v1) Frontend (~3500 lines): - modules/aia: AgentHub + ChatWorkspace (100% prototype restoration) - shared/Chat: AIStreamChat, ThinkingBlock, useAIStream Hook - Update API endpoints to v1 Documentation: - AIA module status guide - Universal capabilities catalog - System overview updates - All module documentation sync Tested: Stream response verified, authentication working Status: AIA V2.0 core completed (85%)
1027 lines
30 KiB
Markdown
1027 lines
30 KiB
Markdown
# ASL 譁<>鍵螟<E98DB5>炊謚譛ｯ騾牙梛
|
||
|
||
> **譁<>｡｣迚域悽<E59F9F>?* V1.0
|
||
> **蛻帛ｻｺ譌･譛滂ｼ?* 2025-11-15
|
||
> **騾ら畑讓｡蝮暦ｼ?* AI 譎ｺ閭ｽ譁<EFBDBD>鍵<EFBFBD><E98DB5>SL<53>?
|
||
> **逶ｮ譬<EFBDAE>ｼ?* 螳壻ｹ牙<EFBDB9>遲帙∝<C280>譁<EFBFBD>､咲ｭ帙∝<C280>譁<EFBFBD>署蜿也噪謚譛ｯ譬亥柱螳樒鴫霍ｯ蠕?
|
||
|
||
---
|
||
|
||
## <20>搭 譁<>｡｣讎りｿｰ
|
||
|
||
ASL 讓｡蝮玲ｶ牙所荳臥ｧ堺ｸ榊酔逧<E98594>枚迪ｮ螟<EFBDAE>炊蝨ｺ譎ｯ<E8AD8E>梧ｯ冗ｧ榊惻譎ｯ譛我ｸ榊酔逧<E98594>橿譛ｯ迚ｹ轤ｹ蜥悟ｮ樒鴫譁ｹ譯茨ｼ?
|
||
|
||
| 蝨ｺ譎ｯ | 霎灘<E99C8E>譬ｼ蠑<EFBDBC> | 譬ｸ蠢<EFBDB8>橿譛?| 荳ｻ隕∵倦謌<E580A6> |
|
||
|------|---------|---------|---------|
|
||
| **譬<>｢俶遭隕∝<E99A95>遲<EFBFBD>** | Excel 譁<>ｻｶ | Excel 隗｣譫<EFBDA3> + LLM 遲幃?| 謇ｹ驥丞､<E4B89E>炊謨育紫 |
|
||
| **蜈ｨ譁<EFBDA8>､咲ｭ<E592B2>** | PDF 蜈ｨ譁<EFBDA8> | PDF 謠仙叙 + LLM 遲幃?| PDF 隗｣譫仙㊥遑ｮ邇?|
|
||
| **蜈ｨ譁<EFBDA8>焚謐ｮ謠仙叙** | PDF 蜈ｨ譁<EFBDA8> | PDF 謠仙叙 + LLM 扈捺桷蛹匁署蜿?| 陦ｨ譬ｼ縲∝<E7B8B2>蠑丞㊥遑ｮ謠仙<E8ACA0>?|
|
||
|
||
---
|
||
|
||
## <20>識 謚譛ｯ譫ｶ譫<EFBDB6>ｻ隗<EFBDBB>
|
||
|
||
```
|
||
笏娯楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? ASL 譁<>鍵螟<E98DB5>炊豬∫ｨ<E288AB> 笏?
|
||
笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏?
|
||
笏懌楳 蝨ｺ譎ｯ 1: 譬<>｢俶遭隕∝<E99A95>遲<EFBFBD>
|
||
笏? 笏披楳 逕ｨ謌ｷ荳贋ｼ<E8B48B> Excel 竊?隗｣譫<EFBDA3> 竊?LLM 謇ｹ驥冗ｭ幃?竊?蟇ｼ蜃ｺ扈捺棡
|
||
笏?
|
||
笏懌楳 蝨ｺ譎ｯ 2: 蜈ｨ譁<EFBDA8>､咲ｭ<E592B2>
|
||
笏? 笏披楳 逕ｨ謌ｷ荳贋ｼ<E8B48B> PDF 竊?PDF 謠仙叙 竊?LLM 遲幃?竊?螟肴<E89E9F>ｸ
|
||
笏?
|
||
笏披楳 蝨ｺ譎ｯ 3: 蜈ｨ譁<EFBDA8>焚謐ｮ謠仙叙
|
||
笏披楳 PDF 竊?謠仙叙 + 扈捺桷蛹?竊?LLM 謠仙叙謨ｰ謐ｮ 竊?莠ｺ蟾･螟肴<E89E9F>ｸ
|
||
|
||
笏娯楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 謚譛ｯ譬亥<E8ADAC>螻よ楔譫<E6A594>ｼ亥<EFBDBC>莠ｫ<E88EA0><EFBDAB> 笏?
|
||
笏懌楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 蜑咲ｫｯ螻? React 19 + Ant Design 5 + xlsx/exceljs 笏?
|
||
笏懌楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 蜷守ｫｯ螻? Node.js (Fastify) + TypeScript 笏?
|
||
笏懌楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 譁<>｡｣螟<EFBDA3>炊螻? Python 蠕ｮ譛榊<E8AD9B>?(extraction_service) 笏?
|
||
笏? 笏懌楳 PyMuPDF: 蠢ｫ騾?PDF 謠仙叙 笏?
|
||
笏? 笏懌楳 Nougat: 闍ｱ譁<EFBDB1>ｧ大ｭｦ譁<EFBDA6>鍵鬮倩ｴｨ驥乗署蜿?箝? 笏?
|
||
笏? 笏披楳 Language Detector: 閾ｪ蜉ｨ隸ｭ險譽豬? 笏?
|
||
笏懌楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? LLM 螻? DeepSeek-V3 + Qwen3 / GPT-5 + Claude-4.5 笏?
|
||
笏懌楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 謨ｰ謐ｮ蠎? PostgreSQL 15 (asl_schema) 笏?
|
||
笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
```
|
||
|
||
---
|
||
|
||
## <20>東 蝨ｺ譎ｯ 1: 譬<>｢俶遭隕∝<E99A95>遲<EFBFBD>
|
||
|
||
### 1.1 謚譛ｯ迚ｹ轤?
|
||
|
||
- **霎灘<E99C8E>譬ｼ蠑<EFBDBC>**: Excel 譁<>ｻｶ (`.xlsx` / `.xls`)
|
||
- **謨ｰ謐ｮ隗<EFBDAE>ｨ｡**: 50-500 遽<>枚迪?謇ｹ谺｡
|
||
- **荳ｻ隕∝ｭ玲ｮｵ**: 譬<>｢倥∵遭隕√．OI縲∽ｽ懆<E68786>∝書陦ｨ蟷ｴ莉ｽ縲∵悄蛻?
|
||
- **螟<>炊驥咲せ**: 謇ｹ驥城ｫ俶譜螟<E8AD9C>炊<EFBFBD>梧裏髴 PDF 隗｣譫<EFBDA3>
|
||
|
||
### 1.2 謚譛ｯ騾牙梛
|
||
|
||
#### 蜑咲ｫｯ<EFBDAB>哘xcel 荳贋ｼ<E8B48B>荳手ｧ｣譫?
|
||
|
||
| 謚譛?| 蠎?| 逕ｨ騾?| 莨伜漢 |
|
||
|------|-----|------|------|
|
||
| **Excel 荳贋ｼ<E8B48B>** | `antd Upload` | 譁<>ｻｶ荳贋ｼ<E8B48B>扈<EFBFBD>ｻｶ | 諡匁郷荳贋ｼ<E8B48B>縲∬ｿ帛ｺｦ譚｡ |
|
||
| **Excel 隗｣譫<EFBDA3>** | `xlsx` / `exceljs` | 蜑咲ｫｯ隗｣譫<EFBDA3> Excel | 郤ｯ蜑咲ｫｯ螟<EFBDAF>炊<EFBFBD>悟ｿｫ騾滄｢<E6BB84>ｧ?|
|
||
| **讓｡譚ｿ鬪瑚ｯ<E7919A>** | 閾ｪ螳壻ｹ蛾ｻ霎<EFBDBB> | 譬｡鬪悟<E9ACAA>蜷榊柱謨ｰ謐ｮ譬ｼ蠑?| 謠仙燕蜿醍鴫譬ｼ蠑城漠隸ｯ |
|
||
|
||
**謗ｨ闕先婿譯茨ｼ啻xlsx` 蠎難ｼ<E99BA3>heetJS<4A>?*
|
||
- 笨?謾ｯ謖<EFBDAF> `.xlsx` 蜥?`.xls` 譬ｼ蠑<EFBDBC>
|
||
- 笨?郤?JavaScript<70>悟燕遶ｯ逶ｴ謗･隗｣譫?
|
||
- 笨?菴鍋ｧｯ蟆擾ｼ<E693BE>600KB<4B>会ｼ梧ｧ閭ｽ螂?
|
||
- 笨?謾ｯ謖∝､ｧ譁<EFBDA7>ｻｶ<EFBDBB><EFBDB6>1000+ 陦鯉ｼ<E9AF89>
|
||
|
||
**莉｣遐∫､ｺ萓具ｼ?*
|
||
```typescript
|
||
import * as XLSX from 'xlsx';
|
||
|
||
function parseExcel(file: File): Promise<Literature[]> {
|
||
return new Promise((resolve, reject) => {
|
||
const reader = new FileReader();
|
||
|
||
reader.onload = (e) => {
|
||
try {
|
||
const data = new Uint8Array(e.target.result as ArrayBuffer);
|
||
const workbook = XLSX.read(data, { type: 'array' });
|
||
|
||
// 隸ｻ蜿也ｬｬ荳荳ｪ蟾･菴懆｡ｨ
|
||
const sheetName = workbook.SheetNames[0];
|
||
const worksheet = workbook.Sheets[sheetName];
|
||
|
||
// 霓ｬ謐｢荳?JSON
|
||
const jsonData = XLSX.utils.sheet_to_json(worksheet);
|
||
|
||
// 譏<>蟆<EFBFBD>ｸｺ譬<EFBDBA>㊥譬ｼ蠑?
|
||
const literatures = jsonData.map((row: any) => ({
|
||
title: row['Title'] || row['譬<>｢<EFBFBD>'],
|
||
abstract: row['Abstract'] || row['鞫倩ｦ<E580A9>'],
|
||
doi: row['DOI'],
|
||
authors: row['Authors'] || row['菴懆?],
|
||
year: row['Year'] || row['蟷ｴ莉ｽ'],
|
||
journal: row['Journal'] || row['譛溷<E8AD9B>'],
|
||
}));
|
||
|
||
resolve(literatures);
|
||
} catch (error) {
|
||
reject(new Error('Excel 隗｣譫仙､ｱ雍･'));
|
||
}
|
||
};
|
||
|
||
reader.onerror = () => reject(new Error('譁<>ｻｶ隸ｻ蜿門､ｱ雍･'));
|
||
reader.readAsArrayBuffer(file);
|
||
});
|
||
}
|
||
```
|
||
|
||
#### 蜷守ｫｯ<EFBDAB>壽音驥冗ｭ幃牙､<E78999><EFBDA4>?
|
||
|
||
**螟<>炊豬∫ｨ具ｼ?*
|
||
```
|
||
Excel 謨ｰ謐ｮ 竊?謇ｹ驥丞<E9A9A5>扈<EFBFBD>ｼ?0-20 遽?扈<>ｼ俄<EFBDBC>?蟷ｶ陦瑚ｰ<E7919A>畑 LLM 竊?豎<>ｻ扈捺<E68988>?
|
||
```
|
||
|
||
**蜈ｳ髞ｮ謚譛ｯ轤ｹ<E8BDA4>?*
|
||
1. **謇ｹ驥丞<E9A9A5>扈<EFBFBD>**<2A>夐∩蜈榊黒谺｡隸ｷ豎りｿ<E3828A>､ｧ<EFBDA4><EFBDA7>10-20 遽?扈<>怙莨?
|
||
2. **蟷ｶ陦悟､<E6829F>炊**<2A>壻ｽｿ逕?`Promise.all` 蟷ｶ陦瑚ｰ<E7919A>畑 LLM
|
||
3. **霑帛ｺｦ謗ｨ騾?*<2A>啗ebSocket 螳樊慮謗ｨ騾∝､<E2889D>炊霑帛ｺ?
|
||
4. **譁ｭ轤ｹ扈ｭ莨<EFBDAD>**<2A>壽髪謖∽ｻｻ蜉｡荳ｭ譁ｭ蜷守ｻｧ扈ｭ
|
||
|
||
**莉｣遐∫､ｺ萓具ｼ?*
|
||
```typescript
|
||
async function batchScreening(
|
||
literatures: Literature[],
|
||
protocol: Protocol,
|
||
progressCallback: (progress: number) => void
|
||
) {
|
||
const batchSize = 15;
|
||
const batches = chunk(literatures, batchSize);
|
||
const results = [];
|
||
|
||
for (let i = 0; i < batches.length; i++) {
|
||
const batch = batches[i];
|
||
|
||
// 蟷ｶ陦悟､<E6829F>炊蠖灘燕謇ｹ谺｡
|
||
const batchResults = await Promise.all(
|
||
batch.map(lit => dualModelScreening(lit, protocol))
|
||
);
|
||
|
||
results.push(...batchResults);
|
||
|
||
// 謗ｨ騾∬ｿ帛ｺ?
|
||
const progress = Math.round(((i + 1) / batches.length) * 100);
|
||
progressCallback(progress);
|
||
}
|
||
|
||
return results;
|
||
}
|
||
```
|
||
|
||
### 1.3 謨ｰ謐ｮ豬?
|
||
|
||
```
|
||
逕ｨ謌ｷ謫堺ｽ<EFBFBD> 蜑咲ｫｯ螟<EFBDAF>炊 蜷守ｫｯ螟<EFBDAF>炊 LLM 螟<>炊
|
||
笏? 笏? 笏? 笏?
|
||
笏懌楳 荳贋ｼ<E8B48B> Excel 笏? 笏? 笏?
|
||
笏? 笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏竊停狽 笏? 笏?
|
||
笏? 笏懌楳 隗｣譫<EFBDA3> Excel 笏? 笏?
|
||
笏? 笏懌楳 鬪瑚ｯ∵<EFBDAF>ｼ蠑<EFBDBC> 笏? 笏?
|
||
笏? 笏懌楳 譏ｾ遉ｺ鬚<EFBDBA>ｧ<EFBFBD> 笏? 笏?
|
||
笏? 笏? 笏? 笏?
|
||
笏? 笏懌楳 謠蝉ｺ､遲幃我ｻｻ蜉? 笏? 笏?
|
||
笏? 笏? 笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏竊停狽 笏?
|
||
笏? 笏? 笏懌楳 菫晏ｭ倅ｻｻ蜉｡ 笏?
|
||
笏? 笏? 笏懌楳 蛻<>ｻ<EFBFBD>ｼ?5 遽?扈<>ｼ<EFBFBD> 笏?
|
||
笏? 笏? 笏? 笏?
|
||
笏? 笏? 笏懌楳 謇ｹ谺｡ 1 笏?
|
||
笏? 笏? 笏? 笏披楳笏笏笏笏笏笏笏笏笏笏笏笏笏竊停狽
|
||
笏? 笏? 笏? 笏懌楳 DeepSeek 遲幃?
|
||
笏? 笏? 笏? 笏懌楳 Qwen3 遲幃?
|
||
笏? 笏? 笏? 笏懌楳 蟇ｹ豈皮ｻ捺棡
|
||
笏? 笏? 笏? 竊絶楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏?
|
||
笏? 笏? 笏懌楳 菫晏ｭ倡ｻ捺棡 笏?
|
||
笏? 笏? 笏? 笏?
|
||
笏? 笏? 笏懌楳 謇ｹ谺｡ 2... 笏?
|
||
笏? 笏? 笏? 笏?
|
||
笏? 笏? 竊絶楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏笏?霑泌屓螳梧紛扈捺棡 笏?
|
||
笏? 竊絶楳笏笏笏笏笏笏笏笏笏笏笏笏笏笏?譏ｾ遉ｺ扈捺棡 笏? 笏?
|
||
笏披楳 莠ｺ蟾･螟肴<E89E9F>ｸ 笏? 笏? 笏?
|
||
```
|
||
|
||
---
|
||
|
||
## <20>東 蝨ｺ譎ｯ 2 & 3: 蜈ｨ譁<EFBDA8>､咲ｭ帑ｸ取焚謐ｮ謠仙<E8ACA0>?
|
||
|
||
### 2.1 謚譛ｯ迚ｹ轤?
|
||
|
||
- **霎灘<E99C8E>譬ｼ蠑<EFBDBC>**: PDF 譁<>ｻｶ<EFBDBB>郁恭譁<E681AD>現蟄ｦ譁<EFBDA6>鍵<EFBFBD><E98DB5>
|
||
- **譁<>ｻｶ迚ｹ轤ｹ**:
|
||
- 遘大ｭｦ隶ｺ譁<EFBDBA><E8AD81>ｼ蠑擾ｼ域<EFBDBC><E59F9F>｢倥∵遭隕√∝ｼ戊ｨ縲∵婿豕輔∫ｻ捺棡縲∬ｮｨ隶ｺ縲∝盾閠<E79BBE>枚迪ｮ<E8BFAA><EFBDAE>
|
||
- 蛹<>性螟肴揩陦ｨ譬ｼ縲∝<E7B8B2>蠑上∝崟陦?
|
||
- 騾壼ｸｸ 10-30 鬘?
|
||
- **螟<>炊驥咲せ**: 鬮伜㊥遑ｮ邇<EFBDAE>署蜿厄ｼ御ｿ晉蕗扈捺桷蜥梧<E89CA5>ｼ蠑<EFBDBC>
|
||
|
||
### 2.2 謚譛ｯ騾牙梛<E78999>啀DF 謠仙叙
|
||
|
||
#### 譬ｸ蠢<EFBDB8>婿譯茨ｼ哢ougat + PyMuPDF 鬘ｺ蠎城剄郤ｧ遲也払 箝?
|
||
|
||
**邇ｰ譛画楔譫<E6A594>**<2A>亥ｷｲ螳樒鴫<E6A892>御ｽ堺ｺ?`extraction_service/`<60>会ｼ<E4BC9A>
|
||
|
||
```python
|
||
# 鬘ｺ蠎城剄郤ｧ遲也払
|
||
def extract_pdf(file_path: str):
|
||
# Step 1: 譽豬玖ｯｭ險
|
||
language = detect_language(file_path)
|
||
|
||
# Step 2: 荳ｭ譁<EFBDAD> PDF 竊?PyMuPDF<44>亥ｿｫ騾滂ｼ<E6BB82>
|
||
if language == 'chinese':
|
||
return extract_pdf_pymupdf(file_path)
|
||
|
||
# Step 3: 闍ｱ譁<EFBDB1> PDF 竊?蟆晁ｯ<E69981> Nougat
|
||
if check_nougat_available():
|
||
result = extract_pdf_nougat(file_path)
|
||
|
||
# 雍ｨ驥乗｣譟･<E8AD9F>磯<EFBFBD>蛟?0.7<EFBFBD>?
|
||
if result['quality_score'] >= 0.7:
|
||
return result # 笨?Nougat 謌仙粥
|
||
|
||
# Step 4: 髯咲ｺｧ蛻?PyMuPDF
|
||
return extract_pdf_pymupdf(file_path)
|
||
```
|
||
|
||
#### 謚譛ｯ蟇ｹ豈?
|
||
|
||
| 譁ｹ譯<EFBDB9> | 莨伜漢 | 蜉｣蜉ｿ | 騾ら畑蝨ｺ譎ｯ |
|
||
|------|------|------|---------|
|
||
| **Nougat** 箝?| 窶?荳謎ｸｺ遘大ｭｦ譁<EFBDA6>鍵隶ｾ隶｡<br>窶?蜈ｬ蠑上∬｡ｨ譬ｼ蜃<EFBDBC>｡ｮ邇<EFBDAE>ｫ?br>窶?霎灘<E99C8E> Markdown 譬ｼ蠑<EFBDBC><br>窶?菫晉蕗譁<E89597>｡｣扈捺桷 | 窶?騾溷ｺｦ諷｢<E8ABB7><EFBDA2>1-2 蛻<>帖/20 鬘ｵ<E9AC98><EFBDB5><br>窶?髴隕?GPU 蜉<>騾?br>窶?蜀<>ｭ伜頃逕ｨ螟ｧ<E89E9F><EFBDA7>4GB<47>?| 闍ｱ譁<EFBDB1>現蟄ｦ譁<EFBDA6>鍵蜈ｨ譁<EFBDA8>署蜿<E7BDB2> |
|
||
| **PyMuPDF** | 窶?騾溷ｺｦ蠢ｫ<E8A0A2>育ｧ堤ｺｧ<EFBDBA>?br>窶?蜀<>ｭ伜頃逕ｨ菴?br>窶?驛ｨ鄂ｲ邂蜊?| 窶?蜈ｬ蠑上∬｡ｨ譬ｼ譏謎ｸ｢螟ｱ<br>窶?郤ｯ譁<EFBDAF>悽霎灘<E99C8E>?br>窶?蟶<>ｱ譏捺ｷｷ荵?| 荳ｭ譁<EFBDAD>枚迪ｮ縲∝ｿｫ騾滄｢<E6BB84>ｧ?|
|
||
| **Adobe API** | 窶?蝠<>ｸ夂ｺｧ蜃<EFBDA7>｡ｮ邇<EFBDAE><br>窶?莠醍ｫｯ螟<EFBDAF>炊 | 窶?髴莉倩ｴｹ<br>窶?鄂醍ｻ應ｾ晁ｵ<E69981><br>窶?髫千ｧ<E58D83>｣朱勦 | 荳肴耳闕撰ｼ域<EFBDBC>譛ｬ鬮假ｼ<E58187> |
|
||
| **Tesseract OCR** | 窶?蠑貅仙<E8B285>雍?br>窶?謾ｯ謖∝､夊ｯｭ險 | 窶?髴隕∝崟蜒城｢<E59F8E>､<EFBFBD>炊<br>窶?蜃<>｡ｮ邇<EFBDAE>ｸ咲ｨｳ螳<EFBDB3> | 謇ｫ謠冗<E8ACA0>?PDF<44>亥､<E4BAA5>会ｼ<E4BC9A> |
|
||
|
||
**謗ｨ闕先婿譯茨ｼ哢ougat<61>井ｸｻ<EFBDB8>?+ PyMuPDF<44>磯剄郤ｧ<E983A4><EFBDA7> 箝?*
|
||
|
||
#### Nougat 譬ｸ蠢<EFBDB8>ｼ伜漢<E4BC9C>亥現蟄ｦ譁<EFBDA6>鍵蝨ｺ譎ｯ<E8AD8E><EFBDAF>
|
||
|
||
```
|
||
笨?荳謎ｸｺ遘大ｭｦ譁<EFBDA6>鍵隶ｾ隶｡
|
||
笏懌楳 隶ｭ扈<EFBDAD>焚謐ｮ<E8AC90>啾rXiv 隶ｺ譁<EFBDBA> + 遘大ｭｦ譛溷<E8AD9B>
|
||
笏懌楳 蜈ｬ蠑剰ｯ<E589B0>悪<EFBFBD>哭aTeX 譬ｼ蠑剰ｾ灘<EFBDBE>
|
||
笏懌楳 陦ｨ譬ｼ菫晉蕗<E69989>哺arkdown 陦ｨ譬ｼ譬ｼ蠑<EFBDBC>
|
||
笏披楳 扈捺桷蛹冶ｾ灘<EFBDBE><E78198>夂ｫ<E5A482>闃ゅ∵ｮｵ關ｽ貂<EFBDBD><E8B282>?
|
||
|
||
笨?霎灘<E99C8E>譬ｼ蠑擾ｼ哺arkdown
|
||
笏懌楳 譬<>｢伜ｱらｺｧ<EFBDBA>? ## ###
|
||
笏懌楳 陦ｨ譬ｼ<E8ADAC>嘶 Header | Data |
|
||
笏懌楳 蜈ｬ蠑擾ｼ?$ formula $$
|
||
笏披楳 蠑慕畑<E68595>喙1] [2] [3]
|
||
|
||
笨?雍ｨ驥剰ｯ<E589B0>ｼｰ譛ｺ蛻ｶ
|
||
笏懌楳 閾ｪ蜉ｨ雍ｨ驥剰ｯ<E589B0><EFBDAF><EFBFBD>?-1<>?
|
||
笏懌楳 菴手ｴｨ驥剰<E9A9A5>蜉ｨ髯咲ｺ?PyMuPDF
|
||
笏披楳 菫晁ｯ∵署蜿匁<E89CBF>蜉溽<E89C89>?
|
||
```
|
||
|
||
#### 螳樒鴫扈<E9B4AB>鰍
|
||
|
||
**譛榊苅譫ｶ譫<EFBDB6>ｼ?*
|
||
```
|
||
Node.js Backend (Port 3001)
|
||
笏?
|
||
笏懌楳 隹<>畑 ExtractionClient.ts
|
||
笏? 笏披楳 HTTP 隸ｷ豎<EFBDB7> 竊?Python 蠕ｮ譛榊<E8AD9B>?
|
||
笏?
|
||
Python Extraction Service (Port 8000)
|
||
笏?
|
||
笏懌楳 /api/extract/pdf
|
||
笏? 笏懌楳 detect_language()
|
||
笏? 笏懌楳 extract_pdf_nougat() 竊?Nougat Model
|
||
笏? 笏披楳 extract_pdf_pymupdf() 竊?PyMuPDF
|
||
笏?
|
||
笏披楳 /api/health
|
||
笏披楳 譽譟?Nougat 蜿ｯ逕ｨ諤?
|
||
```
|
||
|
||
**Node.js 隹<>畑莉｣遐<EFBDA3>ｼ?*
|
||
```typescript
|
||
import { extractionClient } from '@common/document/ExtractionClient';
|
||
|
||
async function extractLiteraturePDF(file: Buffer, filename: string) {
|
||
try {
|
||
// 譁ｹ豕<EFBDB9> 1: 閾ｪ蜉ｨ騾画叫<E794BB>域耳闕撰ｼ<E692B0>
|
||
const result = await extractionClient.extractPdf(
|
||
file,
|
||
filename,
|
||
'auto'
|
||
);
|
||
|
||
// 譁ｹ豕<EFBDB9> 2: 蠑ｺ蛻ｶ菴ｿ逕ｨ Nougat
|
||
// const result = await extractionClient.extractPdf(file, filename, 'nougat');
|
||
|
||
return {
|
||
text: result.text,
|
||
method: result.method, // "nougat" | "pymupdf"
|
||
quality: result.metadata.quality_score,
|
||
pageCount: result.metadata.page_count,
|
||
hasTables: result.metadata.has_tables,
|
||
hasFormulas: result.metadata.has_formulas
|
||
};
|
||
} catch (error) {
|
||
console.error('PDF extraction failed:', error);
|
||
throw error;
|
||
}
|
||
}
|
||
```
|
||
|
||
**Python 謠仙叙莉｣遐<EFBDA3>ｼ?*
|
||
```python
|
||
# extraction_service/services/nougat_extractor.py
|
||
|
||
def extract_pdf_nougat(file_path: str) -> Dict[str, Any]:
|
||
"""
|
||
菴ｿ逕ｨ Nougat 謠仙叙 PDF 譁<>悽
|
||
|
||
蜻ｽ莉､陦瑚ｰ<E7919A>畑<EFBFBD><E79591>
|
||
nougat <pdf_path> -o <output_dir> --markdown --no-skipping
|
||
"""
|
||
cmd = [
|
||
'nougat',
|
||
file_path,
|
||
'-o', output_dir,
|
||
'--markdown', # 霎灘<E99C8E> Markdown 譬ｼ蠑<EFBDBC>
|
||
'--no-skipping' # 荳崎ｷｳ霑<EFBDB3>ｻｻ菴暮｡ｵ髱?
|
||
]
|
||
|
||
# 謇ｧ陦<EFBDA7> Nougat<61>郁ｶ<E98381><EFBDB6>?5 蛻<>帖<EFBFBD>?
|
||
process = subprocess.Popen(cmd, ...)
|
||
stdout, stderr = process.communicate(timeout=300)
|
||
|
||
# 隸ｻ蜿冶ｾ灘<EFBDBE>譁<EFBFBD>ｻｶ<EFBDBB>?mmd<6D>?
|
||
markdown_text = read_output_file()
|
||
|
||
# 雍ｨ驥剰ｯ<E589B0>ｼｰ
|
||
quality_score = evaluate_nougat_quality(markdown_text)
|
||
|
||
return {
|
||
"success": True,
|
||
"method": "nougat",
|
||
"text": markdown_text,
|
||
"format": "markdown",
|
||
"metadata": {
|
||
"quality_score": quality_score,
|
||
"has_tables": detect_tables(markdown_text),
|
||
"has_formulas": detect_formulas(markdown_text)
|
||
}
|
||
}
|
||
```
|
||
|
||
### 2.3 譁<>悽蜷主､<E4B8BB><EFBDA4>?
|
||
|
||
**Nougat 霎灘<E99C8E>莨伜喧<E4BC9C>?*
|
||
```typescript
|
||
function postProcessNougatOutput(markdown: string): ProcessedText {
|
||
return {
|
||
// 蜴溷ｧ<E6BAB7> Markdown
|
||
raw: markdown,
|
||
|
||
// 遶<>闃ょ<E99783>蜑ｲ
|
||
sections: extractSections(markdown), // {abstract, methods, results, ...}
|
||
|
||
// 陦ｨ譬ｼ謠仙叙
|
||
tables: extractTables(markdown),
|
||
|
||
// 蜈ｬ蠑乗署蜿<E7BDB2>
|
||
formulas: extractFormulas(markdown),
|
||
|
||
// 郤ｯ譁<EFBDAF>悽<EFBFBD>亥悉髯､譬ｼ蠑擾ｼ?
|
||
plainText: markdownToPlainText(markdown),
|
||
|
||
// 扈捺桷蛹匁焚謐ｮ<E8AC90>育畑莠<E79591> LLM<4C>?
|
||
structured: {
|
||
title: extractTitle(markdown),
|
||
abstract: extractAbstract(markdown),
|
||
methodology: extractMethodology(markdown),
|
||
results: extractResults(markdown),
|
||
}
|
||
};
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## <20>東 蝨ｺ譎ｯ 4: 譁<>鍵荳玖ｽｽ<EFBDBD><EFBDBD>npaywall API<50>俄ｭ<E4BF84>
|
||
|
||
### 3.1 謚譛ｯ閭梧<E996AD>?
|
||
|
||
**Unpaywall** 譏ｯ荳荳ｪ蜈崎ｴｹ逧<EFBDB9>ｼ謾ｾ闔ｷ蜿厄ｼ<E58E84>pen Access<73>画枚迪?API<50>悟庄莉･<E88E89><EFBDA5>
|
||
- 笨?騾夊ｿ<E5A48A> DOI 譟･隸｢譁<EFBDA2>鍵譏ｯ蜷ｦ譛牙<E8AD9B>雍ｹ蜈ｨ譁?
|
||
- 笨?闔ｷ蜿門粋豕慕<E8B195>?PDF 荳玖ｽｽ體ｾ謗･
|
||
- 笨?螳悟<E89EB3>蜈崎ｴｹ<EFBDB4>梧裏髴莉倩ｴｹ
|
||
- 笨?謨ｰ謐ｮ蠎楢ｦ<E6A5A2><EFBDA6>?3000+ 荳<>ｯ<EFBFBD>枚迪ｮ
|
||
|
||
**螳倡ｽ<E580A1>**: https://unpaywall.org/products/api
|
||
|
||
### 3.2 謚譛ｯ騾牙梛
|
||
|
||
#### API 隹<>畑譁ｹ蠑<EFBDB9>
|
||
|
||
**蝓ｺ遑菫｡諱ｯ<E8ABB1>?*
|
||
- **API 遶ｯ轤ｹ**: `https://api.unpaywall.org/v2/{doi}?email={your_email}`
|
||
- **隸ｷ豎よ婿豕<E5A9BF>**: GET
|
||
- **隶､隸∵婿蠑<E5A9BF>**: 譌<>髴 API Key<65>御ｻ<E5BEA1>怙謠蝉ｾ幃ぐ邂ｱ
|
||
- **騾溽紫髯仙宛**: 100,000 谺?螟ｩ<E89E9F>亥<EFBFBD>雍ｹ<E99B8D>?
|
||
|
||
**遉ｺ萓玖ｯｷ豎ゑｼ?*
|
||
```bash
|
||
curl "https://api.unpaywall.org/v2/10.1038/nature12373?email=YOUR_EMAIL"
|
||
```
|
||
|
||
**蜩榊ｺ皮､ｺ萓具ｼ?*
|
||
```json
|
||
{
|
||
"doi": "10.1038/nature12373",
|
||
"title": "The genome of the woodland strawberry",
|
||
"is_oa": true,
|
||
"oa_status": "gold",
|
||
"best_oa_location": {
|
||
"url": "https://www.nature.com/articles/nature12373.pdf",
|
||
"url_for_pdf": "https://www.nature.com/articles/nature12373.pdf",
|
||
"url_for_landing_page": "https://www.nature.com/articles/nature12373",
|
||
"license": "cc-by",
|
||
"version": "publishedVersion"
|
||
},
|
||
"oa_locations": [...]
|
||
}
|
||
```
|
||
|
||
#### Node.js 螳樒鴫
|
||
|
||
**譛榊苅蟆∬｣<E288AC>ｼ?*
|
||
```typescript
|
||
// backend/src/common/literature/UnpaywallClient.ts
|
||
|
||
import axios from 'axios';
|
||
import { config } from '../../config/env';
|
||
|
||
export interface UnpaywallResult {
|
||
doi: string;
|
||
title: string;
|
||
isOA: boolean; // 譏ｯ蜷ｦ蠑謾ｾ闔ｷ蜿?
|
||
oaStatus: string; // "gold" | "green" | "hybrid" | "bronze" | "closed"
|
||
pdfUrl: string | null; // PDF 荳玖ｽｽ體ｾ謗･
|
||
landingPageUrl: string; // 譁<>鍵鬘ｵ髱｢體ｾ謗･
|
||
license: string | null; // 隶ｸ蜿ｯ蜊剰ｮｮ
|
||
version: string | null; // "publishedVersion" | "acceptedVersion"
|
||
}
|
||
|
||
class UnpaywallClient {
|
||
private baseUrl = 'https://api.unpaywall.org/v2';
|
||
private email: string;
|
||
|
||
constructor(email: string = config.unpaywallEmail) {
|
||
this.email = email;
|
||
}
|
||
|
||
/**
|
||
* 騾夊ｿ<E5A48A> DOI 譟･隸｢譁<EFBDA2>鍵菫｡諱ｯ
|
||
*/
|
||
async getByDoi(doi: string): Promise<UnpaywallResult> {
|
||
try {
|
||
const url = `${this.baseUrl}/${doi}?email=${this.email}`;
|
||
const response = await axios.get(url, {
|
||
timeout: 10000, // 10 遘定ｶ<E5AE9A><EFBDB6>?
|
||
});
|
||
|
||
const data = response.data;
|
||
|
||
// 闔ｷ蜿匁怙菴ｳ荳玖ｽｽ菴咲ｽ?
|
||
const bestOA = data.best_oa_location;
|
||
|
||
return {
|
||
doi: data.doi,
|
||
title: data.title,
|
||
isOA: data.is_oa,
|
||
oaStatus: data.oa_status,
|
||
pdfUrl: bestOA?.url_for_pdf || null,
|
||
landingPageUrl: bestOA?.url_for_landing_page || data.doi_url,
|
||
license: bestOA?.license || null,
|
||
version: bestOA?.version || null,
|
||
};
|
||
} catch (error) {
|
||
if (axios.isAxiosError(error)) {
|
||
if (error.response?.status === 404) {
|
||
throw new Error(`DOI not found: ${doi}`);
|
||
}
|
||
}
|
||
throw new Error(`Unpaywall API error: ${error.message}`);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* 謇ｹ驥乗衍隸｢<E99AB8>亥ｸｦ騾溽紫髯仙宛<E4BB99>?
|
||
*/
|
||
async getBatch(dois: string[]): Promise<UnpaywallResult[]> {
|
||
const results = [];
|
||
|
||
for (const doi of dois) {
|
||
try {
|
||
const result = await this.getByDoi(doi);
|
||
results.push(result);
|
||
|
||
// 騾溽紫髯仙宛<E4BB99>?00ms/隸ｷ豎<EFBDB7>
|
||
await new Promise(resolve => setTimeout(resolve, 100));
|
||
} catch (error) {
|
||
console.error(`Failed to fetch ${doi}:`, error.message);
|
||
results.push(null); // 螟ｱ雍･鬘ｹ譬<EFBDB9>ｮｰ荳ｺ null
|
||
}
|
||
}
|
||
|
||
return results.filter(r => r !== null);
|
||
}
|
||
|
||
/**
|
||
* 荳玖ｽｽ PDF 譁<>ｻｶ
|
||
*/
|
||
async downloadPdf(pdfUrl: string, outputPath: string): Promise<void> {
|
||
try {
|
||
const response = await axios.get(pdfUrl, {
|
||
responseType: 'arraybuffer',
|
||
timeout: 60000, // 1 蛻<>帖雜<E5B896>慮
|
||
});
|
||
|
||
const fs = require('fs');
|
||
fs.writeFileSync(outputPath, response.data);
|
||
} catch (error) {
|
||
throw new Error(`PDF download failed: ${error.message}`);
|
||
}
|
||
}
|
||
}
|
||
|
||
export const unpaywallClient = new UnpaywallClient();
|
||
```
|
||
|
||
**邇ｯ蠅<EFBDAF>序驥城<E9A9A5>鄂ｮ<E98482>?*
|
||
```env
|
||
# .env
|
||
UNPAYWALL_EMAIL=your-email@example.com
|
||
```
|
||
|
||
#### 荳壼苅髮<E88B85><E9ABAE>
|
||
|
||
**蝨ｺ譎ｯ 1<>壽音驥乗｣譟･譁<EFBDA5>鍵譏ｯ蜷ｦ蜿ｯ荳玖ｽｽ**
|
||
```typescript
|
||
async function checkLiteratureAvailability(literatures: Literature[]) {
|
||
const dois = literatures
|
||
.map(lit => lit.doi)
|
||
.filter(doi => doi); // 霑<>ｻ､遨?DOI
|
||
|
||
const results = await unpaywallClient.getBatch(dois);
|
||
|
||
return literatures.map(lit => ({
|
||
...lit,
|
||
downloadable: results.find(r => r.doi === lit.doi)?.isOA || false,
|
||
pdfUrl: results.find(r => r.doi === lit.doi)?.pdfUrl || null,
|
||
}));
|
||
}
|
||
```
|
||
|
||
**蝨ｺ譎ｯ 2<>夂畑謌ｷ轤ｹ蜃ｻ荳玖ｽｽ蜈ｨ譁?*
|
||
```typescript
|
||
async function downloadLiteratureFullText(doi: string) {
|
||
// Step 1: 譟･隸｢ Unpaywall
|
||
const unpaywallResult = await unpaywallClient.getByDoi(doi);
|
||
|
||
if (!unpaywallResult.pdfUrl) {
|
||
throw new Error('隸･譁<EFBDA5>鍵譌<E98DB5>蜈崎ｴｹ蜈ｨ譁<EFBDA8>');
|
||
}
|
||
|
||
// Step 2: 荳玖ｽｽ PDF
|
||
const filename = `${doi.replace(/\//g, '_')}.pdf`;
|
||
const outputPath = `./downloads/${filename}`;
|
||
|
||
await unpaywallClient.downloadPdf(unpaywallResult.pdfUrl, outputPath);
|
||
|
||
// Step 3: 謠仙叙譁<E58F99>悽<EFBFBD>郁ｰ<E98381><EFBDB0>?extraction_service<63>?
|
||
const extractionResult = await extractionClient.extractPdf(
|
||
fs.readFileSync(outputPath),
|
||
filename,
|
||
'auto'
|
||
);
|
||
|
||
return {
|
||
pdfPath: outputPath,
|
||
text: extractionResult.text,
|
||
method: extractionResult.method,
|
||
};
|
||
}
|
||
```
|
||
|
||
### 3.3 蜑咲ｫｯ髮<EFBDAF><E9ABAE>
|
||
|
||
**謇ｹ驥丈ｸ玖ｽｽ謖蛾聴<E89BBE>?*
|
||
```typescript
|
||
// 謇ｹ驥乗｣譟･蜿ｯ荳玖ｽｽ諤?
|
||
async function checkDownloadable(selectedRows: Literature[]) {
|
||
setLoading(true);
|
||
|
||
const results = await api.checkLiteratureAvailability(selectedRows);
|
||
|
||
const downloadableCount = results.filter(r => r.downloadable).length;
|
||
|
||
message.success(`蜿醍鴫 ${downloadableCount} 遽<>庄荳玖ｽｽ蜈ｨ譁㌔);
|
||
setLiteratures(results);
|
||
setLoading(false);
|
||
}
|
||
|
||
// 荳玖ｽｽ蜈ｨ譁<EFBDA8>
|
||
async function downloadFullText(literature: Literature) {
|
||
if (!literature.downloadable) {
|
||
message.warning('隸･譁<EFBDA5>鍵譌<E98DB5>蜈崎ｴｹ蜈ｨ譁<EFBDA8>');
|
||
return;
|
||
}
|
||
|
||
try {
|
||
const result = await api.downloadLiteratureFullText(literature.doi);
|
||
message.success('荳玖ｽｽ謌仙粥');
|
||
|
||
// 謇灘ｼ PDF 譟･逵句<E980B5>?
|
||
openPdfViewer(result.pdfPath);
|
||
} catch (error) {
|
||
message.error(`荳玖ｽｽ螟ｱ雍･: ${error.message}`);
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## <20>剥 陦･蜈<EFBDA5>橿譛ｯ轤ｹ
|
||
|
||
### 4.1 謔ｨ謠仙芦逧<E88AA6>橿譛ｯ轤ｹ諤ｻ扈<EFBDBB>
|
||
|
||
| 謚譛ｯ轤ｹ | 迥ｶ諤?| 隸ｴ譏<EFBDB4> |
|
||
|--------|------|------|
|
||
| 笨?Nougat 讓｡蝙<EFBDA1> | 蟾ｲ螳樒<E89EB3>?| `extraction_service/services/nougat_extractor.py` |
|
||
| 笨?PyMuPDF | 蟾ｲ螳樒<E89EB3>?| `extraction_service/services/pdf_extractor.py` |
|
||
| 笨?鬘ｺ蠎城剄郤ｧ遲也払 | 蟾ｲ螳樒<E89EB3>?| 闍ｱ譁<EFBDB1><E8AD81>Nougat<61>御ｸｭ譁<EFBDAD><E8AD81>PyMuPDF |
|
||
| <20><> Unpaywall API | 髴譁ｰ蠅<EFBDB0> | 譛ｬ譁<EFBDAC>｡｣謠蝉ｾ帛ｮ樒鴫譁ｹ譯?|
|
||
| 笨?Excel 隗｣譫<EFBDA3> | 髴譁ｰ蠅<EFBDB0> | 菴ｿ逕ｨ `xlsx` 蠎難ｼ亥燕遶ｯ<E981B6>?|
|
||
|
||
### 4.2 蜿ｯ閭ｽ驕玲ｼ冗噪謚譛ｯ轤ｹ 箝?
|
||
|
||
#### <20>?<3F>芽｡ｨ譬ｼ謠仙叙蠅槫ｼ?
|
||
|
||
**髣ｮ鬚<EFBDAE>**<2A>哢ougat 陌ｽ辟ｶ菫晉蕗陦ｨ譬ｼ扈捺桷<E68DBA>御ｽ<E5BEA1> LLM 逶ｴ謗･螟<EFBDA5>炊 Markdown 陦ｨ譬ｼ蜿ｯ閭ｽ荳榊㊥遑ｮ縲?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ啜able Transformer**
|
||
```python
|
||
# 菴ｿ逕ｨ蠕ｮ霓ｯ逧?Table Transformer 讓｡蝙<EFBDA1>
|
||
# https://github.com/microsoft/table-transformer
|
||
|
||
from transformers import TableTransformerForObjectDetection
|
||
import torch
|
||
|
||
def extract_tables_enhanced(pdf_path: str):
|
||
"""
|
||
菴ｿ逕ｨ Table Transformer 邊ｾ遑ｮ螳壻ｽ崎｡ｨ譬ｼ
|
||
"""
|
||
model = TableTransformerForObjectDetection.from_pretrained(
|
||
"microsoft/table-transformer-detection"
|
||
)
|
||
|
||
# 譽豬玖｡ｨ譬ｼ菴咲ｽ?
|
||
tables = model.detect_tables(pdf_path)
|
||
|
||
# 謠仙叙豈丈ｸｪ陦ｨ譬ｼ
|
||
for table in tables:
|
||
table_image = crop_table(pdf_path, table.bbox)
|
||
table_data = ocr_table(table_image)
|
||
|
||
return structured_tables
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>啖2.0**<EFBFBD><EFBFBD>VP 髦ｶ谿ｵ Nougat 雜ｳ螟滂ｼ?
|
||
|
||
#### <20>?<3F>牙ｼ慕畑隗｣譫蝉ｸ朱得謗･
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>夂ｧ大ｭｦ譁<EFBFBD>鍵蛹<EFBFBD>性螟ｧ驥丞ｼ慕<EFBFBD>?`[1] [2] [3]`<EFBFBD>碁怙隕∬ｧ｣譫仙ｹｶ體ｾ謗･蛻ｰ蜿り<EFBFBD>枚迪ｮ縲?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ哦ROBID**
|
||
```python
|
||
# GROBID: 蠑貅千ｧ大ｭｦ譁<EFBDA6>鍵隗｣譫仙ｷ･蜈?
|
||
# https://github.com/kermitt2/grobid
|
||
|
||
import requests
|
||
|
||
def parse_references(pdf_path: str):
|
||
"""
|
||
菴ｿ逕ｨ GROBID 隗｣譫仙盾閠<E79BBE>枚迪?
|
||
"""
|
||
with open(pdf_path, 'rb') as f:
|
||
files = {'input': f}
|
||
response = requests.post(
|
||
'http://localhost:8070/api/processFulltextDocument',
|
||
files=files
|
||
)
|
||
|
||
# 霑泌屓扈捺桷蛹也噪蠑慕畑蛻苓｡ｨ
|
||
return response.json()['references']
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>啖2.0**<EFBFBD>磯撼譬ｸ蠢<EFBFBD>粥閭ｽ<EFBFBD>?
|
||
|
||
#### <20>?<3F>牙<EFBFBD>蠑剰ｯ<E589B0>悪荳取ｸｲ譟<EFBDB2>
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>哢ougat 霎灘<E99C8E> LaTeX 蜈ｬ蠑擾ｼ悟燕遶ｯ髴隕∵ｸｲ譟薙?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ哮aTeX / MathJax**
|
||
```typescript
|
||
// 蜑咲ｫｯ貂ｲ譟<EFBDB2> LaTeX 蜈ｬ蠑<EFBDAC>
|
||
import katex from 'katex';
|
||
import 'katex/dist/katex.min.css';
|
||
|
||
function renderFormula(latex: string) {
|
||
return katex.renderToString(latex, {
|
||
throwOnError: false,
|
||
displayMode: true,
|
||
});
|
||
}
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>哺VP**<EFBFBD>域署蜊<EFBFBD>畑謌ｷ菴馴ｪ鯉ｼ<EFBFBD>
|
||
|
||
#### <20>?<3F>臼DF 鬚<>ｧ井ｸ取<EFBDB8><E58F96>ｳ?
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>壻ｺｺ蟾･螟肴<EFBFBD>ｸ譌ｶ髴隕∵衍逵句次譁<EFBFBD>ｼ悟ｹｶ鬮倅ｺｮ譬<EFBFBD>ｳｨ縲?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ啀DF.js + Annotator.js**
|
||
```typescript
|
||
// React 扈<>ｻｶ
|
||
import { Viewer } from '@react-pdf-viewer/core';
|
||
import '@react-pdf-viewer/core/lib/styles/index.css';
|
||
|
||
function PdfViewer({ pdfUrl, annotations }) {
|
||
return (
|
||
<Viewer
|
||
fileUrl={pdfUrl}
|
||
plugins={[
|
||
highlightPlugin({
|
||
highlights: annotations // 鬮倅ｺｮ菴咲ｽｮ
|
||
})
|
||
]}
|
||
/>
|
||
);
|
||
}
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>哺VP**<EFBFBD>域<EFBFBD>ｸ蠢<EFBFBD>粥閭ｽ<EFBFBD><EFBFBD>
|
||
|
||
#### <20>?<3F>画枚迪ｮ蜴ｻ驥?
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>哘xcel 荳贋ｼ<E8B48B>蜿ｯ閭ｽ蛹<EFBDBD>性驥榊､肴枚迪ｮ<E8BFAA>亥酔荳遽<C280>枚迪ｮ荳榊酔迚域悽<E59F9F>峨?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ壼渕莠?DOI 蜥梧<E89CA5><E6A2A7>｢倡噪蜴ｻ驥<EFBDBB>**
|
||
```typescript
|
||
function deduplicateLiteratures(literatures: Literature[]) {
|
||
const seen = new Set();
|
||
|
||
return literatures.filter(lit => {
|
||
// 莨伜<E88EA8>菴ｿ逕ｨ DOI
|
||
if (lit.doi) {
|
||
if (seen.has(lit.doi)) return false;
|
||
seen.add(lit.doi);
|
||
return true;
|
||
}
|
||
|
||
// 蜷ｦ蛻吩ｽｿ逕ｨ譬<EFBDA8>｢假ｼ域<EFBDBC><E59F9F>㊥蛹門錘<E99680><E98C98>
|
||
const normalizedTitle = normalizeTitle(lit.title);
|
||
if (seen.has(normalizedTitle)) return false;
|
||
seen.add(normalizedTitle);
|
||
return true;
|
||
});
|
||
}
|
||
|
||
function normalizeTitle(title: string): string {
|
||
return title
|
||
.toLowerCase()
|
||
.replace(/[^\w\s]/g, '') // 蜴ｻ髯､譬<EFBDA4>せ
|
||
.replace(/\s+/g, ' ') // 隗<>激蛹也ｩｺ譬?
|
||
.trim();
|
||
}
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>哺VP**<EFBFBD>亥ｿ<EFBFBD>｡ｻ蜉溯<EFBFBD><EFBFBD><EFBFBD>
|
||
|
||
#### <20>?<3F>画枚迪ｮ蜈<EFBDAE>焚謐ｮ陦･蜈ｨ
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>哘xcel 荳贋ｼ<E8B48B>逧<EFBFBD>焚謐ｮ蜿ｯ閭ｽ荳榊ｮ梧紛<E6A2A7>育ｼｺ DOI縲∝ｹｴ莉ｽ遲会ｼ峨?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ咾rossref API**
|
||
```typescript
|
||
// 騾夊ｿ<E5A48A><EFBDBF><EFBFBD>｢俶衍隸｢ DOI
|
||
async function enrichMetadata(literature: Literature) {
|
||
if (literature.doi) return literature; // 蟾ｲ譛<EFBDB2> DOI
|
||
|
||
// 隹<>畑 Crossref API
|
||
const response = await axios.get(
|
||
`https://api.crossref.org/works?query.title=${literature.title}`
|
||
);
|
||
|
||
const match = response.data.message.items[0];
|
||
|
||
return {
|
||
...literature,
|
||
doi: match.DOI,
|
||
year: match['published-print']?.['date-parts'][0][0],
|
||
journal: match['container-title'][0],
|
||
};
|
||
}
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>啖1.0**<EFBFBD>亥｢槫ｼｺ蜉溯<EFBFBD><EFBFBD><EFBFBD>
|
||
|
||
#### <20>?<3F>画音螟<E99FB3>炊霑帛ｺｦ謖∽ｹ<E288BD><EFBDB9>?
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>壽音驥冗ｭ幃芽玲慮髟ｿ<EFBFBD><EFBFBD>1000 遽?> 10 蛻<>帖<EFBFBD>会ｼ碁怙謾ｯ謖∵妙轤ｹ扈ｭ莨<EFBDAD>縲?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ啌edis + 莉ｻ蜉｡髦溷<E9ABA6>**
|
||
```typescript
|
||
// 菴ｿ逕ｨ Bull 髦溷<E9ABA6>
|
||
import Queue from 'bull';
|
||
|
||
const screeningQueue = new Queue('literature-screening', {
|
||
redis: { host: 'localhost', port: 6379 }
|
||
});
|
||
|
||
// 豺ｻ蜉<EFBDBB>莉ｻ蜉｡
|
||
screeningQueue.add({
|
||
projectId: 'xxx',
|
||
literatures: [...],
|
||
protocol: {...}
|
||
});
|
||
|
||
// 螟<>炊莉ｻ蜉｡
|
||
screeningQueue.process(async (job) => {
|
||
const { projectId, literatures, protocol } = job.data;
|
||
|
||
for (let i = 0; i < literatures.length; i++) {
|
||
// 螟<>炊蜊慕ｯ<E68595>枚迪ｮ
|
||
await screenLiterature(literatures[i], protocol);
|
||
|
||
// 譖ｴ譁ｰ霑帛ｺｦ
|
||
job.progress((i + 1) / literatures.length * 100);
|
||
}
|
||
});
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>啖1.0**<EFBFBD>井ｽ馴ｪ御ｼ伜喧<EFBFBD><EFBFBD>
|
||
|
||
#### <20>?<3F>蛾漠隸ｯ螟<EFBDAF>炊荳朱㍾隸<E38DBE>
|
||
|
||
**髣ｮ鬚<EFBDAE>**<EFBFBD>哭LM 隹<>畑蜿ｯ閭ｽ螟ｱ雍･<E99B8D>育ｽ醍ｻ懊∬ｶ<E288AC>慮縲<E685AE>剞豬<E5899E>ｼ峨?
|
||
|
||
**隗｣蜀ｳ譁ｹ譯茨ｼ壽欠謨ｰ騾驕ｿ驥崎ｯ?*
|
||
```typescript
|
||
async function retryWithBackoff<T>(
|
||
fn: () => Promise<T>,
|
||
maxRetries: number = 3
|
||
): Promise<T> {
|
||
for (let i = 0; i < maxRetries; i++) {
|
||
try {
|
||
return await fn();
|
||
} catch (error) {
|
||
if (i === maxRetries - 1) throw error;
|
||
|
||
// 謖<>焚騾驕ｿ<E9A995><EFBDBF>1s, 2s, 4s
|
||
const delay = Math.pow(2, i) * 1000;
|
||
await new Promise(resolve => setTimeout(resolve, delay));
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**莨伜<E88EA8>郤ｧ<E983A4>哺VP**<EFBFBD>亥ｿ<EFBFBD>｡ｻ蜉溯<EFBFBD><EFBFBD><EFBFBD>
|
||
|
||
---
|
||
|
||
## <20>投 謚譛ｯ騾牙梛諤ｻ扈<EFBDBB>
|
||
|
||
### MVP 髦ｶ谿ｵ蠢<EFBDB5>画橿譛?
|
||
|
||
| 螻らｺｧ | 謚譛?| 逕ｨ騾?|
|
||
|------|------|------|
|
||
| **蜑咲ｫｯ** | `xlsx` | Excel 隗｣譫<EFBDA3> |
|
||
| **蜑咲ｫｯ** | `PDF.js` | PDF 鬚<>ｧ<EFBFBD> |
|
||
| **蜑咲ｫｯ** | `KaTeX` | 蜈ｬ蠑乗ｸｲ譟<EFBDB2> |
|
||
| **蜷守ｫｯ** | `ExtractionClient` | 隹<>畑 Python 蠕ｮ譛榊<E8AD9B>?|
|
||
| **蜷守ｫｯ** | `UnpaywallClient` | 譁<>鍵荳玖ｽｽ |
|
||
| **Python** | `Nougat` | 闍ｱ譁<EFBDB1> PDF 謠仙叙 |
|
||
| **Python** | `PyMuPDF` | 蠢ｫ騾?PDF 謠仙叙 |
|
||
| **謨ｰ謐ｮ蠎?* | `asl_schema` | 謨ｰ謐ｮ蟄伜お |
|
||
|
||
### V1.0 蠅槫ｼｺ謚譛?
|
||
|
||
| 謚譛?| 逕ｨ騾?|
|
||
|------|------|
|
||
| Crossref API | 蜈<>焚謐ｮ陦･蜈?|
|
||
| Bull Queue | 莉ｻ蜉｡髦溷<E9ABA6> |
|
||
| Redis | 霑帛ｺｦ謖∽ｹ<E288BD><EFBDB9>?|
|
||
|
||
### V2.0 鬮倡ｺｧ謚譛?
|
||
|
||
| 謚譛?| 逕ｨ騾?|
|
||
|------|------|
|
||
| Table Transformer | 陦ｨ譬ｼ邊ｾ遑ｮ謠仙叙 |
|
||
| GROBID | 蠑慕畑隗｣譫<EFBDA3> |
|
||
| Semantic Scholar API | 蟄ｦ譛ｯ蝗ｾ隹ｱ |
|
||
|
||
---
|
||
|
||
## <20>刀 豬玖ｯ墓焚謐ｮ蟄俶叛蟒ｺ隶ｮ
|
||
|
||
譬ｹ謐ｮ ASL 讓｡蝮礼噪譁<E599AA>ｻｶ螟ｹ扈捺桷<E68DBA>梧ｵ玖ｯ墓焚謐ｮ蠎碑ｯ･謾ｾ蝨ｨ<E89DA8><EFBDA8>
|
||
|
||
```
|
||
AIclinicalresearch/docs/03-荳壼苅讓｡蝮<EFBDA1>/ASL-AI譎ｺ閭ｽ譁<EFBDBD>鍵/
|
||
笏披楳笏 05-豬玖ｯ墓枚譯｣/
|
||
笏懌楳笏 01-豬玖ｯ戊ｮ｡蛻<EFBDA1>.md
|
||
笏懌楳笏 02-譬<>｢俶遭隕∝<E99A95>遲帶ｵ玖ｯ慕畑萓<E79591>.md
|
||
笏披楳笏 03-豬玖ｯ墓焚謐ｮ/ 竊?譁ｰ蟒ｺ譁<EFBDBA>ｻｶ螟?
|
||
笏懌楳笏 README.md 竊?隸ｴ譏取枚譯｣
|
||
笏懌楳笏 screening-test-data/
|
||
笏? 笏懌楳笏 literature-list-199.xlsx 竊?199 遽<>枚迪ｮ蛻苓｡?
|
||
笏? 笏懌楳笏 picos-criteria.txt 竊?PICOS 譬<>㊥
|
||
笏? 笏披楳笏 expected-results.json 竊?鬚<>悄扈捺棡<E68DBA>磯≡譬<E289A1>㊥<EFBFBD>?
|
||
笏懌楳笏 pdf-samples/
|
||
笏? 笏懌楳笏 sample-rct-01.pdf
|
||
笏? 笏懌楳笏 sample-cohort-01.pdf
|
||
笏? 笏披楳笏 README.md
|
||
笏披楳笏 extraction-test-data/
|
||
笏披楳笏 README.md
|
||
```
|
||
|
||
**謗ｨ闕千ｻ捺桷<E68DBA>?*
|
||
```
|
||
05-豬玖ｯ墓枚譯｣/
|
||
笏懌楳笏 01-豬玖ｯ戊ｮ｡蛻<EFBDA1>.md
|
||
笏懌楳笏 02-譬<>｢俶遭隕∝<E99A95>遲帶ｵ玖ｯ慕畑萓<E79591>.md
|
||
笏披楳笏 03-豬玖ｯ墓焚謐ｮ/
|
||
笏懌楳笏 README.md 竊?驥崎ｦ<E5B48E>ｼ∬ｯｴ譏取ｵ玖ｯ墓焚謐ｮ譚･貅舌∫沿譚<E6B2BF>∽ｽｿ逕ｨ譁ｹ豕?
|
||
笏懌楳笏 screening/
|
||
笏? 笏懌楳笏 literature-list-199.xlsx
|
||
笏? 笏懌楳笏 picos-criteria.txt
|
||
笏? 笏懌楳笏 inclusion-criteria.txt
|
||
笏? 笏懌楳笏 exclusion-criteria.txt
|
||
笏? 笏披楳笏 gold-standard.json 竊?莠ｺ蟾･譬<EFBDA5>ｳｨ逧<EFBDA8>ｭ｣遑ｮ遲疲｡?
|
||
笏披楳笏 pdf-extraction/
|
||
笏懌楳笏 sample-01-high-quality.pdf
|
||
笏懌楳笏 sample-02-with-tables.pdf
|
||
笏披楳笏 sample-03-chinese.pdf
|
||
```
|
||
|
||
**README.md 遉ｺ萓具ｼ?*
|
||
```markdown
|
||
# ASL 豬玖ｯ墓焚謐ｮ髮?
|
||
|
||
## <20>搭 謨ｰ謐ｮ隸ｴ譏<EFBDB4>
|
||
|
||
### 1. 譬<>｢俶遭隕∝<E99A95>遲帶ｵ玖ｯ墓焚謐ｮ
|
||
- **譁<>ｻｶ**: `literature-list-199.xlsx`
|
||
- **謨ｰ驥<EFBDB0>**: 199 遽<>恭譁<E681AD>現蟄ｦ譁<EFBDA6><E8AD81>?
|
||
- **蟄玲ｮｵ**: 譬<>｢倥∵遭隕√．OI縲∽ｽ懆<E68786>∝ｹｴ莉ｽ縲∵悄蛻?
|
||
- **譚･貅<EFBDA5>**: [謠剰ｿｰ謨ｰ謐ｮ譚･貅疹
|
||
- **迚域揀**: [隸ｴ譏守沿譚<E6B2BF>ｿ｡諱ｯ]
|
||
|
||
### 2. PICOS 譬<>㊥
|
||
- **譁<>ｻｶ**: `picos-criteria.txt`
|
||
- **蜀<>ｮｹ**: Population, Intervention, Comparison, Outcome, Study Design
|
||
- **郤ｳ蜈･譬<EFBDA5>㊥**: 5 譚?
|
||
- **謗帝勁譬<E58B81>㊥**: 8 譚?
|
||
|
||
### 3. 驥第<E9A9A5><E7ACAC>㊥<EFBFBD>井ｺｺ蟾･譬<EFBDA5>ｳｨ扈捺棡<E68DBA>?
|
||
- **譁<>ｻｶ**: `gold-standard.json`
|
||
- **譬<>ｳｨ莠?*: [譬<>ｳｨ荳灘ｮｶ菫｡諱ｯ]
|
||
- **譬<>ｳｨ譌ｶ髣ｴ**: [譌ｶ髣ｴ]
|
||
- **鬚<>悄蜃<E68284>｡ｮ邇?*: 竕?90%
|
||
|
||
## <20>識 菴ｿ逕ｨ譁ｹ豕<EFBDB9>
|
||
|
||
### 霑占｡梧ｵ玖ｯ<E78E96>
|
||
```bash
|
||
npm run test:asl:screening
|
||
```
|
||
|
||
### 隸<>ｼｰ蜃<EFBDB0>｡ｮ邇?
|
||
```bash
|
||
npm run test:asl:evaluate -- --gold-standard gold-standard.json
|
||
```
|
||
|
||
## <20>投 鬚<>悄扈捺棡
|
||
- 郤ｳ蜈･: 45 遽?
|
||
- 謗帝勁: 132 遽?
|
||
- 荳咲｡ｮ螳? 22 遽?
|
||
```
|
||
|
||
---
|
||
|
||
## <20>答 逶ｸ蜈ｳ譁<EFBDB3>｡｣
|
||
|
||
- [雍ｨ驥丈ｿ晞囿荳主庄霑ｽ貅ｯ遲也払](./06-雍ｨ驥丈ｿ晞囿荳主庄霑ｽ貅ｯ遲也払.md)
|
||
- [謨ｰ謐ｮ蠎楢ｮｾ隶｡](./01-謨ｰ謐ｮ蠎楢ｮｾ隶?md)
|
||
- [API 隶ｾ隶｡隗<EFBDA1>激](./02-API隶ｾ隶｡隗<EFBDA1>激.md)
|
||
- [譁<EFBFBD>｡｣謠仙叙蠕ｮ譛榊苅](../../../../extraction_service/README.md)
|
||
|
||
---
|
||
|
||
**譖ｴ譁ｰ譌･蠢<EFBDA5>**<EFBFBD>?
|
||
- 2025-11-15: 蛻帛ｻｺ譁<EFBDBA>｡｣<EFBDA1>悟ｮ壻ｹ牙<EFBDB9>遲帙∝<C280>譁<EFBFBD>､<EFBFBD>炊縲∵枚迪ｮ荳玖ｽｽ謚譛ｯ騾牙梛
|
||
|
||
|
||
|