HaHafeng
4c6eaaecbf
Summary: - Implement async file upload processing (Platform-Only pattern) - Add parseExcelWorker with pg-boss queue - Implement React Query polling mechanism - Add clean data caching (avoid duplicate parsing) - Fix pivot single-value column tuple issue - Optimize performance by 99 percent Technical Details: 1. Async Architecture (Postgres-Only): - SessionService.createSession: Fast upload + push to queue (3s) - parseExcelWorker: Background parsing + save clean data (53s) - SessionController.getSessionStatus: Status query API for polling - React Query Hook: useSessionStatus (auto-serial polling) - Frontend progress bar with real-time feedback 2. Performance Optimization: - Clean data caching: Worker saves processed data to OSS - getPreviewData: Read from clean data cache (0.5s vs 43s, -99 percent) - getFullData: Read from clean data cache (0.5s vs 43s, -99 percent) - Intelligent cleaning: Boundary detection + ghost column/row removal - Safety valve: Max 3000 columns, 5M cells 3. Bug Fixes: - Fix pivot column name tuple issue for single value column - Fix queue name format (colon to underscore: asl:screening -> asl_screening) - Fix polling storm (15+ concurrent requests -> 1 serial request) - Fix QUEUE_TYPE environment variable (memory -> pgboss) - Fix logger import in PgBossQueue - Fix formatSession to return cleanDataKey - Fix saveProcessedData to update clean data synchronously 4. Database Changes: - ALTER TABLE dc_tool_c_sessions ADD COLUMN clean_data_key VARCHAR(1000) - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_rows DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN total_cols DROP NOT NULL - ALTER TABLE dc_tool_c_sessions ALTER COLUMN columns DROP NOT NULL 5. Documentation: - Create Postgres-Only async task processing guide (588 lines) - Update Tool C status document (Day 10 summary) - Update DC module status document - Update system overview document - Update cloud-native development guide Performance Improvements: - Upload + preview: 96s -> 53.5s (-44 percent) - Filter operation: 44s -> 2.5s (-94 percent) - Pivot operation: 45s -> 2.5s (-94 percent) - Concurrent requests: 15+ -> 1 (-93 percent) - Complete workflow (upload + 7 ops): 404s -> 70.5s (-83 percent) Files Changed: - Backend: 15 files (Worker, Service, Controller, Schema, Config) - Frontend: 4 files (Hook, Component, API) - Docs: 4 files (Guide, Status, Overview, Spec) - Database: 4 column modifications - Total: ~1388 lines of new/modified code Status: Fully tested and verified, production ready
98 lines
2.3 KiB
Python
98 lines
2.3 KiB
Python
"""
|
||
数值映射（重编码）操作
|
||
|
||
将分类变量的原始值映射为新值（如：男→1，女→2）。
|
||
"""
|
||
|
||
import pandas as pd
|
||
from typing import Dict, Any, Optional
|
||
|
||
|
||
def apply_recode(
|
||
df: pd.DataFrame,
|
||
column: str,
|
||
mapping: Dict[Any, Any],
|
||
create_new_column: bool = True,
|
||
new_column_name: Optional[str] = None
|
||
) -> pd.DataFrame:
|
||
"""
|
||
应用数值映射
|
||
|
||
Args:
|
||
df: 输入数据框
|
||
column: 要重编码的列名
|
||
mapping: 映射字典，如 {'男': 1, '女': 2}
|
||
create_new_column: 是否创建新列（True）或覆盖原列（False）
|
||
new_column_name: 新列名（create_new_column=True时使用）
|
||
|
||
Returns:
|
||
重编码后的数据框
|
||
|
||
Examples:
|
||
>>> df = pd.DataFrame({'性别': ['男', '女', '男', '女']})
|
||
>>> mapping = {'男': 1, '女': 2}
|
||
>>> result = apply_recode(df, '性别', mapping, True, '性别_编码')
|
||
>>> result['性别_编码'].tolist()
|
||
[1, 2, 1, 2]
|
||
"""
|
||
if df.empty:
|
||
return df
|
||
|
||
# 验证列是否存在
|
||
if column not in df.columns:
|
||
raise KeyError(f"列 '{column}' 不存在")
|
||
|
||
if not mapping:
|
||
raise ValueError('映射字典不能为空')
|
||
|
||
# 确定目标列名
|
||
if create_new_column:
|
||
target_column = new_column_name or f'{column}_编码'
|
||
else:
|
||
target_column = column
|
||
|
||
# 创建结果数据框（避免修改原数据）
|
||
result = df.copy()
|
||
|
||
# 应用映射
|
||
result[target_column] = result[column].map(mapping)
|
||
|
||
# 统计结果
|
||
mapped_count = result[target_column].notna().sum()
|
||
unmapped_count = result[target_column].isna().sum()
|
||
total_count = len(result)
|
||
|
||
print(f'映射完成: {mapped_count} 个值成功映射')
|
||
|
||
if unmapped_count > 0:
|
||
print(f'警告: {unmapped_count} 个值未找到对应映射')
|
||
# 找出未映射的唯一值
|
||
unmapped_mask = result[target_column].isna()
|
||
unmapped_values = result.loc[unmapped_mask, column].unique()
|
||
print(f'未映射的值: {list(unmapped_values)[:10]}') # 最多显示10个
|
||
|
||
# 映射成功率
|
||
success_rate = (mapped_count / total_count * 100) if total_count > 0 else 0
|
||
print(f'映射成功率: {success_rate:.1f}%')
|
||
|
||
return result
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|