此前长期处于未跟踪状态的资产,属 P0-7「版本控制失效」的同类隐患——
一次 git clean -fd 即不可恢复。
1. Scripts/bin/backfill_stock_daily.py(88 行)
个股日线历史回填,按 OVERRIDE_DATE 逐日补 akshare_data.stock_daily。
文档说明了两个关键判据:只跑 stock_daily 不做板块快照(后者的接口只返回
实时数据,用于历史回填等于用今天快照伪造历史行情,会污染题材分析);
以及运行前需清除注入的 http_proxy(否则国内源 ProxyError)。
⚠️ 已知问题(未在本轮修改):第 15 行硬编码
/Applications/FinDataHub.app/Contents/Resources/akshare_fetcher/src
与跨平台方向冲突,应改为经 paths.py 解析。因其为部署环境专用脚本,
本轮不做无验证的改动,留作待办。
2. FinDataHub-应用功能设计与模块结构方案-2026-10-17.html(37K)
10-17 的功能与模块结构设计文档。
573 lines
37 KiB
HTML
573 lines
37 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>FinDataHub 应用功能设计与模块结构方案</title>
|
||
<style>
|
||
:root{
|
||
--bg:#0b0e14; --bg2:#0f141d; --panel:#141a26; --panel2:#1a2130;
|
||
--border:#243044; --border2:#2e3c55;
|
||
--text:#e6edf3; --text-dim:#9aa7b8; --text-faint:#6b7686;
|
||
--red:#e5534b; --red-bg:rgba(229,83,75,.12); --red-border:rgba(229,83,75,.4);
|
||
--green:#3fb950; --green-bg:rgba(63,185,80,.12); --green-border:rgba(63,185,80,.35);
|
||
--amber:#d29922; --amber-bg:rgba(210,153,34,.12); --amber-border:rgba(210,153,34,.4);
|
||
--blue:#58a6ff; --blue-bg:rgba(88,166,255,.12); --blue-border:rgba(88,166,255,.35);
|
||
--purple:#bc8cff; --cyan:#39c5cf;
|
||
--l0:#2f81f7; --l1:#39c5cf; --l2:#a371f7; --l3:#3fb950; --l4:#e3b341; --l5:#e5534b;
|
||
--mono:"SF Mono",ui-monospace,Menlo,Consolas,monospace;
|
||
}
|
||
*{box-sizing:border-box;margin:0;padding:0}
|
||
html{scroll-behavior:smooth}
|
||
body{background:var(--bg);color:var(--text);font-family:-apple-system,BlinkMacSystemFont,"PingFang SC","Microsoft YaHei",sans-serif;line-height:1.7;font-size:15px;-webkit-font-smoothing:antialiased}
|
||
.wrap{max-width:1180px;margin:0 auto;padding:0 28px 80px}
|
||
/* header */
|
||
header{background:linear-gradient(135deg,#0d1b2a 0%,#1b2a3f 45%,#0f1420 100%);border-bottom:1px solid var(--border);padding:52px 0 40px;margin-bottom:34px}
|
||
.h-inner{max-width:1180px;margin:0 auto;padding:0 28px}
|
||
.kicker{display:inline-flex;align-items:center;gap:8px;font-size:12.5px;letter-spacing:.14em;color:var(--blue);text-transform:uppercase;margin-bottom:16px}
|
||
.kicker .dot{width:7px;height:7px;border-radius:50%;background:var(--blue);box-shadow:0 0 10px var(--blue)}
|
||
h1{font-size:30px;font-weight:700;letter-spacing:-.02em;margin-bottom:12px}
|
||
.sub{font-size:14.5px;color:var(--text-dim);max-width:760px}
|
||
.meta{display:flex;flex-wrap:wrap;gap:10px;margin-top:22px}
|
||
.chip{font-size:12px;padding:5px 12px;border-radius:20px;border:1px solid var(--border2);background:rgba(255,255,255,.03);color:var(--text-dim)}
|
||
.chip b{color:var(--text);font-weight:600}
|
||
/* section */
|
||
section{margin-bottom:42px}
|
||
.sec-h{display:flex;align-items:center;gap:12px;margin-bottom:18px;padding-bottom:12px;border-bottom:1px solid var(--border)}
|
||
.sec-num{font-size:13px;font-weight:700;color:var(--blue);font-family:var(--mono);background:var(--blue-bg);border:1px solid var(--blue-border);border-radius:8px;padding:3px 10px;white-space:nowrap}
|
||
.sec-h h2{font-size:20px;font-weight:700;letter-spacing:-.01em}
|
||
/* cards */
|
||
.grid{display:grid;gap:16px}
|
||
.g2{grid-template-columns:repeat(auto-fit,minmax(340px,1fr))}
|
||
.g3{grid-template-columns:repeat(auto-fit,minmax(280px,1fr))}
|
||
.card{background:var(--panel);border:1px solid var(--border);border-radius:14px;padding:22px}
|
||
.card h3{font-size:16px;font-weight:600;margin-bottom:12px;display:flex;align-items:center;gap:8px}
|
||
.card p{color:var(--text-dim);font-size:14px}
|
||
.lead{font-size:15.5px;color:var(--text);margin-bottom:18px}
|
||
.lead strong{color:var(--blue)}
|
||
/* verdict banner */
|
||
.verdict{background:linear-gradient(135deg,var(--red-bg),transparent 60%);border:1px solid var(--red-border);border-left:4px solid var(--red);border-radius:12px;padding:20px 24px;margin-bottom:20px}
|
||
.verdict .t{font-weight:700;font-size:16px;color:var(--red);margin-bottom:8px}
|
||
.verdict p{color:var(--text-dim);font-size:14.5px}
|
||
/* table */
|
||
table{width:100%;border-collapse:collapse;font-size:13.5px}
|
||
th{background:var(--panel2);color:var(--text-dim);font-weight:600;text-align:left;padding:10px 12px;border:1px solid var(--border);white-space:nowrap}
|
||
td{padding:10px 12px;border:1px solid var(--border);vertical-align:top;color:var(--text)}
|
||
tr:nth-child(even) td{background:rgba(255,255,255,.015)}
|
||
.tbl-wrap{overflow-x:auto;border:1px solid var(--border);border-radius:12px;background:var(--panel)}
|
||
.tag{display:inline-block;font-size:11.5px;font-weight:600;padding:2px 9px;border-radius:10px;white-space:nowrap}
|
||
.tag-red{background:var(--red-bg);color:var(--red);border:1px solid var(--red-border)}
|
||
.tag-green{background:var(--green-bg);color:var(--green);border:1px solid var(--green-border)}
|
||
.tag-amber{background:var(--amber-bg);color:var(--amber);border:1px solid var(--amber-border)}
|
||
.tag-blue{background:var(--blue-bg);color:var(--blue);border:1px solid var(--blue-border)}
|
||
.mono{font-family:var(--mono);font-size:12.5px}
|
||
/* layer stack */
|
||
.layer{border-radius:14px;border:1px solid var(--border2);overflow:hidden;margin-bottom:14px;background:var(--panel)}
|
||
.layer-h{display:flex;align-items:center;gap:12px;padding:14px 18px;border-bottom:1px solid var(--border)}
|
||
.layer-badge{font-size:12px;font-weight:700;font-family:var(--mono);color:#fff;padding:4px 11px;border-radius:8px;white-space:nowrap}
|
||
.layer-name{font-weight:700;font-size:15.5px}
|
||
.layer-desc{font-size:12.5px;color:var(--text-dim);margin-left:auto;text-align:right}
|
||
.layer-b{padding:14px 18px;display:flex;flex-wrap:wrap;gap:9px}
|
||
.mod{background:var(--panel2);border:1px solid var(--border);border-radius:9px;padding:9px 13px;font-size:13px}
|
||
.mod b{font-weight:600}
|
||
.mod .role{display:block;font-size:11.5px;color:var(--text-faint);margin-top:2px}
|
||
.arrow{text-align:center;color:var(--text-faint);font-size:12px;margin:-4px 0 8px;font-family:var(--mono)}
|
||
.arrow span{color:var(--text-dim)}
|
||
/* svg deps */
|
||
.svgbox{background:var(--panel);border:1px solid var(--border);border-radius:14px;padding:20px;overflow-x:auto}
|
||
.svgbox svg{display:block;margin:0 auto;max-width:100%;height:auto}
|
||
/* callout */
|
||
.callout{border-radius:12px;padding:16px 20px;margin:14px 0;font-size:14px;border-left:4px solid}
|
||
.co-blue{background:var(--blue-bg);border-color:var(--blue)}
|
||
.co-amber{background:var(--amber-bg);border-color:var(--amber)}
|
||
.co-red{background:var(--red-bg);border-color:var(--red)}
|
||
.co-green{background:var(--green-bg);border-color:var(--green)}
|
||
.co-blue b{color:var(--blue)}.co-amber b{color:var(--amber)}.co-red b{color:var(--red)}.co-green b{color:var(--green)}
|
||
ul.clean{list-style:none;padding:0}
|
||
ul.clean li{padding:7px 0 7px 22px;position:relative;color:var(--text-dim);font-size:14px}
|
||
ul.clean li::before{content:"▸";position:absolute;left:4px;color:var(--blue)}
|
||
ul.clean li b{color:var(--text)}
|
||
.disclaimer{margin-top:48px;padding:18px 22px;border:1px dashed var(--border2);border-radius:12px;font-size:12.5px;color:var(--text-faint);background:var(--bg2)}
|
||
.toc{display:flex;flex-wrap:wrap;gap:8px;margin-bottom:28px}
|
||
.toc a{font-size:12.5px;color:var(--text-dim);text-decoration:none;border:1px solid var(--border2);border-radius:16px;padding:4px 12px;transition:.15s}
|
||
.toc a:hover{color:var(--text);border-color:var(--blue)}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<header>
|
||
<div class="h-inner">
|
||
<div class="kicker"><span class="dot"></span>Architecture Audit · 性能优先 · 模块化高复用</div>
|
||
<h1>FinDataHub 应用功能设计与模块结构方案</h1>
|
||
<p class="sub">以性能优先为核心前提,重构模块边界、收敛重复分叉、下沉公共能力。目标:单一职责、单向依赖、可复用底座,杜绝功能零散分布与重复开发。</p>
|
||
<div class="meta">
|
||
<span class="chip">审计日期 <b>2026-10-17</b></span>
|
||
<span class="chip">源码版本 <b>v1.5.2</b></span>
|
||
<span class="chip">Python <b>3.12</b>(全项目锁定)</span>
|
||
<span class="chip">Swift <b>19,616 行</b></span>
|
||
<span class="chip">Python 服务 <b>6 个 / 30,320 行</b></span>
|
||
<span class="chip">运维脚本 <b>39 个</b></span>
|
||
</div>
|
||
</div>
|
||
</header>
|
||
|
||
<div class="wrap">
|
||
|
||
<div class="toc">
|
||
<a href="#s1">① 审计结论</a>
|
||
<a href="#s2">② 核心功能清单与边界</a>
|
||
<a href="#s3">③ 模块职责与层级</a>
|
||
<a href="#s4">④ 依赖与交互</a>
|
||
<a href="#s5">⑤ 公共能力抽象</a>
|
||
<a href="#s6">⑥ 性能优先落实</a>
|
||
<a href="#s7">⑦ 收敛落地路径</a>
|
||
</div>
|
||
|
||
<!-- ============ 1 审计结论 ============ -->
|
||
<section id="s1">
|
||
<div class="sec-h"><span class="sec-num">01</span><h2>审计结论</h2></div>
|
||
|
||
<div class="verdict">
|
||
<div class="t">核心结论:功能已完整,但结构存在三类系统性重复分叉,且性能关键路径未收敛到统一底座</div>
|
||
<p>本项目「能跑、数据在库」,但低层代码缺陷清理之后暴露出的问题已从「写错代码」转为「写对代码但零散分布、重复开发、关键能力未在生产生效」。方案目标不是重写功能,而是<b>收敛结构</b>——把散落在三个入口、两套后端、三份副本里的能力归位到唯一归属模块。</p>
|
||
</div>
|
||
|
||
<div class="grid g3">
|
||
<div class="card">
|
||
<h3><span class="tag tag-red">分叉 1</span> 快讯双后端双采集器</h3>
|
||
<p class="mono" style="color:var(--text-dim)">newsflash_backend.py(Scripts/bin)↔ kuaixun-service/app<br>newsflash_collect.py(3 源)↔ extend/fetchers(12 源)</p>
|
||
<p style="margin-top:8px">快讯功能存在<b>两套后端实现</b>(`newsflash_backend.py` 注释自陈「来自 finance-terminal/kuaixun-service」,是历史分叉)与<b>两套采集器</b>(3 源直抓 vs 12 源管线)。同一域内双实现,修复必须双份、演进必然漂移。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-red">分叉 2</span> 同步三入口</h3>
|
||
<p class="mono" style="color:var(--text-dim)">sync_bridge.py ↔ pipeline_runner.py ↔ daily_update.sh</p>
|
||
<p style="margin-top:8px">生产实际走 launchd 的 `daily_update.sh` <b>直写 raw 库、不过门控</b>;`sync_bridge.py` 是 App 手动同步入口;`pipeline_runner.py`(号称替代前两者、带四级门控)<b>从未接入 launchd</b>。三条链路并存,治理能力在唯一带门控的入口上却不在生产路径。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-amber">分叉 3</span> 公共能力三处重复</h3>
|
||
<p class="mono" style="color:var(--text-dim)">cache / validator / reporter</p>
|
||
<p style="margin-top:8px">历史上 `cache.py`/`validator.py`/`reporter.py` 在 core、tushare、akshare 三处各存一份。tushare 侧已收敛(574→66 行),akshare 侧是<b>有差异的分支版本</b>,待甄别后统一。`db.py` 下沉 core 已成正确范式,需推广到全部公共模块。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="callout co-blue">
|
||
<b>方案主线(五条):</b>① 建立五层单向依赖架构;② 快讯域收敛到 kuaixun-service 单一实现;③ 同步域收敛到 pipeline_runner 单一入口;④ 公共能力全部下沉 findatahub-core;⑤ 性能关键路径(连接复用 / 批处理 / 心跳 / 缓存 / 索引)在架构层固化为契约。
|
||
</div>
|
||
</section>
|
||
|
||
<!-- ============ 2 核心功能清单与边界 ============ -->
|
||
<section id="s2">
|
||
<div class="sec-h"><span class="sec-num">02</span><h2>核心功能清单与边界说明</h2></div>
|
||
<p class="lead">按业务域划分 7 个功能域。每域明确「做什么」与「<strong>不做什么</strong>」——边界即契约,越界即为缺陷。</p>
|
||
|
||
<div class="tbl-wrap">
|
||
<table>
|
||
<thead><tr><th style="width:120px">功能域</th><th style="width:220px">核心功能</th><th>职责(做)</th><th>边界(不做)</th></tr></thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><b>数据采集</b><br><span class="tag tag-blue">L1 Ingestion</span></td>
|
||
<td>结构化行情采集<br><span class="mono">tushare_fetcher / akshare_fetcher</span></td>
|
||
<td>拉取日/周/月线、财务、资金流、股东、分红等 84+ 张表,落 raw 库(tushare_data / akshare_data)</td>
|
||
<td>不做清洗、不做门控、不做对外服务、不做推送</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>快讯采集</b><br><span class="tag tag-blue">L1 Ingestion</span></td>
|
||
<td>12 源快讯抓取<br><span class="mono">extend/fetchers/*</span></td>
|
||
<td>抓取 cls/新浪/财联社/华尔街见闻/格隆汇/金十等 12 源,进入去重管线</td>
|
||
<td>不做行情数据、不直接对外、不负责推送路由</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>数据治理</b><br><span class="tag tag-purple">L2 Governance</span></td>
|
||
<td>Pipeline DAG + 四级门控<br><span class="mono">pipeline / gate / cleaner</span></td>
|
||
<td>七步流水线编排;gate1 采集/gate2 清洗/gate3 入库/gate4 后置四级质量门控;清洗、插补、离群检测、双源交叉验证;产出 fin_data_clean</td>
|
||
<td>不做采集、不做对外服务;门控是 raw→clean 的唯一闸口</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>数据存储</b><br><span class="tag tag-green">L0 Infra</span></td>
|
||
<td>三库分层 + 连接池<br><span class="mono">db / db_pool</span></td>
|
||
<td>raw(tushare/akshare)→ clean 三级库;连接池、幂等 upsert、日期/排序规则护栏</td>
|
||
<td>不含业务规则、不含字段语义,只提供读写基础设施</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>数据访问</b><br><span class="tag tag-green">L3 Access</span></td>
|
||
<td>SDK 统一查询<br><span class="mono">findatahub-client</span></td>
|
||
<td>统一查询/导出入口(client / api / mcp_server),隔离底层库细节,SQL 注入防护</td>
|
||
<td>不做采集、不做治理;是查询的唯一对外出口</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>快讯服务</b><br><span class="tag tag-amber">L4 Service</span></td>
|
||
<td>去重 + AI 增强 + 推送<br><span class="mono">kuaixun-service/app</span></td>
|
||
<td>四层去重、翻译/摘要/标签/评分、多渠道推送(Bark/钉钉/飞书/Telegram/企微)、REST(:8100) + WebSocket</td>
|
||
<td>不做行情、不做治理;是快讯全生命周期的唯一后端</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>应用客户端</b><br><span class="tag tag-red">L5 UI</span></td>
|
||
<td>macOS 客户端 + Vue 前端<br><span class="mono">App (SwiftUI) / kuaixun-frontend</span></td>
|
||
<td>进程编排、同步触发、快讯展示、数据库管理、调度、监控、节能;对外呈现</td>
|
||
<td>不做采集/清洗逻辑本身,只编排子进程与展示数据</td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>运维可观测</b><br><span class="tag tag-green">横切</span></td>
|
||
<td>审计/迁移/监控<br><span class="mono">Scripts/bin + health/retry</span></td>
|
||
<td>调度覆盖审计、数据迁移、性能快照、健康监控、自动恢复、告警</td>
|
||
<td>横切能力,不参与业务数据流</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div class="callout co-amber">
|
||
<b>边界即护栏:</b>历史上「写对代码但越界」是主要事故源——`db.replace()` 被采集层误用导致增量清全表、清洗器判据命中 akshare 的 `vol==amount` 假数据、门控 reroute 不收敛整批丢弃。方案将「边界」作为一等公民写入模块契约,越界即测试失败。
|
||
</div>
|
||
</section>
|
||
|
||
<!-- ============ 3 模块职责与层级 ============ -->
|
||
<section id="s3">
|
||
<div class="sec-h"><span class="sec-num">03</span><h2>模块职责划分与层级关系</h2></div>
|
||
<p class="lead">五层单向依赖架构,<strong>上层依赖下层、下层不感知上层</strong>;findatahub-core 是横切公共底座,被 L1/L2/L3 共同依赖。</p>
|
||
|
||
<!-- 层 5 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l5)">L5</span>
|
||
<span class="layer-name">表现层 Presentation</span>
|
||
<span class="layer-desc">编排 + 展示,不含业务逻辑</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>App (SwiftUI)</b><span class="role">macOS 客户端 · 进程编排 · 同步触发 · 监控 · 节能</span></span>
|
||
<span class="mod"><b>kuaixun-frontend</b><span class="role">Vue3 快讯 Web 前端</span></span>
|
||
</div>
|
||
</div>
|
||
<div class="arrow">▲ 依赖向下 <span>────────────</span> ▲</div>
|
||
|
||
<!-- 层 4 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l4)">L4</span>
|
||
<span class="layer-name">应用服务层 Service</span>
|
||
<span class="layer-desc">快讯全生命周期唯一后端</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>kuaixun-service/app</b><span class="role">FastAPI · REST(:8100) · WebSocket · 鉴权</span></span>
|
||
<span class="mod"><b>kuaixun-service/extend</b><span class="role">采集管线 · 去重 · AI 增强 · 推送</span></span>
|
||
<span class="mod"><b>kuaixun-service/common</b><span class="role">配置 · 安全栈</span></span>
|
||
</div>
|
||
</div>
|
||
<div class="arrow">▲ 依赖向下 <span>────────────</span> ▲</div>
|
||
|
||
<!-- 层 3 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l3)">L3</span>
|
||
<span class="layer-name">数据访问层 Access</span>
|
||
<span class="layer-desc">统一查询出口</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>findatahub-client</b><span class="role">SDK · client · api · mcp_server · SQL 注入防护</span></span>
|
||
<span class="mod"><b>dal / dal_unified</b><span class="role">快讯查询数据访问层</span></span>
|
||
</div>
|
||
</div>
|
||
<div class="arrow">▲ 依赖向下 <span>────────────</span> ▲</div>
|
||
|
||
<!-- 层 2 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l2)">L2</span>
|
||
<span class="layer-name">数据治理层 Governance</span>
|
||
<span class="layer-desc">raw → clean 唯一闸口</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>pipeline</b><span class="role">七步 DAG 编排 · 并行调度</span></span>
|
||
<span class="mod"><b>gate</b><span class="role">四级质量门控 gate1-4</span></span>
|
||
<span class="mod"><b>cleaner / imputer / outlier</b><span class="role">清洗 · 插补 · 离群</span></span>
|
||
<span class="mod"><b>cross_validator</b><span class="role">双源交叉验证</span></span>
|
||
</div>
|
||
</div>
|
||
<div class="arrow">▲ 依赖向下 <span>────────────</span> ▲</div>
|
||
|
||
<!-- 层 1 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l1)">L1</span>
|
||
<span class="layer-name">数据采集层 Ingestion</span>
|
||
<span class="layer-desc">只拉取 + 落 raw 库</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>tushare_fetcher</b><span class="role">结构化行情采集(84+ 表)</span></span>
|
||
<span class="mod"><b>akshare_fetcher</b><span class="role">结构化行情 + 北交所 + THS</span></span>
|
||
<span class="mod"><b>extend/fetchers</b><span class="role">12 源快讯抓取</span></span>
|
||
</div>
|
||
</div>
|
||
<div class="arrow">▲ 依赖向下 <span>────────────</span> ▲</div>
|
||
|
||
<!-- 层 0 -->
|
||
<div class="layer">
|
||
<div class="layer-h">
|
||
<span class="layer-badge" style="background:var(--l0)">L0</span>
|
||
<span class="layer-name">基础设施层 Infra</span>
|
||
<span class="layer-desc">读写 + 可靠性底座</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>db / db_pool</b><span class="role">连接池 · upsert · 日期/排序护栏</span></span>
|
||
<span class="mod"><b>cache</b><span class="role">SyncCache · TradeDateCache</span></span>
|
||
<span class="mod"><b>retry / health</b><span class="role">重试 · 健康监控</span></span>
|
||
<span class="mod"><b>security</b><span class="role">鉴权 · SQL/SSRF 防护</span></span>
|
||
<span class="mod"><b>MySQL</b><span class="role">tushare_data / akshare_data / fin_data_clean</span></span>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- 横切 -->
|
||
<div class="layer" style="border-color:var(--blue-border)">
|
||
<div class="layer-h" style="border-color:var(--blue-border)">
|
||
<span class="layer-badge" style="background:var(--blue)">CROSS</span>
|
||
<span class="layer-name">横切公共能力层(findatahub-core)</span>
|
||
<span class="layer-desc">被 L1/L2/L3 共同依赖的唯一底座</span>
|
||
</div>
|
||
<div class="layer-b">
|
||
<span class="mod"><b>db / cache / validator / reporter</b><span class="role">唯一实现,禁止各 fetcher 自持副本</span></span>
|
||
<span class="mod"><b>pipeline / gate / cleaner / cross_validator</b><span class="role">治理引擎(L2 的运行时)</span></span>
|
||
<span class="mod"><b>retry / health / stats / correlation</b><span class="role">横切可靠性 + 分析</span></span>
|
||
<span class="mod"><b>context / imputer / outlier / pca / profile</b><span class="role">流水线上下文 + 特征工程</span></span>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="callout co-green">
|
||
<b>层级铁律:</b>① 依赖只允许指向更低的层或横切底座,禁止反向、禁止跨层跳跃、禁止同级横向依赖;② findatahub-core 不依赖任何业务包(零业务 import);③ 每个 fetcher 对 core 的依赖收敛为「薄封装 + 源专属适配」,不复制逻辑。
|
||
</div>
|
||
</section>
|
||
|
||
<!-- ============ 4 依赖与交互 ============ -->
|
||
<section id="s4">
|
||
<div class="sec-h"><span class="sec-num">04</span><h2>模块间依赖与交互方式</h2></div>
|
||
|
||
<p class="lead">依赖分两种:<strong>代码依赖</strong>(import 关系,单向向下)与<strong>运行时交互</strong>(进程/网络/协议)。</p>
|
||
|
||
<div class="svgbox">
|
||
<svg viewBox="0 0 1080 620" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,PingFang SC,sans-serif">
|
||
<defs>
|
||
<marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||
<path d="M0 0 L10 5 L0 10 z" fill="#58a6ff"/>
|
||
</marker>
|
||
<marker id="arrg" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||
<path d="M0 0 L10 5 L0 10 z" fill="#3fb950"/>
|
||
</marker>
|
||
<marker id="arrr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||
<path d="M0 0 L10 5 L0 10 z" fill="#e5534b"/>
|
||
</marker>
|
||
</defs>
|
||
<!-- L5 -->
|
||
<rect x="40" y="30" width="1000" height="70" rx="12" fill="#141a26" stroke="#e5534b" stroke-opacity=".5"/>
|
||
<text x="60" y="58" fill="#e5534b" font-size="14" font-weight="700">L5 表现层</text>
|
||
<text x="60" y="78" fill="#9aa7b8" font-size="12">App (SwiftUI) · kuaixun-frontend (Vue3)</text>
|
||
<!-- L4 -->
|
||
<rect x="40" y="130" width="1000" height="70" rx="12" fill="#141a26" stroke="#e3b341" stroke-opacity=".5"/>
|
||
<text x="60" y="158" fill="#e3b341" font-size="14" font-weight="700">L4 服务层</text>
|
||
<text x="60" y="178" fill="#9aa7b8" font-size="12">kuaixun-service (app/extend/common) — FastAPI :8100 + WebSocket</text>
|
||
<!-- L3 -->
|
||
<rect x="40" y="230" width="1000" height="70" rx="12" fill="#141a26" stroke="#3fb950" stroke-opacity=".5"/>
|
||
<text x="60" y="258" fill="#3fb950" font-size="14" font-weight="700">L3 访问层</text>
|
||
<text x="60" y="278" fill="#9aa7b8" font-size="12">findatahub-client (SDK/MCP) · dal</text>
|
||
<!-- L2 -->
|
||
<rect x="40" y="330" width="1000" height="70" rx="12" fill="#141a26" stroke="#a371f7" stroke-opacity=".5"/>
|
||
<text x="60" y="358" fill="#a371f7" font-size="14" font-weight="700">L2 治理层</text>
|
||
<text x="60" y="378" fill="#9aa7b8" font-size="12">pipeline · gate · cleaner · cross_validator(raw → clean 唯一闸口)</text>
|
||
<!-- L1 -->
|
||
<rect x="40" y="430" width="1000" height="70" rx="12" fill="#141a26" stroke="#39c5cf" stroke-opacity=".5"/>
|
||
<text x="60" y="458" fill="#39c5cf" font-size="14" font-weight="700">L1 采集层</text>
|
||
<text x="60" y="478" fill="#9aa7b8" font-size="12">tushare_fetcher · akshare_fetcher · extend/fetchers (12 源)</text>
|
||
<!-- L0 -->
|
||
<rect x="40" y="530" width="1000" height="70" rx="12" fill="#141a26" stroke="#2f81f7" stroke-opacity=".5"/>
|
||
<text x="60" y="558" fill="#2f81f7" font-size="14" font-weight="700">L0 基础设施</text>
|
||
<text x="60" y="578" fill="#9aa7b8" font-size="12">db/db_pool · cache · retry/health · MySQL(三库)</text>
|
||
|
||
<!-- 代码依赖箭头(蓝色,上层→下层) -->
|
||
<path d="M520 100 L520 128" stroke="#58a6ff" stroke-width="1.5" marker-end="url(#arr)" fill="none" stroke-dasharray="4 3"/>
|
||
<path d="M520 200 L520 228" stroke="#58a6ff" stroke-width="1.5" marker-end="url(#arr)" fill="none" stroke-dasharray="4 3"/>
|
||
<path d="M520 300 L520 328" stroke="#58a6ff" stroke-width="1.5" marker-end="url(#arr)" fill="none" stroke-dasharray="4 3"/>
|
||
<path d="M520 400 L520 428" stroke="#58a6ff" stroke-width="1.5" marker-end="url(#arr)" fill="none" stroke-dasharray="4 3"/>
|
||
<path d="M520 500 L520 528" stroke="#58a6ff" stroke-width="1.5" marker-end="url(#arr)" fill="none" stroke-dasharray="4 3"/>
|
||
<text x="532" y="118" fill="#58a6ff" font-size="11">import</text>
|
||
<text x="532" y="218" fill="#58a6ff" font-size="11">import</text>
|
||
<text x="532" y="318" fill="#58a6ff" font-size="11">import</text>
|
||
<text x="532" y="418" fill="#58a6ff" font-size="11">import</text>
|
||
<text x="532" y="518" fill="#58a6ff" font-size="11">import</text>
|
||
|
||
<!-- 运行时交互(右侧说明) -->
|
||
<rect x="760" y="30" width="270" height="570" rx="12" fill="rgba(88,166,255,.04)" stroke="#243044"/>
|
||
<text x="780" y="58" fill="#e6edf3" font-size="13" font-weight="700">运行时交互(非 import)</text>
|
||
<line x1="780" y1="70" x2="1010" y2="70" stroke="#243044"/>
|
||
<text x="780" y="100" fill="#58a6ff" font-size="12">■ App ↔ Python 子进程</text>
|
||
<text x="780" y="120" fill="#9aa7b8" font-size="11">PythonProcessManager spawn,stdout JSON 协议 + PID 文件</text>
|
||
<text x="780" y="150" fill="#3fb950" font-size="12">■ launchd → pipeline_runner</text>
|
||
<text x="780" y="170" fill="#9aa7b8" font-size="11">日更定时任务,经 L2 门控写 clean(收敛后)</text>
|
||
<text x="780" y="200" fill="#e3b341" font-size="12">■ 前端 ↔ 服务 HTTP/WS</text>
|
||
<text x="780" y="220" fill="#9aa7b8" font-size="11">REST :8100 + WebSocket 实时推送</text>
|
||
<text x="780" y="250" fill="#a371f7" font-size="12">■ SDK ↔ 服务 / 直连 DB</text>
|
||
<text x="780" y="270" fill="#9aa7b8" font-size="11">client 经 API 或直连 MySQL(统一出口)</text>
|
||
<text x="780" y="300" fill="#39c5cf" font-size="12">■ 采集器 ↔ 上游 API</text>
|
||
<text x="780" y="320" fill="#9aa7b8" font-size="11">tushare/akshare/各快讯源,限流 + 重试</text>
|
||
<text x="780" y="350" fill="#2f81f7" font-size="12">■ 所有写库经 db 护栏</text>
|
||
<text x="780" y="370" fill="#9aa7b8" font-size="11">日期规范化 / 排序规则 / 幂等 upsert</text>
|
||
<text x="780" y="410" fill="#e5534b" font-size="12">■ 收敛前禁止的交互(红线)</text>
|
||
<text x="780" y="432" fill="#e5534b" font-size="11">daily_update.sh 直写 raw 不过门控</text>
|
||
<text x="780" y="452" fill="#e5534b" font-size="11">newsflash_backend.py 与 kuaixun-service 双活</text>
|
||
<text x="780" y="472" fill="#e5534b" font-size="11">sync_bridge.py 与 pipeline_runner 双入口</text>
|
||
<text x="780" y="500" fill="#9aa7b8" font-size="11">—— 这三条是本次收敛目标 ——</text>
|
||
</svg>
|
||
</div>
|
||
|
||
<div class="callout co-red">
|
||
<b>三条红色交互(收敛目标):</b>① 生产日更绕过门控直写 raw;② 快讯双后端并存;③ 同步双入口。方案落地后,运行时交互唯一合法形态为:<span class="mono">launchd → pipeline_runner → 门控 → clean</span>,快讯唯一后端 <span class="mono">kuaixun-service</span>。
|
||
</div>
|
||
</section>
|
||
|
||
<!-- ============ 5 公共能力抽象 ============ -->
|
||
<section id="s5">
|
||
<div class="sec-h"><span class="sec-num">05</span><h2>可复用公共能力与组件抽象</h2></div>
|
||
<p class="lead">findatahub-core 是唯一公共底座。所有被 2 个以上消费者使用的模块必须下沉 core,各 fetcher 只保留<b>源专属适配层</b>。</p>
|
||
|
||
<div class="tbl-wrap">
|
||
<table>
|
||
<thead><tr><th style="width:170px">公共能力</th><th style="width:230px">组件</th><th>职责</th><th style="width:180px">当前状态</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>数据库访问</td><td class="mono">BaseDatabaseManager · safe_identifier</td><td>连接池、幂等 upsert/replace、日期规范化护栏、排序规则、表结构迁移</td><td><span class="tag tag-green">已下沉</span> akshare/tushare 薄封装</td></tr>
|
||
<tr><td>数据校验</td><td class="mono">DataValidator · ValidationReport</td><td>字段完整性、类型、OHLC 规则校验</td><td><span class="tag tag-green">已下沉</span> tushare 收敛,akshare 待甄别</td></tr>
|
||
<tr><td>报告生成</td><td class="mono">ReportGenerator · SyncReport</td><td>同步结果报告、表级统计</td><td><span class="tag tag-green">已下沉</span> 同上</td></tr>
|
||
<tr><td>缓存</td><td class="mono">SyncCache · TradeDateCache</td><td>同步状态缓存、交易日历缓存(减少重复同步与 DB 查询)</td><td><span class="tag tag-green">已下沉</span></td></tr>
|
||
<tr><td>流水线编排</td><td class="mono">Pipeline · Step · FetchStepProtocol</td><td>DAG 编排、并行调度、上下文传递</td><td><span class="tag tag-amber">在 core,未接入生产</span></td></tr>
|
||
<tr><td>质量门控</td><td class="mono">gate1-4 · execute_gate</td><td>采集/清洗/入库/后置四级门控</td><td><span class="tag tag-amber">在 core,未接入生产</span></td></tr>
|
||
<tr><td>清洗特征</td><td class="mono">DataCleaner · DataImputer · OutlierDetector</td><td>清洗、缺失插补、离群检测</td><td><span class="tag tag-green">在 core</span></td></tr>
|
||
<tr><td>交叉验证</td><td class="mono">CrossValidator</td><td>双源数据一致性校验</td><td><span class="tag tag-green">在 core</span></td></tr>
|
||
<tr><td>重试/健康</td><td class="mono">RetryManager · HealthMonitor</td><td>分类重试、限流、健康监控</td><td><span class="tag tag-green">在 core</span></td></tr>
|
||
<tr><td>分析统计</td><td class="mono">StatsAnalyzer · CorrelationAnalyzer · PCA</td><td>统计、相关性、降维</td><td><span class="tag tag-green">在 core</span></td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div class="grid g2" style="margin-top:16px">
|
||
<div class="card">
|
||
<h3>下沉范式(db.py 已验证)</h3>
|
||
<p class="mono" style="font-size:12.5px">tushare_fetcher/core/db.py = 42 行薄封装<br>akshare_fetcher/core/db.py = 31 行薄封装</p>
|
||
<p style="margin-top:8px">fetcher 侧只做:<b>继承 + 源专属表结构适配</b>。逻辑全部在 core,改一处全仓生效。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3>组件抽象四原则</h3>
|
||
<ul class="clean">
|
||
<li><b>单一实现</b>:一个能力只存在 core 一处,消费者 import 不复制</li>
|
||
<li><b>薄封装</b>:fetcher 侧仅适配源差异,不重写逻辑</li>
|
||
<li><b>契约先行</b>:公共组件带接口契约测试(如 ensure_market_table)</li>
|
||
<li><b>别名收敛</b>:历史同名文件用 <span class="mono">sys.modules</span> 别名归位,非 from-import 转发(防 monkeypatch 失效)</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<!-- ============ 6 性能优先落实 ============ -->
|
||
<section id="s6">
|
||
<div class="sec-h"><span class="sec-num">06</span><h2>性能优先在架构层面的落实要点</h2></div>
|
||
<p class="lead">性能不是事后调优,而是<b>架构契约</b>。以下要点逐一映射到层与组件,越界即回归失败。</p>
|
||
|
||
<div class="grid g2">
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P1</span> 连接复用(L0)</h3>
|
||
<p>去重写入、批量同步全部<b>复用一个连接</b>,禁止每条新建 4~5 次 TCP 握手(历史 500 条≈2000 次握手)。连接池统一超时 + rollback 归还。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P2</span> 幂等 upsert(L0/L1)</h3>
|
||
<p>增量路径强制 <span class="mono">upsert</span>,<span class="mono">replace</span> 仅限全量快照表。杜绝「增量 DELETE 全表」的 I/O 风暴与数据损毁。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P3</span> 批处理 + 事务边界(L2)</h3>
|
||
<p>Pipeline 整批写、单事务提交,避免逐行 commit;并行步骤产物按步骤名隔离,防串源重算。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P4</span> 心跳替代 DB 轮询(L5)</h3>
|
||
<p>进程健康用<b>心跳文件</b>判定,不用「最后写库时间」(历史 300s 阈值误判 193 次/日)。交易/非交易时段差异化节能退避(30s→3600s)。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P5</span> 缓存层(L0)</h3>
|
||
<p>TradeDateCache 缓存交易日历、SyncCache 缓存同步状态,避免重复拉取与重复同步;MySQL 缓冲池 2GB 命中优先。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P6</span> 索引与查询策略(L3/L4)</h3>
|
||
<p>搜索保留 LIKE 连续子串语义(正确优先),拒绝 ngram 全文索引静默丢结果;<span class="mono">total</span> 单独 COUNT,不扫全表。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P7</span> 异步 I/O(L4)</h3>
|
||
<p>快讯采集、内容增强、推送走 asyncio 并发;限流 + 分类重试(限频/网络/业务分类退避),避免串行阻塞。</p>
|
||
</div>
|
||
<div class="card">
|
||
<h3><span class="tag tag-blue">P8</span> 资源按需启停(L5)</h3>
|
||
<p>MySQLOnDemandManager / PowerSaver:非交易时段降频、按需拉起 mysqld,主进程 UNIX 调用从 5092/s 降至 0.2/s。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="tbl-wrap" style="margin-top:16px">
|
||
<table>
|
||
<thead><tr><th>性能基线</th><th>非交易时段</th><th>交易时段(优化前)</th><th>交易时段(优化后)</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>主进程 UNIX 调用/s</td><td>0.2–33</td><td class="mono" style="color:var(--red)">5092.5</td><td class="mono" style="color:var(--green)">0.2</td></tr>
|
||
<tr><td>采集器 UNIX 调用/s</td><td>0.2</td><td class="mono" style="color:var(--red)">658.1</td><td class="mono" style="color:var(--green)">1.6</td></tr>
|
||
<tr><td>MySQL UNIX 调用/s</td><td>42</td><td class="mono" style="color:var(--red)">8932</td><td class="mono" style="color:var(--green)">48.2</td></tr>
|
||
<tr><td>MySQL RSS</td><td>470 MB</td><td>1870 MB</td><td>2959 MB(缓冲池 2GB + 数据增长,非泄漏)</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<p style="font-size:12.5px;color:var(--text-faint);margin-top:8px">数据来源:`Scripts/bin/perf_snapshot.py`(libproc 采样)。交易/非交易时段必须分别验证,不可直接比较——交易时段高负载是常态非泄漏。</p>
|
||
</section>
|
||
|
||
<!-- ============ 7 收敛落地路径 ============ -->
|
||
<section id="s7">
|
||
<div class="sec-h"><span class="sec-num">07</span><h2>收敛落地路径(结构重构)</h2></div>
|
||
|
||
<div class="tbl-wrap">
|
||
<table>
|
||
<thead><tr><th style="width:60px">批次</th><th style="width:220px">动作</th><th>内容</th><th style="width:140px">优先级</th></tr></thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><b>第 1 批</b></td>
|
||
<td>快讯域收敛</td>
|
||
<td>废弃 <span class="mono">Scripts/bin/newsflash_backend.py</span>(历史分叉),统一到 kuaixun-service;废弃 3 源 <span class="mono">newsflash_collect.py</span>,统一到 extend/fetchers 12 源管线</td>
|
||
<td><span class="tag tag-red">P0 结构</span></td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>第 2 批</b></td>
|
||
<td>同步域收敛</td>
|
||
<td><span class="mono">pipeline_runner.py</span> 装入 launchd,成为 fin_data_clean 唯一写入方;<span class="mono">daily_update.sh</span> 降级为 raw 采集(本就该是 raw);<span class="mono">sync_bridge.py</span> 收敛为 pipeline 的 App 触发入口</td>
|
||
<td><span class="tag tag-red">P0 结构</span></td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>第 3 批</b></td>
|
||
<td>公共能力收敛</td>
|
||
<td>甄别 akshare 侧 cache/validator/reporter 分支版本,合并到 core;治理引擎(pipeline/gate)确认唯一运行时位置</td>
|
||
<td><span class="tag tag-amber">P1</span></td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>第 4 批</b></td>
|
||
<td>超大文件拆分</td>
|
||
<td><span class="mono">akshare fetcher.py</span>(1035)、<span class="mono">newsflash_collect.py</span>(962)、<span class="mono">PythonProcessManager.swift</span>(842)、<span class="mono">DatabaseSetupGuideView.swift</span>(807) 按职责拆 extension</td>
|
||
<td><span class="tag tag-amber">P1</span></td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>第 5 批</b></td>
|
||
<td>契约测试补全</td>
|
||
<td>为公共组件(db/validator/gate/pipeline)补接口契约测试,锁定「边界即护栏」,越界即失败</td>
|
||
<td><span class="tag tag-blue">P2</span></td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div class="callout co-blue">
|
||
<b>落地原则:</b>结构收敛与既有缺陷修复并行,不改功能语义、只改归属。每批先补能证伪的回归用例(确认旧结构下用例失败、新结构下通过),再执行收敛,最后跑全量测试(当前基线 1147 passed)。Swift 侧改动须经 <span class="mono">swift build --disable-sandbox</span> 编译验证后打包部署。
|
||
</div>
|
||
</section>
|
||
|
||
<div class="disclaimer">
|
||
本方案基于 2026-10-17 对 FinDataHub 全仓(App Swift 19,616 行 / 6 个 Python 服务 30,320 行 / 39 个运维脚本)的静态结构取证与历史审计结论(v1.5.2)生成。所引性能数据来自 `perf_snapshot.py`(libproc 采样)与历史审计记录,交易/非交易时段已分别标注。本方案为架构设计与模块结构建议,不涉及任何投资标的判断。<br><br>
|
||
仅供参考,不构成投资建议。
|
||
</div>
|
||
|
||
</div>
|
||
</body>
|
||
</html>
|