feat(market-intel): 新增既有資料橋接預覽
All checks were successful
CD Pipeline / deploy (push) Successful in 1m2s

This commit is contained in:
OoO
2026-05-18 14:19:43 +08:00
parent c021945047
commit bb6a862dbe
9 changed files with 762 additions and 4 deletions

View File

@@ -78,6 +78,7 @@
- 正式環境 V10.101 已白名單部署:備份 `/tmp/codex_market_intel_v10100_predeploy_20260513_103809.tgz`、`/tmp/codex_market_intel_v10101_predeploy_20260513_104053.tgz` 與 seed 前快照 `/tmp/codex_market_platforms_v10100_before_20260513_103809.json`;僅 recreate `momo-app`,未碰 `momo-db`、未使用 `--remove-orphans`。CLI seed 首次因 timestamp not-null rollbackV10.101 修正後成功 insert `momo/pchome/coupang/shopee` 四筆 seedread-only diff 顯示 existing=4、missing=0、matching=4正式頁 console error 0。
- 2026-05-13 Codex 只讀複核:`https://mo.wooo.work/health` 回報 V10.111`/api/market_intel/status` 仍為 `enabled=false`、`crawler_enabled=false`、`write_enabled=false`、`dry_run_only=true`、phase 26`/api/market_intel/platform_seed_db_diff?execute=true&platform=all` 只讀確認 `market_platforms` 已有 `momo/pchome/coupang/shopee` 四筆,`existing_seed_count=4`、`missing_codes=[]`、`database_write_executed=false`。
- 注意:正式端 `/api/market_intel/seed_writer_cli_status?execute=true&platform=all` 仍回傳舊版 `approval_token_hint=APPROVED_MARKET_INTEL_SEED_WRITE_V1` 與固定 token gate 文案,與 main 已入庫的一次性環境 token hardening 不一致;下次正式白名單部署需優先同步 `services/market_intel/seed_writer_cli.py`、`services/market_intel/service.py`、`scripts/market_intel_seed_writer.py`,並 smoke 確認不再回吐 `approval_token_hint`。
- Phase 27 legacy source bridge preview新增 `services/market_intel/legacy_source_bridge.py` 與 `/api/market_intel/legacy_source_bridge`,只讀盤點既有 `promo_products`、`competitor_prices`、`competitor_price_history`,產生導入 `market_*` 的 mapping / dedupe / blocked operation preview預設 `execute=false` 不連 DB`execute=true` 也只做 read-only query不寫 DB、不建立 ORM session、不連外、不掛 schedulerUI 新增「既有資料橋接預覽」panel版本同步至 V10.182。
- Schema smoke`tests/test_market_intel_skeleton.py` 檢查 `Base.metadata` 內含 ADR-035 七張 `market_*` tables。
- Desktop UI QA本機只註冊 `market_intel_bp` 的 Flask harness 載入 `/market_intel`,確認 Phase 15、候選預覽、writer preview、安全 flags、點陣暖紙視覺正常console error 0。
- API QA`/api/market_intel/schema_smoke` 通過 7 張表與 `market_platforms` 必要欄位檢查;`/api/market_intel/platform_seed_writer_plan` 回傳 4 筆 dry-run upsert preview`writes_executed=false`,四平台皆 `blocked_dry_run_only`。

View File

@@ -320,7 +320,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
# ==========================================
# 系統版本與路徑
# ==========================================
SYSTEM_VERSION = "V10.180"
SYSTEM_VERSION = "V10.182"
LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
public_url = PUBLIC_URL # 用於模板顯示

View File

@@ -151,6 +151,7 @@ EwoooC 目前已有 MOMO EDM / 節慶活動資料、`promo_products`、PChome
- 2026-05-12 追加 read-only DB schema probe`/api/market_intel/schema_db_probe` 預設只回 planned不連 DB人工 smoke 才能以明確參數查正式 DB catalog。探針不得使用 `DatabaseManager()`,避免觸發 metadata `create_all()`;不得建立 ORM session、不得寫入、不得 commit。
- 2026-05-12 追加 platform seed DB diff probe`/api/market_intel/platform_seed_db_diff` 預設只回 planned不連 DB人工 smoke 才能以明確參數只讀查詢 `market_platforms`,比對 adapter seed 是否 missing / differs / matches。探針不得使用 `DatabaseManager()`、不得建立 ORM session、不得寫入、不得 commit。
- 2026-05-13 追加 platform seed CLI writer`scripts/market_intel_seed_writer.py` 可在 CLI 明確帶入 `--execute``--apply-real-write` 與確認 token 時,以 SQLAlchemy Core 短 transaction upsert `market_platforms`API 仍不得替使用者執行 DB 寫入,不建立 ORM session、不連外、不掛 scheduler。
- 2026-05-18 追加 legacy source bridge preview`/api/market_intel/legacy_source_bridge` 預設 `execute=false` 只回 planned不連 DB人工 smoke 才能以 `execute=true` 只讀盤點 `promo_products``competitor_prices``competitor_price_history`,產生舊資料導入 `market_*` 的 mapping、dedupe 與 blocked operation preview。此橋接不得寫入 DB、不得建立 ORM session、不得把 PChome 比價快取冒充為活動頁商品、不得掛 scheduler。
### Phase 4Coupang / Shopee Adapter

View File

@@ -19,7 +19,7 @@
| `edm_routes.py` | EDM 與節慶儀表板 | `/edm`, `/festival` |
| `monthly_routes.py` | 月結分析 | `/monthly_summary_analysis`, `/api/monthly_summary_data` |
| `daily_sales_routes.py` | 當日業績 | `/daily_sales`, `/daily_sales/export*` |
| `market_intel_routes.py` | 市場情報 Phase 26 platform seed CLI writer | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` |
| `market_intel_routes.py` | 市場情報 Phase 27 legacy source bridge preview | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/legacy_source_bridge`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` |
| `api_routes.py` | 通用任務與查詢 API | `/api/run_task`, `/api/history/*` |
| `export_routes.py` | 匯出功能 | `/api/export/*` |
| `import_routes.py` | 匯入功能 | `/api/import_excel`, `/api/import/monthly_summary` |

View File

@@ -94,6 +94,19 @@ def market_intel_platform_seed_db_diff():
)
@market_intel_bp.route("/api/market_intel/legacy_source_bridge")
@login_required
def market_intel_legacy_source_bridge():
execute_requested = request.args.get("execute", "false").lower() == "true"
sample_limit = request.args.get("limit", default=5, type=int)
return jsonify(
_service().build_legacy_source_bridge(
execute_requested=execute_requested,
sample_limit=sample_limit,
)
)
@market_intel_bp.route("/api/market_intel/adapters")
@login_required
def market_intel_adapters():

View File

@@ -0,0 +1,439 @@
"""既有資料來源橋接 preview。
本模組只讀取既有 EDM / PChome 資料表的摘要,產生未來導入 market_* 的
去重與映射計畫;不寫入 DB、不建立 ORM session、不掛 scheduler。
"""
from datetime import date, datetime
from decimal import Decimal
from sqlalchemy import create_engine, text
LEGACY_SOURCE_TABLES = (
{
"table": "promo_products",
"source_code": "momo_promo_products",
"platform_code": "momo",
"description": "既有 MOMO EDM / festival 活動商品資料,可作為 market_campaigns 與 market_campaign_products 的導入來源。",
"planned_targets": ["market_campaigns", "market_campaign_products"],
},
{
"table": "competitor_prices",
"source_code": "pchome_competitor_prices",
"platform_code": "pchome",
"description": "既有 PChome 最新比價快取,可作為商品比對與價格威脅背景,不直接冒充活動頁商品。",
"planned_targets": ["market_product_matches"],
},
{
"table": "competitor_price_history",
"source_code": "pchome_competitor_price_history",
"platform_code": "pchome",
"description": "既有 PChome 比價歷史,可在 market 商品與 campaign 建立後作為價格趨勢參考。",
"planned_targets": ["market_product_price_history"],
},
)
BRIDGE_OPERATIONS = (
{
"source_table": "promo_products",
"target_table": "market_campaigns",
"operation": "derive_momo_campaign_from_page_type_batch_and_activity_time",
"dedupe_key": "platform_code + campaign_key",
"write_status": "preview_only",
},
{
"source_table": "promo_products",
"target_table": "market_campaign_products",
"operation": "map_i_code_price_discount_url_into_campaign_product",
"dedupe_key": "campaign_id + platform_code + platform_product_id",
"write_status": "preview_only",
},
{
"source_table": "competitor_prices",
"target_table": "market_product_matches",
"operation": "reuse_pchome_match_score_as_review_seed",
"dedupe_key": "market_product_id + momo_i_code",
"write_status": "preview_only",
},
{
"source_table": "competitor_price_history",
"target_table": "market_product_price_history",
"operation": "defer_until_market_product_and_campaign_exist",
"dedupe_key": "platform_code + platform_product_id + crawled_at",
"write_status": "blocked_until_campaign_product_exists",
},
)
DUPLICATE_CONTROLS = (
{
"key": "momo_campaign_key",
"rule": "campaign_key 使用 momo:{page_type}:{batch_id 或 activity_time_text}:{time_slot},避免 EDM 批次重複建立活動。",
},
{
"key": "momo_product_key",
"rule": "platform_product_id 使用 promo_products.i_code並依 market_campaign_products unique key 去重。",
},
{
"key": "pchome_match_key",
"rule": "PChome 比價資料只進入比對候選,不直接建立活動商品,避免和未來 PChome 活動 crawler 重複。",
},
)
def _safe_value(value):
if isinstance(value, (datetime, date)):
return value.isoformat()
if isinstance(value, Decimal):
return float(value)
return value
def _row_to_dict(row):
return {
key: _safe_value(value)
for key, value in dict(row._mapping).items()
}
def _planned_source_summaries():
return [
{
**source,
"exists": False,
"row_count": 0,
"distinct_entity_count": 0,
"last_seen_at": None,
"breakdown": [],
"sample_rows": [],
"read_status": "planned_no_db_connection",
}
for source in LEGACY_SOURCE_TABLES
]
def _probe_postgresql_table(conn, table_name):
return bool(
conn.execute(
text(
"""
SELECT EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = ANY (current_schemas(false))
AND table_name = :table_name
)
"""
),
{"table_name": table_name},
).scalar()
)
def _probe_sqlite_table(conn, table_name):
return bool(
conn.execute(
text(
"""
SELECT 1
FROM sqlite_master
WHERE type = 'table'
AND name = :table_name
LIMIT 1
"""
),
{"table_name": table_name},
).fetchone()
)
def _table_exists(conn, table_name, database_type):
if database_type == "postgresql":
return _probe_postgresql_table(conn, table_name)
return _probe_sqlite_table(conn, table_name)
def _query_promo_products_summary(conn, sample_limit):
summary = _row_to_dict(
conn.execute(
text(
"""
SELECT
COUNT(*) AS row_count,
COUNT(DISTINCT batch_id) AS batch_count,
COUNT(DISTINCT i_code) AS distinct_entity_count,
MAX(crawled_at) AS last_seen_at
FROM promo_products
"""
)
).fetchone()
)
breakdown = [
_row_to_dict(row)
for row in conn.execute(
text(
"""
SELECT
COALESCE(page_type, 'unknown') AS source_key,
COUNT(*) AS row_count,
COUNT(DISTINCT i_code) AS distinct_entity_count,
MAX(crawled_at) AS last_seen_at
FROM promo_products
GROUP BY COALESCE(page_type, 'unknown')
ORDER BY row_count DESC
LIMIT :sample_limit
"""
),
{"sample_limit": sample_limit},
).fetchall()
]
samples = [
_row_to_dict(row)
for row in conn.execute(
text(
"""
SELECT
page_type,
batch_id,
i_code,
name,
price,
discount_text,
url,
crawled_at
FROM promo_products
ORDER BY crawled_at DESC
LIMIT :sample_limit
"""
),
{"sample_limit": sample_limit},
).fetchall()
]
return summary, breakdown, samples
def _query_competitor_prices_summary(conn, table_name, sample_limit):
summary = _row_to_dict(
conn.execute(
text(
f"""
SELECT
COUNT(*) AS row_count,
COUNT(DISTINCT sku) AS sku_count,
COUNT(DISTINCT competitor_product_id) AS distinct_entity_count,
MAX(crawled_at) AS last_seen_at
FROM {table_name}
"""
)
).fetchone()
)
breakdown = [
_row_to_dict(row)
for row in conn.execute(
text(
f"""
SELECT
COALESCE(source, 'unknown') AS source_key,
COUNT(*) AS row_count,
COUNT(DISTINCT sku) AS sku_count,
COUNT(DISTINCT competitor_product_id) AS distinct_entity_count,
MAX(crawled_at) AS last_seen_at
FROM {table_name}
GROUP BY COALESCE(source, 'unknown')
ORDER BY row_count DESC
LIMIT :sample_limit
"""
),
{"sample_limit": sample_limit},
).fetchall()
]
samples = [
_row_to_dict(row)
for row in conn.execute(
text(
f"""
SELECT
source,
sku,
competitor_product_id,
competitor_product_name,
price,
original_price,
match_score,
crawled_at
FROM {table_name}
ORDER BY crawled_at DESC
LIMIT :sample_limit
"""
),
{"sample_limit": sample_limit},
).fetchall()
]
return summary, breakdown, samples
def _query_source_summary(conn, source, database_type, sample_limit):
table_name = source["table"]
exists = _table_exists(conn, table_name, database_type)
if not exists:
return {
**source,
"exists": False,
"row_count": 0,
"distinct_entity_count": 0,
"last_seen_at": None,
"breakdown": [],
"sample_rows": [],
"read_status": "missing_table",
}
if table_name == "promo_products":
summary, breakdown, samples = _query_promo_products_summary(conn, sample_limit)
else:
summary, breakdown, samples = _query_competitor_prices_summary(
conn,
table_name,
sample_limit,
)
return {
**source,
"exists": True,
"row_count": int(summary.get("row_count") or 0),
"distinct_entity_count": int(summary.get("distinct_entity_count") or 0),
"last_seen_at": summary.get("last_seen_at"),
"metrics": summary,
"breakdown": breakdown,
"sample_rows": samples,
"read_status": "read_only_loaded",
}
def _build_result(
*,
mode,
execute_requested,
read_only_query_executed,
database_connection_opened,
source_summaries,
error_message=None,
):
existing_sources = [
item["table"] for item in source_summaries
if item.get("exists")
]
missing_sources = [
item["table"] for item in source_summaries
if not item.get("exists")
]
total_existing_rows = sum(int(item.get("row_count") or 0) for item in source_summaries)
blocked_reasons = ["legacy_bridge_preview_only", "market_intel_write_still_blocked"]
if not execute_requested:
blocked_reasons.insert(0, "execute_false_planned_only")
if missing_sources:
blocked_reasons.insert(0, "legacy_source_tables_missing")
if error_message:
blocked_reasons.insert(0, "legacy_source_bridge_error")
return {
"mode": mode,
"execute_requested": bool(execute_requested),
"read_only_query_executed": bool(read_only_query_executed),
"database_connection_opened": bool(database_connection_opened),
"database_session_created": False,
"explicit_transaction_opened": False,
"database_write_executed": False,
"database_commit_executed": False,
"external_network_executed": False,
"scheduler_attached": False,
"source_count": len(source_summaries),
"existing_source_count": len(existing_sources),
"existing_sources": existing_sources,
"missing_sources": missing_sources,
"source_tables_ready": not missing_sources if read_only_query_executed else False,
"total_existing_rows": total_existing_rows,
"source_summaries": source_summaries,
"bridge_operations": list(BRIDGE_OPERATIONS),
"duplicate_controls": list(DUPLICATE_CONTROLS),
"writes_executed": False,
"would_write_database": False,
"blocked_reasons": blocked_reasons,
"error_message": error_message,
}
def build_legacy_source_bridge_plan(
*,
execute_requested=False,
database_url=None,
database_type=None,
engine=None,
sample_limit=5,
):
"""建立既有資料來源橋接計畫;預設只回 planned不連 DB。"""
sample_limit = max(1, min(int(sample_limit or 5), 20))
if not execute_requested:
return _build_result(
mode="legacy_source_bridge_planned",
execute_requested=False,
read_only_query_executed=False,
database_connection_opened=False,
source_summaries=_planned_source_summaries(),
)
from config import DATABASE_PATH, DATABASE_TYPE
effective_database_type = (database_type or DATABASE_TYPE or "").lower()
effective_database_url = database_url or DATABASE_PATH
created_engine = False
connection_opened = False
try:
if engine is None:
connect_args = {}
if effective_database_type == "postgresql":
connect_args = {
"connect_timeout": 8,
"options": "-c statement_timeout=15000",
}
engine = create_engine(
effective_database_url,
isolation_level="AUTOCOMMIT",
pool_pre_ping=True,
connect_args=connect_args,
)
created_engine = True
with engine.connect() as conn:
connection_opened = True
source_summaries = [
_query_source_summary(
conn,
source,
effective_database_type,
sample_limit,
)
for source in LEGACY_SOURCE_TABLES
]
return _build_result(
mode="legacy_source_bridge_read_only",
execute_requested=True,
read_only_query_executed=True,
database_connection_opened=connection_opened,
source_summaries=source_summaries,
)
except Exception as exc:
return _build_result(
mode="legacy_source_bridge_error",
execute_requested=True,
read_only_query_executed=False,
database_connection_opened=connection_opened,
source_summaries=_planned_source_summaries(),
error_message=str(exc),
)
finally:
if created_engine:
engine.dispose()

View File

@@ -19,6 +19,7 @@ from services.market_intel.adapters import (
)
from services.market_intel.candidate_preview import build_candidate_preview_from_discovery
from services.market_intel.discovery_runner import ManualDiscoveryRunner
from services.market_intel.legacy_source_bridge import build_legacy_source_bridge_plan
from services.market_intel.migration_blueprint import build_migration_blueprint
from services.market_intel.platform_seed import build_platform_seed_rows
from services.market_intel.platform_seed_db_diff import build_platform_seed_db_diff_plan
@@ -62,7 +63,7 @@ class MarketIntelRuntimeStatus:
class MarketIntelService:
"""市場情報入口服務,先集中 feature gate 與安全狀態。"""
phase = "phase_26_platform_seed_cli_writer"
phase = "phase_27_legacy_source_bridge_preview"
def get_runtime_status(self) -> MarketIntelRuntimeStatus:
return MarketIntelRuntimeStatus(
@@ -279,6 +280,26 @@ class MarketIntelService:
diff["seed_plan_found"] = bool(seed_plan["found"])
return diff
def build_legacy_source_bridge(
self,
*,
execute_requested=False,
sample_limit=5,
engine=None,
database_url=None,
database_type=None,
):
"""回報既有 EDM/PChome 資料源橋接 preview預設不連 DB。"""
bridge = build_legacy_source_bridge_plan(
execute_requested=execute_requested,
sample_limit=sample_limit,
engine=engine,
database_url=database_url,
database_type=database_type,
)
bridge["phase"] = self.phase
return bridge
def build_platform_seed_writer_plan(self, platform_code="all"):
"""建立 platform seed writer dry-run plan不建立 DB session。"""
seed_plan = self.build_platform_seed_plan(platform_code=platform_code)
@@ -377,6 +398,9 @@ class MarketIntelService:
"platform_seed_db_diff_planned_safe": bool(
not self.build_platform_seed_db_diff()["read_only_query_executed"]
),
"legacy_source_bridge_planned_safe": bool(
not self.build_legacy_source_bridge()["read_only_query_executed"]
),
}
ready_for_production_deploy = all(checks.values())
blocked_reasons = [
@@ -499,6 +523,7 @@ class MarketIntelService:
"/api/market_intel/schema_smoke",
"/api/market_intel/schema_db_probe",
"/api/market_intel/platform_seed_db_diff",
"/api/market_intel/legacy_source_bridge",
],
"status": status.to_dict(),
"schema_smoke": schema_smoke,
@@ -512,4 +537,5 @@ class MarketIntelService:
"seed_writer_cli_status": self.build_seed_writer_cli_status(),
"schema_db_probe": self.build_schema_db_probe(),
"platform_seed_db_diff": self.build_platform_seed_db_diff(),
"legacy_source_bridge": self.build_legacy_source_bridge(),
}

View File

@@ -396,6 +396,24 @@
</div>
</div>
<div class="market-intel-panel" data-market-intel-legacy-bridge>
<div class="market-intel-preview-head">
<div>
<p class="market-intel-muted momo-mono mb-1">LEGACY SOURCE / BRIDGE PREVIEW</p>
<h2 class="market-intel-preview-title">既有資料橋接預覽</h2>
</div>
<button class="market-intel-icon-button" type="button" title="重新整理橋接預覽" data-market-intel-legacy-bridge-refresh>
<i class="fas fa-rotate-right" aria-hidden="true"></i>
</button>
</div>
<div class="market-intel-preview-meta" data-market-intel-legacy-bridge-meta>
<span class="market-intel-pill">loading</span>
</div>
<div data-market-intel-legacy-bridge-body>
<div class="market-intel-empty">讀取既有資料橋接預覽中...</div>
</div>
</div>
<div class="market-intel-panel" data-market-intel-migration>
<div class="market-intel-preview-head">
<div>
@@ -460,10 +478,11 @@
const cliRoot = document.querySelector('[data-market-intel-cli]');
const dbProbeRoot = document.querySelector('[data-market-intel-db-probe]');
const seedDiffRoot = document.querySelector('[data-market-intel-seed-diff]');
const legacyBridgeRoot = document.querySelector('[data-market-intel-legacy-bridge]');
const migrationRoot = document.querySelector('[data-market-intel-migration]');
const approvalRoot = document.querySelector('[data-market-intel-approval]');
const deployRoot = document.querySelector('[data-market-intel-deploy]');
if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !migrationRoot && !approvalRoot && !deployRoot) return;
if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !legacyBridgeRoot && !migrationRoot && !approvalRoot && !deployRoot) return;
const meta = root ? root.querySelector('[data-market-intel-preview-meta]') : null;
const body = root ? root.querySelector('[data-market-intel-preview-body]') : null;
@@ -485,6 +504,10 @@
const seedDiffBody = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-body]') : null;
const seedDiffRefresh = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-refresh]') : null;
const seedDiffEndpoint = "{{ url_for('market_intel.market_intel_platform_seed_db_diff') }}?execute=false";
const legacyBridgeMeta = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-meta]') : null;
const legacyBridgeBody = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-body]') : null;
const legacyBridgeRefresh = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-refresh]') : null;
const legacyBridgeEndpoint = "{{ url_for('market_intel.market_intel_legacy_source_bridge') }}?execute=false&limit=5";
const migrationMeta = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-meta]') : null;
const migrationBody = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-body]') : null;
const migrationRefresh = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-refresh]') : null;
@@ -730,6 +753,95 @@
}
};
const renderLegacyBridgeMeta = data => {
legacyBridgeMeta.innerHTML = [
`mode=${data.mode || 'unknown'}`,
`execute=${data.execute_requested ? 'true' : 'false'}`,
`query=${data.read_only_query_executed ? 'yes' : 'no'}`,
`sources=${data.existing_source_count || 0}/${data.source_count || 0}`,
`rows=${data.total_existing_rows || 0}`,
`writes=${data.writes_executed ? 'executed' : 'blocked'}`
].map(item => `<span class="market-intel-pill">${escapeHtml(item)}</span>`).join('');
};
const renderLegacyBridgeBody = data => {
const blockers = (data.blocked_reasons || []).join(' / ');
const sources = data.source_summaries || [];
const operations = data.bridge_operations || [];
const controls = data.duplicate_controls || [];
const renderSource = item => `
<div class="market-intel-check">
<div>
<strong>${escapeHtml(item.table)}</strong>
<small>${escapeHtml(item.description || '')}</small>
<small>target=${escapeHtml((item.planned_targets || []).join(' / '))}</small>
</div>
<span>${item.exists ? escapeHtml(item.row_count || 0) : 'PENDING'}</span>
</div>
`;
const renderOperation = item => `
<article class="market-intel-operation">
<strong>${escapeHtml(item.source_table)}${escapeHtml(item.target_table)}</strong>
<small>${escapeHtml(item.operation)} / ${escapeHtml(item.write_status)}</small>
<small>dedupe=${escapeHtml(item.dedupe_key)}</small>
</article>
`;
const renderControl = item => `
<div class="market-intel-check">
<div>
<strong>${escapeHtml(item.key)}</strong>
<small>${escapeHtml(item.rule)}</small>
</div>
<span>RULE</span>
</div>
`;
legacyBridgeBody.innerHTML = `
<div class="market-intel-empty mb-3">這裡只盤點既有資料來源與導入規則,不搬資料、不寫 market_*。${blockers ? `阻擋:${escapeHtml(blockers)}` : ''}</div>
<div class="market-intel-deploy-grid">
<div data-market-intel-legacy-sources>
<p class="market-intel-deploy-section-title">SOURCE TABLES</p>
<div class="market-intel-check-list">${
sources.length
? sources.map(renderSource).join('')
: '<div class="market-intel-empty">尚未提供來源摘要。</div>'
}</div>
</div>
<div data-market-intel-legacy-operations>
<p class="market-intel-deploy-section-title">BRIDGE OPERATIONS</p>
<div class="market-intel-operation-list">${
operations.length
? operations.map(renderOperation).join('')
: '<div class="market-intel-empty">尚未提供橋接操作。</div>'
}</div>
</div>
<div data-market-intel-legacy-controls>
<p class="market-intel-deploy-section-title">DUPLICATE CONTROLS</p>
<div class="market-intel-check-list">${
controls.length
? controls.map(renderControl).join('')
: '<div class="market-intel-empty">尚未提供去重規則。</div>'
}</div>
</div>
</div>
`;
};
const loadLegacyBridge = async () => {
if (!legacyBridgeMeta || !legacyBridgeBody) return;
legacyBridgeBody.innerHTML = '<div class="market-intel-empty">讀取既有資料橋接預覽中...</div>';
try {
const response = await fetch(legacyBridgeEndpoint, { credentials: 'same-origin' });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const data = await response.json();
renderLegacyBridgeMeta(data);
renderLegacyBridgeBody(data);
} catch (error) {
legacyBridgeMeta.innerHTML = '<span class="market-intel-pill">error</span>';
legacyBridgeBody.innerHTML = `<div class="market-intel-empty">既有資料橋接預覽讀取失敗:${escapeHtml(error.message)}</div>`;
}
};
const renderMigrationMeta = data => {
const seedWriter = data.command_plan && data.command_plan.seed_writer_command
? data.command_plan.seed_writer_command
@@ -979,6 +1091,9 @@
if (seedDiffRefresh) {
seedDiffRefresh.addEventListener('click', loadSeedDiff);
}
if (legacyBridgeRefresh) {
legacyBridgeRefresh.addEventListener('click', loadLegacyBridge);
}
if (migrationRefresh) {
migrationRefresh.addEventListener('click', loadMigration);
}
@@ -993,6 +1108,7 @@
loadCli();
loadDbProbe();
loadSeedDiff();
loadLegacyBridge();
loadMigration();
loadApproval();
loadDeploy();

View File

@@ -393,6 +393,7 @@ def test_market_intel_preview_template_uses_safe_fetch_false_endpoint():
assert "market_intel.market_intel_seed_writer_cli_status" in template
assert "market_intel.market_intel_schema_db_probe" in template
assert "market_intel.market_intel_platform_seed_db_diff" in template
assert "market_intel.market_intel_legacy_source_bridge" in template
assert "market_intel.market_intel_migration_blueprint" in template
assert "market_intel.market_intel_write_approval_runbook" in template
assert "market_intel.market_intel_deployment_readiness" in template
@@ -414,6 +415,164 @@ def test_market_intel_schema_metadata_contains_all_market_tables():
assert set(MARKET_INTEL_TABLES) <= metadata_tables
def test_legacy_source_bridge_default_is_planned_only():
bridge = MarketIntelService().build_legacy_source_bridge()
assert bridge["mode"] == "legacy_source_bridge_planned"
assert bridge["phase"] == "phase_27_legacy_source_bridge_preview"
assert bridge["execute_requested"] is False
assert bridge["read_only_query_executed"] is False
assert bridge["database_connection_opened"] is False
assert bridge["database_session_created"] is False
assert bridge["database_write_executed"] is False
assert bridge["database_commit_executed"] is False
assert bridge["external_network_executed"] is False
assert bridge["scheduler_attached"] is False
assert bridge["writes_executed"] is False
assert bridge["would_write_database"] is False
assert bridge["source_count"] == 3
assert bridge["existing_source_count"] == 0
assert "execute_false_planned_only" in bridge["blocked_reasons"]
assert {item["table"] for item in bridge["source_summaries"]} == {
"promo_products",
"competitor_prices",
"competitor_price_history",
}
def test_legacy_source_bridge_read_only_sqlite_counts_sources():
engine = create_engine("sqlite:///:memory:")
with engine.begin() as conn:
conn.execute(
text(
"""
CREATE TABLE promo_products (
id INTEGER PRIMARY KEY,
batch_id TEXT,
crawled_at TEXT,
time_slot TEXT,
activity_time_text TEXT,
session_time_text TEXT,
i_code TEXT,
name TEXT,
price INTEGER,
discount_text TEXT,
url TEXT,
image_url TEXT,
previous_price INTEGER,
remain_qty INTEGER,
status_change TEXT,
page_type TEXT
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE competitor_prices (
id INTEGER PRIMARY KEY,
sku TEXT,
source TEXT,
price NUMERIC,
original_price NUMERIC,
discount_pct INTEGER,
competitor_product_id TEXT,
competitor_product_name TEXT,
match_score NUMERIC,
crawled_at TEXT
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE competitor_price_history (
id INTEGER PRIMARY KEY,
sku TEXT,
source TEXT,
momo_product_id INTEGER,
momo_price NUMERIC,
price NUMERIC,
original_price NUMERIC,
discount_pct INTEGER,
competitor_product_id TEXT,
competitor_product_name TEXT,
match_score NUMERIC,
crawled_at TEXT
)
"""
)
)
conn.execute(
text(
"""
INSERT INTO promo_products
(batch_id, crawled_at, time_slot, i_code, name, price, discount_text, url, page_type)
VALUES
('batch-1', '2026-05-18 10:00:00', '10:00', 'A001', 'MOMO 商品 A', 399, '8折', 'https://momo/A001', 'edm'),
('batch-1', '2026-05-18 10:01:00', '10:00', 'A002', 'MOMO 商品 B', 499, '9折', 'https://momo/A002', 'edm')
"""
)
)
conn.execute(
text(
"""
INSERT INTO competitor_prices
(sku, source, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at)
VALUES
('A001', 'pchome', 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00')
"""
)
)
conn.execute(
text(
"""
INSERT INTO competitor_price_history
(sku, source, momo_product_id, momo_price, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at)
VALUES
('A001', 'pchome', 1, 399, 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00')
"""
)
)
bridge = MarketIntelService().build_legacy_source_bridge(
execute_requested=True,
engine=engine,
database_type="sqlite",
)
assert bridge["mode"] == "legacy_source_bridge_read_only"
assert bridge["execute_requested"] is True
assert bridge["read_only_query_executed"] is True
assert bridge["database_connection_opened"] is True
assert bridge["database_session_created"] is False
assert bridge["database_write_executed"] is False
assert bridge["database_commit_executed"] is False
assert bridge["external_network_executed"] is False
assert bridge["scheduler_attached"] is False
assert bridge["writes_executed"] is False
assert bridge["would_write_database"] is False
assert bridge["source_tables_ready"] is True
assert bridge["existing_source_count"] == 3
assert bridge["total_existing_rows"] == 4
assert [item["table"] for item in bridge["source_summaries"]] == [
"promo_products",
"competitor_prices",
"competitor_price_history",
]
assert bridge["source_summaries"][0]["row_count"] == 2
assert bridge["source_summaries"][0]["distinct_entity_count"] == 2
assert bridge["source_summaries"][1]["row_count"] == 1
assert bridge["source_summaries"][2]["row_count"] == 1
assert len(bridge["bridge_operations"]) == 4
assert all(
item["write_status"] != "executed"
for item in bridge["bridge_operations"]
)
def test_market_intel_schema_smoke_checks_platform_columns():
smoke = MarketIntelService().build_schema_smoke()["schema_smoke"]
@@ -661,6 +820,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
assert readiness["checks"]["schema_smoke_passed"] is True
assert readiness["checks"]["schema_db_probe_planned_safe"] is True
assert readiness["checks"]["platform_seed_db_diff_planned_safe"] is True
assert readiness["checks"]["legacy_source_bridge_planned_safe"] is True
assert readiness["checks"]["writer_plan_dry_run_only"] is True
assert readiness["writer_plan_summary"]["writes_executed"] is False
assert "readiness_checks_not_all_passed" not in readiness["blocked_reasons"]
@@ -675,6 +835,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
assert "/health" in readiness["production_smoke_targets"]
assert "/api/market_intel/deployment_readiness" in readiness["production_smoke_targets"]
assert "/api/market_intel/platform_seed_db_diff" in readiness["production_smoke_targets"]
assert "/api/market_intel/legacy_source_bridge" in readiness["production_smoke_targets"]
assert readiness["write_approval_runbook"]["ready_for_real_write"] is False
assert readiness["write_approval_runbook"]["writes_executed"] is False
assert readiness["migration_blueprint"]["migration_executed"] is False
@@ -682,6 +843,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
assert readiness["migration_blueprint"]["file_matches_blueprint"] is True
assert readiness["schema_db_probe"]["read_only_query_executed"] is False
assert readiness["platform_seed_db_diff"]["read_only_query_executed"] is False
assert readiness["legacy_source_bridge"]["read_only_query_executed"] is False
def test_write_approval_runbook_is_read_only_and_blocks_real_write():