feat(market-intel): 新增既有資料橋接預覽
All checks were successful
CD Pipeline / deploy (push) Successful in 1m2s
All checks were successful
CD Pipeline / deploy (push) Successful in 1m2s
This commit is contained in:
@@ -78,6 +78,7 @@
|
||||
- 正式環境 V10.101 已白名單部署:備份 `/tmp/codex_market_intel_v10100_predeploy_20260513_103809.tgz`、`/tmp/codex_market_intel_v10101_predeploy_20260513_104053.tgz` 與 seed 前快照 `/tmp/codex_market_platforms_v10100_before_20260513_103809.json`;僅 recreate `momo-app`,未碰 `momo-db`、未使用 `--remove-orphans`。CLI seed 首次因 timestamp not-null rollback,V10.101 修正後成功 insert `momo/pchome/coupang/shopee` 四筆 seed;read-only diff 顯示 existing=4、missing=0、matching=4,正式頁 console error 0。
|
||||
- 2026-05-13 Codex 只讀複核:`https://mo.wooo.work/health` 回報 V10.111;`/api/market_intel/status` 仍為 `enabled=false`、`crawler_enabled=false`、`write_enabled=false`、`dry_run_only=true`、phase 26;`/api/market_intel/platform_seed_db_diff?execute=true&platform=all` 只讀確認 `market_platforms` 已有 `momo/pchome/coupang/shopee` 四筆,`existing_seed_count=4`、`missing_codes=[]`、`database_write_executed=false`。
|
||||
- 注意:正式端 `/api/market_intel/seed_writer_cli_status?execute=true&platform=all` 仍回傳舊版 `approval_token_hint=APPROVED_MARKET_INTEL_SEED_WRITE_V1` 與固定 token gate 文案,與 main 已入庫的一次性環境 token hardening 不一致;下次正式白名單部署需優先同步 `services/market_intel/seed_writer_cli.py`、`services/market_intel/service.py`、`scripts/market_intel_seed_writer.py`,並 smoke 確認不再回吐 `approval_token_hint`。
|
||||
- Phase 27 legacy source bridge preview:新增 `services/market_intel/legacy_source_bridge.py` 與 `/api/market_intel/legacy_source_bridge`,只讀盤點既有 `promo_products`、`competitor_prices`、`competitor_price_history`,產生導入 `market_*` 的 mapping / dedupe / blocked operation preview;預設 `execute=false` 不連 DB,`execute=true` 也只做 read-only query,不寫 DB、不建立 ORM session、不連外、不掛 scheduler;UI 新增「既有資料橋接預覽」panel;版本同步至 V10.182。
|
||||
- Schema smoke:`tests/test_market_intel_skeleton.py` 檢查 `Base.metadata` 內含 ADR-035 七張 `market_*` tables。
|
||||
- Desktop UI QA:本機只註冊 `market_intel_bp` 的 Flask harness 載入 `/market_intel`,確認 Phase 15、候選預覽、writer preview、安全 flags、點陣暖紙視覺正常,console error 0。
|
||||
- API QA:`/api/market_intel/schema_smoke` 通過 7 張表與 `market_platforms` 必要欄位檢查;`/api/market_intel/platform_seed_writer_plan` 回傳 4 筆 dry-run upsert preview,`writes_executed=false`,四平台皆 `blocked_dry_run_only`。
|
||||
|
||||
@@ -320,7 +320,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
|
||||
# ==========================================
|
||||
# 系統版本與路徑
|
||||
# ==========================================
|
||||
SYSTEM_VERSION = "V10.180"
|
||||
SYSTEM_VERSION = "V10.182"
|
||||
LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
|
||||
public_url = PUBLIC_URL # 用於模板顯示
|
||||
|
||||
|
||||
@@ -151,6 +151,7 @@ EwoooC 目前已有 MOMO EDM / 節慶活動資料、`promo_products`、PChome
|
||||
- 2026-05-12 追加 read-only DB schema probe:`/api/market_intel/schema_db_probe` 預設只回 planned,不連 DB;人工 smoke 才能以明確參數查正式 DB catalog。探針不得使用 `DatabaseManager()`,避免觸發 metadata `create_all()`;不得建立 ORM session、不得寫入、不得 commit。
|
||||
- 2026-05-12 追加 platform seed DB diff probe:`/api/market_intel/platform_seed_db_diff` 預設只回 planned,不連 DB;人工 smoke 才能以明確參數只讀查詢 `market_platforms`,比對 adapter seed 是否 missing / differs / matches。探針不得使用 `DatabaseManager()`、不得建立 ORM session、不得寫入、不得 commit。
|
||||
- 2026-05-13 追加 platform seed CLI writer:`scripts/market_intel_seed_writer.py` 可在 CLI 明確帶入 `--execute`、`--apply-real-write` 與確認 token 時,以 SQLAlchemy Core 短 transaction upsert `market_platforms`;API 仍不得替使用者執行 DB 寫入,不建立 ORM session、不連外、不掛 scheduler。
|
||||
- 2026-05-18 追加 legacy source bridge preview:`/api/market_intel/legacy_source_bridge` 預設 `execute=false` 只回 planned,不連 DB;人工 smoke 才能以 `execute=true` 只讀盤點 `promo_products`、`competitor_prices`、`competitor_price_history`,產生舊資料導入 `market_*` 的 mapping、dedupe 與 blocked operation preview。此橋接不得寫入 DB、不得建立 ORM session、不得把 PChome 比價快取冒充為活動頁商品、不得掛 scheduler。
|
||||
|
||||
### Phase 4:Coupang / Shopee Adapter
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
| `edm_routes.py` | EDM 與節慶儀表板 | `/edm`, `/festival` |
|
||||
| `monthly_routes.py` | 月結分析 | `/monthly_summary_analysis`, `/api/monthly_summary_data` |
|
||||
| `daily_sales_routes.py` | 當日業績 | `/daily_sales`, `/daily_sales/export*` |
|
||||
| `market_intel_routes.py` | 市場情報 Phase 26 platform seed CLI writer | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` |
|
||||
| `market_intel_routes.py` | 市場情報 Phase 27 legacy source bridge preview | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/legacy_source_bridge`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` |
|
||||
| `api_routes.py` | 通用任務與查詢 API | `/api/run_task`, `/api/history/*` |
|
||||
| `export_routes.py` | 匯出功能 | `/api/export/*` |
|
||||
| `import_routes.py` | 匯入功能 | `/api/import_excel`, `/api/import/monthly_summary` |
|
||||
|
||||
@@ -94,6 +94,19 @@ def market_intel_platform_seed_db_diff():
|
||||
)
|
||||
|
||||
|
||||
@market_intel_bp.route("/api/market_intel/legacy_source_bridge")
|
||||
@login_required
|
||||
def market_intel_legacy_source_bridge():
|
||||
execute_requested = request.args.get("execute", "false").lower() == "true"
|
||||
sample_limit = request.args.get("limit", default=5, type=int)
|
||||
return jsonify(
|
||||
_service().build_legacy_source_bridge(
|
||||
execute_requested=execute_requested,
|
||||
sample_limit=sample_limit,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@market_intel_bp.route("/api/market_intel/adapters")
|
||||
@login_required
|
||||
def market_intel_adapters():
|
||||
|
||||
439
services/market_intel/legacy_source_bridge.py
Normal file
439
services/market_intel/legacy_source_bridge.py
Normal file
@@ -0,0 +1,439 @@
|
||||
"""既有資料來源橋接 preview。
|
||||
|
||||
本模組只讀取既有 EDM / PChome 資料表的摘要,產生未來導入 market_* 的
|
||||
去重與映射計畫;不寫入 DB、不建立 ORM session、不掛 scheduler。
|
||||
"""
|
||||
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
|
||||
|
||||
LEGACY_SOURCE_TABLES = (
|
||||
{
|
||||
"table": "promo_products",
|
||||
"source_code": "momo_promo_products",
|
||||
"platform_code": "momo",
|
||||
"description": "既有 MOMO EDM / festival 活動商品資料,可作為 market_campaigns 與 market_campaign_products 的導入來源。",
|
||||
"planned_targets": ["market_campaigns", "market_campaign_products"],
|
||||
},
|
||||
{
|
||||
"table": "competitor_prices",
|
||||
"source_code": "pchome_competitor_prices",
|
||||
"platform_code": "pchome",
|
||||
"description": "既有 PChome 最新比價快取,可作為商品比對與價格威脅背景,不直接冒充活動頁商品。",
|
||||
"planned_targets": ["market_product_matches"],
|
||||
},
|
||||
{
|
||||
"table": "competitor_price_history",
|
||||
"source_code": "pchome_competitor_price_history",
|
||||
"platform_code": "pchome",
|
||||
"description": "既有 PChome 比價歷史,可在 market 商品與 campaign 建立後作為價格趨勢參考。",
|
||||
"planned_targets": ["market_product_price_history"],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
BRIDGE_OPERATIONS = (
|
||||
{
|
||||
"source_table": "promo_products",
|
||||
"target_table": "market_campaigns",
|
||||
"operation": "derive_momo_campaign_from_page_type_batch_and_activity_time",
|
||||
"dedupe_key": "platform_code + campaign_key",
|
||||
"write_status": "preview_only",
|
||||
},
|
||||
{
|
||||
"source_table": "promo_products",
|
||||
"target_table": "market_campaign_products",
|
||||
"operation": "map_i_code_price_discount_url_into_campaign_product",
|
||||
"dedupe_key": "campaign_id + platform_code + platform_product_id",
|
||||
"write_status": "preview_only",
|
||||
},
|
||||
{
|
||||
"source_table": "competitor_prices",
|
||||
"target_table": "market_product_matches",
|
||||
"operation": "reuse_pchome_match_score_as_review_seed",
|
||||
"dedupe_key": "market_product_id + momo_i_code",
|
||||
"write_status": "preview_only",
|
||||
},
|
||||
{
|
||||
"source_table": "competitor_price_history",
|
||||
"target_table": "market_product_price_history",
|
||||
"operation": "defer_until_market_product_and_campaign_exist",
|
||||
"dedupe_key": "platform_code + platform_product_id + crawled_at",
|
||||
"write_status": "blocked_until_campaign_product_exists",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
DUPLICATE_CONTROLS = (
|
||||
{
|
||||
"key": "momo_campaign_key",
|
||||
"rule": "campaign_key 使用 momo:{page_type}:{batch_id 或 activity_time_text}:{time_slot},避免 EDM 批次重複建立活動。",
|
||||
},
|
||||
{
|
||||
"key": "momo_product_key",
|
||||
"rule": "platform_product_id 使用 promo_products.i_code,並依 market_campaign_products unique key 去重。",
|
||||
},
|
||||
{
|
||||
"key": "pchome_match_key",
|
||||
"rule": "PChome 比價資料只進入比對候選,不直接建立活動商品,避免和未來 PChome 活動 crawler 重複。",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _safe_value(value):
|
||||
if isinstance(value, (datetime, date)):
|
||||
return value.isoformat()
|
||||
if isinstance(value, Decimal):
|
||||
return float(value)
|
||||
return value
|
||||
|
||||
|
||||
def _row_to_dict(row):
|
||||
return {
|
||||
key: _safe_value(value)
|
||||
for key, value in dict(row._mapping).items()
|
||||
}
|
||||
|
||||
|
||||
def _planned_source_summaries():
|
||||
return [
|
||||
{
|
||||
**source,
|
||||
"exists": False,
|
||||
"row_count": 0,
|
||||
"distinct_entity_count": 0,
|
||||
"last_seen_at": None,
|
||||
"breakdown": [],
|
||||
"sample_rows": [],
|
||||
"read_status": "planned_no_db_connection",
|
||||
}
|
||||
for source in LEGACY_SOURCE_TABLES
|
||||
]
|
||||
|
||||
|
||||
def _probe_postgresql_table(conn, table_name):
|
||||
return bool(
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT EXISTS (
|
||||
SELECT 1
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = ANY (current_schemas(false))
|
||||
AND table_name = :table_name
|
||||
)
|
||||
"""
|
||||
),
|
||||
{"table_name": table_name},
|
||||
).scalar()
|
||||
)
|
||||
|
||||
|
||||
def _probe_sqlite_table(conn, table_name):
|
||||
return bool(
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT 1
|
||||
FROM sqlite_master
|
||||
WHERE type = 'table'
|
||||
AND name = :table_name
|
||||
LIMIT 1
|
||||
"""
|
||||
),
|
||||
{"table_name": table_name},
|
||||
).fetchone()
|
||||
)
|
||||
|
||||
|
||||
def _table_exists(conn, table_name, database_type):
|
||||
if database_type == "postgresql":
|
||||
return _probe_postgresql_table(conn, table_name)
|
||||
return _probe_sqlite_table(conn, table_name)
|
||||
|
||||
|
||||
def _query_promo_products_summary(conn, sample_limit):
|
||||
summary = _row_to_dict(
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) AS row_count,
|
||||
COUNT(DISTINCT batch_id) AS batch_count,
|
||||
COUNT(DISTINCT i_code) AS distinct_entity_count,
|
||||
MAX(crawled_at) AS last_seen_at
|
||||
FROM promo_products
|
||||
"""
|
||||
)
|
||||
).fetchone()
|
||||
)
|
||||
breakdown = [
|
||||
_row_to_dict(row)
|
||||
for row in conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
COALESCE(page_type, 'unknown') AS source_key,
|
||||
COUNT(*) AS row_count,
|
||||
COUNT(DISTINCT i_code) AS distinct_entity_count,
|
||||
MAX(crawled_at) AS last_seen_at
|
||||
FROM promo_products
|
||||
GROUP BY COALESCE(page_type, 'unknown')
|
||||
ORDER BY row_count DESC
|
||||
LIMIT :sample_limit
|
||||
"""
|
||||
),
|
||||
{"sample_limit": sample_limit},
|
||||
).fetchall()
|
||||
]
|
||||
samples = [
|
||||
_row_to_dict(row)
|
||||
for row in conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
page_type,
|
||||
batch_id,
|
||||
i_code,
|
||||
name,
|
||||
price,
|
||||
discount_text,
|
||||
url,
|
||||
crawled_at
|
||||
FROM promo_products
|
||||
ORDER BY crawled_at DESC
|
||||
LIMIT :sample_limit
|
||||
"""
|
||||
),
|
||||
{"sample_limit": sample_limit},
|
||||
).fetchall()
|
||||
]
|
||||
return summary, breakdown, samples
|
||||
|
||||
|
||||
def _query_competitor_prices_summary(conn, table_name, sample_limit):
|
||||
summary = _row_to_dict(
|
||||
conn.execute(
|
||||
text(
|
||||
f"""
|
||||
SELECT
|
||||
COUNT(*) AS row_count,
|
||||
COUNT(DISTINCT sku) AS sku_count,
|
||||
COUNT(DISTINCT competitor_product_id) AS distinct_entity_count,
|
||||
MAX(crawled_at) AS last_seen_at
|
||||
FROM {table_name}
|
||||
"""
|
||||
)
|
||||
).fetchone()
|
||||
)
|
||||
breakdown = [
|
||||
_row_to_dict(row)
|
||||
for row in conn.execute(
|
||||
text(
|
||||
f"""
|
||||
SELECT
|
||||
COALESCE(source, 'unknown') AS source_key,
|
||||
COUNT(*) AS row_count,
|
||||
COUNT(DISTINCT sku) AS sku_count,
|
||||
COUNT(DISTINCT competitor_product_id) AS distinct_entity_count,
|
||||
MAX(crawled_at) AS last_seen_at
|
||||
FROM {table_name}
|
||||
GROUP BY COALESCE(source, 'unknown')
|
||||
ORDER BY row_count DESC
|
||||
LIMIT :sample_limit
|
||||
"""
|
||||
),
|
||||
{"sample_limit": sample_limit},
|
||||
).fetchall()
|
||||
]
|
||||
samples = [
|
||||
_row_to_dict(row)
|
||||
for row in conn.execute(
|
||||
text(
|
||||
f"""
|
||||
SELECT
|
||||
source,
|
||||
sku,
|
||||
competitor_product_id,
|
||||
competitor_product_name,
|
||||
price,
|
||||
original_price,
|
||||
match_score,
|
||||
crawled_at
|
||||
FROM {table_name}
|
||||
ORDER BY crawled_at DESC
|
||||
LIMIT :sample_limit
|
||||
"""
|
||||
),
|
||||
{"sample_limit": sample_limit},
|
||||
).fetchall()
|
||||
]
|
||||
return summary, breakdown, samples
|
||||
|
||||
|
||||
def _query_source_summary(conn, source, database_type, sample_limit):
|
||||
table_name = source["table"]
|
||||
exists = _table_exists(conn, table_name, database_type)
|
||||
if not exists:
|
||||
return {
|
||||
**source,
|
||||
"exists": False,
|
||||
"row_count": 0,
|
||||
"distinct_entity_count": 0,
|
||||
"last_seen_at": None,
|
||||
"breakdown": [],
|
||||
"sample_rows": [],
|
||||
"read_status": "missing_table",
|
||||
}
|
||||
|
||||
if table_name == "promo_products":
|
||||
summary, breakdown, samples = _query_promo_products_summary(conn, sample_limit)
|
||||
else:
|
||||
summary, breakdown, samples = _query_competitor_prices_summary(
|
||||
conn,
|
||||
table_name,
|
||||
sample_limit,
|
||||
)
|
||||
|
||||
return {
|
||||
**source,
|
||||
"exists": True,
|
||||
"row_count": int(summary.get("row_count") or 0),
|
||||
"distinct_entity_count": int(summary.get("distinct_entity_count") or 0),
|
||||
"last_seen_at": summary.get("last_seen_at"),
|
||||
"metrics": summary,
|
||||
"breakdown": breakdown,
|
||||
"sample_rows": samples,
|
||||
"read_status": "read_only_loaded",
|
||||
}
|
||||
|
||||
|
||||
def _build_result(
|
||||
*,
|
||||
mode,
|
||||
execute_requested,
|
||||
read_only_query_executed,
|
||||
database_connection_opened,
|
||||
source_summaries,
|
||||
error_message=None,
|
||||
):
|
||||
existing_sources = [
|
||||
item["table"] for item in source_summaries
|
||||
if item.get("exists")
|
||||
]
|
||||
missing_sources = [
|
||||
item["table"] for item in source_summaries
|
||||
if not item.get("exists")
|
||||
]
|
||||
total_existing_rows = sum(int(item.get("row_count") or 0) for item in source_summaries)
|
||||
blocked_reasons = ["legacy_bridge_preview_only", "market_intel_write_still_blocked"]
|
||||
if not execute_requested:
|
||||
blocked_reasons.insert(0, "execute_false_planned_only")
|
||||
if missing_sources:
|
||||
blocked_reasons.insert(0, "legacy_source_tables_missing")
|
||||
if error_message:
|
||||
blocked_reasons.insert(0, "legacy_source_bridge_error")
|
||||
|
||||
return {
|
||||
"mode": mode,
|
||||
"execute_requested": bool(execute_requested),
|
||||
"read_only_query_executed": bool(read_only_query_executed),
|
||||
"database_connection_opened": bool(database_connection_opened),
|
||||
"database_session_created": False,
|
||||
"explicit_transaction_opened": False,
|
||||
"database_write_executed": False,
|
||||
"database_commit_executed": False,
|
||||
"external_network_executed": False,
|
||||
"scheduler_attached": False,
|
||||
"source_count": len(source_summaries),
|
||||
"existing_source_count": len(existing_sources),
|
||||
"existing_sources": existing_sources,
|
||||
"missing_sources": missing_sources,
|
||||
"source_tables_ready": not missing_sources if read_only_query_executed else False,
|
||||
"total_existing_rows": total_existing_rows,
|
||||
"source_summaries": source_summaries,
|
||||
"bridge_operations": list(BRIDGE_OPERATIONS),
|
||||
"duplicate_controls": list(DUPLICATE_CONTROLS),
|
||||
"writes_executed": False,
|
||||
"would_write_database": False,
|
||||
"blocked_reasons": blocked_reasons,
|
||||
"error_message": error_message,
|
||||
}
|
||||
|
||||
|
||||
def build_legacy_source_bridge_plan(
|
||||
*,
|
||||
execute_requested=False,
|
||||
database_url=None,
|
||||
database_type=None,
|
||||
engine=None,
|
||||
sample_limit=5,
|
||||
):
|
||||
"""建立既有資料來源橋接計畫;預設只回 planned,不連 DB。"""
|
||||
sample_limit = max(1, min(int(sample_limit or 5), 20))
|
||||
if not execute_requested:
|
||||
return _build_result(
|
||||
mode="legacy_source_bridge_planned",
|
||||
execute_requested=False,
|
||||
read_only_query_executed=False,
|
||||
database_connection_opened=False,
|
||||
source_summaries=_planned_source_summaries(),
|
||||
)
|
||||
|
||||
from config import DATABASE_PATH, DATABASE_TYPE
|
||||
|
||||
effective_database_type = (database_type or DATABASE_TYPE or "").lower()
|
||||
effective_database_url = database_url or DATABASE_PATH
|
||||
created_engine = False
|
||||
connection_opened = False
|
||||
|
||||
try:
|
||||
if engine is None:
|
||||
connect_args = {}
|
||||
if effective_database_type == "postgresql":
|
||||
connect_args = {
|
||||
"connect_timeout": 8,
|
||||
"options": "-c statement_timeout=15000",
|
||||
}
|
||||
engine = create_engine(
|
||||
effective_database_url,
|
||||
isolation_level="AUTOCOMMIT",
|
||||
pool_pre_ping=True,
|
||||
connect_args=connect_args,
|
||||
)
|
||||
created_engine = True
|
||||
|
||||
with engine.connect() as conn:
|
||||
connection_opened = True
|
||||
source_summaries = [
|
||||
_query_source_summary(
|
||||
conn,
|
||||
source,
|
||||
effective_database_type,
|
||||
sample_limit,
|
||||
)
|
||||
for source in LEGACY_SOURCE_TABLES
|
||||
]
|
||||
|
||||
return _build_result(
|
||||
mode="legacy_source_bridge_read_only",
|
||||
execute_requested=True,
|
||||
read_only_query_executed=True,
|
||||
database_connection_opened=connection_opened,
|
||||
source_summaries=source_summaries,
|
||||
)
|
||||
except Exception as exc:
|
||||
return _build_result(
|
||||
mode="legacy_source_bridge_error",
|
||||
execute_requested=True,
|
||||
read_only_query_executed=False,
|
||||
database_connection_opened=connection_opened,
|
||||
source_summaries=_planned_source_summaries(),
|
||||
error_message=str(exc),
|
||||
)
|
||||
finally:
|
||||
if created_engine:
|
||||
engine.dispose()
|
||||
@@ -19,6 +19,7 @@ from services.market_intel.adapters import (
|
||||
)
|
||||
from services.market_intel.candidate_preview import build_candidate_preview_from_discovery
|
||||
from services.market_intel.discovery_runner import ManualDiscoveryRunner
|
||||
from services.market_intel.legacy_source_bridge import build_legacy_source_bridge_plan
|
||||
from services.market_intel.migration_blueprint import build_migration_blueprint
|
||||
from services.market_intel.platform_seed import build_platform_seed_rows
|
||||
from services.market_intel.platform_seed_db_diff import build_platform_seed_db_diff_plan
|
||||
@@ -62,7 +63,7 @@ class MarketIntelRuntimeStatus:
|
||||
class MarketIntelService:
|
||||
"""市場情報入口服務,先集中 feature gate 與安全狀態。"""
|
||||
|
||||
phase = "phase_26_platform_seed_cli_writer"
|
||||
phase = "phase_27_legacy_source_bridge_preview"
|
||||
|
||||
def get_runtime_status(self) -> MarketIntelRuntimeStatus:
|
||||
return MarketIntelRuntimeStatus(
|
||||
@@ -279,6 +280,26 @@ class MarketIntelService:
|
||||
diff["seed_plan_found"] = bool(seed_plan["found"])
|
||||
return diff
|
||||
|
||||
def build_legacy_source_bridge(
|
||||
self,
|
||||
*,
|
||||
execute_requested=False,
|
||||
sample_limit=5,
|
||||
engine=None,
|
||||
database_url=None,
|
||||
database_type=None,
|
||||
):
|
||||
"""回報既有 EDM/PChome 資料源橋接 preview;預設不連 DB。"""
|
||||
bridge = build_legacy_source_bridge_plan(
|
||||
execute_requested=execute_requested,
|
||||
sample_limit=sample_limit,
|
||||
engine=engine,
|
||||
database_url=database_url,
|
||||
database_type=database_type,
|
||||
)
|
||||
bridge["phase"] = self.phase
|
||||
return bridge
|
||||
|
||||
def build_platform_seed_writer_plan(self, platform_code="all"):
|
||||
"""建立 platform seed writer dry-run plan,不建立 DB session。"""
|
||||
seed_plan = self.build_platform_seed_plan(platform_code=platform_code)
|
||||
@@ -377,6 +398,9 @@ class MarketIntelService:
|
||||
"platform_seed_db_diff_planned_safe": bool(
|
||||
not self.build_platform_seed_db_diff()["read_only_query_executed"]
|
||||
),
|
||||
"legacy_source_bridge_planned_safe": bool(
|
||||
not self.build_legacy_source_bridge()["read_only_query_executed"]
|
||||
),
|
||||
}
|
||||
ready_for_production_deploy = all(checks.values())
|
||||
blocked_reasons = [
|
||||
@@ -499,6 +523,7 @@ class MarketIntelService:
|
||||
"/api/market_intel/schema_smoke",
|
||||
"/api/market_intel/schema_db_probe",
|
||||
"/api/market_intel/platform_seed_db_diff",
|
||||
"/api/market_intel/legacy_source_bridge",
|
||||
],
|
||||
"status": status.to_dict(),
|
||||
"schema_smoke": schema_smoke,
|
||||
@@ -512,4 +537,5 @@ class MarketIntelService:
|
||||
"seed_writer_cli_status": self.build_seed_writer_cli_status(),
|
||||
"schema_db_probe": self.build_schema_db_probe(),
|
||||
"platform_seed_db_diff": self.build_platform_seed_db_diff(),
|
||||
"legacy_source_bridge": self.build_legacy_source_bridge(),
|
||||
}
|
||||
|
||||
@@ -396,6 +396,24 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="market-intel-panel" data-market-intel-legacy-bridge>
|
||||
<div class="market-intel-preview-head">
|
||||
<div>
|
||||
<p class="market-intel-muted momo-mono mb-1">LEGACY SOURCE / BRIDGE PREVIEW</p>
|
||||
<h2 class="market-intel-preview-title">既有資料橋接預覽</h2>
|
||||
</div>
|
||||
<button class="market-intel-icon-button" type="button" title="重新整理橋接預覽" data-market-intel-legacy-bridge-refresh>
|
||||
<i class="fas fa-rotate-right" aria-hidden="true"></i>
|
||||
</button>
|
||||
</div>
|
||||
<div class="market-intel-preview-meta" data-market-intel-legacy-bridge-meta>
|
||||
<span class="market-intel-pill">loading</span>
|
||||
</div>
|
||||
<div data-market-intel-legacy-bridge-body>
|
||||
<div class="market-intel-empty">讀取既有資料橋接預覽中...</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="market-intel-panel" data-market-intel-migration>
|
||||
<div class="market-intel-preview-head">
|
||||
<div>
|
||||
@@ -460,10 +478,11 @@
|
||||
const cliRoot = document.querySelector('[data-market-intel-cli]');
|
||||
const dbProbeRoot = document.querySelector('[data-market-intel-db-probe]');
|
||||
const seedDiffRoot = document.querySelector('[data-market-intel-seed-diff]');
|
||||
const legacyBridgeRoot = document.querySelector('[data-market-intel-legacy-bridge]');
|
||||
const migrationRoot = document.querySelector('[data-market-intel-migration]');
|
||||
const approvalRoot = document.querySelector('[data-market-intel-approval]');
|
||||
const deployRoot = document.querySelector('[data-market-intel-deploy]');
|
||||
if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !migrationRoot && !approvalRoot && !deployRoot) return;
|
||||
if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !legacyBridgeRoot && !migrationRoot && !approvalRoot && !deployRoot) return;
|
||||
|
||||
const meta = root ? root.querySelector('[data-market-intel-preview-meta]') : null;
|
||||
const body = root ? root.querySelector('[data-market-intel-preview-body]') : null;
|
||||
@@ -485,6 +504,10 @@
|
||||
const seedDiffBody = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-body]') : null;
|
||||
const seedDiffRefresh = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-refresh]') : null;
|
||||
const seedDiffEndpoint = "{{ url_for('market_intel.market_intel_platform_seed_db_diff') }}?execute=false";
|
||||
const legacyBridgeMeta = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-meta]') : null;
|
||||
const legacyBridgeBody = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-body]') : null;
|
||||
const legacyBridgeRefresh = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-refresh]') : null;
|
||||
const legacyBridgeEndpoint = "{{ url_for('market_intel.market_intel_legacy_source_bridge') }}?execute=false&limit=5";
|
||||
const migrationMeta = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-meta]') : null;
|
||||
const migrationBody = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-body]') : null;
|
||||
const migrationRefresh = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-refresh]') : null;
|
||||
@@ -730,6 +753,95 @@
|
||||
}
|
||||
};
|
||||
|
||||
const renderLegacyBridgeMeta = data => {
|
||||
legacyBridgeMeta.innerHTML = [
|
||||
`mode=${data.mode || 'unknown'}`,
|
||||
`execute=${data.execute_requested ? 'true' : 'false'}`,
|
||||
`query=${data.read_only_query_executed ? 'yes' : 'no'}`,
|
||||
`sources=${data.existing_source_count || 0}/${data.source_count || 0}`,
|
||||
`rows=${data.total_existing_rows || 0}`,
|
||||
`writes=${data.writes_executed ? 'executed' : 'blocked'}`
|
||||
].map(item => `<span class="market-intel-pill">${escapeHtml(item)}</span>`).join('');
|
||||
};
|
||||
|
||||
const renderLegacyBridgeBody = data => {
|
||||
const blockers = (data.blocked_reasons || []).join(' / ');
|
||||
const sources = data.source_summaries || [];
|
||||
const operations = data.bridge_operations || [];
|
||||
const controls = data.duplicate_controls || [];
|
||||
const renderSource = item => `
|
||||
<div class="market-intel-check">
|
||||
<div>
|
||||
<strong>${escapeHtml(item.table)}</strong>
|
||||
<small>${escapeHtml(item.description || '')}</small>
|
||||
<small>target=${escapeHtml((item.planned_targets || []).join(' / '))}</small>
|
||||
</div>
|
||||
<span>${item.exists ? escapeHtml(item.row_count || 0) : 'PENDING'}</span>
|
||||
</div>
|
||||
`;
|
||||
const renderOperation = item => `
|
||||
<article class="market-intel-operation">
|
||||
<strong>${escapeHtml(item.source_table)} → ${escapeHtml(item.target_table)}</strong>
|
||||
<small>${escapeHtml(item.operation)} / ${escapeHtml(item.write_status)}</small>
|
||||
<small>dedupe=${escapeHtml(item.dedupe_key)}</small>
|
||||
</article>
|
||||
`;
|
||||
const renderControl = item => `
|
||||
<div class="market-intel-check">
|
||||
<div>
|
||||
<strong>${escapeHtml(item.key)}</strong>
|
||||
<small>${escapeHtml(item.rule)}</small>
|
||||
</div>
|
||||
<span>RULE</span>
|
||||
</div>
|
||||
`;
|
||||
|
||||
legacyBridgeBody.innerHTML = `
|
||||
<div class="market-intel-empty mb-3">這裡只盤點既有資料來源與導入規則,不搬資料、不寫 market_*。${blockers ? `阻擋:${escapeHtml(blockers)}` : ''}</div>
|
||||
<div class="market-intel-deploy-grid">
|
||||
<div data-market-intel-legacy-sources>
|
||||
<p class="market-intel-deploy-section-title">SOURCE TABLES</p>
|
||||
<div class="market-intel-check-list">${
|
||||
sources.length
|
||||
? sources.map(renderSource).join('')
|
||||
: '<div class="market-intel-empty">尚未提供來源摘要。</div>'
|
||||
}</div>
|
||||
</div>
|
||||
<div data-market-intel-legacy-operations>
|
||||
<p class="market-intel-deploy-section-title">BRIDGE OPERATIONS</p>
|
||||
<div class="market-intel-operation-list">${
|
||||
operations.length
|
||||
? operations.map(renderOperation).join('')
|
||||
: '<div class="market-intel-empty">尚未提供橋接操作。</div>'
|
||||
}</div>
|
||||
</div>
|
||||
<div data-market-intel-legacy-controls>
|
||||
<p class="market-intel-deploy-section-title">DUPLICATE CONTROLS</p>
|
||||
<div class="market-intel-check-list">${
|
||||
controls.length
|
||||
? controls.map(renderControl).join('')
|
||||
: '<div class="market-intel-empty">尚未提供去重規則。</div>'
|
||||
}</div>
|
||||
</div>
|
||||
</div>
|
||||
`;
|
||||
};
|
||||
|
||||
const loadLegacyBridge = async () => {
|
||||
if (!legacyBridgeMeta || !legacyBridgeBody) return;
|
||||
legacyBridgeBody.innerHTML = '<div class="market-intel-empty">讀取既有資料橋接預覽中...</div>';
|
||||
try {
|
||||
const response = await fetch(legacyBridgeEndpoint, { credentials: 'same-origin' });
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
const data = await response.json();
|
||||
renderLegacyBridgeMeta(data);
|
||||
renderLegacyBridgeBody(data);
|
||||
} catch (error) {
|
||||
legacyBridgeMeta.innerHTML = '<span class="market-intel-pill">error</span>';
|
||||
legacyBridgeBody.innerHTML = `<div class="market-intel-empty">既有資料橋接預覽讀取失敗:${escapeHtml(error.message)}</div>`;
|
||||
}
|
||||
};
|
||||
|
||||
const renderMigrationMeta = data => {
|
||||
const seedWriter = data.command_plan && data.command_plan.seed_writer_command
|
||||
? data.command_plan.seed_writer_command
|
||||
@@ -979,6 +1091,9 @@
|
||||
if (seedDiffRefresh) {
|
||||
seedDiffRefresh.addEventListener('click', loadSeedDiff);
|
||||
}
|
||||
if (legacyBridgeRefresh) {
|
||||
legacyBridgeRefresh.addEventListener('click', loadLegacyBridge);
|
||||
}
|
||||
if (migrationRefresh) {
|
||||
migrationRefresh.addEventListener('click', loadMigration);
|
||||
}
|
||||
@@ -993,6 +1108,7 @@
|
||||
loadCli();
|
||||
loadDbProbe();
|
||||
loadSeedDiff();
|
||||
loadLegacyBridge();
|
||||
loadMigration();
|
||||
loadApproval();
|
||||
loadDeploy();
|
||||
|
||||
@@ -393,6 +393,7 @@ def test_market_intel_preview_template_uses_safe_fetch_false_endpoint():
|
||||
assert "market_intel.market_intel_seed_writer_cli_status" in template
|
||||
assert "market_intel.market_intel_schema_db_probe" in template
|
||||
assert "market_intel.market_intel_platform_seed_db_diff" in template
|
||||
assert "market_intel.market_intel_legacy_source_bridge" in template
|
||||
assert "market_intel.market_intel_migration_blueprint" in template
|
||||
assert "market_intel.market_intel_write_approval_runbook" in template
|
||||
assert "market_intel.market_intel_deployment_readiness" in template
|
||||
@@ -414,6 +415,164 @@ def test_market_intel_schema_metadata_contains_all_market_tables():
|
||||
assert set(MARKET_INTEL_TABLES) <= metadata_tables
|
||||
|
||||
|
||||
def test_legacy_source_bridge_default_is_planned_only():
|
||||
bridge = MarketIntelService().build_legacy_source_bridge()
|
||||
|
||||
assert bridge["mode"] == "legacy_source_bridge_planned"
|
||||
assert bridge["phase"] == "phase_27_legacy_source_bridge_preview"
|
||||
assert bridge["execute_requested"] is False
|
||||
assert bridge["read_only_query_executed"] is False
|
||||
assert bridge["database_connection_opened"] is False
|
||||
assert bridge["database_session_created"] is False
|
||||
assert bridge["database_write_executed"] is False
|
||||
assert bridge["database_commit_executed"] is False
|
||||
assert bridge["external_network_executed"] is False
|
||||
assert bridge["scheduler_attached"] is False
|
||||
assert bridge["writes_executed"] is False
|
||||
assert bridge["would_write_database"] is False
|
||||
assert bridge["source_count"] == 3
|
||||
assert bridge["existing_source_count"] == 0
|
||||
assert "execute_false_planned_only" in bridge["blocked_reasons"]
|
||||
assert {item["table"] for item in bridge["source_summaries"]} == {
|
||||
"promo_products",
|
||||
"competitor_prices",
|
||||
"competitor_price_history",
|
||||
}
|
||||
|
||||
|
||||
def test_legacy_source_bridge_read_only_sqlite_counts_sources():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE promo_products (
|
||||
id INTEGER PRIMARY KEY,
|
||||
batch_id TEXT,
|
||||
crawled_at TEXT,
|
||||
time_slot TEXT,
|
||||
activity_time_text TEXT,
|
||||
session_time_text TEXT,
|
||||
i_code TEXT,
|
||||
name TEXT,
|
||||
price INTEGER,
|
||||
discount_text TEXT,
|
||||
url TEXT,
|
||||
image_url TEXT,
|
||||
previous_price INTEGER,
|
||||
remain_qty INTEGER,
|
||||
status_change TEXT,
|
||||
page_type TEXT
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE competitor_prices (
|
||||
id INTEGER PRIMARY KEY,
|
||||
sku TEXT,
|
||||
source TEXT,
|
||||
price NUMERIC,
|
||||
original_price NUMERIC,
|
||||
discount_pct INTEGER,
|
||||
competitor_product_id TEXT,
|
||||
competitor_product_name TEXT,
|
||||
match_score NUMERIC,
|
||||
crawled_at TEXT
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE competitor_price_history (
|
||||
id INTEGER PRIMARY KEY,
|
||||
sku TEXT,
|
||||
source TEXT,
|
||||
momo_product_id INTEGER,
|
||||
momo_price NUMERIC,
|
||||
price NUMERIC,
|
||||
original_price NUMERIC,
|
||||
discount_pct INTEGER,
|
||||
competitor_product_id TEXT,
|
||||
competitor_product_name TEXT,
|
||||
match_score NUMERIC,
|
||||
crawled_at TEXT
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO promo_products
|
||||
(batch_id, crawled_at, time_slot, i_code, name, price, discount_text, url, page_type)
|
||||
VALUES
|
||||
('batch-1', '2026-05-18 10:00:00', '10:00', 'A001', 'MOMO 商品 A', 399, '8折', 'https://momo/A001', 'edm'),
|
||||
('batch-1', '2026-05-18 10:01:00', '10:00', 'A002', 'MOMO 商品 B', 499, '9折', 'https://momo/A002', 'edm')
|
||||
"""
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO competitor_prices
|
||||
(sku, source, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at)
|
||||
VALUES
|
||||
('A001', 'pchome', 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00')
|
||||
"""
|
||||
)
|
||||
)
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO competitor_price_history
|
||||
(sku, source, momo_product_id, momo_price, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at)
|
||||
VALUES
|
||||
('A001', 'pchome', 1, 399, 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00')
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
bridge = MarketIntelService().build_legacy_source_bridge(
|
||||
execute_requested=True,
|
||||
engine=engine,
|
||||
database_type="sqlite",
|
||||
)
|
||||
|
||||
assert bridge["mode"] == "legacy_source_bridge_read_only"
|
||||
assert bridge["execute_requested"] is True
|
||||
assert bridge["read_only_query_executed"] is True
|
||||
assert bridge["database_connection_opened"] is True
|
||||
assert bridge["database_session_created"] is False
|
||||
assert bridge["database_write_executed"] is False
|
||||
assert bridge["database_commit_executed"] is False
|
||||
assert bridge["external_network_executed"] is False
|
||||
assert bridge["scheduler_attached"] is False
|
||||
assert bridge["writes_executed"] is False
|
||||
assert bridge["would_write_database"] is False
|
||||
assert bridge["source_tables_ready"] is True
|
||||
assert bridge["existing_source_count"] == 3
|
||||
assert bridge["total_existing_rows"] == 4
|
||||
assert [item["table"] for item in bridge["source_summaries"]] == [
|
||||
"promo_products",
|
||||
"competitor_prices",
|
||||
"competitor_price_history",
|
||||
]
|
||||
assert bridge["source_summaries"][0]["row_count"] == 2
|
||||
assert bridge["source_summaries"][0]["distinct_entity_count"] == 2
|
||||
assert bridge["source_summaries"][1]["row_count"] == 1
|
||||
assert bridge["source_summaries"][2]["row_count"] == 1
|
||||
assert len(bridge["bridge_operations"]) == 4
|
||||
assert all(
|
||||
item["write_status"] != "executed"
|
||||
for item in bridge["bridge_operations"]
|
||||
)
|
||||
|
||||
|
||||
def test_market_intel_schema_smoke_checks_platform_columns():
|
||||
smoke = MarketIntelService().build_schema_smoke()["schema_smoke"]
|
||||
|
||||
@@ -661,6 +820,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
|
||||
assert readiness["checks"]["schema_smoke_passed"] is True
|
||||
assert readiness["checks"]["schema_db_probe_planned_safe"] is True
|
||||
assert readiness["checks"]["platform_seed_db_diff_planned_safe"] is True
|
||||
assert readiness["checks"]["legacy_source_bridge_planned_safe"] is True
|
||||
assert readiness["checks"]["writer_plan_dry_run_only"] is True
|
||||
assert readiness["writer_plan_summary"]["writes_executed"] is False
|
||||
assert "readiness_checks_not_all_passed" not in readiness["blocked_reasons"]
|
||||
@@ -675,6 +835,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
|
||||
assert "/health" in readiness["production_smoke_targets"]
|
||||
assert "/api/market_intel/deployment_readiness" in readiness["production_smoke_targets"]
|
||||
assert "/api/market_intel/platform_seed_db_diff" in readiness["production_smoke_targets"]
|
||||
assert "/api/market_intel/legacy_source_bridge" in readiness["production_smoke_targets"]
|
||||
assert readiness["write_approval_runbook"]["ready_for_real_write"] is False
|
||||
assert readiness["write_approval_runbook"]["writes_executed"] is False
|
||||
assert readiness["migration_blueprint"]["migration_executed"] is False
|
||||
@@ -682,6 +843,7 @@ def test_deployment_readiness_reports_app_only_release_gate():
|
||||
assert readiness["migration_blueprint"]["file_matches_blueprint"] is True
|
||||
assert readiness["schema_db_probe"]["read_only_query_executed"] is False
|
||||
assert readiness["platform_seed_db_diff"]["read_only_query_executed"] is False
|
||||
assert readiness["legacy_source_bridge"]["read_only_query_executed"] is False
|
||||
|
||||
|
||||
def test_write_approval_runbook_is_read_only_and_blocks_real_write():
|
||||
|
||||
Reference in New Issue
Block a user