diff --git a/TODO_NEXT_STEPS.txt b/TODO_NEXT_STEPS.txt index 76a310f..0a0ae9f 100644 --- a/TODO_NEXT_STEPS.txt +++ b/TODO_NEXT_STEPS.txt @@ -78,6 +78,7 @@ - 正式環境 V10.101 已白名單部署:備份 `/tmp/codex_market_intel_v10100_predeploy_20260513_103809.tgz`、`/tmp/codex_market_intel_v10101_predeploy_20260513_104053.tgz` 與 seed 前快照 `/tmp/codex_market_platforms_v10100_before_20260513_103809.json`;僅 recreate `momo-app`,未碰 `momo-db`、未使用 `--remove-orphans`。CLI seed 首次因 timestamp not-null rollback,V10.101 修正後成功 insert `momo/pchome/coupang/shopee` 四筆 seed;read-only diff 顯示 existing=4、missing=0、matching=4,正式頁 console error 0。 - 2026-05-13 Codex 只讀複核:`https://mo.wooo.work/health` 回報 V10.111;`/api/market_intel/status` 仍為 `enabled=false`、`crawler_enabled=false`、`write_enabled=false`、`dry_run_only=true`、phase 26;`/api/market_intel/platform_seed_db_diff?execute=true&platform=all` 只讀確認 `market_platforms` 已有 `momo/pchome/coupang/shopee` 四筆,`existing_seed_count=4`、`missing_codes=[]`、`database_write_executed=false`。 - 注意:正式端 `/api/market_intel/seed_writer_cli_status?execute=true&platform=all` 仍回傳舊版 `approval_token_hint=APPROVED_MARKET_INTEL_SEED_WRITE_V1` 與固定 token gate 文案,與 main 已入庫的一次性環境 token hardening 不一致;下次正式白名單部署需優先同步 `services/market_intel/seed_writer_cli.py`、`services/market_intel/service.py`、`scripts/market_intel_seed_writer.py`,並 smoke 確認不再回吐 `approval_token_hint`。 + - Phase 27 legacy source bridge preview:新增 `services/market_intel/legacy_source_bridge.py` 與 `/api/market_intel/legacy_source_bridge`,只讀盤點既有 `promo_products`、`competitor_prices`、`competitor_price_history`,產生導入 `market_*` 的 mapping / dedupe / blocked operation preview;預設 `execute=false` 不連 DB,`execute=true` 也只做 read-only query,不寫 DB、不建立 ORM session、不連外、不掛 scheduler;UI 新增「既有資料橋接預覽」panel;版本同步至 V10.182。 - Schema smoke:`tests/test_market_intel_skeleton.py` 檢查 `Base.metadata` 內含 ADR-035 七張 `market_*` tables。 - Desktop UI QA:本機只註冊 `market_intel_bp` 的 Flask harness 載入 `/market_intel`,確認 Phase 15、候選預覽、writer preview、安全 flags、點陣暖紙視覺正常,console error 0。 - API QA:`/api/market_intel/schema_smoke` 通過 7 張表與 `market_platforms` 必要欄位檢查;`/api/market_intel/platform_seed_writer_plan` 回傳 4 筆 dry-run upsert preview,`writes_executed=false`,四平台皆 `blocked_dry_run_only`。 diff --git a/config.py b/config.py index c5c8a3b..e00d630 100644 --- a/config.py +++ b/config.py @@ -320,7 +320,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '') # ========================================== # 系統版本與路徑 # ========================================== -SYSTEM_VERSION = "V10.180" +SYSTEM_VERSION = "V10.182" LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log') public_url = PUBLIC_URL # 用於模板顯示 diff --git a/docs/adr/ADR-035-cross-platform-market-campaign-intelligence.md b/docs/adr/ADR-035-cross-platform-market-campaign-intelligence.md index e2a89c3..b9377a5 100644 --- a/docs/adr/ADR-035-cross-platform-market-campaign-intelligence.md +++ b/docs/adr/ADR-035-cross-platform-market-campaign-intelligence.md @@ -151,6 +151,7 @@ EwoooC 目前已有 MOMO EDM / 節慶活動資料、`promo_products`、PChome - 2026-05-12 追加 read-only DB schema probe:`/api/market_intel/schema_db_probe` 預設只回 planned,不連 DB;人工 smoke 才能以明確參數查正式 DB catalog。探針不得使用 `DatabaseManager()`,避免觸發 metadata `create_all()`;不得建立 ORM session、不得寫入、不得 commit。 - 2026-05-12 追加 platform seed DB diff probe:`/api/market_intel/platform_seed_db_diff` 預設只回 planned,不連 DB;人工 smoke 才能以明確參數只讀查詢 `market_platforms`,比對 adapter seed 是否 missing / differs / matches。探針不得使用 `DatabaseManager()`、不得建立 ORM session、不得寫入、不得 commit。 - 2026-05-13 追加 platform seed CLI writer:`scripts/market_intel_seed_writer.py` 可在 CLI 明確帶入 `--execute`、`--apply-real-write` 與確認 token 時,以 SQLAlchemy Core 短 transaction upsert `market_platforms`;API 仍不得替使用者執行 DB 寫入,不建立 ORM session、不連外、不掛 scheduler。 +- 2026-05-18 追加 legacy source bridge preview:`/api/market_intel/legacy_source_bridge` 預設 `execute=false` 只回 planned,不連 DB;人工 smoke 才能以 `execute=true` 只讀盤點 `promo_products`、`competitor_prices`、`competitor_price_history`,產生舊資料導入 `market_*` 的 mapping、dedupe 與 blocked operation preview。此橋接不得寫入 DB、不得建立 ORM session、不得把 PChome 比價快取冒充為活動頁商品、不得掛 scheduler。 ### Phase 4:Coupang / Shopee Adapter diff --git a/routes/README.md b/routes/README.md index 093192a..b247a22 100644 --- a/routes/README.md +++ b/routes/README.md @@ -19,7 +19,7 @@ | `edm_routes.py` | EDM 與節慶儀表板 | `/edm`, `/festival` | | `monthly_routes.py` | 月結分析 | `/monthly_summary_analysis`, `/api/monthly_summary_data` | | `daily_sales_routes.py` | 當日業績 | `/daily_sales`, `/daily_sales/export*` | -| `market_intel_routes.py` | 市場情報 Phase 26 platform seed CLI writer | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` | +| `market_intel_routes.py` | 市場情報 Phase 27 legacy source bridge preview | `/market_intel`, `/market_intel/*`, `/api/market_intel/status`, `/api/market_intel/schema`, `/api/market_intel/schema_smoke`, `/api/market_intel/schema_db_probe`, `/api/market_intel/platform_seed_db_diff`, `/api/market_intel/legacy_source_bridge`, `/api/market_intel/adapters`, `/api/market_intel/dry_run_plan`, `/api/market_intel/discovery_plan`, `/api/market_intel/manual_discovery`, `/api/market_intel/candidate_preview`, `/api/market_intel/platform_seed_plan`, `/api/market_intel/platform_seed_write_guard`, `/api/market_intel/platform_seed_writer_plan`, `/api/market_intel/migration_blueprint`, `/api/market_intel/seed_writer_cli_status`, `/api/market_intel/write_approval_runbook`, `/api/market_intel/deployment_readiness` | | `api_routes.py` | 通用任務與查詢 API | `/api/run_task`, `/api/history/*` | | `export_routes.py` | 匯出功能 | `/api/export/*` | | `import_routes.py` | 匯入功能 | `/api/import_excel`, `/api/import/monthly_summary` | diff --git a/routes/market_intel_routes.py b/routes/market_intel_routes.py index 3cecd87..d06c293 100644 --- a/routes/market_intel_routes.py +++ b/routes/market_intel_routes.py @@ -94,6 +94,19 @@ def market_intel_platform_seed_db_diff(): ) +@market_intel_bp.route("/api/market_intel/legacy_source_bridge") +@login_required +def market_intel_legacy_source_bridge(): + execute_requested = request.args.get("execute", "false").lower() == "true" + sample_limit = request.args.get("limit", default=5, type=int) + return jsonify( + _service().build_legacy_source_bridge( + execute_requested=execute_requested, + sample_limit=sample_limit, + ) + ) + + @market_intel_bp.route("/api/market_intel/adapters") @login_required def market_intel_adapters(): diff --git a/services/market_intel/legacy_source_bridge.py b/services/market_intel/legacy_source_bridge.py new file mode 100644 index 0000000..6118d34 --- /dev/null +++ b/services/market_intel/legacy_source_bridge.py @@ -0,0 +1,439 @@ +"""既有資料來源橋接 preview。 + +本模組只讀取既有 EDM / PChome 資料表的摘要,產生未來導入 market_* 的 +去重與映射計畫;不寫入 DB、不建立 ORM session、不掛 scheduler。 +""" + +from datetime import date, datetime +from decimal import Decimal + +from sqlalchemy import create_engine, text + + +LEGACY_SOURCE_TABLES = ( + { + "table": "promo_products", + "source_code": "momo_promo_products", + "platform_code": "momo", + "description": "既有 MOMO EDM / festival 活動商品資料,可作為 market_campaigns 與 market_campaign_products 的導入來源。", + "planned_targets": ["market_campaigns", "market_campaign_products"], + }, + { + "table": "competitor_prices", + "source_code": "pchome_competitor_prices", + "platform_code": "pchome", + "description": "既有 PChome 最新比價快取,可作為商品比對與價格威脅背景,不直接冒充活動頁商品。", + "planned_targets": ["market_product_matches"], + }, + { + "table": "competitor_price_history", + "source_code": "pchome_competitor_price_history", + "platform_code": "pchome", + "description": "既有 PChome 比價歷史,可在 market 商品與 campaign 建立後作為價格趨勢參考。", + "planned_targets": ["market_product_price_history"], + }, +) + + +BRIDGE_OPERATIONS = ( + { + "source_table": "promo_products", + "target_table": "market_campaigns", + "operation": "derive_momo_campaign_from_page_type_batch_and_activity_time", + "dedupe_key": "platform_code + campaign_key", + "write_status": "preview_only", + }, + { + "source_table": "promo_products", + "target_table": "market_campaign_products", + "operation": "map_i_code_price_discount_url_into_campaign_product", + "dedupe_key": "campaign_id + platform_code + platform_product_id", + "write_status": "preview_only", + }, + { + "source_table": "competitor_prices", + "target_table": "market_product_matches", + "operation": "reuse_pchome_match_score_as_review_seed", + "dedupe_key": "market_product_id + momo_i_code", + "write_status": "preview_only", + }, + { + "source_table": "competitor_price_history", + "target_table": "market_product_price_history", + "operation": "defer_until_market_product_and_campaign_exist", + "dedupe_key": "platform_code + platform_product_id + crawled_at", + "write_status": "blocked_until_campaign_product_exists", + }, +) + + +DUPLICATE_CONTROLS = ( + { + "key": "momo_campaign_key", + "rule": "campaign_key 使用 momo:{page_type}:{batch_id 或 activity_time_text}:{time_slot},避免 EDM 批次重複建立活動。", + }, + { + "key": "momo_product_key", + "rule": "platform_product_id 使用 promo_products.i_code,並依 market_campaign_products unique key 去重。", + }, + { + "key": "pchome_match_key", + "rule": "PChome 比價資料只進入比對候選,不直接建立活動商品,避免和未來 PChome 活動 crawler 重複。", + }, +) + + +def _safe_value(value): + if isinstance(value, (datetime, date)): + return value.isoformat() + if isinstance(value, Decimal): + return float(value) + return value + + +def _row_to_dict(row): + return { + key: _safe_value(value) + for key, value in dict(row._mapping).items() + } + + +def _planned_source_summaries(): + return [ + { + **source, + "exists": False, + "row_count": 0, + "distinct_entity_count": 0, + "last_seen_at": None, + "breakdown": [], + "sample_rows": [], + "read_status": "planned_no_db_connection", + } + for source in LEGACY_SOURCE_TABLES + ] + + +def _probe_postgresql_table(conn, table_name): + return bool( + conn.execute( + text( + """ + SELECT EXISTS ( + SELECT 1 + FROM information_schema.tables + WHERE table_schema = ANY (current_schemas(false)) + AND table_name = :table_name + ) + """ + ), + {"table_name": table_name}, + ).scalar() + ) + + +def _probe_sqlite_table(conn, table_name): + return bool( + conn.execute( + text( + """ + SELECT 1 + FROM sqlite_master + WHERE type = 'table' + AND name = :table_name + LIMIT 1 + """ + ), + {"table_name": table_name}, + ).fetchone() + ) + + +def _table_exists(conn, table_name, database_type): + if database_type == "postgresql": + return _probe_postgresql_table(conn, table_name) + return _probe_sqlite_table(conn, table_name) + + +def _query_promo_products_summary(conn, sample_limit): + summary = _row_to_dict( + conn.execute( + text( + """ + SELECT + COUNT(*) AS row_count, + COUNT(DISTINCT batch_id) AS batch_count, + COUNT(DISTINCT i_code) AS distinct_entity_count, + MAX(crawled_at) AS last_seen_at + FROM promo_products + """ + ) + ).fetchone() + ) + breakdown = [ + _row_to_dict(row) + for row in conn.execute( + text( + """ + SELECT + COALESCE(page_type, 'unknown') AS source_key, + COUNT(*) AS row_count, + COUNT(DISTINCT i_code) AS distinct_entity_count, + MAX(crawled_at) AS last_seen_at + FROM promo_products + GROUP BY COALESCE(page_type, 'unknown') + ORDER BY row_count DESC + LIMIT :sample_limit + """ + ), + {"sample_limit": sample_limit}, + ).fetchall() + ] + samples = [ + _row_to_dict(row) + for row in conn.execute( + text( + """ + SELECT + page_type, + batch_id, + i_code, + name, + price, + discount_text, + url, + crawled_at + FROM promo_products + ORDER BY crawled_at DESC + LIMIT :sample_limit + """ + ), + {"sample_limit": sample_limit}, + ).fetchall() + ] + return summary, breakdown, samples + + +def _query_competitor_prices_summary(conn, table_name, sample_limit): + summary = _row_to_dict( + conn.execute( + text( + f""" + SELECT + COUNT(*) AS row_count, + COUNT(DISTINCT sku) AS sku_count, + COUNT(DISTINCT competitor_product_id) AS distinct_entity_count, + MAX(crawled_at) AS last_seen_at + FROM {table_name} + """ + ) + ).fetchone() + ) + breakdown = [ + _row_to_dict(row) + for row in conn.execute( + text( + f""" + SELECT + COALESCE(source, 'unknown') AS source_key, + COUNT(*) AS row_count, + COUNT(DISTINCT sku) AS sku_count, + COUNT(DISTINCT competitor_product_id) AS distinct_entity_count, + MAX(crawled_at) AS last_seen_at + FROM {table_name} + GROUP BY COALESCE(source, 'unknown') + ORDER BY row_count DESC + LIMIT :sample_limit + """ + ), + {"sample_limit": sample_limit}, + ).fetchall() + ] + samples = [ + _row_to_dict(row) + for row in conn.execute( + text( + f""" + SELECT + source, + sku, + competitor_product_id, + competitor_product_name, + price, + original_price, + match_score, + crawled_at + FROM {table_name} + ORDER BY crawled_at DESC + LIMIT :sample_limit + """ + ), + {"sample_limit": sample_limit}, + ).fetchall() + ] + return summary, breakdown, samples + + +def _query_source_summary(conn, source, database_type, sample_limit): + table_name = source["table"] + exists = _table_exists(conn, table_name, database_type) + if not exists: + return { + **source, + "exists": False, + "row_count": 0, + "distinct_entity_count": 0, + "last_seen_at": None, + "breakdown": [], + "sample_rows": [], + "read_status": "missing_table", + } + + if table_name == "promo_products": + summary, breakdown, samples = _query_promo_products_summary(conn, sample_limit) + else: + summary, breakdown, samples = _query_competitor_prices_summary( + conn, + table_name, + sample_limit, + ) + + return { + **source, + "exists": True, + "row_count": int(summary.get("row_count") or 0), + "distinct_entity_count": int(summary.get("distinct_entity_count") or 0), + "last_seen_at": summary.get("last_seen_at"), + "metrics": summary, + "breakdown": breakdown, + "sample_rows": samples, + "read_status": "read_only_loaded", + } + + +def _build_result( + *, + mode, + execute_requested, + read_only_query_executed, + database_connection_opened, + source_summaries, + error_message=None, +): + existing_sources = [ + item["table"] for item in source_summaries + if item.get("exists") + ] + missing_sources = [ + item["table"] for item in source_summaries + if not item.get("exists") + ] + total_existing_rows = sum(int(item.get("row_count") or 0) for item in source_summaries) + blocked_reasons = ["legacy_bridge_preview_only", "market_intel_write_still_blocked"] + if not execute_requested: + blocked_reasons.insert(0, "execute_false_planned_only") + if missing_sources: + blocked_reasons.insert(0, "legacy_source_tables_missing") + if error_message: + blocked_reasons.insert(0, "legacy_source_bridge_error") + + return { + "mode": mode, + "execute_requested": bool(execute_requested), + "read_only_query_executed": bool(read_only_query_executed), + "database_connection_opened": bool(database_connection_opened), + "database_session_created": False, + "explicit_transaction_opened": False, + "database_write_executed": False, + "database_commit_executed": False, + "external_network_executed": False, + "scheduler_attached": False, + "source_count": len(source_summaries), + "existing_source_count": len(existing_sources), + "existing_sources": existing_sources, + "missing_sources": missing_sources, + "source_tables_ready": not missing_sources if read_only_query_executed else False, + "total_existing_rows": total_existing_rows, + "source_summaries": source_summaries, + "bridge_operations": list(BRIDGE_OPERATIONS), + "duplicate_controls": list(DUPLICATE_CONTROLS), + "writes_executed": False, + "would_write_database": False, + "blocked_reasons": blocked_reasons, + "error_message": error_message, + } + + +def build_legacy_source_bridge_plan( + *, + execute_requested=False, + database_url=None, + database_type=None, + engine=None, + sample_limit=5, +): + """建立既有資料來源橋接計畫;預設只回 planned,不連 DB。""" + sample_limit = max(1, min(int(sample_limit or 5), 20)) + if not execute_requested: + return _build_result( + mode="legacy_source_bridge_planned", + execute_requested=False, + read_only_query_executed=False, + database_connection_opened=False, + source_summaries=_planned_source_summaries(), + ) + + from config import DATABASE_PATH, DATABASE_TYPE + + effective_database_type = (database_type or DATABASE_TYPE or "").lower() + effective_database_url = database_url or DATABASE_PATH + created_engine = False + connection_opened = False + + try: + if engine is None: + connect_args = {} + if effective_database_type == "postgresql": + connect_args = { + "connect_timeout": 8, + "options": "-c statement_timeout=15000", + } + engine = create_engine( + effective_database_url, + isolation_level="AUTOCOMMIT", + pool_pre_ping=True, + connect_args=connect_args, + ) + created_engine = True + + with engine.connect() as conn: + connection_opened = True + source_summaries = [ + _query_source_summary( + conn, + source, + effective_database_type, + sample_limit, + ) + for source in LEGACY_SOURCE_TABLES + ] + + return _build_result( + mode="legacy_source_bridge_read_only", + execute_requested=True, + read_only_query_executed=True, + database_connection_opened=connection_opened, + source_summaries=source_summaries, + ) + except Exception as exc: + return _build_result( + mode="legacy_source_bridge_error", + execute_requested=True, + read_only_query_executed=False, + database_connection_opened=connection_opened, + source_summaries=_planned_source_summaries(), + error_message=str(exc), + ) + finally: + if created_engine: + engine.dispose() diff --git a/services/market_intel/service.py b/services/market_intel/service.py index 4169639..bba9af1 100644 --- a/services/market_intel/service.py +++ b/services/market_intel/service.py @@ -19,6 +19,7 @@ from services.market_intel.adapters import ( ) from services.market_intel.candidate_preview import build_candidate_preview_from_discovery from services.market_intel.discovery_runner import ManualDiscoveryRunner +from services.market_intel.legacy_source_bridge import build_legacy_source_bridge_plan from services.market_intel.migration_blueprint import build_migration_blueprint from services.market_intel.platform_seed import build_platform_seed_rows from services.market_intel.platform_seed_db_diff import build_platform_seed_db_diff_plan @@ -62,7 +63,7 @@ class MarketIntelRuntimeStatus: class MarketIntelService: """市場情報入口服務,先集中 feature gate 與安全狀態。""" - phase = "phase_26_platform_seed_cli_writer" + phase = "phase_27_legacy_source_bridge_preview" def get_runtime_status(self) -> MarketIntelRuntimeStatus: return MarketIntelRuntimeStatus( @@ -279,6 +280,26 @@ class MarketIntelService: diff["seed_plan_found"] = bool(seed_plan["found"]) return diff + def build_legacy_source_bridge( + self, + *, + execute_requested=False, + sample_limit=5, + engine=None, + database_url=None, + database_type=None, + ): + """回報既有 EDM/PChome 資料源橋接 preview;預設不連 DB。""" + bridge = build_legacy_source_bridge_plan( + execute_requested=execute_requested, + sample_limit=sample_limit, + engine=engine, + database_url=database_url, + database_type=database_type, + ) + bridge["phase"] = self.phase + return bridge + def build_platform_seed_writer_plan(self, platform_code="all"): """建立 platform seed writer dry-run plan,不建立 DB session。""" seed_plan = self.build_platform_seed_plan(platform_code=platform_code) @@ -377,6 +398,9 @@ class MarketIntelService: "platform_seed_db_diff_planned_safe": bool( not self.build_platform_seed_db_diff()["read_only_query_executed"] ), + "legacy_source_bridge_planned_safe": bool( + not self.build_legacy_source_bridge()["read_only_query_executed"] + ), } ready_for_production_deploy = all(checks.values()) blocked_reasons = [ @@ -499,6 +523,7 @@ class MarketIntelService: "/api/market_intel/schema_smoke", "/api/market_intel/schema_db_probe", "/api/market_intel/platform_seed_db_diff", + "/api/market_intel/legacy_source_bridge", ], "status": status.to_dict(), "schema_smoke": schema_smoke, @@ -512,4 +537,5 @@ class MarketIntelService: "seed_writer_cli_status": self.build_seed_writer_cli_status(), "schema_db_probe": self.build_schema_db_probe(), "platform_seed_db_diff": self.build_platform_seed_db_diff(), + "legacy_source_bridge": self.build_legacy_source_bridge(), } diff --git a/templates/market_intel/disabled.html b/templates/market_intel/disabled.html index 2dd84b6..7a9828e 100644 --- a/templates/market_intel/disabled.html +++ b/templates/market_intel/disabled.html @@ -396,6 +396,24 @@ +
+
+
+

LEGACY SOURCE / BRIDGE PREVIEW

+

既有資料橋接預覽

+
+ +
+
+ loading +
+
+
讀取既有資料橋接預覽中...
+
+
+
@@ -460,10 +478,11 @@ const cliRoot = document.querySelector('[data-market-intel-cli]'); const dbProbeRoot = document.querySelector('[data-market-intel-db-probe]'); const seedDiffRoot = document.querySelector('[data-market-intel-seed-diff]'); + const legacyBridgeRoot = document.querySelector('[data-market-intel-legacy-bridge]'); const migrationRoot = document.querySelector('[data-market-intel-migration]'); const approvalRoot = document.querySelector('[data-market-intel-approval]'); const deployRoot = document.querySelector('[data-market-intel-deploy]'); - if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !migrationRoot && !approvalRoot && !deployRoot) return; + if (!root && !writerRoot && !cliRoot && !dbProbeRoot && !seedDiffRoot && !legacyBridgeRoot && !migrationRoot && !approvalRoot && !deployRoot) return; const meta = root ? root.querySelector('[data-market-intel-preview-meta]') : null; const body = root ? root.querySelector('[data-market-intel-preview-body]') : null; @@ -485,6 +504,10 @@ const seedDiffBody = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-body]') : null; const seedDiffRefresh = seedDiffRoot ? seedDiffRoot.querySelector('[data-market-intel-seed-diff-refresh]') : null; const seedDiffEndpoint = "{{ url_for('market_intel.market_intel_platform_seed_db_diff') }}?execute=false"; + const legacyBridgeMeta = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-meta]') : null; + const legacyBridgeBody = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-body]') : null; + const legacyBridgeRefresh = legacyBridgeRoot ? legacyBridgeRoot.querySelector('[data-market-intel-legacy-bridge-refresh]') : null; + const legacyBridgeEndpoint = "{{ url_for('market_intel.market_intel_legacy_source_bridge') }}?execute=false&limit=5"; const migrationMeta = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-meta]') : null; const migrationBody = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-body]') : null; const migrationRefresh = migrationRoot ? migrationRoot.querySelector('[data-market-intel-migration-refresh]') : null; @@ -730,6 +753,95 @@ } }; + const renderLegacyBridgeMeta = data => { + legacyBridgeMeta.innerHTML = [ + `mode=${data.mode || 'unknown'}`, + `execute=${data.execute_requested ? 'true' : 'false'}`, + `query=${data.read_only_query_executed ? 'yes' : 'no'}`, + `sources=${data.existing_source_count || 0}/${data.source_count || 0}`, + `rows=${data.total_existing_rows || 0}`, + `writes=${data.writes_executed ? 'executed' : 'blocked'}` + ].map(item => `${escapeHtml(item)}`).join(''); + }; + + const renderLegacyBridgeBody = data => { + const blockers = (data.blocked_reasons || []).join(' / '); + const sources = data.source_summaries || []; + const operations = data.bridge_operations || []; + const controls = data.duplicate_controls || []; + const renderSource = item => ` +
+
+ ${escapeHtml(item.table)} + ${escapeHtml(item.description || '')} + target=${escapeHtml((item.planned_targets || []).join(' / '))} +
+ ${item.exists ? escapeHtml(item.row_count || 0) : 'PENDING'} +
+ `; + const renderOperation = item => ` +
+ ${escapeHtml(item.source_table)} → ${escapeHtml(item.target_table)} + ${escapeHtml(item.operation)} / ${escapeHtml(item.write_status)} + dedupe=${escapeHtml(item.dedupe_key)} +
+ `; + const renderControl = item => ` +
+
+ ${escapeHtml(item.key)} + ${escapeHtml(item.rule)} +
+ RULE +
+ `; + + legacyBridgeBody.innerHTML = ` +
這裡只盤點既有資料來源與導入規則,不搬資料、不寫 market_*。${blockers ? `阻擋:${escapeHtml(blockers)}` : ''}
+
+
+

SOURCE TABLES

+
${ + sources.length + ? sources.map(renderSource).join('') + : '
尚未提供來源摘要。
' + }
+
+
+

BRIDGE OPERATIONS

+
${ + operations.length + ? operations.map(renderOperation).join('') + : '
尚未提供橋接操作。
' + }
+
+
+

DUPLICATE CONTROLS

+
${ + controls.length + ? controls.map(renderControl).join('') + : '
尚未提供去重規則。
' + }
+
+
+ `; + }; + + const loadLegacyBridge = async () => { + if (!legacyBridgeMeta || !legacyBridgeBody) return; + legacyBridgeBody.innerHTML = '
讀取既有資料橋接預覽中...
'; + try { + const response = await fetch(legacyBridgeEndpoint, { credentials: 'same-origin' }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const data = await response.json(); + renderLegacyBridgeMeta(data); + renderLegacyBridgeBody(data); + } catch (error) { + legacyBridgeMeta.innerHTML = 'error'; + legacyBridgeBody.innerHTML = `
既有資料橋接預覽讀取失敗:${escapeHtml(error.message)}
`; + } + }; + const renderMigrationMeta = data => { const seedWriter = data.command_plan && data.command_plan.seed_writer_command ? data.command_plan.seed_writer_command @@ -979,6 +1091,9 @@ if (seedDiffRefresh) { seedDiffRefresh.addEventListener('click', loadSeedDiff); } + if (legacyBridgeRefresh) { + legacyBridgeRefresh.addEventListener('click', loadLegacyBridge); + } if (migrationRefresh) { migrationRefresh.addEventListener('click', loadMigration); } @@ -993,6 +1108,7 @@ loadCli(); loadDbProbe(); loadSeedDiff(); + loadLegacyBridge(); loadMigration(); loadApproval(); loadDeploy(); diff --git a/tests/test_market_intel_skeleton.py b/tests/test_market_intel_skeleton.py index 227bf91..ae1887d 100644 --- a/tests/test_market_intel_skeleton.py +++ b/tests/test_market_intel_skeleton.py @@ -393,6 +393,7 @@ def test_market_intel_preview_template_uses_safe_fetch_false_endpoint(): assert "market_intel.market_intel_seed_writer_cli_status" in template assert "market_intel.market_intel_schema_db_probe" in template assert "market_intel.market_intel_platform_seed_db_diff" in template + assert "market_intel.market_intel_legacy_source_bridge" in template assert "market_intel.market_intel_migration_blueprint" in template assert "market_intel.market_intel_write_approval_runbook" in template assert "market_intel.market_intel_deployment_readiness" in template @@ -414,6 +415,164 @@ def test_market_intel_schema_metadata_contains_all_market_tables(): assert set(MARKET_INTEL_TABLES) <= metadata_tables +def test_legacy_source_bridge_default_is_planned_only(): + bridge = MarketIntelService().build_legacy_source_bridge() + + assert bridge["mode"] == "legacy_source_bridge_planned" + assert bridge["phase"] == "phase_27_legacy_source_bridge_preview" + assert bridge["execute_requested"] is False + assert bridge["read_only_query_executed"] is False + assert bridge["database_connection_opened"] is False + assert bridge["database_session_created"] is False + assert bridge["database_write_executed"] is False + assert bridge["database_commit_executed"] is False + assert bridge["external_network_executed"] is False + assert bridge["scheduler_attached"] is False + assert bridge["writes_executed"] is False + assert bridge["would_write_database"] is False + assert bridge["source_count"] == 3 + assert bridge["existing_source_count"] == 0 + assert "execute_false_planned_only" in bridge["blocked_reasons"] + assert {item["table"] for item in bridge["source_summaries"]} == { + "promo_products", + "competitor_prices", + "competitor_price_history", + } + + +def test_legacy_source_bridge_read_only_sqlite_counts_sources(): + engine = create_engine("sqlite:///:memory:") + with engine.begin() as conn: + conn.execute( + text( + """ + CREATE TABLE promo_products ( + id INTEGER PRIMARY KEY, + batch_id TEXT, + crawled_at TEXT, + time_slot TEXT, + activity_time_text TEXT, + session_time_text TEXT, + i_code TEXT, + name TEXT, + price INTEGER, + discount_text TEXT, + url TEXT, + image_url TEXT, + previous_price INTEGER, + remain_qty INTEGER, + status_change TEXT, + page_type TEXT + ) + """ + ) + ) + conn.execute( + text( + """ + CREATE TABLE competitor_prices ( + id INTEGER PRIMARY KEY, + sku TEXT, + source TEXT, + price NUMERIC, + original_price NUMERIC, + discount_pct INTEGER, + competitor_product_id TEXT, + competitor_product_name TEXT, + match_score NUMERIC, + crawled_at TEXT + ) + """ + ) + ) + conn.execute( + text( + """ + CREATE TABLE competitor_price_history ( + id INTEGER PRIMARY KEY, + sku TEXT, + source TEXT, + momo_product_id INTEGER, + momo_price NUMERIC, + price NUMERIC, + original_price NUMERIC, + discount_pct INTEGER, + competitor_product_id TEXT, + competitor_product_name TEXT, + match_score NUMERIC, + crawled_at TEXT + ) + """ + ) + ) + conn.execute( + text( + """ + INSERT INTO promo_products + (batch_id, crawled_at, time_slot, i_code, name, price, discount_text, url, page_type) + VALUES + ('batch-1', '2026-05-18 10:00:00', '10:00', 'A001', 'MOMO 商品 A', 399, '8折', 'https://momo/A001', 'edm'), + ('batch-1', '2026-05-18 10:01:00', '10:00', 'A002', 'MOMO 商品 B', 499, '9折', 'https://momo/A002', 'edm') + """ + ) + ) + conn.execute( + text( + """ + INSERT INTO competitor_prices + (sku, source, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at) + VALUES + ('A001', 'pchome', 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00') + """ + ) + ) + conn.execute( + text( + """ + INSERT INTO competitor_price_history + (sku, source, momo_product_id, momo_price, price, original_price, competitor_product_id, competitor_product_name, match_score, crawled_at) + VALUES + ('A001', 'pchome', 1, 399, 379, 459, 'PC-A001', 'PChome 商品 A', 0.82, '2026-05-18 11:00:00') + """ + ) + ) + + bridge = MarketIntelService().build_legacy_source_bridge( + execute_requested=True, + engine=engine, + database_type="sqlite", + ) + + assert bridge["mode"] == "legacy_source_bridge_read_only" + assert bridge["execute_requested"] is True + assert bridge["read_only_query_executed"] is True + assert bridge["database_connection_opened"] is True + assert bridge["database_session_created"] is False + assert bridge["database_write_executed"] is False + assert bridge["database_commit_executed"] is False + assert bridge["external_network_executed"] is False + assert bridge["scheduler_attached"] is False + assert bridge["writes_executed"] is False + assert bridge["would_write_database"] is False + assert bridge["source_tables_ready"] is True + assert bridge["existing_source_count"] == 3 + assert bridge["total_existing_rows"] == 4 + assert [item["table"] for item in bridge["source_summaries"]] == [ + "promo_products", + "competitor_prices", + "competitor_price_history", + ] + assert bridge["source_summaries"][0]["row_count"] == 2 + assert bridge["source_summaries"][0]["distinct_entity_count"] == 2 + assert bridge["source_summaries"][1]["row_count"] == 1 + assert bridge["source_summaries"][2]["row_count"] == 1 + assert len(bridge["bridge_operations"]) == 4 + assert all( + item["write_status"] != "executed" + for item in bridge["bridge_operations"] + ) + + def test_market_intel_schema_smoke_checks_platform_columns(): smoke = MarketIntelService().build_schema_smoke()["schema_smoke"] @@ -661,6 +820,7 @@ def test_deployment_readiness_reports_app_only_release_gate(): assert readiness["checks"]["schema_smoke_passed"] is True assert readiness["checks"]["schema_db_probe_planned_safe"] is True assert readiness["checks"]["platform_seed_db_diff_planned_safe"] is True + assert readiness["checks"]["legacy_source_bridge_planned_safe"] is True assert readiness["checks"]["writer_plan_dry_run_only"] is True assert readiness["writer_plan_summary"]["writes_executed"] is False assert "readiness_checks_not_all_passed" not in readiness["blocked_reasons"] @@ -675,6 +835,7 @@ def test_deployment_readiness_reports_app_only_release_gate(): assert "/health" in readiness["production_smoke_targets"] assert "/api/market_intel/deployment_readiness" in readiness["production_smoke_targets"] assert "/api/market_intel/platform_seed_db_diff" in readiness["production_smoke_targets"] + assert "/api/market_intel/legacy_source_bridge" in readiness["production_smoke_targets"] assert readiness["write_approval_runbook"]["ready_for_real_write"] is False assert readiness["write_approval_runbook"]["writes_executed"] is False assert readiness["migration_blueprint"]["migration_executed"] is False @@ -682,6 +843,7 @@ def test_deployment_readiness_reports_app_only_release_gate(): assert readiness["migration_blueprint"]["file_matches_blueprint"] is True assert readiness["schema_db_probe"]["read_only_query_executed"] is False assert readiness["platform_seed_db_diff"]["read_only_query_executed"] is False + assert readiness["legacy_source_bridge"]["read_only_query_executed"] is False def test_write_approval_runbook_is_read_only_and_blocks_real_write():