V10.545 收斂比價覆蓋率口徑

2026-06-01 12:02:22 +08:00
parent 83b813c1b9
commit e4534c2f96
7 changed files with 65 additions and 19 deletions
--- a/TODO_NEXT_STEPS.txt
+++ b/TODO_NEXT_STEPS.txt
@@ -4,6 +4,7 @@
 ================================================================================

 【已完成】
+   - V10.545 收斂 Dashboard 比價覆蓋率口徑：coverage cache 升到 v9，新增身份覆蓋、可用比價、新鮮度、待補身份、過期身份與人工閉環欄位；商品看板和 PChome 覆核頁改只把真正待處理狀態算進「比價覆核」，人工已否決 / 人工單位價 / 需補研究改列為人工閉環；PChome competitor map 只吃有效價格、新鮮、identity_v2 最新 row，資料新鮮度也改看可用比價 row。
   - V10.544 收斂變體安全與 YES 指甲工具線：新增 YES 德悅氏指甲剪附除垢銼刀、腳皮銼腳板、藍寶石銼刀、三面拋光棒與 6/8cm 指甲剪的精準 total-price 線，要求同品牌、同工具名稱、同尺寸與同亮面/霧面/可收納/三面/不掉屑等款式訊號；同步接進 revalidation SQL。新增 MUJI / COCODOR 未知香味差異與 OPI 無型號不同色名 hard veto，HOOOME 暖燈材質差留人工覆核，搜尋詞也會優先帶香味/色名，提升 crawler 精準候選率。
   - V10.543 打通 `rescore_accepted_current` 窄門回刷：已進人工覆核池的候選若命中具名 focused exact 線，可進 `run_retryable_candidate_revalidation()` 重新評分；新增 SK-II 青春露 330ml 兩入、AMIINO 安美諾 30ml、YES 腳指甲剪刀 10.5cm、YES 極細指甲緣硬皮剪刀 9cm 的安全 total-price 線，並補 ANNY / OPI 指甲油型號 code hard veto，避免不同色號被錯配。
   - V10.542 拆開「可用比價覆蓋率」與「身份覆蓋率」：`decision_ready_rate = fresh identity / ACTIVE 商品數`，Dashboard 第一張 KPI 改顯示真正可進入決策、圖表、簡報的比價資料比例；daily / growth / Webcrumbs / OpenClaw payload 同步輸出，避免把身份覆蓋、新鮮率、價格可用率混成單一數字。
--- a/config.py
+++ b/config.py
@@ -402,7 +402,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
 # ==========================================
 # 系統版本與路徑
 # ==========================================
-SYSTEM_VERSION = "V10.544"
+SYSTEM_VERSION = "V10.545"
 LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
 public_url = PUBLIC_URL  # 用於模板顯示

--- a/docs/memory/history_logs.md
+++ b/docs/memory/history_logs.md
@@ -13,6 +13,7 @@
 ## 📅 詳細更新日誌 (考古存檔)

 ### 2026-06-01：PChome 比價新鮮度操作閉環
+- **V10.545 Dashboard 比價覆蓋率口徑收斂**: 商品看板與 PChome 覆核頁把「身份覆蓋率」與「可用比價覆蓋率」拆成明確欄位，coverage cache 更新為 v9 並回傳 `identity_coverage_*`、`pending_identity_count`、`stale_identity_count`、`last_decision_ready_crawled_at`。覆核 KPI 改只計算真正待處理狀態，人工否決 / 人工單位價 / 需補研究另列 `manual_closed_count`，避免人工閉環候選被混進待審總數。Dashboard 的 PChome competitor map 也改成只取新鮮、有效價格、identity_v2 的最新 row，資料新鮮度改看可用比價 row，不再被無效或低信心抓取紀錄撐高。
 - **V10.544 變體安全與 YES 工具線收斂**: 延續近門檻 `low_score` 救回，但把安全邊界補得更硬。新增 YES 德悅氏指甲工具精準線，只有同品牌、同工具線、同尺寸且同亮面/霧面/可收納/三面等關鍵款式時才進 `total_price`，並接入 revalidation SQL。同步新增未知香味差異與無型號指彩色名差異 hard veto：MUJI / COCODOR 不同香型、OPI 無型號不同色名不再被高分誤配；HOOOME 暖燈陶瓷/玻璃/水晶/金屬等材質差保留人工覆核。搜尋詞對護手霜、擴香瓶、無型號指彩優先帶上香味/色名，提升 crawler 找到真同款候選的機率。
 - **V10.543 rescore accepted 窄門回刷與高信心線補強**: `run_retryable_candidate_revalidation()` 追加 `rescore_accepted_current` 窄門，只允許已進人工池且命中具名 focused exact 品線的候選重新評分，仍由 matcher 判定是否可寫正式價差。新增 SK-II 青春露 330ml 兩入、AMIINO 安美諾美白修護霜 30ml、YES 腳指甲剪刀 10.5cm、YES 極細指甲緣硬皮剪刀 9cm 的 total-price 安全線；同時新增指甲油型號衝突防線，ANNY `A10.074.60` vs `A10.500`、OPI 不同 `ISL...` 型號都會 hard veto，不會為了拉覆蓋率誤配色號。
 - **V10.542 可用比價覆蓋率口徑拆分**: `fetch_competitor_coverage()` 新增 `decision_ready_matches` / `decision_ready_rate`，以「高信心 identity 且價格仍新鮮」除以 ACTIVE 商品數，和 `match_rate`（身份覆蓋）及 `fresh_match_rate`（已配對中的新鮮率）分開。Dashboard 第一屏改顯示可用比價覆蓋率，daily / growth / Webcrumbs / OpenClaw payload 同步輸出，避免使用者把舊截圖的低價格可用率、身份覆蓋率與新鮮率混成同一個 KPI。
--- a/routes/dashboard_routes.py
+++ b/routes/dashboard_routes.py
@@ -46,7 +46,7 @@ PCHOME_MATCH_SCORE_FLOOR = 0.76
 REVIEW_STATUS_OPTIONS = [
    {
        'key': 'all',
-        'label': '全部',
+        'label': '全部待處理',
        'statuses': (
            'unit_comparable',
            'refresh_unit_comparable',
@@ -59,9 +59,6 @@ REVIEW_STATUS_OPTIONS = [
            'expired_match',
            'refresh_no_result',
            'no_result',
-            'manual_rejected',
-            'manual_unit_price_required',
-            'manual_needs_research',
            'rescore_accepted_current',
        ),
    },
@@ -69,7 +66,7 @@ REVIEW_STATUS_OPTIONS = [
    {
        'key': 'unit_comparable',
        'label': '需單位價',
-        'statuses': ('unit_comparable', 'refresh_unit_comparable', 'manual_unit_price_required'),
+        'statuses': ('unit_comparable', 'refresh_unit_comparable'),
    },
    {'key': 'identity_veto', 'label': '已排除', 'statuses': ('identity_veto',)},
    {'key': 'recoverable_low_score', 'label': '近門檻可救', 'statuses': ('recoverable_low_score',)},
@@ -398,7 +395,7 @@ def _load_pchome_competitor_map(session, skus):

    try:
        stmt = text("""
-            SELECT
+            SELECT DISTINCT ON (sku)
                sku,
                price,
                original_price,
@@ -419,8 +416,11 @@ def _load_pchome_competitor_map(session, skus):
            WHERE source = 'pchome'
              AND sku IN :skus
              AND (expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP)
+              AND price IS NOT NULL
+              AND price > 0
              AND COALESCE(match_score, 0) >= :match_score_floor
              AND COALESCE(tags, '[]'::jsonb) ? 'identity_v2'
+            ORDER BY sku, crawled_at DESC NULLS LAST
        """).bindparams(bindparam("skus", expanding=True))
        rows = session.execute(
            stmt,
@@ -683,6 +683,16 @@ def _merge_competitor_review_context(overview, review_context):
    overview.update({
        'total_active': int(coverage.get('active_with_price') or overview.get('total_active') or 0),
        'matched_count': int(coverage.get('valid_matches') or overview.get('matched_count') or 0),
+        'identity_coverage_count': int(
+            coverage.get('identity_coverage_matches')
+            or coverage.get('valid_matches')
+            or overview.get('identity_coverage_count')
+            or 0
+        ),
+        'identity_coverage_rate': coverage.get(
+            'identity_coverage_rate',
+            overview.get('identity_coverage_rate') or coverage.get('match_rate', 0),
+        ),
        'match_rate': coverage.get('match_rate', overview.get('match_rate') or 0),
        'fresh_match_count': int(coverage.get('fresh_matches') or 0),
        'fresh_match_rate': coverage.get('fresh_match_rate', 0),
@@ -691,6 +701,7 @@ def _merge_competitor_review_context(overview, review_context):
        'stale_match_count': int(coverage.get('stale_matches') or 0),
        'pending_match_count': int(coverage.get('pending') or overview.get('pending_match_count') or 0),
        'review_queue_count': int(coverage.get('actionable_review_count') or len(review_queue) or 0),
+        'manual_closed_count': int(coverage.get('manual_closed_count') or 0),
        'unit_comparable_count': int(coverage.get('unit_comparable_count') or 0),
        'rescore_accepted_count': int(
            coverage.get('rescore_accepted_count')
@@ -700,6 +711,9 @@ def _merge_competitor_review_context(overview, review_context):
        'review_status_counts': review_status_counts,
        'review_queue': review_queue[:3],
    })
+    last_decision_ready = coverage.get('last_decision_ready_crawled_at')
+    if last_decision_ready:
+        overview['last_pchome_crawled'] = _format_dashboard_dt(last_decision_ready)
    return overview


@@ -1041,7 +1055,7 @@ def _load_competitor_decision_overview(session, latest_items=None):
            (SELECT COUNT(*) FROM compared WHERE gap_pct > -5 AND gap_pct < 5) AS near_count,
            (SELECT COALESCE(ROUND(AVG(gap_pct)::numeric, 1), 0) FROM compared WHERE gap_pct >= 5) AS avg_advantage_gap,
            (SELECT COUNT(*) FROM ai_price_recommendations WHERE strategy = 'product_pick' AND status = 'pending') AS ai_pick_count,
-            (SELECT MAX(crawled_at) FROM competitor_prices WHERE source = 'pchome') AS last_pchome_crawled
+            (SELECT MAX(crawled_at) FROM compared) AS last_pchome_crawled
    """)

    advantage_sql = text(latest_compared_cte + """
@@ -1737,12 +1751,14 @@ def _render_pchome_review_dashboard(
    now_taipei,
    today_start_db,
 ):
+    fresh_review_context = _load_competitor_review_context(session, limit=12)
    overview_hint = _load_cached_competitor_overview_for_review(
        now_taipei,
        [],
        0,
        review_status,
    )
+    _merge_competitor_review_context(overview_hint, fresh_review_context)
    count_total = (
        review_status != 'all'
        or bool(search_query)
@@ -1783,6 +1799,9 @@ def _render_pchome_review_dashboard(
        review_queue_total,
        review_status,
    )
+    _merge_competitor_review_context(competitor_overview, fresh_review_context)
+    if review_queue:
+        competitor_overview['review_queue'] = review_queue[:3]
    review_status_options = _build_review_status_options(competitor_overview)
    total_pages = math.ceil(review_queue_total / per_page) if review_queue_total else 0
    total_products_history = int(competitor_overview.get('total_active') or 0)
--- a/services/competitor_intel_repository.py
+++ b/services/competitor_intel_repository.py
@@ -25,6 +25,11 @@ from sqlalchemy import inspect, text

 PCHOME_MATCH_SCORE_FLOOR = 0.76
 UNIT_COMPARABLE_STATUSES = {"unit_comparable", "refresh_unit_comparable"}
+MANUAL_CLOSED_ATTEMPT_STATUSES = {
+    "manual_rejected",
+    "manual_unit_price_required",
+    "manual_needs_research",
+}
 ACTIONABLE_ATTEMPT_STATUSES = {
    "rescore_accepted_current",
    "unit_comparable",
@@ -38,13 +43,11 @@ ACTIONABLE_ATTEMPT_STATUSES = {
    "expired_match",
    "refresh_no_result",
    "no_result",
-    "manual_rejected",
-    "manual_unit_price_required",
-    "manual_needs_research",
 }
+REVIEW_QUEUE_ATTEMPT_STATUSES = ACTIONABLE_ATTEMPT_STATUSES | MANUAL_CLOSED_ATTEMPT_STATUSES
 REVIEW_STATUS_FILTER_GROUPS = {
    "rescore_accepted": ("rescore_accepted_current",),
-    "unit_comparable": ("unit_comparable", "refresh_unit_comparable", "manual_unit_price_required"),
+    "unit_comparable": ("unit_comparable", "refresh_unit_comparable"),
    "identity_veto": ("identity_veto",),
    "low_score": ("low_score", "refresh_low_score", "recoverable_low_score", "true_low_confidence"),
    "recoverable_low_score": ("recoverable_low_score",),
@@ -705,7 +708,7 @@ def _cached_payload(cache_key: str, producer, ttl_seconds: int = COMPETITOR_INTE

 def fetch_competitor_coverage(engine) -> dict:
    return _cached_payload(
-        f"coverage:v8:floor={PCHOME_MATCH_SCORE_FLOOR}:manual_reviews=1:rescore=1:review_no_fresh=1:decision_ready=1",
+        f"coverage:v9:floor={PCHOME_MATCH_SCORE_FLOOR}:manual_reviews=1:rescore=1:review_no_fresh=1:decision_ready=1:open_queue=1",
        lambda: _fetch_competitor_coverage_uncached(engine),
    )

@@ -724,13 +727,19 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
            "stale_matches": 0,
            "pending": 0,
            "decision_ready_matches": 0,
+            "identity_coverage_matches": 0,
+            "identity_coverage_rate": 0,
+            "pending_identity_count": 0,
+            "stale_identity_count": 0,
            "match_rate": 0,
            "fresh_match_rate": 0,
            "decision_ready_rate": 0,
+            "last_decision_ready_crawled_at": None,
            "attempt_status": {},
            "unit_comparable_count": 0,
            "rescore_accepted_count": 0,
            "actionable_review_count": 0,
+            "manual_closed_count": 0,
            "manual_review_summary": manual_review_summary,
            "manual_review_total": manual_review_summary["total"],
            "manual_accept_count": manual_review_summary["accept_identity"],
@@ -779,7 +788,8 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
        identity_competitor AS (
            SELECT DISTINCT ON (cp.sku)
                cp.sku,
-                cp.expires_at
+                cp.expires_at,
+                cp.crawled_at
            FROM competitor_prices cp
            WHERE cp.source = 'pchome'
              AND cp.price IS NOT NULL
@@ -789,7 +799,7 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
            ORDER BY cp.sku, cp.crawled_at DESC NULLS LAST
        ),
        fresh_competitor AS (
-            SELECT sku
+            SELECT sku, crawled_at
            FROM identity_competitor
            WHERE expires_at IS NULL OR expires_at > CURRENT_TIMESTAMP
        ),
@@ -811,6 +821,9 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
             FROM latest_momo lm
             LEFT JOIN identity_competitor ic ON ic.sku = lm.sku
             WHERE ic.sku IS NULL) AS pending,
+            (SELECT MAX(fc.crawled_at)
+             FROM latest_momo lm
+             JOIN fresh_competitor fc ON fc.sku = lm.sku) AS last_decision_ready_crawled_at,
            COALESCE(la.attempt_status, 'never_attempted') AS attempt_status,
            COUNT(*) AS status_count
        FROM latest_momo lm
@@ -834,6 +847,8 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
    unit_count = sum(statuses.get(status, 0) for status in UNIT_COMPARABLE_STATUSES)
    rescore_accepted_count = int(statuses.get("rescore_accepted_current") or 0)
    actionable_count = sum(statuses.get(status, 0) for status in ACTIONABLE_ATTEMPT_STATUSES)
+    manual_closed_count = sum(statuses.get(status, 0) for status in MANUAL_CLOSED_ATTEMPT_STATUSES)
+    last_decision_ready_crawled_at = rows[0].get("last_decision_ready_crawled_at") if rows else None
    return {
        "active_with_price": active,
        "valid_matches": valid,
@@ -841,13 +856,19 @@ def _fetch_competitor_coverage_uncached(engine) -> dict:
        "stale_matches": stale,
        "pending": pending,
        "decision_ready_matches": fresh,
+        "identity_coverage_matches": valid,
+        "identity_coverage_rate": round(valid / max(active, 1) * 100, 1),
+        "pending_identity_count": pending,
+        "stale_identity_count": stale,
        "match_rate": round(valid / max(active, 1) * 100, 1),
        "fresh_match_rate": round(fresh / max(valid, 1) * 100, 1),
        "decision_ready_rate": round(fresh / max(active, 1) * 100, 1),
+        "last_decision_ready_crawled_at": last_decision_ready_crawled_at,
        "attempt_status": statuses,
        "unit_comparable_count": unit_count,
        "rescore_accepted_count": rescore_accepted_count,
        "actionable_review_count": actionable_count,
+        "manual_closed_count": manual_closed_count,
        "manual_review_summary": manual_review_summary,
        "manual_review_total": manual_review_summary["total"],
        "manual_accept_count": manual_review_summary["accept_identity"],
--- a/templates/dashboard_v2.html
+++ b/templates/dashboard_v2.html
@@ -21,7 +21,7 @@
                <div class="dashboard-kpi-value momo-mono">{{ overview.decision_ready_rate | default(0) }}%</div>
                <div class="dashboard-kpi-sub momo-mono">
                    {{ overview.decision_ready_count | default(0) | number_format }} / {{ overview.total_active | default(total_products) | number_format }} ACTIVE
-                    · 身份 {{ overview.match_rate | default(0) }}%
+                    · 身份 {{ overview.identity_coverage_rate | default(overview.match_rate | default(0)) }}%
                    · 過期 {{ overview.stale_match_count | default(0) | number_format }}
                </div>
            </div>
@@ -48,11 +48,12 @@
                <div class="dashboard-kpi-sub momo-mono">
                    <a class="dashboard-kpi-sub-link" href="{{ url_for('dashboard.index', filter='pchome_review', category=current_category, q=search_query, review_status='rescore_accepted', sort_by='pchome_review', order='desc') }}">重算待覆核 {{ overview.rescore_accepted_count | default(0) | number_format }}</a>
                    · <a class="dashboard-kpi-sub-link" href="{{ url_for('dashboard.index', filter='pchome_review', category=current_category, q=search_query, review_status='unit_comparable', sort_by='pchome_review', order='desc') }}">需單位價 {{ overview.unit_comparable_count | default(0) | number_format }}</a>
+                    · 人工閉環 {{ overview.manual_closed_count | default(0) | number_format }}
                    · 待補抓 {{ overview.pending_match_count | default(0) | number_format }}
                </div>
            </div>
            <div class="dashboard-kpi">
-                <div class="dashboard-kpi-label momo-mono">資料新鮮度</div>
+                <div class="dashboard-kpi-label momo-mono">可用資料新鮮度</div>
                <div class="dashboard-kpi-value momo-mono is-small">{{ '已更新' if overview.last_pchome_crawled else '待更新' }}</div>
                <div class="dashboard-kpi-sub momo-mono">
                    {{ overview.last_pchome_crawled or '尚無 PChome 抓取紀錄' }}
@@ -71,7 +72,7 @@
                <div class="dashboard-backfill-label momo-mono">PCHOME MATCH BACKFILL</div>
                <div class="dashboard-backfill-title">PChome 補抓產線</div>
                <div class="dashboard-backfill-meta momo-mono">
-                    待刷新 {{ overview.stale_match_count | default(0) | number_format }} · 待補抓 {{ overview.pending_match_count | default(0) | number_format }} · 覆核 {{ overview.review_queue_count | default(0) | number_format }} · 重算待覆核 {{ overview.rescore_accepted_count | default(0) | number_format }} · 單位價 {{ overview.unit_comparable_count | default(0) | number_format }}
+                    待刷新 {{ overview.stale_match_count | default(0) | number_format }} · 待補抓 {{ overview.pending_match_count | default(0) | number_format }} · 待處理覆核 {{ overview.review_queue_count | default(0) | number_format }} · 人工閉環 {{ overview.manual_closed_count | default(0) | number_format }} · 單位價 {{ overview.unit_comparable_count | default(0) | number_format }}
                </div>
            </div>
            <div class="dashboard-backfill-progress" aria-hidden="true">
--- a/tests/test_competitor_intel_cache.py
+++ b/tests/test_competitor_intel_cache.py
@@ -75,7 +75,7 @@ def test_competitor_coverage_counts_only_active_product_intersection():
        "def _fetch_manual_review_summary", 1
    )[0]

-    assert "coverage:v8" in source
+    assert "coverage:v9" in source
    assert "rescore_accepted_count" in coverage_source
    assert "(SELECT COUNT(*) FROM valid_competitor) AS valid_matches" not in coverage_source
    assert "identity_competitor AS" in coverage_source
@@ -87,6 +87,9 @@ def test_competitor_coverage_counts_only_active_product_intersection():
    assert "\"stale_matches\": stale" in coverage_source
    assert "\"decision_ready_matches\": fresh" in coverage_source
    assert "\"decision_ready_rate\": round(fresh / max(active, 1) * 100, 1)" in coverage_source
+    assert "\"identity_coverage_matches\": valid" in coverage_source
+    assert "\"manual_closed_count\": manual_closed_count" in coverage_source
+    assert "\"last_decision_ready_crawled_at\": last_decision_ready_crawled_at" in coverage_source
    assert "FROM products p\n            JOIN LATERAL" in coverage_source
    assert "WHERE p.status = 'ACTIVE'" in coverage_source