補齊 AI Smoke 趨勢管理

2026-04-29 23:54:23 +08:00
parent 81159b5b3d
commit 10bbd55f5b
14 changed files with 191 additions and 14 deletions
--- a/CONSTITUTION.md
+++ b/CONSTITUTION.md
@@ -2,7 +2,7 @@

 > 本文件定義專案開發的核心準則與不可違反的規範
 > **建立日期**: 2026-01-12
-> **當前版本**: V10.7 (四 AI Agent 自動化 Smoke 趨勢版)
+> **當前版本**: V10.8 (四 AI Agent 自動化 Smoke 趨勢管理版)
 > **最後更新**: 2026-04-29

 ---
--- a/TODO_NEXT_STEPS.txt
+++ b/TODO_NEXT_STEPS.txt
@@ -7,15 +7,16 @@
   - ADR-018：四 AI Agent 自動化控制面立案。
   - Memory：新增 `docs/memory/ai_automation_closure_20260429.md`。
   - Guide/Skills 替代：新增 `docs/guides/ai_automation_session_sop.md`。
-   - SOT：更新 `docs/AI_INTELLIGENCE_MODULE_SOT.md` 至 V10.7 AI Automation Smoke Trend 架構。
+   - SOT：更新 `docs/AI_INTELLIGENCE_MODULE_SOT.md` 至 V10.8 AI Automation Smoke Management 架構。
   - Codex 規則：更新 `AGENTS.md`、`CONSTITUTION.md`、ADR/memory 索引。
   - Prometheus 指標化：新增 EventRouter / AutoHeal / safe action / replay in-process metrics，並接入 `/metrics`。
   - 線上 smoke dashboard：新增 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`，覆蓋 EventRouter、AutoHeal、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL。
   - Smoke 趨勢保存：`/api/ai-automation/smoke` 每次快檢追加 JSONL 精簡紀錄，dashboard 顯示最近趨勢。
+   - Smoke 趨勢管理：新增 JSONL 匯出、清理與每日摘要。

 【下次待辦】
   - Superset / Grafana 視覺化：`momo_ai_event_router_dispatch_total`、`momo_ai_event_router_latency_ms_*`、`momo_ai_autoheal_action_total`。
-   - Smoke trend 增加手動清理/匯出與每日摘要。
+   - Smoke trend 增加每日摘要 Telegram 推播或排程摘要。

 ================================================================================
                      品牌資產最終處理與維護 (Phase 7) [DONE]
--- a/app.py
+++ b/app.py
@@ -95,9 +95,9 @@ except Exception as e:
    sys_log.error(f"無法檢測磁碟空間: {e}")

 # 🚩 系統版本定義 (備份與顯示用)
-# 🚩 2026-04-29 V10.7: AI Smoke 趨勢保存 — 最近快檢結果 JSONL
-#                  持久化與 dashboard 趨勢視覺化
-SYSTEM_VERSION = "V10.7"
+# 🚩 2026-04-29 V10.8: AI Smoke 趨勢管理 — JSONL 匯出 / 清理 /
+#                  每日摘要接入 dashboard
+SYSTEM_VERSION = "V10.8"

 # ==========================================
 # 🔒 SQL Injection 防護函數
--- a/config.py
+++ b/config.py
@@ -253,7 +253,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
 # ==========================================
 # 系統版本與路徑
 # ==========================================
-SYSTEM_VERSION = "V10.7"
+SYSTEM_VERSION = "V10.8"
 LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
 public_url = PUBLIC_URL  # 用於模板顯示

--- a/docs/AI_INTELLIGENCE_MODULE_SOT.md
+++ b/docs/AI_INTELLIGENCE_MODULE_SOT.md
@@ -1,8 +1,8 @@
 # MOMO PRO — AI 競價情報模組 Single Source of Truth

 > **最後更新**: 2026-04-29 (台北時間)
-> **狀態**: 🟢 四 AI Agent 自動化閉環已落地 — EventRouter / AutoHeal / OpenClaw Memory / ElephantAlpha bridge / Prometheus metrics / Smoke Dashboard / Smoke Trend 具測試覆蓋
-> **適用版本**: V10.7 AI Automation Smoke Trend 架構
+> **狀態**: 🟢 四 AI Agent 自動化閉環已落地 — EventRouter / AutoHeal / OpenClaw Memory / ElephantAlpha bridge / Prometheus metrics / Smoke Dashboard / Smoke Trend Management 具測試覆蓋
+> **適用版本**: V10.8 AI Automation Smoke Management 架構

 ---

@@ -65,6 +65,7 @@ SQL漏斗(~300筆)
 - `/ai_automation_smoke` 提供登入後 smoke dashboard。
 - `/api/ai-automation/smoke` 提供 read-only JSON 狀態，不做外部網路呼叫。
 - Smoke API 會將最近快檢結果保存到 JSONL，dashboard 顯示最近狀態趨勢。
+- Smoke history 支援 JSONL 匯出、清理與每日 OK / Warning / Critical 摘要。

 ---

--- a/docs/adr/ADR-012-agent-action-ladder.md
+++ b/docs/adr/ADR-012-agent-action-ladder.md
@@ -146,7 +146,8 @@ L1 Hermes 掛  → L0 模板直出 + 🟡 「AI 分析暫不可用」
 - 2026-04-29 已補 `/metrics` 匯出：EventRouter dispatch、L2 safe action、Telegram replay、AutoHeal action 與 latency/duration。
 - 2026-04-29 已補 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`：EventRouter、AutoHeal、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL 線上快檢。
 - 2026-04-29 已補 smoke 結果 JSONL 保存與 dashboard 趨勢視覺化。
- 尚未完成：Grafana/Superset 視覺化面板與 smoke 每日摘要。
+- 2026-04-29 已補 smoke history JSONL 匯出、清理與每日摘要。
+- 尚未完成：Grafana/Superset 視覺化面板與每日摘要 Telegram 推播。

 ## References
 - `services/event_router.py` — 分流入口（Phase 1）
--- a/docs/guides/ai_automation_session_sop.md
+++ b/docs/guides/ai_automation_session_sop.md
@@ -27,6 +27,7 @@
 - EventRouter / AutoHeal 變更必須更新 `services/ai_automation_metrics.py` 指標或確認既有指標已覆蓋。
 - AI 自動化閉環變更必須確認 `/api/ai-automation/smoke` 與 `/ai_automation_smoke` 仍能反映新狀態。
 - Smoke dashboard 會保存 JSONL 趨勢；若新增檢查項目，要確保 history compact record 仍保持小而可讀。
+- Smoke history 管理只能操作 `MOMO_AI_AUTOMATION_SMOKE_HISTORY` 指向的 JSONL，不得清理 DB 或 EventRouter queue。
 - L2 action 必須在 `SAFE_ACTIONS` 且可審計、可回放、低副作用。
 - AutoHeal 不得 restart / stop / recreate `momo-db` 或 `momo-postgres`。
 - raw `ai_insights` 寫入後必須 enqueue embedding；若 enqueue 失敗，必須可 backfill。
--- a/docs/memory/ai_automation_closure_20260429.md
+++ b/docs/memory/ai_automation_closure_20260429.md
@@ -12,6 +12,7 @@
 - AI 自動化最小 Prometheus 指標已接入 `/metrics`，來源為 `services/ai_automation_metrics.py`。
 - 線上 smoke dashboard 已接入 `/ai_automation_smoke`，JSON API 為 `/api/ai-automation/smoke`。
 - Smoke API 會保存最近快檢 JSONL 趨勢，dashboard 顯示 OK / Warning / Critical 最近分布。
+- Smoke history 已支援 JSONL 匯出、清理與每日摘要；清理只影響 smoke history，不碰 DB 或 EventRouter queue。

 ## 已落地範圍

@@ -26,12 +27,14 @@
 - `/metrics` 已匯出 EventRouter dispatch、latency、safe action、Telegram replay、AutoHeal action 與 duration 指標。
 - Smoke dashboard read-only 檢查 EventRouter queue、AutoHeal protected resources、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL，不做外部網路呼叫。
 - Smoke history 只保存精簡紀錄，不保存完整 details，避免長期檔案膨脹與敏感資訊堆積。
+- Export API 回傳 `application/x-ndjson`，clear API 只刪除 `MOMO_AI_AUTOMATION_SMOKE_HISTORY` 指向檔案。

 ## 驗證紀錄

 - 2026-04-29 AI metrics 批次：`26 passed`。
 - 2026-04-29 AI smoke dashboard 批次：`2 passed`（單檔 smoke service），後續核心組需持續納入。
 - 2026-04-29 AI smoke trend 批次：`5 passed`（smoke + metrics）。
+- 2026-04-29 AI smoke management 批次：`7 passed`（smoke + metrics）。
 - 2026-04-29 L2 安全記憶批次：`24 passed`。
 - collect-only：`48 tests collected`。
 - `git diff --check` 已通過。
--- a/docs/memory/history_logs.md
+++ b/docs/memory/history_logs.md
@@ -27,6 +27,7 @@
 - **可觀測性落地**: `/metrics` 匯出 EventRouter dispatch/latency、safe action、Telegram replay、AutoHeal action/duration 指標。
 - **Smoke Dashboard**: 新增 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`，提供四 Agent 閉環 read-only 快檢。
 - **Smoke 趨勢保存**: Smoke API 追加 JSONL 精簡紀錄，dashboard 顯示最近 OK / Warning / Critical 趨勢。
+- **Smoke 趨勢管理**: Dashboard 增加 JSONL 匯出、清理與每日摘要，清理範圍限定 smoke history 檔。

 ### 2026-04-28~29：Phase 3e 重構大戰 + daily_sales cache 隱形 bug 根除
 - **app.py 縮減 -10.8%**: 7,386 → 6,590 行，11 commits 全綠零 502。
--- a/routes/README.md
+++ b/routes/README.md
@@ -14,7 +14,7 @@
 |------|------|----------|
 | `dashboard_routes.py` | 商品看板首頁 | `/` |
 | `sales_routes.py` | 業績分析與 ABC 明細 | `/sales_analysis`, `/growth_analysis`, `/abc_analysis/detail`, `/api/sales_analysis/*` |
-| `system_public_routes.py` | 無 prefix 公開系統頁與監控 | `/health`, `/metrics`, `/ai_automation_smoke`, `/api/ai-automation/smoke`, `/settings`, `/system_settings`, `/logs`, `/api/logs`, `/api/backup` |
+| `system_public_routes.py` | 無 prefix 公開系統頁與監控 | `/health`, `/metrics`, `/ai_automation_smoke`, `/api/ai-automation/smoke*`, `/settings`, `/system_settings`, `/logs`, `/api/logs`, `/api/backup` |
 | `system_routes.py` | 內部系統維護 API | `/api/system/*` |
 | `edm_routes.py` | EDM 與節慶儀表板 | `/edm`, `/festival` |
 | `monthly_routes.py` | 月結分析 | `/monthly_summary_analysis`, `/api/monthly_summary_data` |
--- a/routes/system_public_routes.py
+++ b/routes/system_public_routes.py
@@ -211,6 +211,26 @@ def ai_automation_smoke_api():
    return jsonify(collect_ai_automation_smoke())


+@system_public_bp.route('/api/ai-automation/smoke/history/export')
+@login_required
+def ai_automation_smoke_history_export():
+    """Export compact smoke history JSONL."""
+    from services.ai_automation_smoke_service import export_smoke_history_jsonl
+    export = export_smoke_history_jsonl()
+    response = Response(export["content"], mimetype='application/x-ndjson; charset=utf-8')
+    response.headers["Content-Disposition"] = "attachment; filename=ai_automation_smoke_history.jsonl"
+    response.headers["X-Smoke-History-Count"] = str(export["count"])
+    return response
+
+
+@system_public_bp.route('/api/ai-automation/smoke/history/clear', methods=['POST'])
+@login_required
+def ai_automation_smoke_history_clear():
+    """Clear local compact smoke history JSONL."""
+    from services.ai_automation_smoke_service import clear_smoke_history
+    return jsonify(clear_smoke_history())
+
+
@system_public_bp.route('/logs')
 def show_logs():
    return render_template('logs.html')
--- a/services/ai_automation_smoke_service.py
+++ b/services/ai_automation_smoke_service.py
@@ -60,13 +60,17 @@ def _compact_history_record(result: Dict[str, Any]) -> Dict[str, Any]:
    }


+def _read_history_lines() -> List[str]:
+    with _HISTORY_LOCK:
+        with open(_HISTORY_PATH, "r", encoding="utf-8") as fh:
+            return fh.readlines()
+
+
 def _load_history(limit: int = 20) -> List[Dict[str, Any]]:
    if limit <= 0:
        return []
    try:
-        with _HISTORY_LOCK:
-            with open(_HISTORY_PATH, "r", encoding="utf-8") as fh:
-                lines = fh.readlines()
+        lines = _read_history_lines()
    except FileNotFoundError:
        return []

@@ -108,9 +112,50 @@ def _history_summary(records: List[Dict[str, Any]]) -> Dict[str, Any]:
        "counts": counts,
        "recent": records,
        "latest": records[-1] if records else None,
+        "daily": _daily_summary(records),
    }


+def _daily_summary(records: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    by_day: Dict[str, Dict[str, int]] = {}
+    for record in records:
+        day = str(record.get("generated_at") or "unknown")[:10]
+        bucket = by_day.setdefault(day, {"ok": 0, "warning": 0, "critical": 0, "total": 0})
+        status = record.get("status", "critical")
+        if status in bucket:
+            bucket[status] += 1
+        bucket["total"] += 1
+    return [
+        {"date": day, **counts}
+        for day, counts in sorted(by_day.items())[-14:]
+    ]
+
+
+def export_smoke_history_jsonl() -> Dict[str, Any]:
+    try:
+        lines = _read_history_lines()
+    except FileNotFoundError:
+        lines = []
+    return {
+        "content": "".join(lines),
+        "count": sum(1 for line in lines if line.strip()),
+        "path": _HISTORY_PATH,
+    }
+
+
+def clear_smoke_history() -> Dict[str, Any]:
+    try:
+        cleared = _count_jsonl_lines(_HISTORY_PATH)
+        with _HISTORY_LOCK:
+            try:
+                os.remove(_HISTORY_PATH)
+            except FileNotFoundError:
+                cleared = 0
+        return {"cleared": cleared, "path": _HISTORY_PATH}
+    except Exception as exc:
+        return {"cleared": 0, "path": _HISTORY_PATH, "error": str(exc)[:300]}
+
+
 def _event_router_check() -> Dict[str, Any]:
    try:
        from services import event_router
--- a/templates/ai_automation_smoke.html
+++ b/templates/ai_automation_smoke.html
@@ -114,6 +114,12 @@
            <button id="refreshBtn" class="btn btn-light mt-3">
                <i class="fas fa-sync-alt me-2"></i>重新檢查
            </button>
+            <a class="btn btn-outline-light mt-3 ms-lg-2" href="/api/ai-automation/smoke/history/export">
+                <i class="fas fa-file-export me-2"></i>匯出 JSONL
+            </a>
+            <button id="clearHistoryBtn" class="btn btn-outline-warning mt-3 ms-lg-2">
+                <i class="fas fa-broom me-2"></i>清理趨勢
+            </button>
        </div>
    </div>
 </div>
@@ -138,6 +144,33 @@
    </div>
 </div>

+<div class="card smoke-card mb-4">
+    <div class="card-body">
+        <div class="d-flex flex-column flex-md-row justify-content-between gap-2 mb-3">
+            <div>
+                <h5 class="mb-1">每日摘要</h5>
+                <div class="text-muted small">依最近保存紀錄彙整，方便快速觀察是否有連續 warning / critical。</div>
+            </div>
+        </div>
+        <div class="table-responsive">
+            <table class="table align-middle mb-0">
+                <thead>
+                    <tr>
+                        <th>日期</th>
+                        <th>OK</th>
+                        <th>Warning</th>
+                        <th>Critical</th>
+                        <th>Total</th>
+                    </tr>
+                </thead>
+                <tbody id="dailySummaryRows">
+                    <tr><td colspan="5" class="text-muted">等待資料...</td></tr>
+                </tbody>
+            </table>
+        </div>
+    </div>
+</div>
+
 <div class="smoke-grid" id="checkGrid"></div>
 {% endblock %}

@@ -197,6 +230,24 @@ function renderTrend(history) {
             title="${escapeHtml((item.generated_at || '-') + ' · ' + (item.status || 'unknown'))}">
        </div>
    `).join('');
+    renderDailySummary(history.daily || []);
+}
+
+function renderDailySummary(rows) {
+    const body = document.getElementById('dailySummaryRows');
+    if (!rows.length) {
+        body.innerHTML = '<tr><td colspan="5" class="text-muted">尚無每日摘要。</td></tr>';
+        return;
+    }
+    body.innerHTML = rows.slice().reverse().map(row => `
+        <tr>
+            <td>${escapeHtml(row.date)}</td>
+            <td class="text-success fw-semibold">${row.ok || 0}</td>
+            <td class="text-warning fw-semibold">${row.warning || 0}</td>
+            <td class="text-danger fw-semibold">${row.critical || 0}</td>
+            <td>${row.total || 0}</td>
+        </tr>
+    `).join('');
 }

 async function loadSmoke() {
@@ -223,6 +274,24 @@ async function loadSmoke() {
 }

 document.getElementById('refreshBtn').addEventListener('click', loadSmoke);
+document.getElementById('clearHistoryBtn').addEventListener('click', async () => {
+    if (!confirm('確定要清理 AI Smoke 趨勢紀錄？這只會刪除本頁 JSONL history，不會影響 DB 或事件資料。')) {
+        return;
+    }
+    const btn = document.getElementById('clearHistoryBtn');
+    btn.disabled = true;
+    try {
+        const res = await fetchWithCSRF('/api/ai-automation/smoke/history/clear', {method: 'POST'});
+        if (!res.ok) throw new Error(`HTTP ${res.status}`);
+        const data = await res.json();
+        showToast(`已清理 ${data.cleared || 0} 筆 Smoke 趨勢紀錄`, 'success');
+        await loadSmoke();
+    } catch (err) {
+        showToast(`清理失敗：${err.message}`, 'error');
+    } finally {
+        btn.disabled = false;
+    }
+});
 loadSmoke();
 </script>
 {% endblock %}
--- a/tests/test_ai_automation_smoke_service.py
+++ b/tests/test_ai_automation_smoke_service.py
@@ -58,3 +58,38 @@ def test_collect_ai_automation_smoke_persists_recent_history(tmp_path, monkeypat
    assert second["history"]["counts"]["ok"] == 2
    assert third["history"]["counts"]["ok"] == 2
    assert len(history_path.read_text(encoding="utf-8").strip().splitlines()) == 2
+
+
+def test_smoke_history_export_and_clear(tmp_path, monkeypatch):
+    from services import ai_automation_smoke_service as smoke
+
+    history_path = tmp_path / "smoke_history.jsonl"
+    history_path.write_text(
+        '{"generated_at":"2026-04-29T01:00:00","status":"ok"}\n'
+        '{"generated_at":"2026-04-29T02:00:00","status":"warning"}\n',
+        encoding="utf-8",
+    )
+    monkeypatch.setattr(smoke, "_HISTORY_PATH", str(history_path))
+
+    export = smoke.export_smoke_history_jsonl()
+    cleared = smoke.clear_smoke_history()
+
+    assert export["count"] == 2
+    assert '"status":"warning"' in export["content"]
+    assert cleared["cleared"] == 2
+    assert not history_path.exists()
+
+
+def test_smoke_history_daily_summary():
+    from services import ai_automation_smoke_service as smoke
+
+    summary = smoke._history_summary([
+        {"generated_at": "2026-04-28T23:00:00", "status": "ok"},
+        {"generated_at": "2026-04-29T01:00:00", "status": "warning"},
+        {"generated_at": "2026-04-29T02:00:00", "status": "critical"},
+    ])
+
+    assert summary["daily"] == [
+        {"date": "2026-04-28", "ok": 1, "warning": 0, "critical": 0, "total": 1},
+        {"date": "2026-04-29", "ok": 0, "warning": 1, "critical": 1, "total": 2},
+    ]