保存 AI Smoke 趨勢紀錄

CONSTITUTION.md

@@ -2,7 +2,7 @@
 
 > 本文件定義專案開發的核心準則與不可違反的規範
 > **建立日期**: 2026-01-12
-> **當前版本**: V10.6 (四 AI Agent 自動化 Smoke Dashboard 版)
+> **當前版本**: V10.7 (四 AI Agent 自動化 Smoke 趨勢版)
 > **最後更新**: 2026-04-29
 
 ---

TODO_NEXT_STEPS.txt

@@ -7,14 +7,15 @@
    - ADR-018：四 AI Agent 自動化控制面立案。
    - Memory：新增 `docs/memory/ai_automation_closure_20260429.md`。
    - Guide/Skills 替代：新增 `docs/guides/ai_automation_session_sop.md`。
-   - SOT：更新 `docs/AI_INTELLIGENCE_MODULE_SOT.md` 至 V10.6 AI Automation Smoke Dashboard 架構。
+   - SOT：更新 `docs/AI_INTELLIGENCE_MODULE_SOT.md` 至 V10.7 AI Automation Smoke Trend 架構。
    - Codex 規則：更新 `AGENTS.md`、`CONSTITUTION.md`、ADR/memory 索引。
    - Prometheus 指標化：新增 EventRouter / AutoHeal / safe action / replay in-process metrics，並接入 `/metrics`。
    - 線上 smoke dashboard：新增 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`，覆蓋 EventRouter、AutoHeal、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL。
+   - Smoke 趨勢保存：`/api/ai-automation/smoke` 每次快檢追加 JSONL 精簡紀錄，dashboard 顯示最近趨勢。
 
 【下次待辦】
    - Superset / Grafana 視覺化：`momo_ai_event_router_dispatch_total`、`momo_ai_event_router_latency_ms_*`、`momo_ai_autoheal_action_total`。
-   - Smoke dashboard 增加最近一次實際 smoke test 結果保存與趨勢圖。
+   - Smoke trend 增加手動清理/匯出與每日摘要。
 
 ================================================================================
                       品牌資產最終處理與維護 (Phase 7) [DONE]

app.py

@@ -95,9 +95,9 @@ except Exception as e:
     sys_log.error(f"無法檢測磁碟空間: {e}")
 
 # 🚩 系統版本定義 (備份與顯示用)
-# 🚩 2026-04-29 V10.6: AI 自動化 Smoke Dashboard — EventRouter / AutoHeal /
-#                  NemoTron / OpenClaw / ElephantAlpha 線上閉環快檢
-SYSTEM_VERSION = "V10.6"
+# 🚩 2026-04-29 V10.7: AI Smoke 趨勢保存 — 最近快檢結果 JSONL
+#                  持久化與 dashboard 趨勢視覺化
+SYSTEM_VERSION = "V10.7"
 
 # ==========================================
 # 🔒 SQL Injection 防護函數

config.py

@@ -253,7 +253,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
 # ==========================================
 # 系統版本與路徑
 # ==========================================
-SYSTEM_VERSION = "V10.6"
+SYSTEM_VERSION = "V10.7"
 LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
 public_url = PUBLIC_URL  # 用於模板顯示
 

docs/AI_INTELLIGENCE_MODULE_SOT.md

@@ -1,8 +1,8 @@
 # MOMO PRO — AI 競價情報模組 Single Source of Truth
 
 > **最後更新**: 2026-04-29 (台北時間)
-> **狀態**: 🟢 四 AI Agent 自動化閉環已落地 — EventRouter / AutoHeal / OpenClaw Memory / ElephantAlpha bridge / Prometheus metrics / Smoke Dashboard 具測試覆蓋
-> **適用版本**: V10.6 AI Automation Smoke Dashboard 架構
+> **狀態**: 🟢 四 AI Agent 自動化閉環已落地 — EventRouter / AutoHeal / OpenClaw Memory / ElephantAlpha bridge / Prometheus metrics / Smoke Dashboard / Smoke Trend 具測試覆蓋
+> **適用版本**: V10.7 AI Automation Smoke Trend 架構
 
 ---
 
@@ -64,6 +64,7 @@ SQL漏斗(~300筆)
 - `/metrics` 匯出 `momo_ai_autoheal_action_total` 與 `momo_ai_autoheal_duration_ms_count/sum/max`。
 - `/ai_automation_smoke` 提供登入後 smoke dashboard。
 - `/api/ai-automation/smoke` 提供 read-only JSON 狀態，不做外部網路呼叫。
+- Smoke API 會將最近快檢結果保存到 JSONL，dashboard 顯示最近狀態趨勢。
 
 ---
 

docs/adr/ADR-012-agent-action-ladder.md

@@ -145,7 +145,8 @@ L1 Hermes 掛  → L0 模板直出 + 🟡 「AI 分析暫不可用」
 - L3 已擴展為 OpenClaw + ElephantAlpha：OpenClaw 負責策略/記憶，ElephantAlpha 負責 orchestration/HITL/AutoHeal bridge。
 - 2026-04-29 已補 `/metrics` 匯出：EventRouter dispatch、L2 safe action、Telegram replay、AutoHeal action 與 latency/duration。
 - 2026-04-29 已補 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`：EventRouter、AutoHeal、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL 線上快檢。
-- 尚未完成：Grafana/Superset 視覺化面板與 smoke 結果趨勢保存。
+- 2026-04-29 已補 smoke 結果 JSONL 保存與 dashboard 趨勢視覺化。
+- 尚未完成：Grafana/Superset 視覺化面板與 smoke 每日摘要。
 
 ## References
 - `services/event_router.py` — 分流入口（Phase 1）

docs/guides/ai_automation_session_sop.md

@@ -26,6 +26,7 @@
 - Telegram 失敗必須可暫存與 replay。
 - EventRouter / AutoHeal 變更必須更新 `services/ai_automation_metrics.py` 指標或確認既有指標已覆蓋。
 - AI 自動化閉環變更必須確認 `/api/ai-automation/smoke` 與 `/ai_automation_smoke` 仍能反映新狀態。
+- Smoke dashboard 會保存 JSONL 趨勢；若新增檢查項目，要確保 history compact record 仍保持小而可讀。
 - L2 action 必須在 `SAFE_ACTIONS` 且可審計、可回放、低副作用。
 - AutoHeal 不得 restart / stop / recreate `momo-db` 或 `momo-postgres`。
 - raw `ai_insights` 寫入後必須 enqueue embedding；若 enqueue 失敗，必須可 backfill。

docs/memory/ai_automation_closure_20260429.md

@@ -11,6 +11,7 @@
 - ElephantAlpha 只負責 orchestration / HITL / AutoHeal bridge，不可繞過 ADR-011、ADR-012、ADR-013。
 - AI 自動化最小 Prometheus 指標已接入 `/metrics`，來源為 `services/ai_automation_metrics.py`。
 - 線上 smoke dashboard 已接入 `/ai_automation_smoke`，JSON API 為 `/api/ai-automation/smoke`。
+- Smoke API 會保存最近快檢 JSONL 趨勢，dashboard 顯示 OK / Warning / Critical 最近分布。
 
 ## 已落地範圍
 
@@ -24,11 +25,13 @@
 - L2 `agent_actions.py` 的 `flag_for_human_review`、`route_to_km`、`mark_for_relearn` 已從 stub 改為可審計 OpenClaw memory 寫入。
 - `/metrics` 已匯出 EventRouter dispatch、latency、safe action、Telegram replay、AutoHeal action 與 duration 指標。
 - Smoke dashboard read-only 檢查 EventRouter queue、AutoHeal protected resources、NemoTron fallback、OpenClaw embedding queue、ElephantAlpha HITL，不做外部網路呼叫。
+- Smoke history 只保存精簡紀錄，不保存完整 details，避免長期檔案膨脹與敏感資訊堆積。
 
 ## 驗證紀錄
 
 - 2026-04-29 AI metrics 批次：`26 passed`。
 - 2026-04-29 AI smoke dashboard 批次：`2 passed`（單檔 smoke service），後續核心組需持續納入。
+- 2026-04-29 AI smoke trend 批次：`5 passed`（smoke + metrics）。
 - 2026-04-29 L2 安全記憶批次：`24 passed`。
 - collect-only：`48 tests collected`。
 - `git diff --check` 已通過。

docs/memory/history_logs.md

@@ -26,6 +26,7 @@
 - **L2 action 落地**: `flag_for_human_review`、`route_to_km`、`mark_for_relearn` 改為可審計 OpenClaw memory 寫入。
 - **可觀測性落地**: `/metrics` 匯出 EventRouter dispatch/latency、safe action、Telegram replay、AutoHeal action/duration 指標。
 - **Smoke Dashboard**: 新增 `/ai_automation_smoke` 與 `/api/ai-automation/smoke`，提供四 Agent 閉環 read-only 快檢。
+- **Smoke 趨勢保存**: Smoke API 追加 JSONL 精簡紀錄，dashboard 顯示最近 OK / Warning / Critical 趨勢。
 
 ### 2026-04-28~29：Phase 3e 重構大戰 + daily_sales cache 隱形 bug 根除
 - **app.py 縮減 -10.8%**: 7,386 → 6,590 行，11 commits 全綠零 502。

services/ai_automation_smoke_service.py

@@ -7,6 +7,8 @@ are meant for a fast dashboard/API sanity check, not for deep production probes.
 from __future__ import annotations
 
 import os
+import json
+import threading
 from datetime import datetime
 from typing import Any, Dict, List
 
@@ -17,6 +19,12 @@ from database.manager import get_session
 
 
 STATUS_RANK = {"ok": 0, "warning": 1, "critical": 2}
+_HISTORY_PATH = os.getenv(
+    "MOMO_AI_AUTOMATION_SMOKE_HISTORY",
+    os.path.join(os.path.dirname(os.path.dirname(__file__)), "data", "ai_automation_smoke_history.jsonl"),
+)
+_HISTORY_LIMIT = int(os.getenv("MOMO_AI_AUTOMATION_SMOKE_HISTORY_LIMIT", "200"))
+_HISTORY_LOCK = threading.Lock()
 
 
 def _check(name: str, status: str, summary: str, details: Dict[str, Any] | None = None) -> Dict[str, Any]:
@@ -36,6 +44,73 @@ def _count_jsonl_lines(path: str) -> int:
         return 0
 
 
+def _compact_history_record(result: Dict[str, Any]) -> Dict[str, Any]:
+    return {
+        "generated_at": result.get("generated_at"),
+        "status": result.get("status", "critical"),
+        "summary": result.get("summary", {}),
+        "checks": [
+            {
+                "name": item.get("name"),
+                "status": item.get("status"),
+                "summary": item.get("summary"),
+            }
+            for item in result.get("checks", [])
+        ],
+    }
+
+
+def _load_history(limit: int = 20) -> List[Dict[str, Any]]:
+    if limit <= 0:
+        return []
+    try:
+        with _HISTORY_LOCK:
+            with open(_HISTORY_PATH, "r", encoding="utf-8") as fh:
+                lines = fh.readlines()
+    except FileNotFoundError:
+        return []
+
+    records = []
+    for line in lines[-limit:]:
+        try:
+            records.append(json.loads(line))
+        except json.JSONDecodeError:
+            continue
+    return records
+
+
+def _append_history(result: Dict[str, Any]) -> None:
+    record = _compact_history_record(result)
+    os.makedirs(os.path.dirname(_HISTORY_PATH), exist_ok=True)
+    with _HISTORY_LOCK:
+        existing = []
+        try:
+            with open(_HISTORY_PATH, "r", encoding="utf-8") as fh:
+                existing = fh.readlines()
+        except FileNotFoundError:
+            pass
+
+        existing.append(json.dumps(record, ensure_ascii=False, default=str) + "\n")
+        if len(existing) > _HISTORY_LIMIT:
+            existing = existing[-_HISTORY_LIMIT:]
+
+        with open(_HISTORY_PATH, "w", encoding="utf-8") as fh:
+            fh.writelines(existing)
+
+
+def _history_summary(records: List[Dict[str, Any]]) -> Dict[str, Any]:
+    counts = {"ok": 0, "warning": 0, "critical": 0}
+    for record in records:
+        status = record.get("status", "critical")
+        if status in counts:
+            counts[status] += 1
+    return {
+        "counts": counts,
+        "recent": records,
+        "latest": records[-1] if records else None,
+    }
+
+
 def _event_router_check() -> Dict[str, Any]:
     try:
         from services import event_router
@@ -187,7 +262,7 @@ def _elephant_hitl_check() -> Dict[str, Any]:
         return _check("ElephantAlpha HITL", "critical", f"ElephantAlpha smoke 失敗：{exc}")
 
 
-def collect_ai_automation_smoke() -> Dict[str, Any]:
+def collect_ai_automation_smoke(*, record_history: bool = True, history_limit: int = 20) -> Dict[str, Any]:
     checks: List[Dict[str, Any]] = [
         _event_router_check(),
         _autoheal_check(),
@@ -196,7 +271,7 @@ def collect_ai_automation_smoke() -> Dict[str, Any]:
         _elephant_hitl_check(),
     ]
     worst = max(checks, key=lambda item: STATUS_RANK.get(item["status"], 2))["status"]
-    return {
+    result = {
         "status": worst,
         "version": SYSTEM_VERSION,
         "generated_at": datetime.now().isoformat(timespec="seconds"),
@@ -208,3 +283,10 @@ def collect_ai_automation_smoke() -> Dict[str, Any]:
             "total": len(checks),
         },
     }
+    if record_history:
+        try:
+            _append_history(result)
+        except Exception as exc:
+            result["history_error"] = str(exc)[:300]
+    result["history"] = _history_summary(_load_history(limit=history_limit))
+    return result

templates/ai_automation_smoke.html

@@ -80,6 +80,23 @@
         overflow: auto;
         white-space: pre-wrap;
     }
+
+    .trend-strip {
+        display: grid;
+        grid-template-columns: repeat(auto-fit, minmax(14px, 1fr));
+        gap: 6px;
+        align-items: end;
+    }
+
+    .trend-dot {
+        height: 34px;
+        border-radius: 8px;
+        opacity: .92;
+    }
+
+    .trend-dot.status-ok { background: #22c55e; }
+    .trend-dot.status-warning { background: #f59e0b; }
+    .trend-dot.status-critical { background: #ef4444; }
 </style>
 {% endblock %}
 
@@ -108,6 +125,19 @@
     <div class="col-md-3"><div class="card smoke-card"><div class="card-body"><div class="text-muted small">Generated</div><div class="fs-6 fw-semibold" id="generatedAt">-</div></div></div></div>
 </div>
 
+<div class="card smoke-card mb-4">
+    <div class="card-body">
+        <div class="d-flex flex-column flex-md-row justify-content-between gap-2 mb-3">
+            <div>
+                <h5 class="mb-1">最近 Smoke 趨勢</h5>
+                <div class="text-muted small">每次 API 快檢會保存一筆精簡紀錄，保留最近 200 筆。</div>
+            </div>
+            <div class="text-muted small" id="historySummary">等待資料...</div>
+        </div>
+        <div class="trend-strip" id="trendStrip"></div>
+    </div>
+</div>
+
 <div class="smoke-grid" id="checkGrid"></div>
 {% endblock %}
 
@@ -134,6 +164,7 @@ function renderSmoke(data) {
     document.getElementById('warningCount').textContent = data.summary.warning ?? 0;
     document.getElementById('criticalCount').textContent = data.summary.critical ?? 0;
     document.getElementById('generatedAt').textContent = data.generated_at || '-';
+    renderTrend(data.history || {});
 
     const grid = document.getElementById('checkGrid');
     grid.innerHTML = data.checks.map(item => `
@@ -150,6 +181,24 @@ function renderSmoke(data) {
     `).join('');
 }
 
+function renderTrend(history) {
+    const recent = history.recent || [];
+    const counts = history.counts || {ok: 0, warning: 0, critical: 0};
+    document.getElementById('historySummary').textContent =
+        `OK ${counts.ok || 0} / Warning ${counts.warning || 0} / Critical ${counts.critical || 0}`;
+
+    const strip = document.getElementById('trendStrip');
+    if (!recent.length) {
+        strip.innerHTML = '<div class="text-muted small">尚無歷史紀錄。</div>';
+        return;
+    }
+    strip.innerHTML = recent.map(item => `
+        <div class="trend-dot status-${item.status || 'critical'}"
+             title="${escapeHtml((item.generated_at || '-') + ' · ' + (item.status || 'unknown'))}">
+        </div>
+    `).join('');
+}
+
 async function loadSmoke() {
     const btn = document.getElementById('refreshBtn');
     btn.disabled = true;

tests/test_ai_automation_smoke_service.py

@@ -32,7 +32,29 @@ def test_collect_ai_automation_smoke_uses_worst_status(monkeypatch):
     monkeypatch.setattr(smoke, "_embedding_queue_check", lambda: smoke._check("embedding", "critical", "boom"))
     monkeypatch.setattr(smoke, "_elephant_hitl_check", lambda: smoke._check("elephant", "ok", "ok"))
 
-    result = smoke.collect_ai_automation_smoke()
+    result = smoke.collect_ai_automation_smoke(record_history=False)
 
     assert result["status"] == "critical"
     assert result["summary"] == {"ok": 3, "warning": 1, "critical": 1, "total": 5}
+
+
+def test_collect_ai_automation_smoke_persists_recent_history(tmp_path, monkeypatch):
+    from services import ai_automation_smoke_service as smoke
+
+    history_path = tmp_path / "smoke_history.jsonl"
+    monkeypatch.setattr(smoke, "_HISTORY_PATH", str(history_path))
+    monkeypatch.setattr(smoke, "_HISTORY_LIMIT", 2)
+    monkeypatch.setattr(smoke, "_event_router_check", lambda: smoke._check("event", "ok", "ok"))
+    monkeypatch.setattr(smoke, "_autoheal_check", lambda: smoke._check("autoheal", "ok", "ok"))
+    monkeypatch.setattr(smoke, "_nemotron_check", lambda: smoke._check("nemotron", "ok", "ok"))
+    monkeypatch.setattr(smoke, "_embedding_queue_check", lambda: smoke._check("embedding", "ok", "ok"))
+    monkeypatch.setattr(smoke, "_elephant_hitl_check", lambda: smoke._check("elephant", "ok", "ok"))
+
+    first = smoke.collect_ai_automation_smoke(history_limit=5)
+    second = smoke.collect_ai_automation_smoke(history_limit=5)
+    third = smoke.collect_ai_automation_smoke(history_limit=5)
+
+    assert first["status"] == "ok"
+    assert second["history"]["counts"]["ok"] == 2
+    assert third["history"]["counts"]["ok"] == 2
+    assert len(history_path.read_text(encoding="utf-8").strip().splitlines()) == 2