From 8bd44b11310b0ee937664739fea255135d3c22a1 Mon Sep 17 00:00:00 2001 From: OoO Date: Thu, 30 Apr 2026 09:02:29 +0800 Subject: [PATCH] =?UTF-8?q?=E4=BF=AE=E5=BE=A9=20CD=20sync=20=E5=BE=8C?= =?UTF-8?q?=E6=9C=AA=20reload?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitea/workflows/cd.yaml | 3 ++- TODO_NEXT_STEPS.txt | 1 + docs/guides/devops_handbook.md | 2 +- docs/memory/ai_automation_closure_20260429.md | 1 + docs/memory/history_logs.md | 1 + tests/test_cd_health_check.py | 7 +++++++ 6 files changed, 13 insertions(+), 2 deletions(-) diff --git a/.gitea/workflows/cd.yaml b/.gitea/workflows/cd.yaml index 42535c7..61120c1 100644 --- a/.gitea/workflows/cd.yaml +++ b/.gitea/workflows/cd.yaml @@ -172,10 +172,11 @@ jobs: run: | # ROOT CAUSE FIX (2026-04-28): 從 `restart` 改為 `up -d` # restart 對「不存在的容器」會失敗 → 任一外力清掉容器後下次 sync 必爆 502 - # up -d 對 image 未變的容器是 no-op（只觸發熱掛載），對不存在則自動 create + # up -d 對不存在容器會自動 create；restart 讓 Python process 重新載入 bind-mounted 檔案 ssh -i ~/.ssh/id_deploy ollama@192.168.0.188 \ "cd /home/ollama/momo-pro && \ docker compose up -d --no-deps momo-app scheduler telegram-bot && \ + docker compose restart momo-app scheduler telegram-bot && \ echo '✅ 三容器已就緒（app/scheduler/telegram-bot）'" # ── 模式 B：重建 Docker Image（Dockerfile / requirements.txt 變動） ── diff --git a/TODO_NEXT_STEPS.txt b/TODO_NEXT_STEPS.txt index 4b462f2..0e3c514 100644 --- a/TODO_NEXT_STEPS.txt +++ b/TODO_NEXT_STEPS.txt @@ -19,6 +19,7 @@ - Prometheus scrape 修復：active monitoring stack 新增 `momo-app` scrape job，目標 `momo-pro-system:80/metrics`。 - Gunicorn preload 修復：`post_fork` 略過 Flask/Werkzeug request-bound LocalProxy，避免 worker boot fail。 - CD 健康檢查強化：改為 internal container health + external `mo.wooo.work` 雙檢查，重試窗延長到約 3 分鐘。 + - CD Sync reload 修復：rsync 後明確 `docker compose restart momo-app scheduler telegram-bot`，避免檔案已同步但 process 仍跑舊版。 【下次待辦】 - 觀察 Prometheus scrape 後 `momo_ai_*` 是否在事件發生後產生時間序列。 diff --git a/docs/guides/devops_handbook.md b/docs/guides/devops_handbook.md index e086f59..195a393 100644 --- a/docs/guides/devops_handbook.md +++ b/docs/guides/devops_handbook.md @@ -71,7 +71,7 @@ ### 3. CI/CD 報錯 `No such container: momo-pro-system` - **原因**: Sync 模式使用硬編碼容器名，但容器可能已被重建或改名。 -- **修復**: 使用 `docker compose restart ` 取代直接重啟容器。 +- **修復**: 先 `docker compose up -d --no-deps momo-app scheduler telegram-bot` 確保容器存在，再 `docker compose restart momo-app scheduler telegram-bot` 讓 bind-mounted Python 檔案重新載入。 ### 4. Telegram Bot 代碼更新無效 (幽靈容器) - **原因**: `docker-compose.yml` 遺漏了 `/app/routes` 的 Volume 掛載。 diff --git a/docs/memory/ai_automation_closure_20260429.md b/docs/memory/ai_automation_closure_20260429.md index ca063c1..eab92c1 100644 --- a/docs/memory/ai_automation_closure_20260429.md +++ b/docs/memory/ai_automation_closure_20260429.md @@ -19,6 +19,7 @@ - 2026-04-30 active Prometheus 補 `momo-app` scrape job，目標 `momo-pro-system:80/metrics`；Prometheus 需加入 `momo-network` 才能解析 app container DNS。 - 2026-04-30 發現並修復 `gunicorn.conf.py` `post_fork` 掃到 Flask/Werkzeug LocalProxy 導致 worker boot fail 的問題。 - 2026-04-30 CD 健康檢查曾因 rebuild 後短暫 502 太早失敗；已改為 internal `docker exec momo-pro-system /health` + external `https://mo.wooo.work/health` 雙檢查，重試約 3 分鐘。 +- 2026-04-30 CD Sync 模式曾只 rsync + `docker compose up -d`，導致 host 檔案已是新版但 gunicorn process 仍跑舊版；已補 `docker compose restart momo-app scheduler telegram-bot`。 ## 已落地範圍 diff --git a/docs/memory/history_logs.md b/docs/memory/history_logs.md index 10e5e1f..b204cdd 100644 --- a/docs/memory/history_logs.md +++ b/docs/memory/history_logs.md @@ -32,6 +32,7 @@ - **Grafana AI 觀測**: 新增 `MOMO AI Automation Overview` provisioning dashboard，覆蓋 EventRouter、safe action、replay、AutoHeal Prometheus 指標。 - **Grafana 線上載入與 scrape 修復**: 188 active Grafana 載入 4 dashboards；active Prometheus 補 `momo-app` scrape job，並修復 gunicorn preload LocalProxy boot crash。 - **CD 健康檢查強化**: Gitea Actions health check 改為 internal container health + external URL 雙檢查，降低 rebuild 後短暫 502 誤判。 +- **CD Sync reload 修復**: rsync 後明確 restart 三容器，避免 bind-mounted Python 檔案更新但 gunicorn/scheduler/bot process 未 reload。 ### 2026-04-28~29：Phase 3e 重構大戰 + daily_sales cache 隱形 bug 根除 - **app.py 縮減 -10.8%**: 7,386 → 6,590 行，11 commits 全綠零 502。 diff --git a/tests/test_cd_health_check.py b/tests/test_cd_health_check.py index 8f823a1..8f0b83f 100644 --- a/tests/test_cd_health_check.py +++ b/tests/test_cd_health_check.py @@ -20,3 +20,10 @@ def test_cd_health_check_validates_internal_and_external_health(): assert "http://127.0.0.1:80/health" in workflow assert "https://mo.wooo.work/health" in workflow assert 'internal=$INTERNAL_CODE, external=$EXTERNAL_CODE' in workflow + + +def test_cd_sync_mode_restarts_after_rsync_to_reload_python_processes(): + workflow = CD_WORKFLOW.read_text(encoding="utf-8") + + assert "docker compose up -d --no-deps momo-app scheduler telegram-bot" in workflow + assert "docker compose restart momo-app scheduler telegram-bot" in workflow