diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 51f674c4..ec42280a 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -45299,6 +45299,43 @@ production browser smoke: - 不得把 Wazuh route `200`、transport `6`、Dashboard index pattern `3`、Dashboard 可開或 UI 卡片可見宣稱為全主機納管恢復。 - 不得宣稱 active response、host write、agent re-enroll、restart、secret patch、Kali active scan 或 runtime gate 已批准。 +## 2026-06-26 — 08:59 post-reboot declaration guard / SOP v1.72 + +**時間與來源**： +- 2026-06-26 08:59 Asia/Taipei。 +- 來源：新增 `scripts/reboot-recovery/post-reboot-declaration-guard.py`，讀取 `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 的 key/value summary。 + +**完成內容**： +- 新增 `awoooi_post_reboot_declaration_guard_v1`，將重啟後可宣稱 / 禁止宣稱狀態機器可讀化。 +- guard 預設跑 live read-only summary；也可用 `--summary-file` 讀既有 artifact。 +- guard 支援 `--proposed`，可在文件、AI agent、owner packet 或對外狀態更新前驗證某個宣告是否可講。 +- `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 升至 v1.12，將 declaration guard 放在 summary 後、dispatch 前。 +- `docs/runbooks/FULL-STACK-COLD-START-SOP.md` 升至 v1.72，新增 declaration guard baseline。 +- `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md` 更新為 `DONE_WITH_DECLARATION_GUARD_V172`。 + +**只讀驗證結果**： +- `POST_REBOOT_DECLARATION_GUARD_OK status=allowed_with_boundary_blockers` +- 允許宣稱：`SERVICE_RECOVERY_GREEN`、`PRODUCT_DATA_GREEN`、`BACKUP_CORE_GREEN`、`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。 +- 禁止宣稱：`DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED`。 +- `--proposed DR_COMPLETE --proposed WAZUH_REGISTRY_RECOVERED` 必須被拒絕。 + +**做過的命令類型**： +- 只讀：post-reboot readiness summary、declaration guard、next-gate dispatch、owner-packet JSON、contract guard、source guards。 +- 寫入：repo script / docs-only。 +- 未做：host / Docker / systemd / Nginx / firewall / K8s / DB / Wazuh runtime 寫操作；未讀 secret 明文；未送 owner request；未寫 escrow marker；未執行 active response。 + +**目前判定**： +- Declaration guard automation：`0% -> 100%`。 +- Reboot service / data / backup readiness remains `GREEN`。 +- Overall declaration remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。 +- Runtime repair / owner request sent / credential marker write / Wazuh registry accepted：仍 `0%`。 + +**仍 blocked / 不得宣稱**： +- DR credential escrow evidence missing `5`。 +- 188 host hygiene 維護窗口仍未執行。 +- Wazuh manager registry accepted remains `0`。 +- 不得宣稱 `DR_COMPLETE`、188 host fully green、Wazuh registry recovered、runtime action authorized、owner request sent、owner response accepted。 + ## 2026-06-26 — 08:40 post-reboot owner-packet contract guard / SOP v1.70 **時間與來源**： diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 337e524c..14e52a65 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.71 +> Version: v1.72 > Last updated: 2026-06-26 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. @@ -10,7 +10,7 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -若只是重啟後要快速判斷能不能宣稱恢復，先跑機器可讀摘要：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color`。此腳本會呼叫一頁式總檢查、188 host hygiene checklist 與 Wazuh no-false-green repo gates，並把 delegated logs 留在 `/tmp/awoooi-post-reboot-readiness-*`。若 summary 顯示 `SERVICE_GREEN=1` 但 `NEXT_REQUIRED_GATES` 仍非空，接著跑 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color`，把 DR escrow、188 hygiene、Wazuh registry 三條 blocker 轉成 owner / evidence / forbidden-action dispatch checklist；需要機器可讀 intake 時，再跑 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --output /tmp/awoooi-post-reboot-owner-packets.json` 產生 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，並立刻跑 `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json`。dispatch / packet / guard 均固定 `DISPATCH_AUTHORIZED=0`、`REQUEST_SENT_COUNT=0`、`OWNER_RESPONSE_ACCEPTED=0`、`HOST_WRITE_AUTHORIZED=0`、`SECRET_VALUE_COLLECTION_ALLOWED=0`、`RUNTIME_GATE=0`；guard 未通過時不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / 188 host hygiene / Wazuh registry complete。需要人工展開時，再跑 `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 +若只是重啟後要快速判斷能不能宣稱恢復，先跑機器可讀摘要：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color`。此腳本會呼叫一頁式總檢查、188 host hygiene checklist 與 Wazuh no-false-green repo gates，並把 delegated logs 留在 `/tmp/awoooi-post-reboot-readiness-*`。接著跑 `scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color`，把 summary 轉成 allowed / forbidden declaration，避免把服務綠誤報成 DR complete、188 host fully green、Wazuh registry recovered 或 runtime authorized。若 summary 顯示 `SERVICE_GREEN=1` 但 `NEXT_REQUIRED_GATES` 仍非空，再跑 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color`，把 DR escrow、188 hygiene、Wazuh registry 三條 blocker 轉成 owner / evidence / forbidden-action dispatch checklist；需要機器可讀 intake 時，再跑 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --output /tmp/awoooi-post-reboot-owner-packets.json` 產生 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，並立刻跑 `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json`。dispatch / packet / guard 均固定 `DISPATCH_AUTHORIZED=0`、`REQUEST_SENT_COUNT=0`、`OWNER_RESPONSE_ACCEPTED=0`、`HOST_WRITE_AUTHORIZED=0`、`SECRET_VALUE_COLLECTION_ALLOWED=0`、`RUNTIME_GATE=0`；guard 未通過時不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / 188 host hygiene / Wazuh registry complete。需要人工展開時，再跑 `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 2026-06-26 07:47 machine-readable readiness summary：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 已驗證可用，artifact dir `/tmp/awoooi-post-reboot-readiness-20260626-074702`。摘要輸出 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=3`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_SERVICE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=1`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_RUNTIME_GATE=0`、`RUNTIME_ACTION_AUTHORIZED=0`。目前 `OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，`NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export`。這是每次重啟後的第一層 operator / AI agent 判定格式。 @@ -22,6 +22,8 @@ 2026-06-26 08:47 Wazuh registry detail baseline：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 已把 Wazuh repo-side coverage / runtime gate 的細節納入固定 key/value：`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_RUNTIME_GATE=0`。`scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` 的 `wazuh_manager_registry_export` gate 會把這些狀態放入 `CURRENT_EVIDENCE`。判讀鐵律：route `200`、transport `6`、Dashboard index pattern `3` 都不是 manager registry accepted；全主機納管與 Dashboard API 修復仍需 owner evidence / registry export / acceptance record。 +2026-06-26 08:59 declaration guard baseline：`scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color` 將 summary 轉成 `awoooi_post_reboot_declaration_guard_v1`，目前 status 為 `allowed_with_boundary_blockers`。允許宣稱 `SERVICE_RECOVERY_GREEN`、`PRODUCT_DATA_GREEN`、`BACKUP_CORE_GREEN` 與 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`；禁止宣稱 `DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED`。若用 `--proposed DR_COMPLETE` 或 `--proposed WAZUH_REGISTRY_RECOVERED` 測試，guard 必須以 `blocked_false_green_proposal` 拒絕。 + 2026-06-26 07:39 live quick-check refresh：`scripts/reboot-recovery/post-start-quick-check.sh --no-color` 完整跑完，四主機 ping / SSH 全部 OK，delegated cold-start 為 `PASS=89 WARN=0 BLOCKED=0`，wrapper 總結為 `POST_START_QUICK_CHECK PASS=38 WARN=3 BLOCKED=0`、warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`、`RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。MOMO health `V10.701`，daily snapshot `109061` rows / `2025-07-01..2026-06-24`，current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`，latest import job `57 completed`。StockPlatform freshness `status=ok`、latest trading date `2026-06-25`，price / chips / margin / AI recommendations 均為 `2026-06-25`。Backup-status 07:39 顯示 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。Public routes extended list 全部回 expected 2xx/3xx。110 CPU attribution 顯示 load 約 `5.19 / 4.66 / 4.91`，CPU idle 多數樣本 `80%+`，目前負載來自 Gitea / ClickHouse / Docker / Kafka / StockPlatform / AWOOOI API / Sentry 等正常平台工作，不是 orphan Chrome。這一輪 allowed declaration：主機、K3s、服務、網站、產品資料 freshness、備份核心與 offsite freshness 綠；forbidden declaration：DR complete、credential escrow complete、188 host fully green、Wazuh registry recovered。 2026-06-26 07:19 follow-up：`gitea/main` 已包含前一輪 SOP 文件 commit `1fd5e2a8`，ArgoCD `awoooi-prod` 讀回 `Synced / Healthy`，revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`，API `2/2`、Web `2/2`、Worker `1/1`，pods `restart=0`。重跑 full cold-start 仍是 `PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。直接 public route 讀回：AWOOOI API `200`、AWOOOI Web `307`、VibeWork `200`、AwoooGo `200`、MOMO health `200`、Stock freshness `200`、Bitan `200`、Gitea `200`、Harbor `200`、Registry `/v2/` expected `401`、Sentry expected `302`、SigNoz `200`、Langfuse `200`。188 blocker 精準分類：`pg_lsclusters` 顯示 host PostgreSQL `14/main` down，`systemctl status postgresql@14-main` 顯示 `invalid primary checkpoint record` 與 `PANIC: could not locate a valid checkpoint record`；`certbot.service` 顯示 `sentry.wooo.work` renew rate-limited，`snap.certbot.renew.service` 顯示 challenge failed；`awoooi-startup.service` 曾嘗試以 root 執行 `pg_resetwal` 並失敗。本輪不執行 `pg_resetwal`、不 `reset-failed`、不重啟 service；188 需用獨立維護窗口、rollback owner、restore/source-of-truth plan 處理，詳見 `docs/runbooks/HOST-188-HYGIENE-MAINTENANCE-RUNBOOK.md`，並可先跑 `scripts/reboot-recovery/188-host-hygiene-maintenance-checklist.sh --no-color` 取得只讀 preflight。110 load 已降到約 `4.83 / 4.82 / 5.52`，top CPU 是 active AWOOOI Web `turbo build` / Docker buildx；Swap 仍滿但 memory available 約 `41Gi`，本輪不手動清 swap。整體宣告仍是 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。 diff --git a/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md b/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md index a2678c24..c01b3b4a 100644 --- a/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md +++ b/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md @@ -1,6 +1,6 @@ # 主機重啟後一頁式總檢查 -> Version: v1.11 +> Version: v1.12 > Last updated: 2026-06-26 Asia/Taipei > Scope: 110 / 120 / 121 / 188 post-reboot service recovery. 112 Kali / Wazuh / active scan 不屬於本流程。 @@ -10,7 +10,7 @@ 每次 110 / 120 / 121 / 188 任一台主機開機、關機、重啟、斷電恢復、VMware console fsck、Docker / K3s 大量重排後，都先跑本頁，再決定是否宣稱恢復。 -最新基準：2026-06-26 08:47 Wazuh registry detail in post-reboot summary。`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=1`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。接著 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` 將 `NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export` 展成三個 owner / evidence / forbidden-action checklist；Wazuh checklist 的 `CURRENT_EVIDENCE` 會保留 registry accepted、coverage scope、direct active、no transport、SSH blocked、route、transport、Dashboard API 與 index pattern 狀態，避免把 route `200` 或 transport `6` 誤報成 registry recovered。`scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color` 進一步轉成 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，固定 `dispatch_authorized=0`、`request_sent_count=0`、`owner_response_accepted_count=0`、`host_write_authorized=0`、`secret_value_collection_allowed=0`、`runtime_gate_count=0`；`scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json` 鎖定三個 P0 gate、所有 `0 / false` 邊界、禁用 secret payload / runtime action 與 no-false-green 規則。Cold-start `PASS=89 WARN=0 BLOCKED=0`；MOMO `V10.701`、latest import job `57 completed`、`DB_DAILY_FRESHNESS 1|2026-06-24`；StockPlatform `/api/v1/system/freshness` 為 `status=ok`、`latest_trading_date=2026-06-25`、blockers `[]`；backup-status 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`last_backup_all=2026-06-26 02:31:02`。DR 仍因 `escrow_missing=5` 不可宣稱 complete。188 host hygiene 與 Wazuh manager registry 仍是 service green 之外的獨立 blocker。 +最新基準：2026-06-26 08:59 post-reboot declaration guard。`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=1`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。`scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color` 會把 summary 轉成 allowed / forbidden declaration：目前允許宣稱服務、產品資料、備份核心與 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`；禁止宣稱 `DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED`。接著 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` 將 `NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export` 展成三個 owner / evidence / forbidden-action checklist；Wazuh checklist 的 `CURRENT_EVIDENCE` 會保留 registry accepted、coverage scope、direct active、no transport、SSH blocked、route、transport、Dashboard API 與 index pattern 狀態，避免把 route `200` 或 transport `6` 誤報成 registry recovered。`scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color` 進一步轉成 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，固定 `dispatch_authorized=0`、`request_sent_count=0`、`owner_response_accepted_count=0`、`host_write_authorized=0`、`secret_value_collection_allowed=0`、`runtime_gate_count=0`；`scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json` 鎖定三個 P0 gate、所有 `0 / false` 邊界、禁用 secret payload / runtime action 與 no-false-green 規則。Cold-start `PASS=89 WARN=0 BLOCKED=0`；MOMO `V10.701`、latest import job `57 completed`、`DB_DAILY_FRESHNESS 1|2026-06-24`；StockPlatform `/api/v1/system/freshness` 為 `status=ok`、`latest_trading_date=2026-06-25`、blockers `[]`；backup-status 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`last_backup_all=2026-06-26 02:31:02`。DR 仍因 `escrow_missing=5` 不可宣稱 complete。188 host hygiene 與 Wazuh manager registry 仍是 service green 之外的獨立 blocker。 本頁只回答四件事： @@ -61,7 +61,21 @@ scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color - `RUNTIME_ACTION_AUTHORIZED=0`：本流程沒有授權 runtime 寫操作。 - `OVERALL_DECLARATION`：本輪可使用的最高宣告。 -summary 顯示 `SERVICE_GREEN=1` 但仍有 `NEXT_REQUIRED_GATES` 時，接著跑下一 Gate 派工 checklist： +summary 顯示 `SERVICE_GREEN=1` 後，先跑宣告 guard，確認本輪可以講什麼、不能講什麼： + +```bash +scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color +``` + +這支 guard 只讀取 summary evidence，並輸出本輪允許與禁止宣稱的邊界。若要測試某個說法是否允許，可用 `--proposed`： + +```bash +scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color --proposed DR_COMPLETE --proposed WAZUH_REGISTRY_RECOVERED +``` + +在目前基準下，`DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED` 都必須被拒絕；任何人、任何 AI agent 或任何文件若提出這些宣告，必須先補對應 evidence / owner acceptance。 + +summary 顯示 `SERVICE_GREEN=1` 但仍有 `NEXT_REQUIRED_GATES` 時，再跑下一 Gate 派工 checklist： ```bash scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index b89172b3..cfb9daa1 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -15,7 +15,7 @@ | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 07:19 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-26 06:58 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-26 02:31:02`。DR remains blocked on real non-secret credential escrow evidence IDs; do not write placeholder markers or paste secret values. | | P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 current CPU pressure is attributable to active AWOOOI Web `turbo build` / Docker buildx, and previous orphan Chrome groups remain cleared. 2026-06-26 07:19 StockPlatform `/api/v1/system/freshness` returned `200`; 07:01 freshness payload was `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.699`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. | -| P3 docs / automation contracts | DONE_WITH_WAZUH_DETAIL_SUMMARY_V171 | 100% | Workplan, SOP v1.71, machine-readable post-reboot readiness summary with Wazuh registry detail fields, post-reboot next-gate dispatch checklist, owner-packet JSON generator, owner-packet contract guard, one-page post-start quick check v1.11, route retry gate, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, 188 host hygiene read-only checklist, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Summary / dispatch now carry Wazuh coverage scope, direct active, no transport, SSH blocked, route, transport, Dashboard API connection and index OK fields while keeping manager registry accepted / runtime gate at `0`; route `200` and transport `6` are explicitly not accepted as full Wazuh recovery. Live 110 script sync remains a separate approved live-write gate; do not claim it here. | +| P3 docs / automation contracts | DONE_WITH_DECLARATION_GUARD_V172 | 100% | Workplan, SOP v1.72, post-reboot declaration guard, machine-readable post-reboot readiness summary with Wazuh registry detail fields, post-reboot next-gate dispatch checklist, owner-packet JSON generator, owner-packet contract guard, one-page post-start quick check v1.12, route retry gate, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, 188 host hygiene read-only checklist, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Declaration guard now machine-checks allowed / forbidden recovery statements: service/data/backup green may be declared, while `DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED` and `RUNTIME_ACTION_AUTHORIZED` remain forbidden until evidence gates close. Live 110 script sync remains a separate approved live-write gate; do not claim it here. | 2026-06-26 07:47 machine-readable summary baseline: `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` stores delegated logs under `/tmp/awoooi-post-reboot-readiness-20260626-074702` and returns `SERVICE_GREEN=1`, `PRODUCT_DATA_GREEN=1`, `BACKUP_CORE_GREEN=1`, `DR_ESCROW_BLOCKED=1`, `ESCROW_MISSING_COUNT=5`, `HOST_188_SERVICE_GREEN=1`, `HOST_188_HYGIENE_BLOCKED=1`, `WAZUH_ROUTE_CODE=200`, `WAZUH_TRANSPORT_COUNT=6`, `WAZUH_MANAGER_REGISTRY_ACCEPTED=0`, `WAZUH_RUNTIME_GATE=0`, `RUNTIME_ACTION_AUTHORIZED=0`, `OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, and `NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export`. This is now the preferred first operator/AI-agent entrypoint after reboot because it separates service health from DR, host hygiene, and security registry evidence. @@ -27,6 +27,8 @@ 2026-06-26 08:47 Wazuh registry detail summary baseline: post-reboot readiness summary now emits `WAZUH_COVERAGE_SCOPE`, `WAZUH_DIRECT_ACTIVE`, `WAZUH_NO_TRANSPORT`, `WAZUH_SSH_BLOCKED`, `WAZUH_DASHBOARD_API_CONNECTION`, and `WAZUH_DASHBOARD_INDEX_OK` alongside existing route / transport / registry fields. Current read-only truth is coverage scope `6`, direct active `2`, no transport `1`, SSH blocked `3`, route `200`, transport `6`, Dashboard API `pending_or_spinning`, index OK `3`, manager registry accepted `0`, runtime gate `0`. This is a security evidence blocker, not a reboot service blocker. +2026-06-26 08:59 declaration guard baseline: `scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color` emits `schema_version=awoooi_post_reboot_declaration_guard_v1`, status `allowed_with_boundary_blockers`, allowed declarations `BACKUP_CORE_GREEN`、`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`PRODUCT_DATA_GREEN`、`SERVICE_RECOVERY_GREEN`, and forbidden declarations `DR_COMPLETE`、`HOST_188_FULLY_GREEN`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED`. Proposed false-green declarations are rejected before they can enter LOGBOOK / owner packets / external status updates. + 2026-06-26 07:39 live quick-check refresh supersedes the 07:19 row for current operator status. `scripts/reboot-recovery/post-start-quick-check.sh --no-color` returned `POST_START_QUICK_CHECK PASS=38 WARN=3 BLOCKED=0`, warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`, result `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. Delegated cold-start returned `PASS=89 WARN=0 BLOCKED=0`; four reboot-scope hosts ping/SSH were OK; AWOOOI / VibeWork / AwoooGo / 2026FIFA / Agent Bounty / MOMO / Stock / Bitan / TsenYang / VTuber / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse / AIOps routes returned expected 2xx/3xx. MOMO `V10.701` has job `57 completed`, daily freshness `1|2026-06-24`, and current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`. StockPlatform freshness is `ok` through `2026-06-25` with price / chips / margin / AI recommendations current. Backup core remains green: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`; DR still has `escrow_missing=5`. 110 load around `5.19 / 4.66 / 4.91` is attributable to normal platform processes, not orphan Chrome. 188 host hygiene remains blocked by failed host PostgreSQL / certbot / startup units and must use the dedicated maintenance runbook and read-only checklist. 2026-06-25 19:06 post-CD wrapper readback supersedes the 18:53 wording: consecutive main pushes created a deploy storm where older deploy markers were superseded by later commits. Latest production truth is deploy marker `d8ca8224 chore(cd): deploy 9dbe044 [skip ci]`, ArgoCD `Synced / Healthy`, API/Web/Worker image tag `9dbe044ea1e8e3894ccbeb5ed760bb124b87f7be`, direct route smoke 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan and expected route-gate statuses for MOMO / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse / AIOps, and wrapper `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.690`; AwoooGo / Stock transient 502 reads cleared after upstream warmup and five consecutive route reads returned `200`; 110 load is around `14.51 / 12.34 / 11.42`, with Gitea Actions cache save / `zstdmt` / `tar`, StockPlatform headless Chrome smoke / CI, Gitea, AWOOOI API, ClickHouse, Docker, and platform services visible, not an AWOOOI service blocker. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain. Wazuh route readback is now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but manager registry accepted remains `0`, so Wazuh is a security registry evidence blocker rather than a reboot service blocker. diff --git a/scripts/reboot-recovery/post-reboot-declaration-guard.py b/scripts/reboot-recovery/post-reboot-declaration-guard.py new file mode 100755 index 00000000..7c62c5de --- /dev/null +++ b/scripts/reboot-recovery/post-reboot-declaration-guard.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Guard post-reboot recovery declarations against false green claims. + +Read-only by design. It consumes post-reboot-readiness-summary.sh output, or +runs that summary when no file is provided. It never repairs services, writes +markers, sends owner requests, or changes host/runtime state. +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from datetime import datetime +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[2] +SUMMARY_SCRIPT = ROOT / "scripts" / "reboot-recovery" / "post-reboot-readiness-summary.sh" + +SCHEMA_VERSION = "awoooi_post_reboot_declaration_guard_v1" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Validate what may be declared after reboot recovery.", + ) + parser.add_argument( + "--summary-file", + type=Path, + help="Read an existing post-reboot readiness summary key/value file.", + ) + parser.add_argument( + "--proposed", + action="append", + default=[], + help="Optional declaration to validate. Repeatable.", + ) + parser.add_argument( + "--json", + action="store_true", + help="Print JSON instead of a compact status line.", + ) + parser.add_argument( + "--output", + type=Path, + help="Write JSON payload to this path.", + ) + parser.add_argument( + "--no-color", + action="store_true", + help="Pass --no-color when running the readiness summary.", + ) + return parser.parse_args() + + +def run_summary(no_color: bool) -> str: + cmd = [str(SUMMARY_SCRIPT)] + if no_color: + cmd.append("--no-color") + completed = subprocess.run( + cmd, + cwd=ROOT, + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) + if completed.returncode not in (0, 2): + raise SystemExit( + "POST_REBOOT_DECLARATION_GUARD_FAILED " + f"summary_rc={completed.returncode}\n{completed.stdout}" + ) + return completed.stdout + + +def load_summary(args: argparse.Namespace) -> str: + if args.summary_file: + return args.summary_file.read_text(encoding="utf-8") + return run_summary(no_color=args.no_color) + + +def parse_summary(text: str) -> dict[str, str]: + values: dict[str, str] = {} + for raw_line in text.splitlines(): + line = raw_line.strip() + if not line or "=" not in line: + continue + key, value = line.split("=", 1) + values[key.strip()] = value.strip() + return values + + +def truthy(value: str | None) -> bool: + return value in {"1", "true", "True", "yes", "YES"} + + +def nonzero(value: str | None) -> bool: + if value is None or value in {"", "unknown"}: + return False + try: + return int(value) != 0 + except ValueError: + return False + + +def split_csv(value: str | None) -> list[str]: + if not value or value == "none": + return [] + return [item.strip() for item in value.split(",") if item.strip()] + + +def build_payload(summary: dict[str, str], proposed: list[str]) -> dict[str, Any]: + allowed: list[str] = [] + forbidden: list[dict[str, str]] = [] + + service_green = truthy(summary.get("SERVICE_GREEN")) + product_data_green = truthy(summary.get("PRODUCT_DATA_GREEN")) + backup_core_green = truthy(summary.get("BACKUP_CORE_GREEN")) + dr_escrow_blocked = truthy(summary.get("DR_ESCROW_BLOCKED")) or nonzero( + summary.get("ESCROW_MISSING_COUNT") + ) + host_188_hygiene_blocked = truthy(summary.get("HOST_188_HYGIENE_BLOCKED")) + wazuh_registry_accepted = truthy(summary.get("WAZUH_MANAGER_REGISTRY_ACCEPTED")) + runtime_authorized = truthy(summary.get("RUNTIME_ACTION_AUTHORIZED")) + next_required_gates = split_csv(summary.get("NEXT_REQUIRED_GATES")) + overall_declaration = summary.get("OVERALL_DECLARATION", "unknown") + + if service_green: + allowed.append("SERVICE_RECOVERY_GREEN") + else: + forbidden.append( + { + "declaration": "SERVICE_RECOVERY_GREEN", + "reason": "service_green_not_1", + } + ) + + if product_data_green: + allowed.append("PRODUCT_DATA_GREEN") + else: + forbidden.append( + { + "declaration": "PRODUCT_DATA_GREEN", + "reason": "product_data_green_not_1", + } + ) + + if backup_core_green: + allowed.append("BACKUP_CORE_GREEN") + else: + forbidden.append( + { + "declaration": "BACKUP_CORE_GREEN", + "reason": "backup_core_green_not_1", + } + ) + + if overall_declaration != "unknown": + allowed.append(overall_declaration) + + if dr_escrow_blocked: + forbidden.append( + { + "declaration": "DR_COMPLETE", + "reason": f"escrow_missing_count:{summary.get('ESCROW_MISSING_COUNT', 'unknown')}", + } + ) + else: + allowed.append("DR_COMPLETE") + + if host_188_hygiene_blocked: + forbidden.append( + { + "declaration": "HOST_188_FULLY_GREEN", + "reason": "host_188_hygiene_blocked:1", + } + ) + else: + allowed.append("HOST_188_FULLY_GREEN") + + if not wazuh_registry_accepted: + forbidden.append( + { + "declaration": "WAZUH_REGISTRY_RECOVERED", + "reason": "wazuh_manager_registry_accepted:0", + } + ) + else: + allowed.append("WAZUH_REGISTRY_RECOVERED") + + if not runtime_authorized: + forbidden.append( + { + "declaration": "RUNTIME_ACTION_AUTHORIZED", + "reason": "runtime_action_authorized:0", + } + ) + else: + allowed.append("RUNTIME_ACTION_AUTHORIZED") + + allowed_set = set(allowed) + forbidden_map = {item["declaration"]: item["reason"] for item in forbidden} + rejected_proposed = [ + {"declaration": item, "reason": forbidden_map[item]} + for item in proposed + if item in forbidden_map + ] + accepted_proposed = [item for item in proposed if item in allowed_set] + unknown_proposed = [ + item + for item in proposed + if item not in allowed_set and item not in forbidden_map + ] + + status = "blocked_service_recovery" if not service_green else "allowed_with_boundary_blockers" + if rejected_proposed: + status = "blocked_false_green_proposal" + + return { + "schema_version": SCHEMA_VERSION, + "generated_at": datetime.now().astimezone().isoformat(timespec="seconds"), + "source": { + "summary_script": str(SUMMARY_SCRIPT.relative_to(ROOT)), + "artifact_dir": summary.get("ARTIFACT_DIR", "unknown"), + "overall_declaration": overall_declaration, + }, + "status": status, + "allowed_declarations": sorted(set(allowed)), + "forbidden_declarations": forbidden, + "next_required_gates": next_required_gates, + "counts": { + "allowed_count": len(set(allowed)), + "forbidden_count": len(forbidden), + "next_required_gate_count": len(next_required_gates), + "proposed_count": len(proposed), + "accepted_proposed_count": len(accepted_proposed), + "rejected_proposed_count": len(rejected_proposed), + "unknown_proposed_count": len(unknown_proposed), + }, + "proposed": { + "accepted": accepted_proposed, + "rejected": rejected_proposed, + "unknown": unknown_proposed, + }, + "evidence": { + "service_green": summary.get("SERVICE_GREEN", "unknown"), + "product_data_green": summary.get("PRODUCT_DATA_GREEN", "unknown"), + "backup_core_green": summary.get("BACKUP_CORE_GREEN", "unknown"), + "escrow_missing_count": summary.get("ESCROW_MISSING_COUNT", "unknown"), + "host_188_hygiene_blocked": summary.get("HOST_188_HYGIENE_BLOCKED", "unknown"), + "wazuh_manager_registry_accepted": summary.get( + "WAZUH_MANAGER_REGISTRY_ACCEPTED", + "unknown", + ), + "wazuh_route_code": summary.get("WAZUH_ROUTE_CODE", "unknown"), + "wazuh_transport_count": summary.get("WAZUH_TRANSPORT_COUNT", "unknown"), + "wazuh_coverage_scope": summary.get("WAZUH_COVERAGE_SCOPE", "unknown"), + "wazuh_direct_active": summary.get("WAZUH_DIRECT_ACTIVE", "unknown"), + "wazuh_no_transport": summary.get("WAZUH_NO_TRANSPORT", "unknown"), + "wazuh_ssh_blocked": summary.get("WAZUH_SSH_BLOCKED", "unknown"), + "wazuh_dashboard_api_connection": summary.get( + "WAZUH_DASHBOARD_API_CONNECTION", + "unknown", + ), + "wazuh_dashboard_index_ok": summary.get("WAZUH_DASHBOARD_INDEX_OK", "unknown"), + "runtime_action_authorized": summary.get("RUNTIME_ACTION_AUTHORIZED", "unknown"), + }, + "runtime_write_performed": False, + } + + +def main() -> int: + args = parse_args() + summary = parse_summary(load_summary(args)) + payload = build_payload(summary, args.proposed) + serialized = json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(serialized + "\n", encoding="utf-8") + + if args.json: + print(serialized) + else: + prefix = "POST_REBOOT_DECLARATION_GUARD_OK" + if payload["proposed"]["rejected"]: + prefix = "POST_REBOOT_DECLARATION_GUARD_REJECTED" + elif payload["status"] == "blocked_service_recovery": + prefix = "POST_REBOOT_DECLARATION_GUARD_BLOCKED" + print( + f"{prefix} " + f"status={payload['status']} " + f"allowed={payload['counts']['allowed_count']} " + f"forbidden={payload['counts']['forbidden_count']} " + f"next_gates={payload['counts']['next_required_gate_count']} " + f"rejected_proposed={payload['counts']['rejected_proposed_count']}" + ) + + if payload["proposed"]["rejected"]: + return 2 + if payload["status"] == "blocked_service_recovery": + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main())