wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 4 Packages Projects Releases Wiki Activity

docs(ops): record reboot blocker follow-up [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-26 07:23:32 +08:00
parent 7f204ca71b
commit 12d93b7583
3 changed files with 37 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
docs/LOGBOOK.md
View File

@@ -44721,3 +44721,33 @@ production browser smoke:
**不得宣稱**：
- 不得宣稱 `DR_COMPLETE`、credential escrow complete、Wazuh host registry recovered、188 host fully green、雙機 Codex workspace fully synchronized、或所有主機衛生問題已解決。
## 2026-06-26 — 07:19 follow-up：SOP v1.65 與 blocker 精準分層
**時間與來源**：
- 2026-06-26 07:18-07:19 Asia/Taipei。
- 來源：110 cold-start wrapper、120 `kubectl get` read-only、188 `systemctl status` / `pg_lsclusters` / `docker ps` read-only、public curl route readback、110 backup-status。
**只讀證據**：
- ArgoCD：`awoooi-prod=Synced / Healthy`，revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`；API `2/2`、Web `2/2`、Worker `1/1`，pods restart `0`。
- Cold-start：`PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。
- Public routes：AWOOOI API `200`、AWOOOI Web `307`、VibeWork `200`、AwoooGo `200`、MOMO health `200`、Stock freshness `200`、Bitan `200`、Gitea `200`、Harbor `200`、Registry `/v2/` expected `401`、Sentry expected `302`、SigNoz `200`、Langfuse `200`。
- Backup：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`integrity_stale=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。
- 188 PostgreSQL：`pg_lsclusters` 顯示 host cluster `14/main` down；`systemctl status postgresql@14-main` 顯示 `invalid primary checkpoint record` 與 `PANIC: could not locate a valid checkpoint record`。
- 188 certbot：`certbot.service` 顯示 `sentry.wooo.work` renew rate-limited；`snap.certbot.renew.service` 顯示 challenge failed。
- 188 startup：`awoooi-startup.service` 顯示曾嘗試以 root 執行 `pg_resetwal` 並失敗；此類資料修復必須獨立維護窗口處理，不得納入自動重啟恢復。
- 110 CPU：load 約 `4.83 / 4.82 / 5.52`，top CPU 是 AWOOOI Web `turbo build` / Docker buildx；Swap 滿但 memory available 約 `41Gi`，本輪未手動清 swap。
**做過的命令類型**：
- 只讀：`systemctl status`、`pg_lsclusters`、`docker ps`、`free`、`ps`、`kubectl get`、curl、backup-status、cold-start wrapper。
- 寫入：只更新 repo 文件；沒有 host / Docker / systemd / Nginx / firewall / K8s / DB / Wazuh runtime 寫操作。
**目前判定**：
- Reboot / cold-start / public route / core backup：`GREEN`。
- 正式狀態仍是 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，不是 `DR_COMPLETE`。
- 188 是 `SERVICE_GREEN_HOST_HYGIENE_BLOCKED`：產品容器與 public routes 正常，但 host PostgreSQL / certbot / startup unit 需要維護窗口。
**下一個 P0**：
- 建立 188 維護窗口與 rollback owner：決定 host PostgreSQL `14/main` 是廢棄 cluster、需從 backup restore、或需 break-glass WAL/FS 修復；不得直接 `pg_resetwal`。
- 建立 certbot / DNS challenge 修復窗口：先確認 SAN / wildcard / public gateway owner evidence，再處理 rate-limit 後的 renew。
- 補五個 credential escrow non-secret evidence marker，讓 DR scorecard 從 `BLOCKED` 轉為可驗收。

4
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.64
> Version: v1.65
> Last updated: 2026-06-26 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -12,6 +12,8 @@
若只是重啟後要快速判斷能不能宣稱恢復，先跑一頁式總檢查：`scripts/reboot-recovery/post-start-quick-check.sh --no-color`，並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。
2026-06-26 07:19 follow-up：`gitea/main` 已包含前一輪 SOP 文件 commit `1fd5e2a8`，ArgoCD `awoooi-prod` 讀回 `Synced / Healthy`，revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`，API `2/2`、Web `2/2`、Worker `1/1`，pods `restart=0`。重跑 full cold-start 仍是 `PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。直接 public route 讀回：AWOOOI API `200`、AWOOOI Web `307`、VibeWork `200`、AwoooGo `200`、MOMO health `200`、Stock freshness `200`、Bitan `200`、Gitea `200`、Harbor `200`、Registry `/v2/` expected `401`、Sentry expected `302`、SigNoz `200`、Langfuse `200`。188 blocker 精準分類：`pg_lsclusters` 顯示 host PostgreSQL `14/main` down，`systemctl status postgresql@14-main` 顯示 `invalid primary checkpoint record` 與 `PANIC: could not locate a valid checkpoint record`；`certbot.service` 顯示 `sentry.wooo.work` renew rate-limited，`snap.certbot.renew.service` 顯示 challenge failed；`awoooi-startup.service` 曾嘗試以 root 執行 `pg_resetwal` 並失敗。本輪不執行 `pg_resetwal`、不 `reset-failed`、不重啟 service；188 需用獨立維護窗口、rollback owner、restore/source-of-truth plan 處理。110 load 已降到約 `4.83 / 4.82 / 5.52`，top CPU 是 active AWOOOI Web `turbo build` / Docker buildx；Swap 仍滿但 memory available 約 `41Gi`，本輪不手動清 swap。整體宣告仍是 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。
2026-06-26 07:02 全主機 live refresh：`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。110 `systemctl=running`、failed units `0`，但 load `5.83 / 7.26 / 5.77` 且 top CPU 是 AWOOOI Web `next build`，Swap 仍 `7.8Gi / 7.8Gi`；這是 CI/build 壓力，不是 orphan Chrome 或 Docker 事故。120 / 121 `systemctl=running`、K3s active，nodes `mon` / `mon1` 均為 Ready。ArgoCD `awoooi-prod` 在 06:57 曾短暫 `OutOfSync / Progressing`，因 deploy marker `52f61da4` rollout 正在替換 API/Web/Worker；07:00 後已穩定為 `Synced / Healthy`，API `2/2`、Web `2/2`、Worker `1/1`，API/Web 仍跨 `mon` / `mon1`。重跑 live cold-start：`PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。StockPlatform `/api/v1/system/freshness` 曾在容器剛重啟約 35 秒時短暫 `502`，後續連續讀回皆 `200` 且 `status=ok`、`latest_trading_date=2026-06-25`、blockers `[]`；這類 rollout warmup 只有連續失敗才算 blocker。MOMO health 是 `V10.699`，cold-start direct evidence 仍顯示 current-month parity `15383 / 15383` 截至 `2026-06-24`，daily freshness `1|2026-06-24`。Backup status 06:58：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。188 產品容器健康，但 host `systemctl=degraded` 仍是真實 host hygiene blocker：`awoooi-startup.service`、`postgresql@14-main.service`、`certbot.service`、`snap.certbot.renew.service` failed。112 Wazuh manager/indexer/dashboard active，ports `1514 / 1515 / 55000` listen，但 production Wazuh route 仍回報 `disabled_waiting_iwooos_wazuh_owner_gate`、`configured=false`、manager registry accepted `0`、runtime gate `0`。111 / 168 可連線，但兩邊 AWOOOI dev workspaces 皆 ahead 17 且 HEAD 不同（`111=56c83257`、`168=59485d51`）；Mac Mini `/System/Volumes/Data` 只剩約 `3.2Gi`。目前 service recovery 宣告維持 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，host hygiene / DR escrow / Wazuh registry / workstation capacity 明確列為 service green 之外的 blocker。
2026-06-26 06:50-06:55 188 host hygiene read-only triage：188 product services remain green, but host `systemctl` is still `degraded` and must not be smoothed into full host green. Failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`. Evidence shows the host PostgreSQL cluster `14/main` is down in `pg_lsclusters`, while product DB / exporters still respond through containerized services; therefore `pg_isready` or `pg_up=1` cannot substitute for host cluster health. The 188 startup service detected `could not locate a valid checkpoint record` on 2026-06-23 and attempted `pg_resetwal` as root, which failed; v1.63 treats PostgreSQL checkpoint/WAL errors as break-glass only and the repo-side startup script now fails closed instead of running `pg_resetwal`. Certbot renew for `sentry.wooo.work` is also failing and hit ACME rate-limit / challenge failure, but the public cert is still valid until 2026-07-09 16:03:40 UTC. Current declaration: `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` for 188, while overall service recovery remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`.

8
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,11 +11,11 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 07:02 all-host read-only refresh confirms service recovery is green after a short deploy warmup window. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Cold-start rerun returned `PASS=87 WARN=0 BLOCKED=0`, result `GREEN`; public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan and core tools return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.699`; cold-start MOMO direct evidence keeps current-month parity `15383 / 15383` through `2026-06-24`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 06:58 reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 06:57 ArgoCD and StockPlatform each had a short rollout/warmup transient (`awoooi-prod Progressing`, Stock freshness single 502) but both cleared on consecutive readbacks. 188 remains explicitly split into product-service green and host hygiene blocker: failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, `snap.certbot.renew.service`; host PostgreSQL cluster is down while product DB containers/exporters are up; certbot renewal is failing but public cert remains valid until 2026-07-09 UTC. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 07:02 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `52f61da4b3727f08e41c474a188859025d4c60a2`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. |
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 07:19 read-only follow-up confirms service recovery remains green after the SOP commit reached production. Cold-start rerun returned `PASS=87 WARN=0 BLOCKED=0`, result `GREEN`; ArgoCD `awoooi-prod Synced / Healthy` at revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`; API/Web/Worker pods are Running with restart `0`; public routes return expected statuses including AWOOOI API `200`, AWOOOI Web `307`, VibeWork `200`, AwoooGo `200`, MOMO health `200`, Stock freshness `200`, Bitan `200`, Gitea `200`, Harbor `200`, Registry `/v2/` expected `401`, Sentry expected `302`, SigNoz `200`, Langfuse `200`. Backup-status 07:18 remains 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 188 remains product-service green but host-hygiene blocked: host PostgreSQL `14/main` has `invalid primary checkpoint record` / `could not locate a valid checkpoint record`, certbot renew for `sentry.wooo.work` is rate-limited / challenge-failed, and `awoooi-startup.service` previously attempted `pg_resetwal` as root. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space remain workstation blockers, not reboot service blockers. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 07:19 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-26 06:58 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-26 02:31:02`。DR remains blocked on real non-secret credential escrow evidence IDs; do not write placeholder markers or paste secret values. |
| P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 current CPU pressure is attributable to active AWOOOI Web `next build`, and previous orphan Chrome groups remain cleared. 2026-06-26 07:01 StockPlatform `/api/v1/system/freshness` returned `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.699`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. |
| P3 docs / automation contracts | DONE_WITH_ROUTE_RETRY_V164 | 100% | Workplan, SOP v1.64, one-page post-start quick check v1.6, route retry gate, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Live 110 script sync remains a separate approved live-write gate; do not claim it here. |
| P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 current CPU pressure is attributable to active AWOOOI Web `turbo build` / Docker buildx, and previous orphan Chrome groups remain cleared. 2026-06-26 07:19 StockPlatform `/api/v1/system/freshness` returned `200`; 07:01 freshness payload was `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.699`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. |
| P3 docs / automation contracts | DONE_WITH_ROUTE_RETRY_V165 | 100% | Workplan, SOP v1.65, one-page post-start quick check v1.6, route retry gate, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Live 110 script sync remains a separate approved live-write gate; do not claim it here. |
2026-06-25 19:06 post-CD wrapper readback supersedes the 18:53 wording: consecutive main pushes created a deploy storm where older deploy markers were superseded by later commits. Latest production truth is deploy marker `d8ca8224 chore(cd): deploy 9dbe044 [skip ci]`, ArgoCD `Synced / Healthy`, API/Web/Worker image tag `9dbe044ea1e8e3894ccbeb5ed760bb124b87f7be`, direct route smoke 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan and expected route-gate statuses for MOMO / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse / AIOps, and wrapper `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.690`; AwoooGo / Stock transient 502 reads cleared after upstream warmup and five consecutive route reads returned `200`; 110 load is around `14.51 / 12.34 / 11.42`, with Gitea Actions cache save / `zstdmt` / `tar`, StockPlatform headless Chrome smoke / CI, Gitea, AWOOOI API, ClickHouse, Docker, and platform services visible, not an AWOOOI service blocker. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain. Wazuh route readback is now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but manager registry accepted remains `0`, so Wazuh is a security registry evidence blocker rather than a reboot service blocker.