docs(ops): record all-host reboot refresh [skip ci]

This commit is contained in:
ogt
2026-06-26 07:07:52 +08:00
parent 5151f78260
commit 1fd5e2a8b0
3 changed files with 47 additions and 6 deletions

View File

@@ -44682,3 +44682,42 @@ production browser smoke:
- Backup status110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0``core_blockers=0``escrow_missing=5`
- Full cold-start`PASS=83 WARN=0 BLOCKED=0`
- 判定production deploy / security mirror / cold-start 均已綠DR complete 仍等待五個 credential escrow markerArgoCD fully healthy 仍等待下一次 03:00 `km-vectorize` 官方成功。
## 2026-06-26 — 全主機重啟後 07:02 live refresh 與 SOP v1.64
**時間與來源**
- 2026-06-26 06:57-07:02 Asia/Taipei。
- 來源:本機 curl、110 `/home/wooo/scripts/full-stack-cold-start-check.sh`、110 `/backup/scripts/backup-status.sh --no-notify`、120 `sudo -n kubectl get` read-only、各主機 SSH read-only `systemctl` / `df` / `docker ps` / `ps` / TLS readback。
**只讀證據**
- Host reachability`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。
- 110`systemctl is-system-running=running`failed units `0`load `5.83 / 7.26 / 5.77`top CPU 是 AWOOOI Web `next build`Swap `7.8Gi / 7.8Gi`,本輪未手動清 swap。
- 120 / 121`systemctl=running``k3s=active`nodes `mon` / `mon1` Ready control-plane。
- K3s / ArgoCD06:57 曾因 deploy marker `52f61da4` rollout 短暫 `OutOfSync / Progressing`07:00 後 `awoooi-prod=Synced / Healthy`revision `52f61da4b3727f08e41c474a188859025d4c60a2`API `2/2`、Web `2/2`、Worker `1/1`API/Web 跨 `mon` / `mon1`
- CronJob`km-vectorize schedule=0 3 * * *``lastSchedule=2026-06-25T19:00:00Z``lastSuccess=2026-06-25T19:00:14Z`;保留的舊 failed Job 仍只是歷史 evidence。
- Cold-start重跑 `PASS=87 WARN=0 BLOCKED=0`result `GREEN`
- Public routesAWOOOI API / Web、VibeWork、AwoooGo、MOMO health、Stock freshness、Bitan 連續讀回 200Registry `/v2/` 仍是 expected 401。
- AWOOOI API health`healthy / prod / mock_mode=false`PostgreSQL / Redis / OpenClaw / SigNoz / Ollama routes up。
- MOMO`https://mo.wooo.work/health``V10.699`cold-start direct evidence 顯示 current-month parity `15383 / 15383` through `2026-06-24``MOMO_DAILY_FRESHNESS 1|2026-06-24`
- StockPlatform`/api/v1/system/freshness` 一開始在容器剛重啟約 35 秒時曾出現單次 502後續連續讀回皆 200`status=ok``latest_trading_date=2026-06-25`、blockers `[]`price / chips / margin / AI recommendations 皆為 `2026-06-25`
- Backup06:58 backup-status 顯示 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0``core_blockers=0``offsite_fresh=1``rclone_gdrive_fresh=1``last_backup_all=2026-06-26 02:31:02``escrow_missing=5`
- 188產品容器健康但 host `systemctl=degraded`failed units 仍是 `awoooi-startup.service``postgresql@14-main.service``certbot.service``snap.certbot.renew.service`。Host PostgreSQL `14/main` 仍因 checkpoint/WAL failure downcertbot renew 仍被 ACME rate-limit / challenge failure 擋住shared `sentry.wooo.work` certificate valid until `2026-07-09 16:03:40 UTC`
- 112Wazuh manager / indexer / dashboard activeports `1514 / 1515 / 55000` listenproduction `/api/iwooos/wazuh``/api/v1/iwooos/wazuh``200 disabled_waiting_iwooos_wazuh_owner_gate``configured=false`、runtime gate `0`
- 工作站MacBook Pro 111 可連線,`/Users/ooo/codex-workspaces/awoooi-dev` 位於 HEAD `56c83257`、ahead 17Mac Mini 168 可連線,`/Users/ogt/codex-workspaces/awoooi-dev` 位於 HEAD `59485d51`、ahead 17Mac Mini `/System/Volumes/Data` 可用空間約 `3.2Gi`
**做過的命令類型**
- 只讀ping / SSH port check / `systemctl is-system-running` / `systemctl --failed` / `df` / `free` / `ps` / `docker ps` / `kubectl get` / curl / OpenSSL cert readback / backup-status / cold-start scorecard。
- 寫入:只更新 repo 文件;沒有 host / Docker / systemd / Nginx / firewall / K8s / DB / Wazuh runtime 寫操作。
**目前判定**
- Host / service / product data / backup recovery`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`
- SOP 有效性:服務恢復流程有效,且已能區分 deploy warmup transient、route availability、data freshness、backup core、DR escrow、Wazuh registry、host hygiene 與 workstation capacity。
**仍 blocked**
- DR credential escrow evidence missing `5``restic_repository_password``offsite_provider_credentials``break_glass_admin_credentials``dns_registrar_recovery``oauth_ai_provider_recovery`
- 188 host hygienehost PostgreSQL `14/main` checkpoint/WAL failure、certbot renew failure、startup unit failed不得用 `reset-failed` 假綠。
- Wazuhmanager registry accepted `0`Dashboard/API owner evidence 未完成route 200 不是 agent registry recovery。
- Workstation111/168 AWOOOI dev workspace HEAD driftMac Mini local disk only about `3.2Gi` free不能跑大型 build/test 戰役。
**不得宣稱**
- 不得宣稱 `DR_COMPLETE`、credential escrow complete、Wazuh host registry recovered、188 host fully green、雙機 Codex workspace fully synchronized、或所有主機衛生問題已解決。

View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.63
> Version: v1.64
> Last updated: 2026-06-26 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -12,6 +12,8 @@
若只是重啟後要快速判斷能不能宣稱恢復,先跑一頁式總檢查:`scripts/reboot-recovery/post-start-quick-check.sh --no-color`,並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。
2026-06-26 07:02 全主機 live refresh`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。110 `systemctl=running`、failed units `0`,但 load `5.83 / 7.26 / 5.77` 且 top CPU 是 AWOOOI Web `next build`Swap 仍 `7.8Gi / 7.8Gi`;這是 CI/build 壓力,不是 orphan Chrome 或 Docker 事故。120 / 121 `systemctl=running`、K3s activenodes `mon` / `mon1` 均為 Ready。ArgoCD `awoooi-prod` 在 06:57 曾短暫 `OutOfSync / Progressing`,因 deploy marker `52f61da4` rollout 正在替換 API/Web/Worker07:00 後已穩定為 `Synced / Healthy`API `2/2`、Web `2/2`、Worker `1/1`API/Web 仍跨 `mon` / `mon1`。重跑 live cold-start`PASS=87 WARN=0 BLOCKED=0`result `GREEN`。StockPlatform `/api/v1/system/freshness` 曾在容器剛重啟約 35 秒時短暫 `502`,後續連續讀回皆 `200``status=ok``latest_trading_date=2026-06-25`、blockers `[]`;這類 rollout warmup 只有連續失敗才算 blocker。MOMO health 是 `V10.699`cold-start direct evidence 仍顯示 current-month parity `15383 / 15383` 截至 `2026-06-24`daily freshness `1|2026-06-24`。Backup status 06:58110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0``core_blockers=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02``escrow_missing=5`。188 產品容器健康,但 host `systemctl=degraded` 仍是真實 host hygiene blocker`awoooi-startup.service``postgresql@14-main.service``certbot.service``snap.certbot.renew.service` failed。112 Wazuh manager/indexer/dashboard activeports `1514 / 1515 / 55000` listen但 production Wazuh route 仍回報 `disabled_waiting_iwooos_wazuh_owner_gate``configured=false`、manager registry accepted `0`、runtime gate `0`。111 / 168 可連線,但兩邊 AWOOOI dev workspaces 皆 ahead 17 且 HEAD 不同(`111=56c83257``168=59485d51`Mac Mini `/System/Volumes/Data` 只剩約 `3.2Gi`。目前 service recovery 宣告維持 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`host hygiene / DR escrow / Wazuh registry / workstation capacity 明確列為 service green 之外的 blocker。
2026-06-26 06:50-06:55 188 host hygiene read-only triage188 product services remain green, but host `systemctl` is still `degraded` and must not be smoothed into full host green. Failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`. Evidence shows the host PostgreSQL cluster `14/main` is down in `pg_lsclusters`, while product DB / exporters still respond through containerized services; therefore `pg_isready` or `pg_up=1` cannot substitute for host cluster health. The 188 startup service detected `could not locate a valid checkpoint record` on 2026-06-23 and attempted `pg_resetwal` as root, which failed; v1.63 treats PostgreSQL checkpoint/WAL errors as break-glass only and the repo-side startup script now fails closed instead of running `pg_resetwal`. Certbot renew for `sentry.wooo.work` is also failing and hit ACME rate-limit / challenge failure, but the public cert is still valid until 2026-07-09 16:03:40 UTC. Current declaration: `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` for 188, while overall service recovery remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`.
2026-06-26 06:40-06:44 全主機 read-only refresh`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。核心 reboot scope 維持 green110 `systemctl=running`、failed units `0`Docker / Gitea / Harbor / Prometheus / Alertmanager 可用120 / 121 `systemctl=running`、failed units `0`K3s nodes `mon` / `mon1` Ready188 產品容器與 PostgreSQL / Redis / MOMO / SignOz 可用。ArgoCD `awoooi-prod` 已從先前 degraded 收斂為 `Synced / Healthy`revision `b2945ab9f716d9d685434ae0e67b9318414b27fe``km-vectorize` official 03:00 台北時間 run 成功,`lastSuccess=2026-06-25T19:00:14Z`。Public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.690`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status remains core green with `escrow_missing=5`. Boundaries: 188 host still has failed units `awoooi-startup.service`, `certbot.service`, `postgresql@14-main.service`, `snap.certbot.renew.service` that require host hygiene cleanup; 112 Wazuh services / ports are active but Wazuh manager registry accepted remains `0`; 111 / 168 Codex workspaces are reachable but have different local HEADs on the same ahead branch; Mac Mini free space is about `3.4Gi`. Current service verdict remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DR_COMPLETE` or `Wazuh recovered`.

View File

@@ -11,11 +11,11 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 06:40-06:55 all-host read-only refresh confirms the 2026-06-25 21:14 green baseline still holds and adds 112/111/168 reachability. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Core scope services are green: cold-start `PASS=89 WARN=0 BLOCKED=0`; public routes return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health `V10.690`, latest import job `57 completed`, daily freshness `1|2026-06-24`; StockPlatform freshness `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 188 is now explicitly split into product-service green and host hygiene blocker: failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, `snap.certbot.renew.service`; host PostgreSQL cluster is down while product DB containers/exporters are up; certbot renewal is failing but public cert remains valid until 2026-07-09 UTC. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 06:44 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 19:17 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`2026-06-25 19:19 offsite escrow report shows script presence OK, rclone configured, full and partial rclone markers present, `PASS=8 WARN=5 BLOCKED=0`, `ESCROW_MISSING_COUNT=5`; DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 orphan Chrome CPU pressure is cleared, and StockPlatform cron-source drift is repaired. 2026-06-25 21:13 StockPlatform `/api/v1/system/freshness` returned `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.690`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `DB_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. |
| P3 docs / automation contracts | DONE_WITH_ROUTE_RETRY_V161 | 100% | Workplan, SOP v1.61, one-page post-start quick check v1.6, route retry gate, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform 21:00 / 21:10 natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Live 110 script sync remains a separate approved live-write gate; do not claim it here. |
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 07:02 all-host read-only refresh confirms service recovery is green after a short deploy warmup window. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Cold-start rerun returned `PASS=87 WARN=0 BLOCKED=0`, result `GREEN`; public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan and core tools return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.699`; cold-start MOMO direct evidence keeps current-month parity `15383 / 15383` through `2026-06-24`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 06:58 reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 06:57 ArgoCD and StockPlatform each had a short rollout/warmup transient (`awoooi-prod Progressing`, Stock freshness single 502) but both cleared on consecutive readbacks. 188 remains explicitly split into product-service green and host hygiene blocker: failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, `snap.certbot.renew.service`; host PostgreSQL cluster is down while product DB containers/exporters are up; certbot renewal is failing but public cert remains valid until 2026-07-09 UTC. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 07:02 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `52f61da4b3727f08e41c474a188859025d4c60a2`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-26 06:58 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-26 02:31:02`。DR remains blocked on real non-secret credential escrow evidence IDs; do not write placeholder markers or paste secret values. |
| P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 current CPU pressure is attributable to active AWOOOI Web `next build`, and previous orphan Chrome groups remain cleared. 2026-06-26 07:01 StockPlatform `/api/v1/system/freshness` returned `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.699`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. |
| P3 docs / automation contracts | DONE_WITH_ROUTE_RETRY_V164 | 100% | Workplan, SOP v1.64, one-page post-start quick check v1.6, route retry gate, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Live 110 script sync remains a separate approved live-write gate; do not claim it here. |
2026-06-25 19:06 post-CD wrapper readback supersedes the 18:53 wording: consecutive main pushes created a deploy storm where older deploy markers were superseded by later commits. Latest production truth is deploy marker `d8ca8224 chore(cd): deploy 9dbe044 [skip ci]`, ArgoCD `Synced / Healthy`, API/Web/Worker image tag `9dbe044ea1e8e3894ccbeb5ed760bb124b87f7be`, direct route smoke 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan and expected route-gate statuses for MOMO / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse / AIOps, and wrapper `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.690`; AwoooGo / Stock transient 502 reads cleared after upstream warmup and five consecutive route reads returned `200`; 110 load is around `14.51 / 12.34 / 11.42`, with Gitea Actions cache save / `zstdmt` / `tar`, StockPlatform headless Chrome smoke / CI, Gitea, AWOOOI API, ClickHouse, Docker, and platform services visible, not an AWOOOI service blocker. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain. Wazuh route readback is now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but manager registry accepted remains `0`, so Wazuh is a security registry evidence blocker rather than a reboot service blocker.