wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 4 Packages Projects Releases Wiki Activity

docs(ops): record momo source and workstation baseline [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-24 20:48:16 +08:00
parent b07486b7f2
commit 5dbacbd4d5
3 changed files with 100 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/LOGBOOK.md
View File

@@ -1,3 +1,31 @@
## 2026-06-24｜MOMO V10.646 source-file absence 與雙機 Codex 基準收斂
**背景**：重啟恢復後，MOMO 的服務健康、程式版本、資料新鮮度與 MacBook / Mac Mini 開發基準不能混在一起判斷。`https://mo.wooo.work/health` 回 healthy 不代表業務資料已到今天；反過來，資料 stale 也不代表正式站仍跑舊版。
**Readback**：
- MOMO public health：`{"database":"postgresql","status":"healthy","version":"V10.646"}`。
- Gitea truth：`wooo/ewoooc` `main=7cfca9375445ea03d6f5d10512d0276a20914d25`，`config.py` `SYSTEM_VERSION = "V10.646"`；正式站版本與 Gitea main 一致。
- Mac Mini `/Users/ogt/codex-workspaces/momo-pro-dev`：branch `codex/momo-current-main-dev-base-20260624`、commit `7cfca9375445ea03d6f5d10512d0276a20914d25`、dirty `0`、`SYSTEM_VERSION = "V10.646"`。
- MacBook Pro `/Users/ooo/codex-workspaces/momo-pro-dev`：branch `codex/momo-current-main-dev-base-20260624`、commit `7cfca9375445ea03d6f5d10512d0276a20914d25`、dirty `0`、`SYSTEM_VERSION = "V10.646"`。
- Gitea remote branch：`codex/momo-current-main-dev-base-20260624` 已推上 `wooo/ewoooc`，指向 `7cfca9375445ea03d6f5d10512d0276a20914d25`；沒有快轉 `dev`，沒有觸發 production deploy。
- Safe handoff artifacts 已同步到 MacBook：Start Here SHA256 `c5c9dcb6b9a0f9a472d6c29ab2c04af3840a06f77e11bf633c246525fe6dfef2`，workstation dashboard SHA256 `48bdc4bb878c6bc87a735ecba93277ce2eee5b331a09b022492307ef5efd6a6b`；dashboard JSON parse 通過。
**MOMO data freshness evidence**：
- DB parity 仍成立：current-month `daily_sales_snapshot` / `realtime_sales_monthly` 都是 `10936` rows，日期範圍 `2026-06-01..2026-06-17`。
- Data freshness 仍不成立：`MOMO_DAILY_FRESHNESS 7|2026-06-17`。
- Mac Mini 候選檔 read-only inspection：`/Users/ogt/momo-pro-system/即時業績_當日_20260112.xlsx` 實際日期欄只有 `2025-07-01..2025-07-02`；iCloud `即時業績全月.xlsx` 只有 `2025-06-01..2025-06-30`；其他候選為空表頭。
- MacBook 候選檔 read-only inspection：`/Users/ooo/codex-workspaces/momo-pro-dev/即時業績_當日_20260112.xlsx` 同樣只有 `2025-07-01..2025-07-02`；`/Users/ooo/Downloads/即時業績202506全月(新線別).xlsx` 為 header-only，不是可匯入的新來源。
**SOP 更新**：
- `FULL-STACK-COLD-START-SOP.md` 升至 `v1.35`。
- 新增 `MOMO_RELEASE_CURRENT`、`MOMO_DB_PARITY`、`MOMO_DATA_FRESH`、`MOMO_SOURCE_AVAILABLE` 四段判定。
- Done Criteria 補上：MOMO 程式版本要對齊 Gitea source-of-truth；業務資料 freshness 必須符合 SLO；DB parity 不能單獨代表資料已恢復。
**邊界**：
- 可宣稱：MOMO service / release version 已恢復且對齊 Gitea main；Mac Mini / MacBook Pro 可從同一條 MOMO Codex current-main baseline 開發。
- 不可宣稱：MOMO data current、full-stack green、DR complete。
- 本輪沒有使用或保存先前聊天中的密碼，沒有同步 auth / SQLite / sessions / raw conversations / `.env` / runtime volumes / raw `.git`，沒有匯入舊 Excel，沒有 truncate / restore DB，沒有 force push。
## 2026-06-24｜188 nginx-exporter 與 CD monitoring coverage gate 收斂
**背景**：`2ec7f6f4 fix(ops): harden heartbeat and momo alert noise` 已由 CD 回寫 deploy marker `622bc372 chore(cd): deploy 2ec7f6f [skip ci]`，production API health 也回 `200 healthy`。但 Gitea CD `#3294` 的 `post-deploy-checks` 步驟仍標 Failure，根因不是 API/Web rollout 失敗，而是 `scripts/generate_monitoring.py --check` 看到 Prometheus job `nginx-exporter` down：`192.168.0.188:9113` connection refused。

88
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.34
> Version: v1.35
> Last updated: 2026-06-24 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -10,29 +10,31 @@
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
2026-06-24 20:17 notification-noise hardening and 188 `nginx-exporter` recovery supersede the earlier 11:35 wording where it discusses heartbeat / MOMO alert behavior and monitoring coverage. The service and data readiness gates below are refreshed by the 20:17 live cold-start scorecard:
2026-06-24 20:42 MOMO source-file and Codex dual-workstation readback supersede the earlier 20:17 wording where it discusses MOMO version / MacBook workspace state. The service and data readiness gates below are refreshed by the 20:17 live cold-start scorecard plus 20:42 source/workstation evidence:
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED.
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; latest deployment marker 622bc372 points runtime image to 2ec7f6f4 and production API health returns healthy. CD #3294 still has a historical Failure record because post-deploy monitoring coverage saw 188 nginx-exporter down before the exporter restore.
MOMO state: mo.wooo.work health is healthy on version V10.645; current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
Google Drive state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56.
MOMO release state: mo.wooo.work health is healthy on version V10.646, matching Gitea main commit 7cfca9375445ea03d6f5d10512d0276a20914d25. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at the same commit with dirty=0.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
Google Drive / source-file state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56. Mac Mini and MacBook candidate spreadsheets were also read-only inspected: the local current daily candidate only contains 2025-07-01..2025-07-02, the iCloud full-month candidate only contains 2025-06-01..2025-06-30, and MacBook candidates are either header-only or the same 2025-07-01..2025-07-02 dataset. These are not legitimate newer sources.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved.
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO code release is current with Gitea main V10.646; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence.
Forbidden declaration: full-stack green, MOMO data current, DR complete, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
```
2026-06-24 13:33 Codex workstation continuity readback:
2026-06-24 20:42 Codex workstation continuity readback:
```text
MacBook Pro 192.168.0.111 can now authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624.
AwoooGo MacBook dev workspace is ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0.
MacBook project-window sync now reports projects=6, ready=3, blocked=3. Ready projects are AWOOOI, MOMO Pro, and AwoooGo.
Safe handoff artifacts still match 9/9 by SHA-256. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied.
MacBook Pro 192.168.0.111 can authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624.
MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty=0.
MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty=0.
AwoooGo MacBook dev workspace remains ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0.
Safe handoff artifacts still match by SHA-256 after Start Here / workstation dashboard refresh: Start Here c5c9dcb6b9a0f9a472d6c29ab2c04af3840a06f77e11bf633c246525fe6dfef2; dashboard 48bdc4bb878c6bc87a735ecba93277ce2eee5b331a09b022492307ef5efd6a6b. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied.
This improves workstation continuity after host reboot / operator relocation, but does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence.
```
@@ -212,8 +214,10 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete
```text
110 / 120 / 121 / 188 HOST_READY = yes
Core public services SERVICE_READY = yes
MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.646 and matches Gitea main commit 7cfca9375445ea03d6f5d10512d0276a20914d25
MOMO_DB_PARITY = yes
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 11:35
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 20:42
MOMO_SOURCE_AVAILABLE = no, because Drive intake has no newer 即時業績_當日 source and Mac Mini / MacBook candidate files only contain old or header-only data
FULL_STACK_GREEN = no, because cold-start scorecard is PASS=86 WARN=0 BLOCKED=1
DR_COMPLETE = no, because credential escrow evidence is incomplete
```
@@ -224,22 +228,23 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete
重啟後若需要從 Mac Mini / MacBook Pro 繼續 Codex 開發，必須另外確認 Codex safe handoff artifacts，不得把服務恢復與 Codex raw 對話同步混為一談。
2026-06-24 13:03 Asia/Taipei readback：
2026-06-24 20:42 Asia/Taipei readback：
```text
MacBook Pro 192.168.0.111 SSH = OK
Safe artifacts synced = 9/9 SHA-256 matched
Safe artifacts synced = Start Here SHA c5c9dcb6b9a0f9a472d6c29ab2c04af3840a06f77e11bf633c246525fe6dfef2, dashboard SHA 48bdc4bb878c6bc87a735ecba93277ce2eee5b331a09b022492307ef5efd6a6b
Start Here readback = registry_ready 3, registry_blocked 8, latest_dev_on_gitea 3, production_on_gitea 8, raw_history_sync False
Workstation dashboard readback = artifact_sync_synced 2, artifact_sync_blocked 0
MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, dev commit 76a89a70986b7428704a12ffbb7180f159db151f, dirty 0
AwoooGo MacBook workspace = blocked by Gitea auth / visibility gate
Workstation dashboard readback = artifact_sync_synced 2, artifact_sync_blocked 0, MOMO current main baseline ready 2
MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty 0
MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty 0
AwoooGo MacBook workspace = ready on dev commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty 0
```
允許宣告：
```text
Mac Mini / MacBook Pro 已同步 Codex 開工入口與治理 snapshot。
MOMO Pro 可以在 MacBook Pro 從 Gitea dev workspace 開工；實作前仍需從 dev 切 codex/<task>。
MOMO Pro 可以在 Mac Mini / MacBook Pro 從 Gitea current-main Codex baseline 開工；實作前仍需從 codex/momo-current-main-dev-base-20260624 切新的 codex/<task>。
```
禁止宣告：
@@ -247,7 +252,7 @@ MOMO Pro 可以在 MacBook Pro 從 Gitea dev workspace 開工；實作前仍需
```text
raw Codex / ChatGPT 歷史聊天已同步。
所有產品都能雙機同步開發。
AwoooGo MacBook workspace ready。
把 MOMO Pro 程式版本 V10.646 當成 MOMO 業務資料已更新。
2026FIFA / Agent Bounty owner preflight 已通過。
```
@@ -1973,6 +1978,51 @@ NO-GO: silence monitoring coverage or mark CD green without target recovery evid
NO-GO: prune Docker volumes or delete exporter state not owned by this SOP.
```
### 14.33 2026-06-24 MOMO V10.646 / source-file absence / dual-workstation baseline
2026-06-24 的第七段變更是把 MOMO 的「程式版本最新」與「業務資料不新」拆成兩個獨立 gate，並把 Mac Mini / MacBook Pro 的 MOMO Codex 工作區固定到 Gitea main 最新基準。這避免重啟後出現兩種誤判：看到 `/health` 最新版就宣稱資料已更新，或看到資料 stale 就誤以為服務仍是舊版。
| 項目 | 20:42 MOMO / workstation baseline |
|------|-----------------------------------|
| SOP version | `v1.35` |
| MOMO public health | `https://mo.wooo.work/health` returns healthy, version `V10.646` |
| Gitea main truth | `wooo/ewoooc` `main=7cfca9375445ea03d6f5d10512d0276a20914d25`, `SYSTEM_VERSION = "V10.646"` |
| Mac Mini workspace | `/Users/ogt/codex-workspaces/momo-pro-dev`, branch `codex/momo-current-main-dev-base-20260624`, commit `7cfca9375445ea03d6f5d10512d0276a20914d25`, dirty `0` |
| MacBook workspace | `/Users/ooo/codex-workspaces/momo-pro-dev`, branch `codex/momo-current-main-dev-base-20260624`, commit `7cfca9375445ea03d6f5d10512d0276a20914d25`, dirty `0` |
| Remote baseline branch | `wooo/ewoooc` `codex/momo-current-main-dev-base-20260624` points to `7cfca9375445ea03d6f5d10512d0276a20914d25` |
| DB parity | current-month `daily_sales_snapshot` and `realtime_sales_monthly` match at `10936` rows, range `2026-06-01..2026-06-17` |
| Data freshness | `MOMO_DAILY_FRESHNESS 7|2026-06-17`; still a hard blocker |
| Source candidates inspected | Mac Mini current daily file contains only `2025-07-01..2025-07-02`; iCloud full-month file contains only `2025-06-01..2025-06-30`; MacBook candidates are header-only or the same `2025-07-01..2025-07-02` file |
| Declaration limit | 可宣稱 MOMO release current 與 Codex dual-workstation baseline ready；不可宣稱 MOMO data current 或 full-stack green |
MOMO post-reboot 判定必須同時回答四個問題：
```text
MOMO_RELEASE_CURRENT = yes/no
MOMO_DB_PARITY = yes/no
MOMO_DATA_FRESH = yes/no
MOMO_SOURCE_AVAILABLE = yes/no
```
解除 MOMO data freshness blocker 的唯一安全路徑：
```text
1. 新的合法 即時業績_當日 source file 出現在預期 Drive intake，或 owner 提供可驗證的 source-evidence reference。
2. 匯入 job 成功，且同步 realtime_sales_monthly 失敗時不得標 completed。
3. source file movement / archive evidence 證明該檔只處理一次。
4. daily_sales_snapshot 與 realtime_sales_monthly row count / date bounds 一致。
5. MOMO_DAILY_FRESHNESS <= 2。
```
禁止把以下情境當成解除 blocker：
```text
NO-GO: 用舊 archive、iCloud 舊月檔、header-only 檔或測試檔重複匯入。
NO-GO: 把 V10.646 health 當成資料日期已到今天。
NO-GO: 把 current-month parity 當成 data freshness。
NO-GO: truncate 或 restore 整庫來製造新鮮度。
```
### 14.22 重啟後時間軸驗證
每次重啟後照時間軸推進，不要等到最後才一次判定。
@@ -2002,7 +2052,9 @@ All must be true:
- AWOOOI API and Web reachable through NodePort/VIP.
- Alertmanager E2E webhook succeeds.
- cron/CronJob schedules are active, unsuspended, and verified.
- MOMO release version matches Gitea source-of-truth for the intended deployment branch.
- momo `daily_sales_snapshot` 與 `realtime_sales_monthly` 在最新匯入日期範圍內筆數一致。
- momo business data freshness is within the declared SLO, and the latest import source evidence is legitimate; DB parity alone is not enough.
- Sentry and SignOz are either healthy or explicitly in controlled backlog recovery.
- High-load batch services are capped or delayed.
- Runners are guarded and released last.

4
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -14,8 +14,8 @@
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 20:17 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 11:20 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 96% | Public route/TLS, API/Web route, momo health `V10.645`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. However MOMO latest business date is `2026-06-17`; stale age is `7` days as of 20:17. Drive pending folder has `0` matching files in repeated scheduler checks and archive latest `2026-06-18T01:30:39Z` is already imported by job `56`, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.34, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 96% | Public route/TLS, API/Web route, momo health `V10.646` matching Gitea main `7cfca9375445ea03d6f5d10512d0276a20914d25`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. However MOMO latest business date is `2026-06-17`; stale age is `7` days as of 20:42. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.35, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.646 / Gitea main / dual-workstation Codex baseline readback, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 20:17, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked.