docs(ops): record momo drive auth recovery readback [skip ci]
This commit is contained in:
ogt
@@ -1,3 +1,23 @@
|
||||
## 2026-06-25｜09:37 MOMO Drive auth fail-closed deploy / cold-start readback
|
||||
|
||||
**背景**：09:05 live refresh 顯示 MOMO 資料 freshness 仍停在 `2026-06-17`，且 Google Drive token metadata 缺失。後續 read-only log 追查確認 2026-06-25 00:37 後 scheduler 因缺 JSON token 嘗試 browser OAuth，無頭環境回 `could not locate runnable browser`，但舊程式仍把它折成「沒有找到待匯入檔案」成功。這會讓排程假綠，必須先修 code boundary。
|
||||
|
||||
**Read-only / release evidence**：
|
||||
- MOMO local fix in `/Users/ogt/codex-workspaces/momo-pro-dev`：`e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 fix(import): fail auto import on drive auth failure`。
|
||||
- 修補內容：`GoogleDriveService` 新增 metadata-only `last_error_kind` / `last_error`；`auto_import_from_drive()` 在 Drive auth/API 失敗時回 `success=false`、`failed_count=1`、`connection_error=true`，只在 Drive listing 成功且回空清單時保留「沒有找到待匯入的檔案」成功。
|
||||
- MOMO tests：`pytest tests/test_auto_import_failure_boundaries.py -q` → `4 passed`；`python3 -m py_compile services/google_drive_service.py services/import_service.py tests/test_auto_import_failure_boundaries.py` OK；`git diff --check` OK。
|
||||
- Gitea write / CD：MOMO `main` fast-forwarded to `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`；Gitea Actions `cd.yaml #910` returned `Success` in `1m9s` at `2026-06-25 09:35:20 +08:00`。這是正式 CD lane，不是本視窗手動 SSH restart。
|
||||
- 188 deploy readback：`/home/ollama/momo-pro/services/import_service.py` and `momo-scheduler:/app/services/import_service.py` both contain the fail-closed marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`；`google_drive_service.py` contains `last_error_kind`。`momo-scheduler` and `momo-telegram-bot` were restarted by CD and are healthy。
|
||||
- MOMO public health after deploy：`https://mo.wooo.work/health` returns healthy / PostgreSQL / `V10.657`。
|
||||
- Full cold-start after deploy at `2026-06-25 09:37:52 CST` remains expected `PASS=87 WARN=1 BLOCKED=1`。Hosts / K3s / public routes / AWOOOI API / backup surfaces are available; MOMO scheduler has `SCHEDULER_REGISTERED 5` and recent activity; current-month table parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`。
|
||||
- Backup readback remains green for core backup: 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`integrity_stale=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`escrow_missing=5`。
|
||||
|
||||
**判定**：
|
||||
- 可宣稱：MOMO import pipeline no longer falsely reports Drive auth failure as no-file success after CD `#910`; websites/routes/K3s/backups are available for this evidence set。
|
||||
- 不可宣稱：full-stack green、MOMO data current、Google Drive token repaired、new source file imported、credential escrow complete、DR complete。Hard blocker remains `188 momo source file absent while daily sales data stale` with `MOMO_DAILY_FRESHNESS 8|2026-06-17`; token/writeback remains a separate WARN and must not be repaired by reading token content。
|
||||
|
||||
**邊界**：本輪手動操作為 local code edit / tests / Gitea push / CD readback / read-only SSH verification。沒有手動 SSH 修改 188、沒有手動 Docker restart、沒有 Nginx / firewall / K8s / ArgoCD 操作，沒有 Wazuh / 112 / SOC 修改，沒有讀取或保存 secret。
|
||||
|
||||
## 2026-06-25｜09:05 live cold-start / backup / MOMO token read-only refresh
|
||||
|
||||
**背景**：2026-06-24 23:33 已確認 SOP 能正確阻擋 MOMO source absence，但今天已是 2026-06-25，不能沿用昨日證據。本輪只做 read-only refresh 與文件同步，不碰 Wazuh / 112，不做 Docker、Nginx、firewall、K8s、ArgoCD 或 host runtime 寫操作。
|
||||
|
||||
@@ -16,12 +16,13 @@
|
||||
> 2026-06-24 23:15 Codex live-sync gate readback: read-only deploy parity check correctly blocks because repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; installer remains a live write requiring explicit approval.
|
||||
> 2026-06-24 23:33 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start still blocks only on MOMO source absence / data freshness, not backup.
|
||||
> 2026-06-25 09:05 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start is `PASS=87 WARN=1 BLOCKED=1` because MOMO business data is stale and MOMO Google Drive token metadata is missing.
|
||||
> 2026-06-25 09:37 Codex MOMO deploy readback: MOMO `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`, Gitea Actions `cd.yaml #910` succeeded, and 188 host / `momo-scheduler` source now fail closed on Google Drive auth/API failure. Backup/offsite remains green; full-stack still blocks on MOMO data freshness and `escrow_missing=5`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-25 09:05 Backup / Offsite / Escrow Live Status
|
||||
## 2026-06-25 09:37 Backup / Offsite / Escrow Live Status
|
||||
|
||||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:05 Asia/Taipei.
|
||||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:33 Asia/Taipei; cold-start rerun at 09:37.
|
||||
|
||||
- 110 backup health: `13/13 fresh failed=0`。
|
||||
- 188 backup health: `2/2 fresh failed=0`。
|
||||
@@ -30,6 +31,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
|
||||
- Last aggregate backup: `2026-06-25 02:35:09`。
|
||||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||||
- Full-stack service release blocker remains separate: cold-start `PASS=87 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
|
||||
- MOMO code boundary now covers both failure modes: `cd.yaml #904` makes monthly sync failure fail the import job and prevents Drive file movement; `cd.yaml #910` makes Drive auth/API failure return `success=false` instead of a no-file success.
|
||||
|
||||
| Gate | Status | Evidence |
|
||||
|------|--------|----------|
|
||||
@@ -39,6 +41,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
|
||||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||||
| MOMO Drive token metadata | WARN | Host and container token metadata paths are missing; no token content was read. |
|
||||
| MOMO Drive auth false-green | FIX_DEPLOYED | Gitea Actions `cd.yaml #910` success; 188 host and scheduler container source include fail-closed marker. |
|
||||
| Service full green | NO-GO | Blocked by MOMO source absence / data freshness; token metadata warning also requires owner-gated evidence. |
|
||||
|
||||
---
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.45
|
||||
> Version: v1.46
|
||||
> Last updated: 2026-06-25 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -10,22 +10,22 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
2026-06-25 09:05 live read-only refresh supersedes the 2026-06-24 23:33 scorecard wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO source absence / business data freshness and DR remains blocked by missing credential escrow evidence. A new WARN shows MOMO Google Drive token ownership/writeback is not confirmed because the token metadata path is missing from both host and container views; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
2026-06-25 09:37 live read-only refresh supersedes the 09:05 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO source absence / business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is now fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: auth/API failure now fails closed instead of being reported as an empty folder. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
|
||||
```text
|
||||
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
|
||||
Live cold-start read-only check: 2026-06-25 09:05 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Live cold-start read-only check: 2026-06-25 09:37 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Repo-side cold-start v1.42 live read-only run: New MOMO fields remain MOMO_SOURCE_EMPTY_EVIDENCE_LINES=21, MOMO_IMPORT_CONFIG=當日業績匯入|即時業績_當日, MOMO_LATEST_IMPORT_JOB=56|completed|即時業績_當日.xlsx|2026-06-18T11:41:00.853176|2026-06-18T11:42:02.309425|10936|10936|0. The only BLOCKED text is "188 momo source file absent while daily sales data stale". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
|
||||
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
|
||||
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
|
||||
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. `ollama_local` had a short cooldown in the direct health JSON and is not the current release blocker. 09:05 route smoke shows awoooi API=200, `/zh-TW/iwooos`=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, aiops=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
|
||||
MOMO release state: mo.wooo.work health is healthy on version V10.655. Gitea main fast-forwarded to 84035906aba0e5e190d031a13cfd9b47a8cd1f73 and Gitea Actions cd.yaml #904 completed Success for the import-boundary fix; later version readback confirms the service is current enough to separate release health from business-data freshness.
|
||||
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
|
||||
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current.
|
||||
Google Drive / source-file state: 2026-06-25 09:05 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker remains source absence / stale business data; the token missing state is a separate WARN until owner-provided token/writeback evidence is restored.
|
||||
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:05 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
|
||||
Google Drive / source-file state: 2026-06-25 09:37 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker remains source absence / stale business data; the token missing state is a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
|
||||
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:33 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
|
||||
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
|
||||
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
|
||||
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the import-boundary fix; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence; MOMO Google Drive token/writeback evidence needs a separate non-secret owner-gated repair.
|
||||
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the monthly-sync failure boundary and Drive-auth fail-closed fixes; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence; MOMO Google Drive token/writeback evidence needs a separate non-secret owner-gated repair.
|
||||
Forbidden declaration: full-stack green, MOMO data current, DR complete, credential escrow complete, 110 live monitor synced, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
|
||||
```
|
||||
|
||||
|
||||
@@ -11,13 +11,13 @@
|
||||
|
||||
| Area | Status | Completion | Evidence |
|
||||
|------|--------|------------|----------|
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 09:05 live cold-start returned `PASS=87 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.655`, 110 / 188 runtime and backup checks are green。Remaining hard service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. New warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 09:37 live cold-start returned `PASS=87 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.657`, 110 / 188 runtime and backup checks are green。MOMO Gitea `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`; `cd.yaml #910` succeeded and deployed a fail-closed Drive auth/API boundary into 188 host source and `momo-scheduler` container source. Remaining hard service blocker is still MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-25 09:05 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, VIP `192.168.0.125` is present, node filesystem / disk-pressure / readonly events are `0`, and latest `km-vectorize-29705460-55rgs` completed. |
|
||||
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 09:05 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。DR remains blocked on real non-secret credential escrow evidence IDs. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 97% | Public route/TLS, API/Web route, MOMO health `V10.655`, MOMO main / CD `#904` import-boundary fix, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 09:05. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is now a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE_V142_REPO_ONLY | 100% | Workplan, SOP v1.45, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 97% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 09:37. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE_V142_REPO_ONLY | 100% | Workplan, SOP v1.46, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 09:05, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.655`, but the latest repo-side live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker remains `188 momo source file absent while daily sales data stale`; the token state is a separate WARN and not a reason to read token contents. This is repo-side source-of-truth evidence and not yet a claim that the 110 live monitor script was deployed. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 09:37, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest repo-side live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker remains `188 momo source file absent while daily sales data stale`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`, but this code fix does not create new business data. This is repo-side source-of-truth evidence and not yet a claim that the 110 live monitor script was deployed. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
|
||||
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user