fix(k8s): enforce workload topology spread

2026-06-13 12:52:09 +08:00
parent 0c2e9a590b
commit 1361da04db
6 changed files with 36 additions and 14 deletions
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -33861,3 +33861,19 @@ production browser smoke:
 1. 繼續推進 S4.9 owner response 真實回覆資料包，必填 owner role / team、decision、decision reason、affected scope、redacted evidence refs、followup owner；驗收前維持 `0 / false`。
 2. 持續盤點高價值配置控管，優先納入 Nginx、K8s manifest、ArgoCD app、Gitea workflow、registry / Harbor、Sentry / SigNoz / Alertmanager、public gateway、AI provider route、資料庫 migration 與 secrets injection 流程。
 3. 任何主機維護、Kali 更新、Nginx / Docker / firewall / active scan 仍需獨立維護窗口與人工批准，不得由治理頁或 AwoooP approval 直接替代。
+
+## 2026-06-13 — API / Web topology spread hardening candidate
+
+**背景**：
+- 12:43 cold-start 仍是 `PASS=83 WARN=0 BLOCKED=0`，服務面可用。
+- Live K3s 顯示 `awoooi-api` 兩顆副本都在 120；`awoooi-web` 仍分散在 120 / 121。
+- Live deployment 已有 `topologySpreadConstraints`，但 `whenUnsatisfiable=ScheduleAnyway` 只是軟偏好，所以 scheduler 合法接受同節點。
+
+**修正內容**：
+- `k8s/awoooi-prod/06-deployment-api.yaml`、`05-deployment-web.yaml`、`08-deployment-worker.yaml` 改為 `minDomains: 2` + `whenUnsatisfiable: DoNotSchedule`。
+- 目標是 120 / 121 都 Ready 時，API / Web replicas=2 必須跨節點；Worker 單副本仍可跑，未來多副本時也要分散。
+- `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` 與 reboot workplan 同步更新，避免再把 soft spread 誤當 workload balanced guarantee。
+
+**邊界**：
+- 本輪是 GitOps manifest candidate，不手動刪 Pod、不直接 `kubectl patch` live、不 Docker restart、不 Nginx reload、不 firewall 變更。
+- 完成前仍只能說 service / cold-start green；不可宣稱 API workload balanced。
--- a/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md
+++ b/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md
@@ -176,17 +176,19 @@ sudo kubectl top pods -A --sort-by=cpu
 - `k8s/awoooi-prod/06-deployment-api.yaml`
 - `k8s/awoooi-prod/08-deployment-worker.yaml`

-策略是 `maxSkew: 1`、`topologyKey: kubernetes.io/hostname`、`whenUnsatisfiable: ScheduleAnyway`。這會明確偏好 120 / 121 分散，但在單節點維護或另一台短暫不可用時不會把 rollout 卡死。
+2026-06-13 12:43 live refresh 發現 `ScheduleAnyway` 仍可能讓 `awoooi-api` 兩顆副本合法同落 120；cold-start 仍綠，但不能宣稱 API workload balanced。
+
+修正策略改為 `maxSkew: 1`、`minDomains: 2`、`topologyKey: kubernetes.io/hostname`、`whenUnsatisfiable: DoNotSchedule`。120 / 121 都 Ready 時，API / Web replicas=2 必須跨節點；Worker 單副本仍可跑，未來擴副本時也必須分散。

 驗證：

 - `kubectl kustomize k8s/awoooi-prod` 可渲染三個 `topologySpreadConstraints`。
 - `git diff --check` 通過。
 - ArgoCD `awoooi-prod` 已同步到 revision `acaae99986aee2e1f5630984981ccb0f2b676bb8`，operation `Succeeded`。
- Live deployment 的 API / Web / Worker 均有 `topologySpreadConstraints`。
- `awoooi-api`：120 / 121 各一顆。
- `awoooi-web`：120 / 121 各一顆。
- `awoooi-worker`：單副本在 121，符合單副本判定。
+- Live deployment 的 API / Web / Worker 均需有 `topologySpreadConstraints`。
+- `awoooi-api` replicas=2：120 / 121 各一顆，否則是 `SERVICE_OK_PLACEMENT_DRIFT`。
+- `awoooi-web` replicas=2：120 / 121 各一顆。
+- `awoooi-worker` replicas=1：可在任一節點；若 replicas > 1，必須跨 120 / 121 分散。
 - Rollout 後冷啟動：2026-06-13 00:34 final scorecard `PASS=83 WARN=0 BLOCKED=0`。00:33 曾因 110 `known_hosts` 對 120 / 188 過期而短暫 blocked，已在備份後刷新 host key trust。

 補充：rolling update 當下 API 新 Pod 先集中到 121，因為舊 Pod 正在 120 Terminating；後續以單顆 stateless API Pod 受控重排，最終達到 120 / 121 各一顆。沒有執行全 namespace delete、Docker restart、Nginx reload 或 firewall 變更。
@@ -195,9 +197,9 @@ sudo kubectl top pods -A --sort-by=cpu

 | Workload | 目標 |
 |----------|------|
-| `awoooi-api` replicas=2 | 120 / 121 各一個 |
-| `awoooi-web` replicas=2 | 120 / 121 各一個 |
-| `awoooi-worker` replicas=1 | 可留任一節點，但需有重啟恢復證據 |
+| `awoooi-api` replicas=2 | 120 / 121 各一個，`whenUnsatisfiable=DoNotSchedule` |
+| `awoooi-web` replicas=2 | 120 / 121 各一個，`whenUnsatisfiable=DoNotSchedule` |
+| `awoooi-worker` replicas=1 | 可留任一節點；多副本時 `whenUnsatisfiable=DoNotSchedule` |
 | ArgoCD / Velero / kube-system 單副本 | 不急搬；先保持穩定 |
 | DaemonSet | 120 / 121 各一個，已符合 |

--- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
+++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
@@ -56,6 +56,7 @@ Full cold-start may be declared green only for the latest verified evidence set.
 | 2026-06-13 live refresh | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 00:13 backup status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`; 00:33 cold-start exposed 110 `known_hosts` drift for 120 / 188, fixed after backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; 00:34 final cold-start: `PASS=83 WARN=0 BLOCKED=0`; live K3s has `mon` / `mon1` Ready, API/Web are split 120 / 121. 188 host is `degraded` only because `certbot.service` and `snap.certbot.renew.service` failed; ArgoCD remains Degraded because `km-vectorize` CronJob last success is stale. Manual Job `km-vectorize-codex-002709` did not leave verified completion evidence, so this remains open. |
 | 2026-06-13 `km-vectorize` health remediation | IN_PROGRESS_90 | 00:50 live readback: CronJob `lastScheduleTime=2026-06-12T11:00:00Z`, `lastSuccessfulTime=2026-06-04T11:00:37Z`; retained 6/2, 6/3, 6/4 Jobs are `Complete`, latest visible pod log returned `embed-all: 200 {"total":32,"success":32,"failed":0}`. Gitea main `47ee96b0` and ArgoCD sync now corrected live spec to `schedule=0 3 * * *`, `timeZone=Asia/Taipei`, with Job/Pod labels `app/component/environment/phase/system`. 01:04 cold-start is `PASS=83 WARN=0 BLOCKED=0`. Next gate is the official 03:00 CronJob success readback. |
 | 2026-06-13 post-CD trust / workload verification | SERVICE_GREEN_CD_GUARDRAIL_HELD | Gitea main advanced to deploy marker `e4a349bc chore(cd): deploy 414413a [skip ci]`; ArgoCD revision is `e4a349bc`, sync `Synced`, health still `Degraded` only by `km-vectorize` stale success. Live K3s image readback uses `414413a59268eedd391648f112e228716dd05362`; API pods split `mon1` / `mon`, Web pods split `mon` / `mon1`, Worker is single replica on `mon`. 01:28 `/home/wooo/.ssh/known_hosts` mtime remains `2026-06-13 01:20:02 +0800` with 120 / 188 entries present; deploy-specific `/home/wooo/.ssh/deploy_known_hosts` mtime is `01:24:05`, proving CD fix `80e6ec1a` stopped clobbering global trust. 01:26 cold-start: `PASS=83 WARN=0 BLOCKED=0`. |
+| 2026-06-13 API placement hardening | IN_PROGRESS | 12:43 live refresh showed cold-start `PASS=83 WARN=0 BLOCKED=0`, but API replicas `2/2` were on 120 even though topology spread existed. Root cause: `whenUnsatisfiable=ScheduleAnyway` is a soft preference. GitOps candidate changes API/Web/Worker to `minDomains=2` + `DoNotSchedule`; completion requires ArgoCD sync, rollout readback, public route smoke, and cold-start rerun. |

 ---

--- a/k8s/awoooi-prod/05-deployment-web.yaml
+++ b/k8s/awoooi-prod/05-deployment-web.yaml
@@ -33,11 +33,12 @@ spec:
        system: awoooi
        environment: prod
    spec:
-      # 2026-06-13 Codex: 明確偏好跨 120 / 121 分散，但不在單節點維護時卡死 rollout。
+      # 2026-06-13 Codex: 120 / 121 皆 Ready 時強制跨節點分散，避免 replicas=2 合法同落單節點。
      topologySpreadConstraints:
        - maxSkew: 1
+          minDomains: 2
          topologyKey: kubernetes.io/hostname
-          whenUnsatisfiable: ScheduleAnyway
+          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: awoooi-web
--- a/k8s/awoooi-prod/06-deployment-api.yaml
+++ b/k8s/awoooi-prod/06-deployment-api.yaml
@@ -35,11 +35,12 @@ spec:
        system: awoooi
        environment: prod
    spec:
-      # 2026-06-13 Codex: 明確偏好跨 120 / 121 分散，但不在單節點維護時卡死 rollout。
+      # 2026-06-13 Codex: 120 / 121 皆 Ready 時強制跨節點分散，避免 replicas=2 合法同落單節點。
      topologySpreadConstraints:
        - maxSkew: 1
+          minDomains: 2
          topologyKey: kubernetes.io/hostname
-          whenUnsatisfiable: ScheduleAnyway
+          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: awoooi-api
--- a/k8s/awoooi-prod/08-deployment-worker.yaml
+++ b/k8s/awoooi-prod/08-deployment-worker.yaml
@@ -41,11 +41,12 @@ spec:
        environment: prod
        component: signal-processor
    spec:
-      # 2026-06-13 Codex: Worker 目前 min=1，但 HPA 擴到多副本時優先跨節點分散。
+      # 2026-06-13 Codex: Worker 目前 min=1；擴到多副本時必須跨 120 / 121 分散。
      topologySpreadConstraints:
        - maxSkew: 1
+          minDomains: 2
          topologyKey: kubernetes.io/hostname
-          whenUnsatisfiable: ScheduleAnyway
+          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: awoooi-worker