fix(backup): correct receipt aggregate promql
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m1s
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
AI 技術雷達監控 / ai-technology-watch (push) Successful in 48s
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m1s
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
AI 技術雷達監控 / ai-technology-watch (push) Successful in 48s
This commit is contained in:
@@ -53859,7 +53859,7 @@ production browser smoke:
|
||||
- Gitea main `0acda9f72 fix(backup): add alert receipt contract` 已經由後續 main deploy 帶入 runtime;CD `#4527` 讀回 job succeeded、deploy marker `374d4e4 -> dc4f0ec`、production deploy readback matched、success notification。
|
||||
- 受控同步新版 `backup-health-textfile-exporter.py` 到 110 `/home/wooo/scripts/` 與 188 `/home/ollama/scripts/`,均先備份舊檔;立即刷新 `backup_health.prom`,110 讀回 `awoooi_backup_alert_receipt_stage_expected_info` 88 條、188 讀回 12 條。
|
||||
- `scripts/ops/deploy-alerts.sh` dry-run 後正式套用:遠端規則備份、hash 驗證、Prometheus reload 與 key-rule verifier 成功;Prometheus 已載入 165 條規則。
|
||||
- runtime readback 確認 `BackupAlertReceiptMetricMissing` / `BackupAlertReceiptMetricMissing188` inactive,`BackupAlertReceiptStageMissing` rule health ok。初版逐 stage alert 會產生 100 條 pending,已立刻改成每 `host / receipt_channel` 一條聚合 alert,明細保留在 `awoooi_backup_alert_receipt_stage_fresh == 0`。
|
||||
- runtime readback 確認 `BackupAlertReceiptMetricMissing` / `BackupAlertReceiptMetricMissing188` inactive,`BackupAlertReceiptStageMissing` rule health ok。初版逐 stage alert 會產生 100 條 pending,已立刻改成每 `host / receipt_channel` 一條聚合 alert,明細保留在 `awoooi_backup_alert_receipt_stage_fresh == 0`;聚合 alert expression 必須使用 `== bool 0`,否則 PromQL 會把每個缺口 series 的值保留為 `0`,`sum(...)` 後仍為 `0`。
|
||||
|
||||
**驗證結果**:
|
||||
- `python3.11 -m pytest scripts/ops/tests/test_backup_health_textfile_exporter.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py ops/runner/test_cd_controlled_runtime_profile.py -q -p no:cacheprovider`:`65 passed`。
|
||||
|
||||
@@ -1649,7 +1649,7 @@ groups:
|
||||
runbook: "部署新版 scripts/ops/backup-health-textfile-exporter.py 到 188,刷新 /home/ollama/node_exporter_textfiles/backup_health.prom;不得送測試 Telegram、不得讀 token、不得 restore 到 production。"
|
||||
|
||||
- alert: BackupAlertReceiptStageMissing
|
||||
expr: sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == 0) > 0
|
||||
expr: sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == bool 0) > 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
|
||||
@@ -144,7 +144,7 @@ def static_check(path: Path, baseline_path: Path) -> list[str]:
|
||||
rule = _require_alert(alerts, "BackupAlertReceiptStageMissing")
|
||||
_require_contains(
|
||||
str(rule.get("expr", "")),
|
||||
"sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == 0) > 0",
|
||||
"sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == bool 0) > 0",
|
||||
"BackupAlertReceiptStageMissing expr",
|
||||
)
|
||||
text = _annotation_text(rule)
|
||||
|
||||
@@ -190,7 +190,7 @@ def _escrow_required_alerts(prometheus_url: str, host: str, timeout: int) -> lis
|
||||
def _receipt_stage_required_alerts(prometheus_url: str, host: str, timeout: int) -> list[RequiredAlert]:
|
||||
expr = (
|
||||
"sum by (host, receipt_channel) "
|
||||
f'(awoooi_backup_alert_receipt_stage_fresh{{host="{host}"}} == 0) > 0'
|
||||
f'(awoooi_backup_alert_receipt_stage_fresh{{host="{host}"}} == bool 0) > 0'
|
||||
)
|
||||
rows = _prom_query(prometheus_url, expr, timeout)
|
||||
required: list[RequiredAlert] = []
|
||||
|
||||
@@ -363,7 +363,7 @@ def test_backup_alerts_cover_188_gitea_mirror_subtree() -> None:
|
||||
assert "BackupAlertReceiptMetricMissing" in alerts
|
||||
assert "BackupAlertReceiptStageMissing" in alerts
|
||||
assert "awoooi_backup_alert_receipt_expected_info" in alerts
|
||||
assert "sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == 0) > 0" in alerts
|
||||
assert "sum by (host, receipt_channel) (awoooi_backup_alert_receipt_stage_fresh == bool 0) > 0" in alerts
|
||||
assert "$labels.receipt_channel" in alerts
|
||||
assert "$value" in alerts
|
||||
assert "required_alert/scope/stage/required_notification_type" in alerts
|
||||
|
||||
Reference in New Issue
Block a user