fix(aiops): route backup decisions through ssh
This commit is contained in:
@@ -1344,6 +1344,43 @@ Architecture Review 發現的安全要求(2026-04-11):
|
|||||||
|
|
||||||
3. **群組 B 工具需 trust_score >= 0.8**(硬編碼守衛)
|
3. **群組 B 工具需 trust_score >= 0.8**(硬編碼守衛)
|
||||||
|
|
||||||
|
### Host/Backup SSH Route Invariants (2026-05-01)
|
||||||
|
|
||||||
|
`backup_failure` is a host-layer category. Keep it aligned anywhere
|
||||||
|
`host_resource` is routed, especially:
|
||||||
|
|
||||||
|
- `DecisionManager`: non-`kubectl` actions must route to SSH MCP before
|
||||||
|
`parse_kubectl_action()`. Otherwise SSH diagnosis strings with shell syntax
|
||||||
|
are blocked as `forbidden_shell_metachar`.
|
||||||
|
- `DecisionManager`: `kubectl` actions from `host_resource` or
|
||||||
|
`backup_failure` must be blocked and escalated to emergency intervention.
|
||||||
|
- `AutoRepairService`: host/backup incidents must not fall back to K8s
|
||||||
|
rollout Playbooks.
|
||||||
|
|
||||||
|
Runtime baseline for host/backup repair:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n awoooi-prod get secret ssh-mcp-key awoooi-repair-ssh-key awoooi-repair-known-hosts
|
||||||
|
|
||||||
|
kubectl -n awoooi-prod exec deploy/awoooi-api -- sh -lc '
|
||||||
|
ls -l /run/secrets/ssh_mcp_key /etc/ssh-mcp/known_hosts \
|
||||||
|
/etc/repair-ssh/id_ed25519 /etc/repair-known-hosts/known_hosts
|
||||||
|
'
|
||||||
|
|
||||||
|
kubectl -n awoooi-prod exec deploy/awoooi-api -- sh -lc '
|
||||||
|
for h in 192.168.0.110 192.168.0.120 192.168.0.121; do
|
||||||
|
ssh -i /run/secrets/ssh_mcp_key -o BatchMode=yes \
|
||||||
|
-o StrictHostKeyChecking=yes -o ConnectTimeout=5 wooo@$h "echo OK:$h"
|
||||||
|
done
|
||||||
|
ssh -i /run/secrets/ssh_mcp_key -o BatchMode=yes \
|
||||||
|
-o StrictHostKeyChecking=yes -o ConnectTimeout=5 ollama@192.168.0.188 "echo OK:188"
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
`awoooi-executor` RBAC must include read-only backup evidence:
|
||||||
|
`jobs.batch`, `cronjobs.batch`, PVCs, and Velero backup resources. It may patch
|
||||||
|
`statefulsets.apps` / `daemonsets.apps` only for safe rollout restart.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🚀 Sprint C — DR 備份與恢復 (2026-04-11) ✅
|
## 🚀 Sprint C — DR 備份與恢復 (2026-04-11) ✅
|
||||||
@@ -1503,4 +1540,3 @@ ssh-mcp-key ✅ (ssh_mcp_key + known_hosts)
|
|||||||
|
|
||||||
### Runbook
|
### Runbook
|
||||||
`docs/runbooks/ssh-mcp-setup.md`
|
`docs/runbooks/ssh-mcp-setup.md`
|
||||||
|
|
||||||
|
|||||||
@@ -786,6 +786,31 @@ kubectl -n awoooi-prod logs -l app=awoooi-api --tail=50 | \
|
|||||||
| `Permission denied (publickey)` | known_hosts 缺少該主機 | Pod exec SSH 看錯誤訊息 |
|
| `Permission denied (publickey)` | known_hosts 缺少該主機 | Pod exec SSH 看錯誤訊息 |
|
||||||
| `Load key ... Permission denied` | fsGroup 未設定 | Pod exec `ls -la /etc/repair-ssh/` |
|
| `Load key ... Permission denied` | fsGroup 未設定 | Pod exec `ls -la /etc/repair-ssh/` |
|
||||||
| `Connection refused/timeout` | NetworkPolicy 封鎖 22 | Pod exec `ssh -v` 看連線過程 |
|
| `Connection refused/timeout` | NetworkPolicy 封鎖 22 | Pod exec `ssh -v` 看連線過程 |
|
||||||
|
| `forbidden_shell_metachar` 且 action 是 `ssh ... '...'` | host/backup category 沒在 DecisionManager kubectl parser 前路由 SSH | 查 `alert_category` 是否為 `backup_failure`,確認 `_is_host_layer_ssh_category()` 覆蓋 |
|
||||||
|
|
||||||
|
### Telegram 按鈕 E2E 檢查 (2026-05-01)
|
||||||
|
|
||||||
|
告警卡片按鈕不是純 UI。每個按鈕都必須能在
|
||||||
|
`callback_action_spec.yaml` 找到 callback pattern,並經
|
||||||
|
`callback_dispatcher.py` 路由到實際 handler。
|
||||||
|
|
||||||
|
| 卡片/情境 | 必要按鈕 | 預期處理 |
|
||||||
|
|-----------|----------|----------|
|
||||||
|
| Approval / LLM action | approve, reject, details, ignore | 寫 approval decision、執行或拒絕、查詳情、忽略告警 |
|
||||||
|
| Auto repair unavailable / emergency | investigate, escalate/assign, rollback when applicable | 通知人工/AI Agent 介入,不可靜默 |
|
||||||
|
| Drift TYPE-4D | view diff, adopt, rollback, ignore | 看 diff、採納變更、回滾、忽略 |
|
||||||
|
| Backup / host diagnosis | restart only when rule allows, charts/logs/details, cleanup when safe | 不得提供 K8s-only repair button 當 host/backup 主動作 |
|
||||||
|
| Post-verification degraded/failed | rollback proposal, investigate, details | 不自動 rollback,需人工或 emergency AI Agent 接手 |
|
||||||
|
|
||||||
|
Regression test target: button callback names emitted by `telegram_gateway.py`
|
||||||
|
must stay in sync with `callback_action_spec.yaml`; stale buttons are a
|
||||||
|
production bug because Telegram cards can outlive code deploys.
|
||||||
|
|
||||||
|
Provider name drift is also a ghost-button bug. `callback_action_spec.yaml`
|
||||||
|
may use friendly names (`k8s`, `ssh`), but dispatcher must normalize to actual
|
||||||
|
registered MCP providers (`kubernetes`, `ssh_host`) before `get_provider()`.
|
||||||
|
`backup_failure` cards must expose read-only diagnostics before any write
|
||||||
|
action: host disk, backup jobs, and Velero backup status.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -22,7 +22,7 @@
|
|||||||
# description: <說明>
|
# description: <說明>
|
||||||
|
|
||||||
version: "1.0"
|
version: "1.0"
|
||||||
last_updated: "2026-04-14"
|
last_updated: "2026-05-01"
|
||||||
|
|
||||||
actions:
|
actions:
|
||||||
# ==========================================================================
|
# ==========================================================================
|
||||||
@@ -188,6 +188,53 @@ actions:
|
|||||||
timeout_sec: 1
|
timeout_sec: 1
|
||||||
description: "返回飛輪儀表板 URL"
|
description: "返回飛輪儀表板 URL"
|
||||||
|
|
||||||
|
backup_check_host_disk:
|
||||||
|
label: "查主機磁碟"
|
||||||
|
emoji: "💾"
|
||||||
|
risk: low
|
||||||
|
callback_format: info
|
||||||
|
category: backup_failure
|
||||||
|
mcp:
|
||||||
|
provider: ssh
|
||||||
|
tool: ssh_get_disk_usage
|
||||||
|
params:
|
||||||
|
host: "{labels.instance}"
|
||||||
|
reply_format: code
|
||||||
|
timeout_sec: 8
|
||||||
|
description: "備份失敗時檢查主機磁碟容量與 Docker 目錄大小"
|
||||||
|
|
||||||
|
backup_check_jobs:
|
||||||
|
label: "查備份 Job"
|
||||||
|
emoji: "📦"
|
||||||
|
risk: low
|
||||||
|
callback_format: info
|
||||||
|
category: backup_failure
|
||||||
|
mcp:
|
||||||
|
provider: k8s
|
||||||
|
tool: kubectl_get
|
||||||
|
params:
|
||||||
|
namespace: "awoooi-prod"
|
||||||
|
resource: "jobs"
|
||||||
|
reply_format: truncated
|
||||||
|
timeout_sec: 8
|
||||||
|
description: "列出 awoooi-prod 內的備份相關 Job 狀態"
|
||||||
|
|
||||||
|
backup_check_velero:
|
||||||
|
label: "查 Velero"
|
||||||
|
emoji: "🧰"
|
||||||
|
risk: low
|
||||||
|
callback_format: info
|
||||||
|
category: backup_failure
|
||||||
|
mcp:
|
||||||
|
provider: k8s
|
||||||
|
tool: kubectl_get
|
||||||
|
params:
|
||||||
|
namespace: "velero"
|
||||||
|
resource: "backups.velero.io"
|
||||||
|
reply_format: truncated
|
||||||
|
timeout_sec: 8
|
||||||
|
description: "列出 Velero backup CR 狀態"
|
||||||
|
|
||||||
# ==========================================================================
|
# ==========================================================================
|
||||||
# 寫類按鈕(有副作用,4-part nonce callback)
|
# 寫類按鈕(有副作用,4-part nonce callback)
|
||||||
# ==========================================================================
|
# ==========================================================================
|
||||||
|
|||||||
@@ -35,6 +35,18 @@ import yaml
|
|||||||
logger = structlog.get_logger(__name__)
|
logger = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
_PROVIDER_ALIASES = {
|
||||||
|
"k8s": "kubernetes",
|
||||||
|
"ssh": "ssh_host",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_provider_name(provider_name: str) -> str:
|
||||||
|
"""Normalize legacy callback spec provider names to registered MCP providers."""
|
||||||
|
|
||||||
|
return _PROVIDER_ALIASES.get(provider_name, provider_name)
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Data Types
|
# Data Types
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -262,14 +274,15 @@ async def dispatch_action(
|
|||||||
|
|
||||||
# MCP registry dispatch
|
# MCP registry dispatch
|
||||||
from src.plugins.mcp.registry import get_provider
|
from src.plugins.mcp.registry import get_provider
|
||||||
provider = get_provider(spec.mcp_provider)
|
provider_name = _resolve_provider_name(spec.mcp_provider)
|
||||||
|
provider = get_provider(provider_name)
|
||||||
if not provider:
|
if not provider:
|
||||||
duration = (time.perf_counter() - start) * 1000
|
duration = (time.perf_counter() - start) * 1000
|
||||||
return DispatchResult(
|
return DispatchResult(
|
||||||
success=False, action=action_name, incident_id=incident_id,
|
success=False, action=action_name, incident_id=incident_id,
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
result_text=f"{spec.emoji} {spec.label} 失敗:MCP provider '{spec.mcp_provider}' 未註冊",
|
result_text=f"{spec.emoji} {spec.label} 失敗:MCP provider '{provider_name}' 未註冊",
|
||||||
error=f"provider_not_found: {spec.mcp_provider}",
|
error=f"provider_not_found: {provider_name}",
|
||||||
duration_ms=duration,
|
duration_ms=duration,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -85,6 +85,22 @@ def _should_escalate_auto_approve_rejection(reason: Any) -> bool:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
_HOST_LAYER_SSH_CATEGORIES = {"infrastructure", "host_resource", "backup_failure"}
|
||||||
|
_NON_K8S_HOST_CATEGORIES = {"host_resource", "backup_failure"}
|
||||||
|
|
||||||
|
|
||||||
|
def _is_host_layer_ssh_category(category: str | None) -> bool:
|
||||||
|
"""Return True when DecisionManager must route non-kubectl actions to SSH."""
|
||||||
|
|
||||||
|
return (category or "") in _HOST_LAYER_SSH_CATEGORIES
|
||||||
|
|
||||||
|
|
||||||
|
def _is_non_k8s_host_category(category: str | None) -> bool:
|
||||||
|
"""Return True for host/backup alerts that must not auto-run kubectl."""
|
||||||
|
|
||||||
|
return (category or "") in _NON_K8S_HOST_CATEGORIES
|
||||||
|
|
||||||
|
|
||||||
async def _escalate_decision_auto_repair_unavailable(
|
async def _escalate_decision_auto_repair_unavailable(
|
||||||
*,
|
*,
|
||||||
incident: Incident,
|
incident: Incident,
|
||||||
@@ -1990,36 +2006,36 @@ class DecisionManager:
|
|||||||
except Exception as _rescue_err:
|
except Exception as _rescue_err:
|
||||||
logger.debug("target_rescue_skipped", error=str(_rescue_err))
|
logger.debug("target_rescue_skipped", error=str(_rescue_err))
|
||||||
|
|
||||||
# ADR-073 Phase 3-2: infrastructure 告警 (Docker/Host) → SSH MCP routing (2026-04-12 ogt)
|
# ADR-073 Phase 3-2: infrastructure/host/backup 告警 → SSH MCP routing.
|
||||||
# alert_category = "infrastructure" 表示 Docker 告警,非 kubectl action → SSH
|
# alert_category = "backup_failure" uses the same host-layer path as AutoRepairService.
|
||||||
# P1-1 fix 2026-04-12: 必須在 kubectl safety guard 之前 routing,否則 docker 指令被 _action_safe=False 攔截
|
# P1-1 fix 2026-04-12: 必須在 kubectl safety guard 之前 routing,否則 docker 指令被 _action_safe=False 攔截
|
||||||
_alert_category = getattr(incident, "alert_category", None) or ""
|
_alert_category = getattr(incident, "alert_category", None) or ""
|
||||||
if _alert_category in {"infrastructure", "host_resource"} and action and not action.startswith("kubectl"):
|
if _is_host_layer_ssh_category(_alert_category) and action and not action.startswith("kubectl"):
|
||||||
await self._ssh_execute(incident, token, action, _target)
|
await self._ssh_execute(incident, token, action, _target)
|
||||||
return
|
return
|
||||||
|
|
||||||
# 2026-04-15 ogt: host_resource 告警(HostHighCpuLoad 等)不是 K8s workload 問題
|
# 2026-04-15 ogt: host_resource/backup_failure 告警不是 K8s workload 問題
|
||||||
# 不得執行 kubectl 操作,改降級人工審核
|
# 不得執行 kubectl 操作,改降級人工審核
|
||||||
# 根因:原本只擋了 infrastructure,忘記 host_resource 也不走 K8s
|
# 根因:原本只擋了 infrastructure,忘記 host_resource 也不走 K8s
|
||||||
if _alert_category == "host_resource" and action and action.startswith("kubectl"):
|
if _is_non_k8s_host_category(_alert_category) and action and action.startswith("kubectl"):
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"auto_execute_blocked_host_resource_no_k8s",
|
"auto_execute_blocked_host_layer_no_k8s",
|
||||||
incident_id=incident.incident_id,
|
incident_id=incident.incident_id,
|
||||||
alert_category=_alert_category,
|
alert_category=_alert_category,
|
||||||
action=action[:80],
|
action=action[:80],
|
||||||
reason="host_resource 告警不應執行 K8s kubectl 操作,降級人工審核",
|
reason="host/backup 告警不應執行 K8s kubectl 操作,降級人工審核",
|
||||||
)
|
)
|
||||||
token.state = DecisionState.READY
|
token.state = DecisionState.READY
|
||||||
token.proposal_data["auto_executed"] = False
|
token.proposal_data["auto_executed"] = False
|
||||||
token.proposal_data["mcp_all_failed"] = True
|
token.proposal_data["mcp_all_failed"] = True
|
||||||
token.proposal_data["blocked_reason"] = "host_resource 告警禁止 K8s kubectl,請人工排查主機"
|
token.proposal_data["blocked_reason"] = f"{_alert_category} 告警禁止 K8s kubectl,請人工排查主機/備份"
|
||||||
await self._save_token(token)
|
await self._save_token(token)
|
||||||
_fire_and_forget(
|
_fire_and_forget(
|
||||||
_escalate_decision_auto_repair_unavailable(
|
_escalate_decision_auto_repair_unavailable(
|
||||||
incident=incident,
|
incident=incident,
|
||||||
token=token,
|
token=token,
|
||||||
failure_reason=token.proposal_data["blocked_reason"],
|
failure_reason=token.proposal_data["blocked_reason"],
|
||||||
attempted_actions="auto_execute -> host_resource_k8s_block -> emergency_intervention",
|
attempted_actions=f"auto_execute -> {_alert_category}_k8s_block -> emergency_intervention",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
_fire_and_forget(_push_decision_to_telegram(incident, token.proposal_data))
|
_fire_and_forget(_push_decision_to_telegram(incident, token.proposal_data))
|
||||||
|
|||||||
@@ -4,7 +4,11 @@ from src.api.v1.webhooks import (
|
|||||||
_should_bypass_alertmanager_llm,
|
_should_bypass_alertmanager_llm,
|
||||||
_should_use_alertmanager_rule_first,
|
_should_use_alertmanager_rule_first,
|
||||||
)
|
)
|
||||||
from src.services.decision_manager import _should_escalate_auto_approve_rejection
|
from src.services.decision_manager import (
|
||||||
|
_is_host_layer_ssh_category,
|
||||||
|
_is_non_k8s_host_category,
|
||||||
|
_should_escalate_auto_approve_rejection,
|
||||||
|
)
|
||||||
from src.services.telegram_gateway import _format_resolved_guard_stamp
|
from src.services.telegram_gateway import _format_resolved_guard_stamp
|
||||||
|
|
||||||
|
|
||||||
@@ -84,6 +88,18 @@ def test_manual_gate_reasons_escalate_to_emergency_intervention():
|
|||||||
assert _should_escalate_auto_approve_rejection("critical_operation") is False
|
assert _should_escalate_auto_approve_rejection("critical_operation") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_backup_failure_routes_to_decision_ssh_before_kubectl_parser():
|
||||||
|
assert _is_host_layer_ssh_category("backup_failure") is True
|
||||||
|
assert _is_host_layer_ssh_category("host_resource") is True
|
||||||
|
assert _is_host_layer_ssh_category("kubernetes") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_backup_failure_blocks_k8s_auto_execute():
|
||||||
|
assert _is_non_k8s_host_category("backup_failure") is True
|
||||||
|
assert _is_non_k8s_host_category("host_resource") is True
|
||||||
|
assert _is_non_k8s_host_category("infrastructure") is False
|
||||||
|
|
||||||
|
|
||||||
def test_resolved_guard_stamp_without_timestamp_is_clean():
|
def test_resolved_guard_stamp_without_timestamp_is_clean():
|
||||||
assert _format_resolved_guard_stamp(None) == "✅ 此事件已解決"
|
assert _format_resolved_guard_stamp(None) == "✅ 此事件已解決"
|
||||||
|
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ from src.services.callback_dispatcher import (
|
|||||||
list_actions_for_category,
|
list_actions_for_category,
|
||||||
load_action_registry,
|
load_action_registry,
|
||||||
_lookup_context,
|
_lookup_context,
|
||||||
|
_resolve_provider_name,
|
||||||
_resolve_template,
|
_resolve_template,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -68,6 +69,11 @@ class TestRegistryLoading:
|
|||||||
assert spec and spec.callback_format == "info", \
|
assert spec and spec.callback_format == "info", \
|
||||||
f"{qa} should use info format"
|
f"{qa} should use info format"
|
||||||
|
|
||||||
|
def test_legacy_provider_aliases_resolve_to_registered_names(self):
|
||||||
|
assert _resolve_provider_name("k8s") == "kubernetes"
|
||||||
|
assert _resolve_provider_name("ssh") == "ssh_host"
|
||||||
|
assert _resolve_provider_name("prometheus") == "prometheus"
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Category filtering
|
# Category filtering
|
||||||
@@ -91,6 +97,16 @@ class TestCategoryFiltering:
|
|||||||
assert any(a.callback_format == "info" for a in actions), "需至少 1 個查類"
|
assert any(a.callback_format == "info" for a in actions), "需至少 1 個查類"
|
||||||
assert any(a.callback_format == "nonce" for a in actions), "需至少 1 個寫類"
|
assert any(a.callback_format == "nonce" for a in actions), "需至少 1 個寫類"
|
||||||
|
|
||||||
|
def test_backup_failure_has_read_only_diagnostics(self):
|
||||||
|
actions = list_actions_for_category("backup_failure")
|
||||||
|
names = {a.name for a in actions}
|
||||||
|
assert {
|
||||||
|
"backup_check_host_disk",
|
||||||
|
"backup_check_jobs",
|
||||||
|
"backup_check_velero",
|
||||||
|
}.issubset(names)
|
||||||
|
assert all(a.callback_format == "info" for a in actions)
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Template variable resolution
|
# Template variable resolution
|
||||||
|
|||||||
@@ -12,19 +12,31 @@ Live e2e 用 `HostBackupFailed` 打 Alertmanager 後發現 aged backup 告警會
|
|||||||
|
|
||||||
### 完成
|
### 完成
|
||||||
- `_should_use_alertmanager_rule_first()` / `_should_bypass_alertmanager_llm()` 納入 `backup_failure`,備份失敗 YAML `SSH_DIAGNOSE` 不再被 LLM 覆蓋成 K8s 動作。
|
- `_should_use_alertmanager_rule_first()` / `_should_bypass_alertmanager_llm()` 納入 `backup_failure`,備份失敗 YAML `SSH_DIAGNOSE` 不再被 LLM 覆蓋成 K8s 動作。
|
||||||
|
- `DecisionManager` SSH route 與 `AutoRepairService` 分類對齊:`backup_failure` 非 kubectl action 先走 SSH MCP,不再落入 `parse_kubectl_action()` 後被 `forbidden_shell_metachar` 擋下。
|
||||||
|
- `DecisionManager` host/backup K8s block 納入 `backup_failure`,若 LLM 或 Playbook 產生 kubectl 動作,直接走 emergency escalation,而不是對備份告警誤做 K8s 修復。
|
||||||
- `AutoRepairService` 追加 host/backup Playbook guard:主機/備份 incident 若匹配到 K8s rollout 類 Playbook,阻擋為 `HOST_BACKUP_K8S_PLAYBOOK`,改走緊急介入。
|
- `AutoRepairService` 追加 host/backup Playbook guard:主機/備份 incident 若匹配到 K8s rollout 類 Playbook,阻擋為 `HOST_BACKUP_K8S_PLAYBOOK`,改走緊急介入。
|
||||||
- `AutoRepairService` post-verification rollback guard:host/backup 或非 K8s Playbook 驗證失敗時,不再合成 `kubectl rollout restart deployment/{target}`,改走 emergency escalation,且不自動 resolve incident。
|
- `AutoRepairService` post-verification rollback guard:host/backup 或非 K8s Playbook 驗證失敗時,不再合成 `kubectl rollout restart deployment/{target}`,改走 emergency escalation,且不自動 resolve incident。
|
||||||
- `EmergencyEscalationService` 沿用既有 `APPROVAL_ESCALATED` DB enum 寫 AOL,避免緊急通道因新 enum 未 migration 而留痕失敗。
|
- `EmergencyEscalationService` 沿用既有 `APPROVAL_ESCALATED` DB enum 寫 AOL,避免緊急通道因新 enum 未 migration 而留痕失敗。
|
||||||
- 補 `phase25_knowledge_enum_names.sql`,讓 `AUTO_RUNBOOK` / `ANTI_PATTERN` enum name 可寫入 PG,修復 auto runbook KM 沉澱失敗。
|
- 補 `phase25_knowledge_enum_names.sql`,讓 `AUTO_RUNBOOK` / `ANTI_PATTERN` enum name 可寫入 PG,修復 auto runbook KM 沉澱失敗。
|
||||||
- `NodeExporterDown` Prometheus rule `auto_repair` 改為 `true`,與 YAML rule catalog 的 exporter restart 策略一致。
|
- `NodeExporterDown` Prometheus rule `auto_repair` 改為 `true`,與 YAML rule catalog 的 exporter restart 策略一致。
|
||||||
|
- `awoooi-executor` RBAC 補 backup/DR 診斷權限:PVC、Jobs/CronJobs、Velero resources read-only,以及 StatefulSet/DaemonSet safe rollout patch。
|
||||||
|
- NetworkPolicy 補 K3s master/worker `22/tcp` egress,讓 SSH MCP 可以覆蓋 120/121,不只 110/188。
|
||||||
|
- Telegram category buttons 補 provider alias 正規化:`k8s` → `kubernetes`、`ssh` → `ssh_host`,避免按鈕畫出來後 dispatcher 找不到 MCP provider。
|
||||||
|
- `backup_failure` 補三個 read-only 診斷按鈕:查主機磁碟、查備份 Job、查 Velero;備份告警不再只有通用批准/拒絕/詳情。
|
||||||
- 補 `backup_failure` NO_ACTION / SSH_DIAGNOSE 單元測試。
|
- 補 `backup_failure` NO_ACTION / SSH_DIAGNOSE 單元測試。
|
||||||
|
|
||||||
### 驗證
|
### 驗證
|
||||||
- `python3 -m py_compile apps/api/src/api/v1/webhooks.py` 通過。
|
- `python3 -m py_compile apps/api/src/api/v1/webhooks.py` 通過。
|
||||||
|
- `python3 -m py_compile apps/api/src/services/decision_manager.py apps/api/src/services/callback_dispatcher.py` 通過。
|
||||||
- `cd apps/api && pytest tests/test_alertmanager_rule_bypass.py tests/test_telegram_ai_automation_block.py tests/test_ai_router_diagnose_fallback.py -q` → 24 passed。
|
- `cd apps/api && pytest tests/test_alertmanager_rule_bypass.py tests/test_telegram_ai_automation_block.py tests/test_ai_router_diagnose_fallback.py -q` → 24 passed。
|
||||||
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 27 passed。
|
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 27 passed。
|
||||||
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 29 passed。
|
- `cd apps/api && pytest tests/test_auto_repair_service.py tests/test_alertmanager_rule_bypass.py -q` → 29 passed。
|
||||||
|
- `cd apps/api && pytest tests/test_alertmanager_rule_bypass.py tests/test_callback_dispatcher.py tests/test_telegram_button_consistency.py -q` → 56 passed。
|
||||||
- YAML parse `ops/monitoring/alerts-unified.yml`、`apps/api/alert_rules.yaml` 通過。
|
- YAML parse `ops/monitoring/alerts-unified.yml`、`apps/api/alert_rules.yaml` 通過。
|
||||||
|
- YAML parse `callback_action_spec.yaml`、`07-rbac.yaml`、`02-network-policy.yaml` 通過。
|
||||||
|
- Live Secret/mount 檢查:`ssh-mcp-key`、`awoooi-repair-ssh-key`、`awoooi-repair-known-hosts` 存在且掛載可讀。
|
||||||
|
- Live SSH MCP key 檢查:`wooo@192.168.0.110`、`ollama@192.168.0.188` OK;`wooo@192.168.0.120/121` 已通過 host key,但 remote `authorized_keys` 尚未納入該公鑰,回 `Permission denied (publickey,password)`。
|
||||||
|
- Live RBAC apply 被 Argo 依 Git 狀態拉回;`07-rbac.yaml` 需推上 Gitea 由 Argo 同步後再驗 `can-i`。
|
||||||
|
|
||||||
## 2026-04-30 | ADR-104 Playbook 版本化 lineage
|
## 2026-04-30 | ADR-104 Playbook 版本化 lineage
|
||||||
|
|
||||||
|
|||||||
@@ -140,6 +140,48 @@ MoWoooWorkDown → Jaccard 匹配 momo-app-down-repair → SSH ollama@192.168.0.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Appendix B — Backup Failure Route Parity (2026-05-01)
|
||||||
|
|
||||||
|
`HostBackupFailed` / backup 類告警的 `alert_category` 是 `backup_failure`。它必須在所有 host-layer 自動化路徑與 `host_resource` 同級處理:
|
||||||
|
|
||||||
|
| Layer | 必須行為 |
|
||||||
|
|-------|----------|
|
||||||
|
| Alertmanager rule-first | YAML `SSH_DIAGNOSE` / `NO_ACTION` 不進 LLM 覆寫 |
|
||||||
|
| AutoRepairService | `backup_failure` 視為 host-layer,拒絕 K8s Playbook fallback |
|
||||||
|
| DecisionManager | 非 `kubectl` 動作在 kubectl parser 前路由 SSH MCP |
|
||||||
|
| DecisionManager K8s guard | `backup_failure` 產生 `kubectl` 時降級 emergency escalation |
|
||||||
|
| Telegram buttons | `backup_failure` 顯示只讀診斷按鈕:主機磁碟、備份 Job、Velero 狀態 |
|
||||||
|
|
||||||
|
2026-05-01 根因:DecisionManager SSH route 只含 `infrastructure` / `host_resource`,漏掉 `backup_failure`,導致 `ssh 192.168.0.110 '...;...'` 類只讀診斷動作掉進 `parse_kubectl_action()`,被 `forbidden_shell_metachar` 擋下。
|
||||||
|
|
||||||
|
同日按鈕審計也發現 category button 的 friendly provider name 會漂移:`callback_action_spec.yaml` 使用 `k8s` / `ssh`,但 MCP registry 實際名稱是 `kubernetes` / `ssh_host`。Dispatcher 必須正規化 provider alias,否則卡片按鈕會顯示但執行時變成 `provider_not_found`。
|
||||||
|
|
||||||
|
### Runtime 權限基準
|
||||||
|
|
||||||
|
- K8s Secret:
|
||||||
|
- `awoooi-repair-ssh-key` 掛載 `/etc/repair-ssh/`
|
||||||
|
- `awoooi-repair-known-hosts` 掛載 `/etc/repair-known-hosts/`
|
||||||
|
- `ssh-mcp-key` 掛載 `/run/secrets/ssh_mcp_key` 與 `/etc/ssh-mcp/known_hosts`
|
||||||
|
- Remote `authorized_keys`:
|
||||||
|
- `wooo@192.168.0.110`
|
||||||
|
- `wooo@192.168.0.120`
|
||||||
|
- `wooo@192.168.0.121`
|
||||||
|
- `ollama@192.168.0.188`
|
||||||
|
- NetworkPolicy egress:
|
||||||
|
- `192.168.0.110:22`
|
||||||
|
- `192.168.0.120:22`
|
||||||
|
- `192.168.0.121:22`
|
||||||
|
- `192.168.0.188:22`
|
||||||
|
- `awoooi-executor` RBAC:
|
||||||
|
- read `jobs.batch`, `cronjobs.batch`
|
||||||
|
- read `persistentvolumeclaims`
|
||||||
|
- read Velero `backups`, `backupstoragelocations`, `backuprepositories`, `podvolumebackups`, `podvolumerestores`, `restores`, `schedules`
|
||||||
|
- patch `statefulsets.apps` / `daemonsets.apps` only for safe rollout restart
|
||||||
|
|
||||||
|
If SSH MCP fails, the incident must not silently become a manual approval card; it must raise the emergency intervention path with the exact SSH failure reason when available.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 首席架構師 Review 記錄 (2026-04-05)
|
## 首席架構師 Review 記錄 (2026-04-05)
|
||||||
|
|
||||||
評分:**72/100 → 修正後 88/100**
|
評分:**72/100 → 修正後 88/100**
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# AWOOOI 正式環境零信任網路策略
|
# AWOOOI 正式環境零信任網路策略
|
||||||
# 負責人: CIO
|
# 負責人: CIO
|
||||||
# 版本: v1.5
|
# 版本: v1.6
|
||||||
# 日期: 2026-04-14
|
# 日期: 2026-05-01
|
||||||
# 變更:
|
# 變更:
|
||||||
|
# - v1.6: 新增 K3s node 120/121 SSH egress,供 SSH MCP 主機診斷/修復
|
||||||
# - v1.5: 新增 keepalived VIP 192.168.0.125/32 ArgoCD NodePort 30443 egress(修復 heartbeat probe)
|
# - v1.5: 新增 keepalived VIP 192.168.0.125/32 ArgoCD NodePort 30443 egress(修復 heartbeat probe)
|
||||||
# - v1.4: 新增 ArgoCD MCP egress(argocd namespace port 80/443)
|
# - v1.4: 新增 ArgoCD MCP egress(argocd namespace port 80/443)
|
||||||
# - v1.3: 新增 192.168.0.111 Ollama 主機 (M1 Pro),移除 188 的 Ollama port
|
# - v1.3: 新增 192.168.0.111 Ollama 主機 (M1 Pro),移除 188 的 Ollama port
|
||||||
@@ -168,7 +169,7 @@ spec:
|
|||||||
- protocol: TCP
|
- protocol: TCP
|
||||||
port: 8080
|
port: 8080
|
||||||
|
|
||||||
# 允許訪問 K8s API (Executor 執行 kubectl 指令)
|
# 允許訪問 K8s API + K3s master SSH (Executor 執行 kubectl/host diagnosis)
|
||||||
# 2026-03-23 修復: Y 按鈕執行超時
|
# 2026-03-23 修復: Y 按鈕執行超時
|
||||||
# 重要: ClusterIP (10.43.0.1:443) 會路由到實際端點 (192.168.0.120:6443)
|
# 重要: ClusterIP (10.43.0.1:443) 會路由到實際端點 (192.168.0.120:6443)
|
||||||
# 必須同時允許兩者,否則流量會被 192.168.0.0/16 排除規則阻擋
|
# 必須同時允許兩者,否則流量會被 192.168.0.0/16 排除規則阻擋
|
||||||
@@ -180,8 +181,11 @@ spec:
|
|||||||
port: 443
|
port: 443
|
||||||
- to:
|
- to:
|
||||||
- ipBlock:
|
- ipBlock:
|
||||||
cidr: 192.168.0.120/32 # K3s Master 實際 API Server 端點 + ArgoCD NodePort
|
cidr: 192.168.0.120/32 # K3s Master 實際 API Server 端點 + ArgoCD NodePort + SSH MCP
|
||||||
ports:
|
ports:
|
||||||
|
# SSH MCP — K3s master host diagnosis/repair path
|
||||||
|
- protocol: TCP
|
||||||
|
port: 22
|
||||||
- protocol: TCP
|
- protocol: TCP
|
||||||
port: 6443
|
port: 6443
|
||||||
# ArgoCD MCP NodePort (2026-04-11): ClusterIP DNAT 跨 namespace 不穩定,改用 NodePort
|
# ArgoCD MCP NodePort (2026-04-11): ClusterIP DNAT 跨 namespace 不穩定,改用 NodePort
|
||||||
@@ -221,6 +225,9 @@ spec:
|
|||||||
- ipBlock:
|
- ipBlock:
|
||||||
cidr: 192.168.0.121/32
|
cidr: 192.168.0.121/32
|
||||||
ports:
|
ports:
|
||||||
|
# SSH MCP — K3s worker host diagnosis/repair path
|
||||||
|
- protocol: TCP
|
||||||
|
port: 22
|
||||||
- protocol: TCP
|
- protocol: TCP
|
||||||
port: 6443
|
port: 6443
|
||||||
- protocol: TCP
|
- protocol: TCP
|
||||||
|
|||||||
@@ -62,6 +62,11 @@ rules:
|
|||||||
resources: ["services", "configmaps"]
|
resources: ["services", "configmaps"]
|
||||||
verbs: ["get", "list", "watch"]
|
verbs: ["get", "list", "watch"]
|
||||||
|
|
||||||
|
# 2026-05-01: backup/disk diagnostics need PVC visibility; read-only only.
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["persistentvolumeclaims"]
|
||||||
|
verbs: ["get", "list"]
|
||||||
|
|
||||||
- apiGroups: ["networking.k8s.io"]
|
- apiGroups: ["networking.k8s.io"]
|
||||||
resources: ["ingresses"]
|
resources: ["ingresses"]
|
||||||
verbs: ["get", "list", "watch"]
|
verbs: ["get", "list", "watch"]
|
||||||
@@ -89,6 +94,23 @@ rules:
|
|||||||
resources: ["statefulsets", "daemonsets"]
|
resources: ["statefulsets", "daemonsets"]
|
||||||
verbs: ["get", "list", "watch"]
|
verbs: ["get", "list", "watch"]
|
||||||
|
|
||||||
|
# 2026-05-01: HostBackupFailed / VeleroBackupFailed diagnosis needs backup job status.
|
||||||
|
- apiGroups: ["batch"]
|
||||||
|
resources: ["jobs", "cronjobs"]
|
||||||
|
verbs: ["get", "list", "watch"]
|
||||||
|
|
||||||
|
# 2026-05-01: Velero backup status is read-only evidence for backup_failure alerts.
|
||||||
|
- apiGroups: ["velero.io"]
|
||||||
|
resources:
|
||||||
|
- backups
|
||||||
|
- backupstoragelocations
|
||||||
|
- backuprepositories
|
||||||
|
- podvolumebackups
|
||||||
|
- podvolumerestores
|
||||||
|
- restores
|
||||||
|
- schedules
|
||||||
|
verbs: ["get", "list", "watch"]
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# 寫入權限 (Write) - 僅限故障排除操作
|
# 寫入權限 (Write) - 僅限故障排除操作
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
@@ -104,6 +126,11 @@ rules:
|
|||||||
resources: ["deployments"]
|
resources: ["deployments"]
|
||||||
verbs: ["patch"]
|
verbs: ["patch"]
|
||||||
|
|
||||||
|
# 2026-05-01: allow the same safe rollout restart primitive on controller types.
|
||||||
|
- apiGroups: ["apps"]
|
||||||
|
resources: ["statefulsets", "daemonsets"]
|
||||||
|
verbs: ["patch"]
|
||||||
|
|
||||||
# Scale Deployments (擴縮容)
|
# Scale Deployments (擴縮容)
|
||||||
- apiGroups: ["apps"]
|
- apiGroups: ["apps"]
|
||||||
resources: ["deployments/scale"]
|
resources: ["deployments/scale"]
|
||||||
|
|||||||
Reference in New Issue
Block a user