Your Name 2ce722bda9 feat(heartbeat): full K8s pod lifecycle state machine + regression tests ...

P0 #3 (徹底長期修系列) — 把 daily report 的 pod 健康判斷從「ready=False 一律告警」
升級到完整 K8s pod lifecycle state machine：

| Phase | 行為 |
|-------|------|
| Succeeded / Completed | 跳過（CronJob/Job 跑完正常） |
| Failed | 必告警 |
| Unknown | 必告警 |
| Pending <5min | 跳過（剛 schedule 合理） |
| Pending >=5min | 告警「image pull / scheduling 卡住」|
| Running ready=True | 健康，跳過 |
| Running ready=False <2min | 跳過（剛起來 probe 還沒過）|
| Running ready=False >=2min | 告警「readiness probe fail / 啟動異常」|
| restarts >=3 | 必告警（無論 phase）|

實作：
- PodInfo 加 start_time: Optional[str]（從 .status.startTime）
- _get_pod_status kubectl custom-columns 加 STARTTIME
- _build_warnings 完整 state machine + 閾值常數

regression test (test_heartbeat_pod_state_machine.py 13 個) 覆蓋每個 phase
+ 邊界條件，含 2026-05-02 統帥截圖鐵證重現（3 個 drift-scanner Succeeded
pod 不該觸發「需關注 3 項」假警報）。

Tests: 13 passed (新增 test_heartbeat_pod_state_machine.py)

接續 a38d9112（單純 Succeeded skip），這次徹底處理 Pending/Failed/Unknown
+ 時間閾值 + 沒 start_time 的保守告警。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-03 01:44:58 +08:00

..

agents

fix(execution-feedback): 修復系統自動化反饋完全斷鏈的三層 P0 故障

2026-04-25 03:29:38 +08:00

integration

fix(km): add phase25 knowledge enum labels

2026-05-01 11:03:03 +08:00

llm_testing

fix(api): 修復全部 lint 錯誤 (ruff --fix)

2026-03-26 16:06:20 +08:00

__init__.py

fix(hermes): ANTHROPIC_API_KEY 注入 + solver 信心度修法 A + 12-Agent 治理文件

2026-04-25 02:33:43 +08:00

conftest.py

docs(logbook): ADR-059 Gitea Webhook 遷移完成記錄

2026-04-05 14:45:02 +08:00

e2e_network_test.py

refactor(api): Phase 22 P1 模組化修復 - Router→Service 封裝

2026-03-31 16:25:00 +08:00

factories.py

feat(test): B5 整合測試框架 — 真實 DB, 5/5 通過

2026-04-10 11:22:57 +08:00

test_action_parser_safety.py

fix(aiops): close ssh and telegram remediation gaps

2026-05-01 16:53:02 +08:00

test_action_parsing.py

fix(aiops): close ssh and telegram remediation gaps

2026-05-01 16:53:02 +08:00

test_agent_loop_foundation.py

feat(aiops): structure agent loop shadow output

2026-05-01 15:09:57 +08:00

test_agent_step_timeouts.py

fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana

2026-04-27 08:15:53 +08:00

test_ai_router_diagnose_fallback.py

feat(ai-router): 推翻 A2 鐵律 — DIAGNOSE primary 改 Ollama 本地優先

2026-04-29 11:39:36 +08:00

test_ai_router_failover_integration.py

fix(flywheel): unblock action safety and Claude fallback

2026-04-29 21:51:18 +08:00

test_ai_router_feedback.py

refactor(tests): 技術債清零 — 移除 FakeRepo/FakeSession Mock DB 違規

2026-04-22 01:33:30 +08:00

test_ai_rule_catalog_write.py

feat(flywheel): W1 PR-P1 + ADR-091 T1 — 飛輪 80→90 第一波

2026-04-29 10:44:39 +08:00

test_aider_event_models.py

fix(imports): CI 環境 import path 統一為 src.*（移除 apps.api.src.* PEP 420 假依賴）

2026-04-20 04:13:02 +08:00

test_aider_event_processor.py

fix(tests): 移 DB 測試到 integration 層修復 CI asyncpg 密碼錯誤

2026-04-22 01:41:34 +08:00

test_aider_event_service.py

feat(service): aider_event_service — classify + signal_data builder (uses existing debounce)

2026-04-20 19:40:01 +08:00

test_aider_events_api.py

feat(api): POST /api/v1/aider/events HMAC webhook + Redis stream push

2026-04-20 19:40:01 +08:00

test_alert_grouping_service.py

feat(adr-076): 戰術 B 四大 Task 全部完成 — 告警聚合+重試+自動報告

2026-04-14 14:39:14 +08:00

test_alert_rule_engine_validation.py

fix(auto): use action parser for repair gates

2026-04-30 14:06:09 +08:00

test_alertmanager_rule_bypass.py

fix(aiops): suppress repeated llm alert loops

2026-05-01 13:02:07 +08:00

test_anomaly_counter.py

fix(api): 修復 34 個 Ruff lint 錯誤

2026-03-29 15:27:49 +08:00

test_aol_to_catalog_writeback_job.py

feat(flywheel): W2 三件 + KMWriter critic 修法（1635 tests 全綠）

2026-04-29 19:44:04 +08:00

test_approval_execution_retry.py

feat(adr-076): 戰術 B 四大 Task 全部完成 — 告警聚合+重試+自動報告

2026-04-14 14:39:14 +08:00

test_approval_field_alignment.py

fix(api): 修復 34 個 Ruff lint 錯誤

2026-03-29 15:27:49 +08:00

test_auto_repair_service.py

fix(aiops): escalate failed host verification

2026-05-01 10:47:42 +08:00

test_callback_dispatcher_llm.py

feat(dispatch): B2 LLM 動態 MCP 派發安全閘 + telegram_gateway LLM 按鈕流程

2026-04-27 15:22:31 +08:00

test_callback_dispatcher.py

fix(aiops): persist emergency intervention traces

2026-05-01 20:34:33 +08:00

test_check_trust_drift_commit_outside_context_poc.py

fix(governance): 修治理告警 4 個 silent failure + Prom sentinel 連鎖

2026-05-03 00:18:57 +08:00

test_circuit_breaker.py

feat(api): Wave 1 安全網 - Circuit Breaker + Global Repair Cooldown

2026-03-29 15:48:03 +08:00

test_classify_alert_early.py

fix(alert): fingerprint 加 alertname 防跨告警指紋衝突 + 補入缺漏心跳分類

2026-04-12 22:50:20 +08:00

test_config_url_validation.py

feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復

2026-04-27 08:11:40 +08:00

test_consensus_integration.py

feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復

2026-04-27 08:11:40 +08:00

test_cs1_auto_execute.py

fix(auto): use action parser for repair gates

2026-04-30 14:06:09 +08:00

test_cs3_auto_execute.py

fix(auto): use action parser for repair gates

2026-04-30 14:06:09 +08:00

test_decision_fusion.py

feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復

2026-04-27 08:11:40 +08:00

test_decision_manager_bare_metal_kubectl_guard.py

fix(decision): block kubectl actions on bare_metal host alerts

2026-05-02 17:41:28 +08:00

test_decision_manager_docker_prune_routing.py

feat(aiops): add ssh_docker_prune to auto-repair flywheel for disk-full alerts

2026-05-02 12:31:37 +08:00

test_destructive_patterns.py

fix(auto): use action parser for repair gates

2026-04-30 14:06:09 +08:00

test_diagnosis_aggregator_stub.py

feat(p3.1-t2-patha): DiagnosisAggregator 路徑 A + Solver F4 critical reject + 對齊測試

2026-04-27 14:42:29 +08:00

test_emergency_escalation_service.py

fix(aiops): persist emergency intervention traces

2026-05-01 20:34:33 +08:00

test_failover_alerter.py

fix(telegram): route alerts to SRE war room

2026-04-30 15:01:23 +08:00

test_failover_e2e_dispatch.py

feat(wave5-p2): GovernanceAgent 4 項自檢 + Ollama 健康告警規則 + Prometheus metrics 整合

2026-04-26 20:56:19 +08:00

test_failure_watcher.py

feat(api): Phase 18.6 E2E 測試驗證 (40 tests)

2026-03-31 12:16:54 +08:00

test_feature_flags.py

feat(adr-080): Phase 0 防護欄建立 — AI 自主化飛輪啟動

2026-04-15 12:44:53 +08:00

test_gap_a4_placeholder_resolution.py

fix(GAP-A4): 規則 Action 模板 placeholder 解析修復 — 解開 8.3h 飛輪沉默

2026-04-14 18:43:29 +08:00

test_gemini_provider_security.py

fix(security): keep Gemini key out of request URLs

2026-04-29 22:56:12 +08:00

test_get_incident_type.py

fix(review): 首席架構師 Code Review 修補 — I1 get_incident_type 邏輯修正 + 測試補全

2026-04-11 21:33:19 +08:00

test_gitea_webhook.py

feat(t0): Task A 按鈕一致性測試 + Task C Gitea→Telegram 通知收尾

2026-04-26 20:17:17 +08:00

test_global_repair_cooldown.py

fix(tests): 首席架構師審查修復 - 測試套件 + DI 強化 (96/100 OUTSTANDING)

2026-04-01 11:11:50 +08:00

test_golden_regression.py

feat(wave2+km): LLM 動態按鈕啟用 + KM 自動寫入 + AI Router dead code 標記

2026-04-28 15:27:33 +08:00

test_governance_agent.py

fix(test): align governance self_failure assertions with nested payload schema

2026-05-03 00:05:04 +08:00

test_heartbeat_pod_state_machine.py

feat(heartbeat): full K8s pod lifecycle state machine + regression tests

2026-05-03 01:44:58 +08:00

test_host_repair_agent.py

refactor(api): Re-Review S1/S2/S3 改善 — 消除重複+防禦性驗證+測試隔離

2026-04-07 11:17:40 +08:00

test_incident_timeline_service.py

feat(flywheel): expose incident processing timeline

2026-04-29 23:38:30 +08:00

test_intent_classifier.py

fix(approval): 修復 Y/n 簽核按鈕無動作問題 (Phase 22 P1)

2026-03-31 16:16:16 +08:00

test_kb_rot_cleaner_schedule.py

feat(p3.2+adr-100): Model Version Tracker + SLO 自治 + KB rot cleaner

2026-04-27 14:54:19 +08:00

test_km_playbook_feedback_loop.py

feat(flywheel): W2 三件 + KMWriter critic 修法（1635 tests 全綠）

2026-04-29 19:44:04 +08:00

test_km_writer_backfill_reconciler.py

fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化

2026-04-29 10:44:39 +08:00

test_km_writer_idempotent.py

fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化

2026-04-29 10:44:39 +08:00

test_km_writer.py

fix(critic-review): KMWriter 名實統一 + Alertmanager 修抑制 + drift checker AST 化

2026-04-29 10:44:39 +08:00

test_learning_chain_e2e.py

feat(wave4-5): P1.3+P1.4 真接線 + Ollama_188 provider 註冊 + quota atomic 修復

2026-04-26 20:44:19 +08:00

test_learning_service.py

fix(tests): 移除 Mock 違規 - test_learning_service.py

2026-03-31 12:20:29 +08:00

test_lifespan_failover_wiring.py

feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警

2026-04-26 20:18:33 +08:00

test_llm_tier1_schema.py

fix(api): 修復全部 lint 錯誤 (ruff --fix)

2026-03-26 16:06:20 +08:00

test_matched_playbook_id_e2e.py

feat(flywheel): W1 PR-P1 + ADR-091 T1 — 飛輪 80→90 第一波

2026-04-29 10:44:39 +08:00

test_mcp_tool_registry.py

feat(adr-081): Phase 1 感官縱深 — 8D 情報蒐集 + 執行後驗證

2026-04-15 13:08:38 +08:00

test_model_regression.py

fix(test): test_model_regression 加 @pytest.mark.integration（需 Ollama 服務）

2026-04-12 13:32:42 +08:00

test_model_version_probe.py

fix(security): keep Gemini key out of request URLs

2026-04-29 22:56:12 +08:00

test_model_version_tracker.py

fix(p3.2): model_version_tracker 改 pure unit test + probe 改善

2026-04-27 14:58:46 +08:00

test_notification_matrix_group_cutover.py

fix(telegram): close remaining DM alert routes

2026-04-30 23:02:17 +08:00

test_nvidia_chat.py

refactor(ai): 模組化重構 - NVIDIA chat 移至 NvidiaProvider

2026-03-29 20:49:23 +08:00

test_nvidia_provider.py

fix(test): nvidia → openclaw_nemo 對齊 RATE_LIMITS/COST_LIMITS key (I3)

2026-04-03 14:00:21 +08:00

test_ollama_auto_recovery.py

feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警

2026-04-26 20:18:33 +08:00

test_ollama_failover_manager.py

fix(failover): 188 完全移出 routing chain，備援只用 Gemini

2026-04-27 15:47:41 +08:00

test_ollama_health_monitor.py

feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警

2026-04-26 20:18:33 +08:00

test_openclaw_cache_key.py

fix(aiops): suppress repeated llm alert loops

2026-05-01 13:02:07 +08:00

test_openclaw_nemo_provider.py

fix(flywheel): fallback on OpenClaw degraded responses

2026-04-29 22:38:57 +08:00

test_operation_parser_ssh.py

fix(approval): route SSH actions through SSHProvider on manual approve

2026-05-02 12:31:37 +08:00

test_p0_diagnose_routing.py

feat(ai-router): 推翻 A2 鐵律 — DIAGNOSE primary 改 Ollama 本地優先

2026-04-29 11:39:36 +08:00

test_p2_db_fixes.py

feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復

2026-04-27 08:11:40 +08:00

test_p3_tier1_integrations.py

test(p3.1-t1): test_p3_tier1_integrations 對應 model_rollback + resource_resolver 整合

2026-04-27 08:17:59 +08:00

test_phase2_fallback.py

fix(aiops): fallback and escalate automation blockers

2026-04-30 14:13:57 +08:00

test_phase22_nemotron_collab.py

feat(web): OpenClaw 風格龍蝦 SVG + 三色狀態燈號 + 測試修正

2026-04-09 08:55:21 +08:00

test_phase25_auto_harvesting.py

test(phase25): Phase 25 P1/P2 source code inspection tests (36 tests)

2026-04-05 00:14:50 +08:00

test_phase25_drift_detection.py

test(phase25): Phase 25 P1/P2 source code inspection tests (36 tests)

2026-04-05 00:14:50 +08:00

test_playbook_generator.py

feat(playbook): version generated playbooks

2026-04-30 23:59:39 +08:00

test_playbook_service.py

feat(playbook): version generated playbooks

2026-04-30 23:59:39 +08:00

test_playbook_ssh_extraction.py

feat(adr-076): Task 3.3 — SSH 修復 KM 萃取（補齊飛輪雙手）

2026-04-14 15:19:54 +08:00

test_post_execution_verifier.py

feat(adr-081): Phase 1 感官縱深 — 8D 情報蒐集 + 執行後驗證

2026-04-15 13:08:38 +08:00

test_pre_decision_investigator.py

feat(adr-081): Phase 1 感官縱深 — 8D 情報蒐集 + 執行後驗證

2026-04-15 13:08:38 +08:00

test_prompt_validation.py

test: skip slow LLM prompt validation tests to fix CI timeout

2026-03-31 14:17:36 +08:00

test_provider_version_alerter.py

test(p3.2): provider_version_alerter 單元測試（6 passed）

2026-04-27 19:56:51 +08:00

test_rag_provider.py

fix(mcp): P1 修復 - DI 一致性 + 測試補充 + 配置優化

2026-03-29 16:23:30 +08:00

test_redis_multisig.py

fix(lint): ruff auto-fix + lewooogo-core src 加入 git

2026-03-23 23:51:37 +08:00

test_report_generation_service.py

feat(adr-076): 戰術 B 四大 Task 全部完成 — 告警聚合+重試+自動報告

2026-04-14 14:39:14 +08:00

test_rule_engine_auto_execute.py

fix(auto): use action parser for repair gates

2026-04-30 14:06:09 +08:00

test_rule_to_playbook_migrator.py

feat(flywheel): W1 PR-R1 規則→Playbook 遷移 + PR-K1 timeline 防禦 ALTER

2026-04-29 10:49:25 +08:00

test_sanitization_service.py

feat(adr-081): Phase 1 感官縱深 — 8D 情報蒐集 + 執行後驗證

2026-04-15 13:08:38 +08:00

test_secret_redactor.py

security(secrets): 替換 test fixture 真 TG bot token 為假值

2026-04-20 04:23:09 +08:00

test_self_healing_validator_integration.py

feat(flywheel): W2 三件 + KMWriter critic 修法（1635 tests 全綠）

2026-04-29 19:44:04 +08:00

test_sentry_webhook_signature.py

test(p3.1-t2): Sentry Webhook 簽章驗證 dedicated tests

2026-04-27 08:24:59 +08:00

test_shadow_auto_approve.py

feat(webhook): shadow-run auto_approve.evaluate + 補 metadata kwarg

2026-04-27 16:00:00 +08:00

test_smart_router.py

fix(test): ComplexityScorer 模型名稱更新 llama3.2:3b → gemma3:4b

2026-04-09 09:01:59 +08:00

test_solver_recommended_actions.py

feat(p3.1-t2-patha): DiagnosisAggregator 路徑 A + Solver F4 critical reject + 對齊測試

2026-04-27 14:42:29 +08:00

test_ssh_provider_docker_prune.py

feat(aiops): add ssh_docker_prune to auto-repair flywheel for disk-full alerts

2026-05-02 12:31:37 +08:00

test_ssh_provider_tools.py

fix(aiops): close ssh and telegram remediation gaps

2026-05-01 16:53:02 +08:00

test_telegram_adr050.py

fix(test): 更新 TestHistoryMessageFormat 適配 Phase 27 雙層策略

2026-04-10 01:12:00 +08:00

test_telegram_ai_automation_block.py

feat(flywheel): surface ai automation and code review

2026-04-30 00:09:25 +08:00

test_telegram_button_consistency.py

fix(telegram): route alerts to SRE war room

2026-04-30 15:01:23 +08:00

test_telegram_gateway_llm_buttons.py

fix(telegram): P0 async race + P1 short_id 碰撞 + P0 incident_id 修復

2026-04-27 19:56:51 +08:00

test_telegram_integration.py

test(telegram): 修正測試斷言符合 ADR-075 中文風險等級

2026-04-12 21:08:48 +08:00

test_telegram_message_templates.py

feat(flywheel): surface ai automation and code review

2026-04-30 00:09:25 +08:00

test_terminal_service.py

fix(approval): 修復 Y/n 簽核按鈕無動作問題 (Phase 22 P1)

2026-03-31 16:16:16 +08:00

test_terminal.py

fix(security): 體健修復 — 7項 Critical/Major 安全問題全修

2026-04-22 01:27:39 +08:00

test_trust_drift_watchdog.py

test(governance): trust_drift_watchdog dedicated tests

2026-04-27 08:24:37 +08:00

test_wave8_fusion_fixes.py

feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復

2026-04-27 08:11:40 +08:00

test_wave8_remaining_blockers.py

fix(test): align governance self_failure assertions with nested payload schema

2026-05-03 00:05:04 +08:00