fix(reboot): support windows99 vmx locator via host
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m19s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled

This commit is contained in:
Your Name
2026-07-03 12:51:11 +08:00
parent d74516f2cd
commit 7b02c452e3
5 changed files with 137 additions and 4 deletions

View File

@@ -54296,3 +54296,26 @@ production browser smoke:
**下一步**
- 跑 full focused verifier / runner pressure guard / `git diff --check` 後 commit / push 到 Gitea main等待 CD deploy marker 後讀回 production priority / SLO並接續取得 111 VMX source external verified backup artifact 或更強 identity evidence再做 scoped relink / restore dry-run。
## 2026-07-03 — P0-006 Windows99 VMX locator via-host closure
**完成內容**
- `scripts/reboot-recovery/locate-windows99-vmx-source.sh` 新增 `--via-host` / `WINDOWS99_VIA_HOST` 支援,讓官方 locator 可從目前工作站經 110 no-secret SSH path 執行,而不是卡在本機直連 99 public-key auth missing。
- live 執行 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --timeout 8 --max-auth-users 1` 成功:`locator_collection_status=collected_windows99_vmx_source_locator_stdout``expected_vmx_path_present_count=0``candidate_source_count=0``source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`
- locator 輸出維持 public-safe / no-secret 邊界:`secret_value_read=false``remote_write_performed=false``vm_power_change_performed=false``raw_path_output=false``path_fingerprint_only=true`。結果比前一個 identity metadata scanner 更保守,只輸出 fingerprint不輸出 raw path。
- SOP 已更新 v1.116 / v5.8:下一步不是猜 generic Ubuntu candidate而是取得既有 VMX source 或 verified backup path然後做 scoped relink / restore dry-run + post-verifier。
**本地驗證結果**
- `bash -n scripts/reboot-recovery/locate-windows99-vmx-source.sh`:通過。
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py -q -p no:cacheprovider``5 passed`
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_locate_check.py scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py -q -p no:cacheprovider``7 passed`
- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest apps/api/tests/test_windows99_vmx_source_locator_readback_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider``21 passed`
- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .``git diff --check`:通過。
**仍維持**
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart沒有 workflow_dispatch沒有 DROP / TRUNCATE / restore / prune沒有啟動或關閉 VM。
**下一步**
- commit / push 到 Gitea maindeploy 後讀回 production locator endpoint 與 priority readback。接續 P0 是提供 / 找到 111 verified VMX source 或 backup path再產生 scoped dry-run不直接 relink 或 power on。

View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.115
> Version: v1.116
> Last updated: 2026-07-03 Asia/Taipei
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal此納入不代表授權任何破壞性 runtime apply。
@@ -62,6 +62,8 @@ v1.114 Windows99 VMX package public redaction boundary rulepublic `/api/v1/ag
v1.115 Windows99 VMX source locate check-mode rule2026-07-03 12:34 deploy 後 production readback 已更新為 `status=blocked_reboot_auto_recovery_slo_not_ready``active_blocker_count=10``readiness_percent=60``primary_blocker=reboot_event_required_host_unreachable``can_claim_all_services_recovered_within_target=false`priority readback 同步顯示 `windows99_vmx_source_repair_action_stage=source_repair_check_mode_ready``windows99_vmx_source_repair_next_executable_step=locate_or_relink_missing_vmx_source_check_mode``windows99_vmx_source_locator_status=check_mode_locator_ready_windows99_vmx_source_required`locator endpoint 顯示 `missing_vmx_aliases=["111"]` 與 safe command `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`。定位 111 VMX source 時固定使用 no-secret locator / collector本輪 live check-mode 結果expected 111 VMX path `exists=0``candidate_vmx_count=6``alias_hint_candidate_count=0``identity_target_hint_count=0``identity_unassigned_ubuntu_candidate_count=1``locate_status=single_identity_unassigned_ubuntu_candidate_requires_confirmation`。該 generic Ubuntu candidate 不能直接 relink先前 MAC / neighbor evidence 未證明它是 111且 99 local named backup roots 與 110 bounded backup roots 未找到 111 VMX artifact。下一步只能是取得外部 verified backup artifact 或更強身份證據後建立 scoped relink / restore dry-run再跑 post-verifier不得用猜測候選直接 relink、power on 或宣稱 SLA 完成。
v1.116 Windows99 VMX locator via-host execution rule若目前工作站直連 99 缺 public-key auth但 110 到 99 的 no-secret SSH path 已驗證,官方 locator 必須支援 `--via-host`,固定可執行命令為 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1`。2026-07-03 live readback`locator_collection_status=collected_windows99_vmx_source_locator_stdout``expected_vmx_path_present_count=0``candidate_source_count=0``source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found``safe_next_step=provide_existing_vmx_source_or_backup_path_then_rerun_locator_check_mode_no_write`,且 `secret_value_read=false``remote_write_performed=false``vm_power_change_performed=false``raw_path_output=false``path_fingerprint_only=true`。此結果把下一步收斂為「提供既有 VMX source 或 verified backup path」仍不得 relink generic VM、啟動 VM、restore 到 production 或讀密碼。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。

View File

@@ -46,7 +46,7 @@
- 188 可達,但 `systemd_state=degraded``awoooi-startup.service failed`
- `reboot_event_required_host_unreachable` 仍為 primary blocker因此 10 分鐘 SLO 尚未證明。
固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required``missing_vmx_aliases=["111"]``safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect``secret_value_read=false``remote_write_performed=false``vm_power_change_allowed=false`本輪 live check-mode 結果為 expected 111 VMX path `exists=0``candidate_vmx_count=6``alias_hint_candidate_count=0``identity_target_hint_count=0``identity_unassigned_ubuntu_candidate_count=1``locate_status=single_identity_unassigned_ubuntu_candidate_requires_confirmation`。唯一 generic Ubuntu candidate 還不能安全 relink因為 display name / alias 沒有 111 identity hint先前 MAC / neighbor evidence 也未證明它是 111且 99 local named backup roots 與 110 bounded backup roots 尚未找到 111 VMX artifact。下一步只能取得外部 verified backup artifact 或更強 identity evidence,再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true``check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required``missing_vmx_aliases=["111"]``safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect``secret_value_read=false``remote_write_performed=false``vm_power_change_allowed=false`目前工作站必須加 `--via-host wooo@192.168.0.110 --users wooo --max-auth-users 1` 走已驗證的 110 no-secret pathlive 結果為 `locator_collection_status=collected_windows99_vmx_source_locator_stdout``expected_vmx_path_present_count=0``candidate_source_count=0``source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found``raw_path_output=false``path_fingerprint_only=true`。下一步只能取得既有 VMX source 或 external verified backup path,再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true``check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
### 五主機全貌

View File

@@ -8,6 +8,8 @@ CONNECT_TIMEOUT="${WINDOWS99_CONNECT_TIMEOUT:-3}"
SSH_TIMEOUT="${WINDOWS99_SSH_TIMEOUT:-3}"
REMOTE_LOCATOR_TIMEOUT="${WINDOWS99_REMOTE_LOCATOR_TIMEOUT:-60}"
SSH_PORT="${WINDOWS99_SSH_PORT:-22}"
VIA_HOST="${WINDOWS99_VIA_HOST:-}"
VIA_HOST_PORT="${WINDOWS99_VIA_PORT:-22}"
MAX_AUTH_USERS="${WINDOWS99_MAX_AUTH_USERS:-}"
MAX_AUTH_USERS_EXPLICIT=0
if [[ -n "${WINDOWS99_MAX_AUTH_USERS:-}" ]]; then
@@ -30,7 +32,7 @@ is_positive_int() {
}
usage() {
printf '%s\n' "usage: $0 [--check|--collect] [--host HOST] [--users 'u1 u2'] [--timeout SECONDS] [--max-auth-users N]"
printf '%s\n' "usage: $0 [--check|--collect] [--host HOST] [--users 'u1 u2'] [--via-host USER@HOST] [--timeout SECONDS] [--max-auth-users N]"
}
while [[ $# -gt 0 ]]; do
@@ -51,6 +53,14 @@ while [[ $# -gt 0 ]]; do
SSH_USERS=(${1:-})
SSH_USERS_EXPLICIT=1
;;
--via-host)
shift
VIA_HOST="${1:-}"
;;
--via-port)
shift
VIA_HOST_PORT="${1:-22}"
;;
--timeout)
shift
CONNECT_TIMEOUT="${1:-5}"
@@ -175,15 +185,48 @@ SSH_OPTS=(
-p "${SSH_PORT}"
)
VIA_SSH_OPTS=(
-o BatchMode=yes
-o PreferredAuthentications=publickey
-o PubkeyAuthentication=yes
-o PasswordAuthentication=no
-o KbdInteractiveAuthentication=no
-o NumberOfPasswordPrompts=0
-o ConnectTimeout="${SSH_TIMEOUT}"
-o ConnectionAttempts=1
-o GSSAPIAuthentication=no
-o LogLevel=ERROR
-o StrictHostKeyChecking=no
-o UserKnownHostsFile="${KNOWN_HOSTS_FILE}"
-p "${VIA_HOST_PORT}"
)
quote_words() {
printf '%q ' "$@"
}
run_ssh_timed() {
local user="$1"
local timeout_seconds="$2"
local timeout_wrapper_seconds="$((timeout_seconds + 1))"
local inner_command=""
shift 2
if ! is_positive_int "${timeout_seconds}"; then
timeout_seconds="${SSH_TIMEOUT}"
timeout_wrapper_seconds="$((SSH_TIMEOUT + 1))"
fi
if [[ -n "${VIA_HOST}" ]]; then
local inner_ssh=(ssh "${SSH_OPTS[@]}" -o ConnectTimeout="${timeout_seconds}" "${user}@${TARGET_HOST}" "$@")
inner_command="$(quote_words "${inner_ssh[@]}")"
if [[ "${SSH_TIMEOUT_WRAPPER}" == "timeout" ]]; then
timeout "${timeout_wrapper_seconds}s" ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")"
elif [[ "${SSH_TIMEOUT_WRAPPER}" == "gtimeout" ]]; then
gtimeout "${timeout_wrapper_seconds}s" ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")"
else
ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")"
fi
return $?
fi
if [[ "${SSH_TIMEOUT_WRAPPER}" == "timeout" ]]; then
timeout "${timeout_wrapper_seconds}s" ssh "${SSH_OPTS[@]}" -o ConnectTimeout="${timeout_seconds}" "${user}@${TARGET_HOST}" "$@"
elif [[ "${SSH_TIMEOUT_WRAPPER}" == "gtimeout" ]]; then
@@ -277,6 +320,7 @@ fi
printf '%s\n' "schema_version=windows99_vmx_source_locator_collector_v1"
printf '%s\n' "dry_run=${DRY_RUN}"
printf '%s\n' "target_host=${TARGET_HOST}"
printf '%s\n' "via_host=${VIA_HOST:-none}"
printf '%s\n' "target_host_alias=99"
printf '%s\n' "target_vm_alias=111"
printf '%s\n' "connect_timeout_seconds=${CONNECT_TIMEOUT}"

View File

@@ -50,6 +50,8 @@ def test_locator_contract_forbids_secret_runtime_actions_and_vmx_content_reads()
assert "PasswordAuthentication=no" in bash_text
assert "KbdInteractiveAuthentication=no" in bash_text
assert "NumberOfPasswordPrompts=0" in bash_text
assert "--via-host" in bash_text
assert "via_host=" in bash_text
assert "in_memory_stdin_scriptblock" in bash_text
assert "[Console]::In.ReadToEnd()" in bash_text
assert "Get-ChildItem" in ps_text
@@ -163,7 +165,7 @@ def test_collect_mode_accepts_valid_remote_locator_stdout(tmp_path: Path) -> Non
"""
#!/usr/bin/env bash
joined="$*"
if [[ "$joined" == *"echo AWOOOI_WINDOWS99_SSH_READY"* ]]; then
if [[ "$joined" == *"AWOOOI_WINDOWS99_SSH_READY"* ]]; then
printf '%s\n' 'AWOOOI_WINDOWS99_SSH_READY'
exit 0
fi
@@ -197,3 +199,65 @@ def test_collect_mode_accepts_valid_remote_locator_stdout(tmp_path: Path) -> Non
assert "remote_locator_output_begin" in result.stdout
assert "source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found" in result.stdout
assert "remote_locator_output_end" in result.stdout
def test_collect_mode_supports_no_secret_via_host_path(tmp_path: Path) -> None:
fake_bin = tmp_path / "bin"
fake_bin.mkdir()
_write_executable(
fake_bin / "nc",
"""
#!/usr/bin/env bash
[[ "${!#}" == "22" ]]
""",
)
_write_executable(
fake_bin / "ssh",
"""
#!/usr/bin/env bash
joined="$*"
if [[ "$joined" == *"AWOOOI_WINDOWS99_SSH_READY"* ]]; then
printf '%s\n' 'AWOOOI_WINDOWS99_SSH_READY'
exit 0
fi
if [[ "$joined" == *"powershell"* ]]; then
cat >/dev/null
printf '%s\n' 'EXPECTED_VMX alias=111 index=1 present=0 fingerprint=abc'
printf '%s\n' 'AWOOOI_WINDOWS99_VMX_SOURCE_LOCATOR=1'
printf '%s\n' 'schema_version=windows99_vmx_source_locator_v1'
printf '%s\n' 'MODE=Check'
printf '%s\n' 'target_host_alias=99'
printf '%s\n' 'target_vm_alias=111'
printf '%s\n' 'expected_vmx_path_count=1'
printf '%s\n' 'expected_vmx_path_present_count=0'
printf '%s\n' 'candidate_source_count=0'
printf '%s\n' 'source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found'
printf '%s\n' 'file_content_read=false'
printf '%s\n' 'remote_write_performed=false'
exit 0
fi
exit 255
""",
)
result = _run_locator(
fake_bin,
"--collect",
"--via-host",
"wooo@192.168.0.110",
"--users",
"wooo",
"--max-auth-users",
"1",
)
assert result.returncode == 0
values = _key_values(result.stdout)
assert values["via_host"] == "wooo@192.168.0.110"
assert values["remote_locator_attempted"] == "1"
assert values["locator_collection_status"] == (
"collected_windows99_vmx_source_locator_stdout"
)
assert values["secret_value_read"] == "false"
assert values["remote_write_performed"] == "false"
assert values["vm_power_change_performed"] == "false"