# 110/188 Host Resource Baseline

> 2026-05-05 ogt + Codex
> Scope: 110 DevOps/Sentry host and 188 AI/Web host Docker Compose resource limits.

## Current Live Finding

### 110

| Service | Live Limit | Live Usage Snapshot | Verdict |
|---|---:|---:|---|
| Sentry ClickHouse | 2 CPU / 8 GiB, merge pool 4 | capped near 2 cores after pool 8 -> 4 restart | Do not lower memory. CPU quota intentionally slows background merge so Sentry cannot dominate 110. If backlog grows, inspect `MergeMutate` and Sentry high-volume features before raising it. |
| Sentry Kafka | 2 CPU / 3 GiB | ~40-55% CPU / 2.5 GiB (84%) | Memory is close to pressure. Do not reduce memory. |
| Sentry taskworker | 2 CPU / 2 GiB, concurrency 2 | ~120-181% CPU after restart | Concurrency reduced from 4 to 2 after Kafka lag cleared. Watch Sentry task latency before further changes. |
| Sentry taskbroker | 1 CPU / 512 MiB | ~70-98% CPU / 160 MiB | CPU is tight; increasing may improve backlog but can raise host load. |
| Sentry taskscheduler | 0.5 CPU / 512 MiB | ~13% CPU / 387 MiB (76%) | Memory is tight; alert at 85% before it stalls. |
| Sentry redis | 0.5 CPU / 512 MiB | ~15-30% CPU / 19 MiB | Live and compose cap are aligned. |
| Sentry uptime-checker | 0.5 CPU / 512 MiB | ~26-30% CPU / 43-187 MiB | Capped after it showed sustained background CPU. |
| Gitea | 3 CPU / 3 GiB | ~4% CPU / 2.18 GiB (73%) | Good cap; memory headroom is not huge. |
| GitHub/Gitea runners | unlimited systemd services | one runner had WatchdogSec=5min and 8,490 restarts; `act` CI containers caused load spikes | Must be monitored outside Docker. Remove bad watchdog drop-in and apply per-runner CPU/Memory quotas. |
| node-exporter | 1 CPU / 256 MiB | ~0-5% CPU / 8 MiB | Good after disabling expensive `arp`, `netclass`, and `netdev` collectors. |
| cadvisor | 1 CPU / 512 MiB | ~0% CPU / 27 MiB | Good safety cap. |
| Harbor / Langfuse / Prometheus / Grafana / Nginx | mostly unlimited | currently low | Needs staged limits, but not during Sentry backlog drain. |

### 188

| Service | Live Limit | Live Usage Snapshot | Verdict |
|---|---:|---:|---|
| momo-scheduler | 2 CPU / 2 GiB | ~0.3% CPU / 163 MiB after crawler burst | CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU. |
| SignOz ClickHouse | 4 CPU / 24 GiB | ~93-133% CPU / 1.1 GiB | Healthy enough; keep current cap. |
| SignOz Zookeeper | 1 CPU / 2 GiB | ~8-18% CPU / 1.09 GiB | OK. |
| cadvisor | 1.5 CPU / 1 GiB | ~0% CPU / 28 MiB | Good. |
| litellm | 1 CPU / 1 GiB | ~0.5-0.9% CPU / 780 MiB | Good cap; keep stateless mode and do not re-add `DATABASE_URL`. |
| momo-pro-system | 2 CPU / 2 GiB | ~1-2% CPU / 740 MiB | Good cap; startup cache prewarm must stay single-flight. |
| momo-scheduler | 2 CPU / 2 GiB | ~0.3% CPU / 105-163 MiB after crawler burst | CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU. |
| momo-telegram-bot | 0.5 CPU / 512 MiB | ~0.7% CPU / 66 MiB | Good cap. |
| momo-db | 2 CPU / 4 GiB | DB had short CPU bursts, then ~0.6-29% with no active long query | Good cap; current bursts are query/workload, not limit pressure. |
| Monitoring tools / websites / exporters | mostly unlimited | low | Add caps gradually with textfile alerts watching pressure. |

## Baseline Policy

Use these thresholds for alerting and AI triage:

| Signal | Threshold | Action |
|---|---:|---|
| host load5 / core | > 1.5 for 15m | Critical SSH diagnosis; classify top service before repair. |
| Docker container CPU | > 2 cores for 10m | Warning diagnosis; check limit, backlog, and workload type. |
| Docker container CPU | > 4 cores for 15m | Critical diagnosis; never generic restart. |
| Docker memory / limit | > 85% for 10m | Warning; raise memory or reduce workload, never lower the limit. |
| Docker CPU or memory limit | missing for 30m | Warning; add service-specific caps before the next spike. |
| Docker restarts | > 5 in 15m | Critical; pull logs and fix crash signature. |
| Systemd runner restarts | > 2 in 15m | Critical; inspect watchdog/drop-ins and active CI jobs. |
| Systemd runner WatchdogSec | > 0 for 10m | Warning; GitHub Actions runner should not be killed by systemd watchdog. |
| Systemd runner quota | CPU or memory unlimited for 30m | Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host. |
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. |

## Rules

1. Do not reduce ClickHouse CPU/memory while merges or Kafka backlog are draining.
2. Do not set ClickHouse `background_pool_size` below the three MergeTree thresholds unless all thresholds are explicitly lower than `pool * ratio`.
3. Do not use lower memory limits as a load-shedding tool. That creates OOM/restart loops.
4. For Chrome crawlers, cap concurrency first; CPU caps are only the safety rail.
5. For Kafka/Snuba, treat high CPU as backlog digestion unless lag stops decreasing.
6. For monitoring tools, caps are required, but every cap must be paired with self-monitoring.
7. Every Docker Compose host must emit `docker_container_cpu_cores`, `docker_container_memory_*`, and a restart counter via node-exporter textfile.
8. Important Docker services must have both CPU and memory guardrails; missing limit alerts should drive service-specific compose fixes, not generic restarts.
9. Disable node-exporter collectors that are slow or failing on each host; exporter scrape time is part of the resource baseline.
10. Runner services are part of the resource baseline even when jobs run outside Docker. Export `systemd_unit_*` metrics for all `actions.runner.*` services.
11. CI runner limits must be set at the systemd unit level. Docker container limits do not protect host load from runner listener, worker, and `act` helper processes.
12. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
13. Stale Gitea Actions cleanup must be dry-run first. Use `--apply` only when the container is older than its workflow/job policy threshold and has no recent useful logs.

## Next Safe Rollout Order

1. Deploy `scripts/ops/docker-stats-textfile-exporter.py` to 110 and 188 textfile collector cron.
2. Reload Prometheus rules with the new Docker CPU/memory/restart baseline alerts.
3. Persist live limits in the owning compose files before considering the host repaired; live `docker update` alone is not durable.
4. Observe 110 for one drain window after node-exporter collector trim and taskworker concurrency 2. Kafka lag is now near zero; if ClickHouse remains high, tune merge/query behavior or reduce high-volume Sentry features, not Kafka memory.
5. Tune `momo-scheduler` crawler concurrency on 188; keep 2 CPU / 2 GiB until success rate and latency prove it is too low.
6. Fix 188 Elephant Alpha/OpenClaw allowed-action drift before enabling resource auto-repair beyond diagnosis.
7. Add modest caps to currently unlimited low-risk services in small batches. Do not alert every unlimited auxiliary container at once; promote candidates only after 24h usage data.
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode.
9. Fix 110 runner services with sudo-capable host maintenance:

```bash
sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply
```

## Known Anti-Patterns

- `docker restart` as a response to sustained CPU.
- Lowering ClickHouse merge pool without validating ClickHouse 25.x thresholds.
- Capping Kafka below current working set during backlog recovery.
- Treating "no alert" as healthy when cAdvisor or textfile exporters are missing.
- Letting monitoring collectors spend seconds per scrape; this turns observability into load.
- Leaving self-hosted runners unlimited on the same host as Sentry/ClickHouse/Gitea.
- Applying live `docker update` without persisting the same guardrail in compose/systemd/IaC.
