Guilgo Blog

From alert to fix (with a brake). When Prometheus fires DiskPressure or a flood of Failed pods, Telegram is not remediation. It is a page. If you want triage and, in safe cases, an automatic fix, you need a local gate and a short allowlist — not a cloud agent that, on its own, writes a polished report without kubectl, Docker, or SSH into your LAN.

This post covers a design decision and a real homelab implementation: Kubernetes (or k3s), kube-prometheus-stack (Prometheus + Alertmanager), a webhook orchestrator (n8n or a small HTTP service), and Docker on the node. You also want one notification, not three messages saying the same thing.

It continues the proactive homelab SOC (Alertmanager → n8n as inbox) and the same operational point as containers are not a security boundary: host disk cleanup does not happen from a Job whose /tmp is not the node’s.

You do not need remote automations or a cloud agent as the executor. You need code on the node, and a handbrake.

The gap we actually had to close

After a classic homelab incident (root disk filled by scanner temp files, DiskPressure taint, thousands of Failed pods, workloads stuck Pending), three holes were obvious:

1. Incomplete alerts — the default stack pages CrashLoop / NotReady, but not always “this node is about to choke”.
2. Manual investigation — a diagnostic script existed; nobody ran it when the alert fired.
3. Noise and duplicates — Alertmanager and the webhook (n8n) could both hit the same channel; the template even claimed “automatic investigation” while doing nothing.

The temptation is to wire an LLM agent to every firing. The reality: a cloud agent, on its own, does not have kubectl, Docker, or SSH access to your LAN. It can write a report; it cannot delete host temp files or unstick the scheduler. That takes code on the node.

Architecture: two layers, start at layer 0

Prometheus ──► Alertmanager
                    │
                    ├─► Direct Telegram     (non-remediable alerts only)
                    │
                    └─► Webhook (n8n or similar)
                              │
                              ├─► Allowlist + firing?
                              │         │
                              │         └─► Local HTTP :PORT /remediate
                              │                   │
                              │                   └─► script on the HOST
                              │                         · investigate
                              │                         · apply allowlist
                              │                         · one Telegram report
                              │
                              └─► Telegram inbox     (only what AM does not cover,
                                                     or a short “RESOLVED” for allowlist)

Layer 0 (this post): investigation + deterministic remediation behind an allowlist.

Layer 1 (optional, later): a local agent (for example with cwd in the repo) only if layer 0 does not close the incident.

Not recommended as the executor: cloud agents or remote automations that “fix” the node.

Remediation allowlist principles

1. Allowlist, not “do whatever you think”. Only scoped, repeatable, low-impact actions.
2. Disk remediation runs on the host, not inside a Kubernetes Job whose /tmp is not the node’s.
3. One notification path per alert family. If Alertmanager already Telegrams CrashLoop, the webhook does not send it again. If the allowlist remediates DiskPressure, the only useful message is the post-action report (plus a short “RESOLVED” if you want it).
4. Cooldown per groupKey / fingerprint — stops loops on every groupInterval.
5. A single lock (flock) — cooldown stops the same alert from re-firing the script; the lock stops two different alerts from cleaning at once.
6. Verify after acting (df, DiskPressure condition, Failed count). Do not declare victory because you launched an rm.

Example allowlist (tune to your lab)

docker builder prune -af is not reversible: it drops rebuildable cache. It belongs on the allowlist because it is scoped and low-impact, not because you can undo it.

Also human-only: crontab, security daemons, firewall, raising limits blindly.

A reasonable Failed threshold on a small node: > 20 (tune it; a bad incident shows hundreds or thousands).

What you need to build

1. Prometheus rules that measure the real failure

On top of the chart defaults, it is worth having expressions for:

• root disk usage above a warning threshold
• kube_node_status_condition{condition="DiskPressure",status="true"}
• NoSchedule taints
• count(kube_pod_status_phase{phase="Failed"}) above thresholds
• long-lived Pending pods that never schedule

Set severity and useful summary / description annotations. A runbook_url helps humans; the allowlist does not depend on it.

2. Alertmanager: two receivers, no overlap

• Telegram receiver — alertnames for non-remediable workloads (CrashLoop, NotReady, JobFailed, informational OOM…).
• Webhook receiver — warning|critical to your orchestrator (http://ORCHESTRATOR:5678/webhook/... or similar).

Allowlist alerts should not also go to Alertmanager’s Telegram receiver if the script is going to send the report. Otherwise you get double (or triple) pages.

Use HTML/Markdown templates that exist in your Alertmanager version: missing helpers produce message text is empty.

3. Investigation + remediation script on the host

A bash (or Python) script that:

1. Takes a type (DiskPressure, PodsMassFailed, auto, …) and flags --remediate / --dry-run / --no-telegram.
2. Collects state: df, node conditions, pod counts, leftover temp-file patterns.
3. If --remediate and the type is on the allowlist, runs only those actions, under a lock:

flock -n /run/soc-remediation.lock \
  /path/investigate.sh DiskPressure --remediate

4. Re-measures and sends one channel summary (cause, before/after, what ran, what was skipped).

If you launch it as a Kubernetes Job, mount the script via hostPath for investigation only. Host /tmp cleanup and host docker must run as a node process (systemd user/system, or the same user that operates the lab).

4. Tiny HTTP webhook on the host

A minimal service (Python stdlib is enough) on 0.0.0.0:PORT:

• GET /health
• POST /remediate with the Alertmanager JSON
  • ignore resolved
  • map alertname → script type
  • cooldown files under /tmp/...
  • return 202 and run the script in the background (so the orchestrator timeout does not kill the call)

The webhook must not be exposed to the internet. Restrict it to the lab LAN/VLAN and, preferably, require a shared token or a signature before accepting /remediate. Listening on 0.0.0.0 is acceptable only if the firewall already filters who can talk to that port.

Example mapping:

NodeDiskPressureActive, NodeHighDiskUsage  → DiskPressure
NodeTaintNoSchedule, PodPendingTaint       → NodeTaint
PodsMassFailedCritical / Warning           → PodsMassFailed*

5. Orchestrator (n8n or similar)

On the Alertmanager webhook:

1. Parse status, commonLabels.alertname, alerts[].
2. Telegram dedupe:
   • allowlist + firing → do not send a short message (the script will report).
   • allowlist + resolved → a short “RESOLVED” (optional).
   • alertnames Alertmanager already Telegrams → do not repeat.
   • everything else → inbox.
3. If allowlist + firing → POST http://HOST_LAN:PORT/remediate with the original body.

The orchestrator container must reach the host’s LAN IP (not the container’s 127.0.0.1). Same networking caveat as in the Alertmanager → n8n SOC step: the pod must reach the webhook, and the orchestrator must reach the node.

6. A service that survives reboots

systemd --user (with linger) or a system unit:

[Service]
ExecStart=/usr/bin/python3 /path/soc-remediate-webhook.py
Environment=KUBECONFIG=/path/.kube/config
Restart=on-failure

Implementation guide

1. Write and test the script by hand bash investigate.sh DiskPressure --dry-run --no-telegram Then without --dry-run in a test environment.

2. Start the webhook and check curl -s http://HOST:PORT/health curl -s -X POST http://HOST:PORT/remediate?dry_run=1 -H 'Content-Type: application/json' -H 'Authorization: Bearer TOKEN' -d @sample-alert.json

3. Split receivers in AlertmanagerConfig / config Keep allowlist alerts off direct Telegram; webhook for high severities.

4. Wire the orchestrator with both branches (single Telegram path + remediate).

5. Force a test alert (or a synthetic POST to the webhook path) and confirm:

   • one report message
   • actions in the script log
   • cooldown: a second POST does not re-run for N hours
   • lock: a second alert type does not start another cleanup in parallel

6. Document the allowlist in the lab repo (what is in / what is out). Whoever inherits the cluster should not have to guess.

Sample JSON (no secrets)

{
  "status": "firing",
  "groupKey": "test:{alertname=\"NodeHighDiskUsage\"}",
  "commonLabels": {
    "alertname": "NodeHighDiskUsage",
    "severity": "warning"
  },
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "NodeHighDiskUsage",
        "severity": "warning"
      },
      "annotations": {
        "summary": "Root disk above warning threshold"
      }
    }
  ]
}

Lessons learned (do not repeat these)

• Claiming “starting automatic investigation” in the Telegram template without wiring the script.
• Remediating from a container without the host /tmp (or Docker socket) and believing you cleaned the node.
• Leaving set -o pipefail + ls /tmp/pattern-* with no matches: ls exits 2 and kills the script. Use find or shopt -s nullglob.
• Spawning a cloud agent for every flapping KubePodNotReady: expensive and useless without host data.
• Declaring the incident closed without checking runtime (replica 1/1, service HTTP, DiskPressure=False).

What you get

• Response time measured in minutes, not “whenever someone looks at Telegram”.
• Predictable, auditable actions (allowlist log + report).
• Less noise: one good message beats three mediocre ones.
• A clean base if you later want a local agent: only when the allowlist is not enough.

What this post is not

It is not a substitute for backups, correctly sized disks, or hygiene in the jobs themselves (the scanner should clean temp files on exit; cron/allowlist is a safety net). It is also not “AI that runs the cluster”: it is an executable runbook with a handbrake.

If your lab already has Prometheus, Alertmanager, and a webhook, you can stand up layer 0 in an afternoon. The value is not the language model. It is not lying in the template, and touching only what is on the list.

Next step

When the allowlist cannot close the incident, the next step is not handing an LLM unrestricted access. It is escalating diagnosis to a local agent with equally scoped tools and permissions.