work

When the internet knocks on every SSH port

Chippy’s June 2026 security pass: mass SSH brute force, soft VPS posture, and the boring fix list — including emails when people had to move.

When the internet knocks on every SSH port

In June 2026 I had Chippy (our agent stack) run a read-only security pass across the Mac mini, Contabo boxes, server-hm, and the tailnet — not because something cute broke in a blog post, but because the logs were already screaming.

What it found (facts, not vibes)

From the audit artifacts we kept:

  • SSH was under constant fire. On the SG and EU VPSes alone, a 24h window showed thousands of failed passwords and invalid-user attempts (order of ~9k–12k failed passwords per host, more invalid users).
  • Posture was soft where it mattered: root login + password auth still on, firewalls idle on those boxes, X11 forwarding still enabled.
  • Extra surface: public Uptime Kuma port, too many 0.0.0.0 Docker binds on server-hm, tailnet nodes exposing services that should never face every peer by default.
  • Monitoring existed (Tailscale, OpenObserve, Uptime Kuma). Detection stack did not (no CrowdSec/Wazuh-class HIDS on that pass).

No clean “we’re owned” smoking gun showed up in that pass. The uncomfortable part was simpler: we would not have noticed many compromises because nothing was watching properly.

What “defend with an agent” actually meant

Not movie hacking. The useful loop was:

  1. Inventory — hosts, open ports, auth counts, Docker binds, DNS edges
  2. Prioritize — SSH lock-down and public binds before fancy scanners
  3. Write it down — report + artifacts under security/audit-… so humans could act
  4. Escalate to people — when access, ownership, or vendor surface was unclear, that meant emails: host owners, teammates on shared infra, and where relevant provider/abuse channels with logs attached — not a silent patch-and-pray

The agent was good at the boring bulk: pull ss/auth counts, nmap the obvious edges, draft the remediation order. Humans still owned keys, firewall changes, and anything that could lock us out.

Lessons I still use

| Bad instinct | Better move |

|---|---|

| “Green monitoring = safe” | Monitoring ≠ detection |

| “SSH is fine, we have keys” | Measure failed auth volume; assume scanners found you |

| “We’ll harden later” | Password auth + root + open 22 is later’s outage |

| “Agent will handle security” | Agent drafts; humans approve lockouts |

If you run personal + company infra on the same tailnet, treat the agent as a night-shift junior on-call with perfect notes — not as a cowboy counter-offensive.

Full technical findings from that run lived in the internal audit report dated 2026-06-08. This post is the public version: pressure was real, the fix list was boring, and boring is what keeps the lights on.

← All writing