I didn't know it was happening. I was in bed. The server was running. Somewhere in a data centre, automated scripts from dozens of IP addresses were hammering the login endpoint, probing SSH, scanning every open port they could find. By the time I opened my laptop the next morning, 23,000 events had been logged in a single night.
That was the moment I stopped treating security as something I'd deal with later.
What I Actually Found in the Logs
The server ran Fail2ban, but I'd barely configured it beyond the defaults. The auth log was the first thing I checked — thousands of lines of Failed password for root from [IP] and Invalid user admin from [IP]. Different usernames, rotating IPs, same pattern: automated credential stuffing, running through dictionaries of common passwords.
The web access log told a different story. Hundreds of POST requests to /login, /wp-login.php (even though I wasn't running WordPress), /admin, /phpmyadmin. Scanners checking for software I didn't have. SQL injection strings appearing in query parameters. User-agent strings that made no attempt to look like a browser.
"The attacks had been running for weeks. I just hadn't been looking."
That was the part that hit hardest. This wasn't a targeted attack — it was automated background noise. Every server on the internet gets this. I just had no visibility into it.
What Types of Attacks Were Hitting
Attack types in that first log review:
- SSH brute force — dictionary attacks on port 22, rotating IPs to avoid basic rate limits
- HTTP credential stuffing — POST floods against login endpoints with common username/password pairs
- Port scanning — systematic probing to map what's running and where
- Vulnerability scanning — probing for known CVEs, default paths, exposed admin interfaces
- SQL injection probes — automated injection strings in GET and POST parameters
None of them were sophisticated. That's what surprised me most. They didn't need to be — they just needed to run long enough, on enough targets, to find one that wasn't properly defended.
What I Did Next
I tightened the UFW firewall rules, locked SSH to key-only authentication, and configured Fail2ban with shorter ban times and lower retry thresholds. Then I started writing scripts to surface the data in a way I could actually read — not raw log files, but counts, patterns, top offending IPs, attack timelines.
One script became ten. Ten became a dashboard. That dashboard became Sentinel.
The number that started at 23,000 kept climbing as I added more servers and extended the monitoring window. I stopped being surprised by the volume and started being interested in the patterns — which attack types were most common at which times, which IPs appeared across multiple servers, which endpoints were being probed before a full attack sequence started.
At the time of writing: 277,000+ blocked attack events across the fleet. Each one logged, categorised, and used to improve the detection rules.
What This Changed for Me
Before that night, I treated security as something to add once the features were built. After it, I understood that security is the foundation — you can't add it on top of a running system without gaps. It has to be designed in from the start.
The other thing it changed: I stopped assuming a quiet server was a safe server. Quiet just means you're not looking. The attacks are always there. The question is whether you're watching.