I find it somewhat annoying and concerning that an essential #security tool like #fail2ban is broken on #ubuntu#linux 24.04 #noble since the end of February and there still is no update in sight.
Someone is using Amazon EC2 instances to try to flood my little Forgejo server with bogus and repeating requests. Added these IP addresses to my firewall. Le sigh.
If you've ever looked at SSH server logs you know what I'm about to say: Any SSH server connected to the public Internet is getting bombarded by constant attempts to log in. Not just a few of them. A lot of them. Sometimes even dozens per second. And this problem is not going away; it is, in fact, getting worse. And attackers' behavior is changing.
The graph attached to this post shows the number of attempted SSH logins per day to one of @cloudlab s clusters over a four-year period. It peaks at about 3.4 million login attempts per day.
This is part of a study we did on our production system, using logs of more than 640 million login attempts, covering more than 1,500 hosts on our side and observing more than 840 thousand incoming IP addresses.
A paper presenting our analysis and a new, highly effective means to block SSH brute force attacks ("Where The Wild Things Are: Brute-Force SSH Attacks In The Wild And How To Stop Them") will be presented next week at #NSDI24 by @sachindhke . The full paper is at https://www.flux.utah.edu/paper/singh-nsdi24
OK, so what can we do about all these SSH brute force attacks?
We have a plan - actually, not just a plan, we run this in production on one of the @cloudlab clusters.
Let's start with this observation: if attackers are using a broad set of usernames, then we can use these username sets as a sort of signature. About half of attacking IP addresses only try one username, but that also means that about half are trying more than one - in fact some individual IP addresses tried more than 10,000 usernames!
What we do is this: we find sets of usernames that are used by more than one attacking IP address (actually it's a bit more complicated that this, details in the paper). This gives us "dictionaries" of usernames that are only used by attackers, and not any of our real users. We collect these dictionaries from the logfiles of a bunch of SSH servers, and combine them to form a Username Block List (UBL).
Now, all we have to do is: as soon as we see an IP address try a username from this UBL, we block it. That simple. We call this Dictionary Based Blocking (DBB).
How well does this work? We used logs from our clusters containing a total of 213 million login attempts, and it blocked 99.5% of all attempts, generating a false positives (accidentally locking out a real user) at a rate of just one about every five days.
But what about #fail2ban, you might ask? That's another method people use to block attacks against SSH (and other services) by locking out addresses that fail to log in more than X times in Y minutes. Well, with it's default settings, it only blocks about 66% of attacks, and it generates more than 5x as many false positives (graph attached). As it turns out, there is no way to tune fail2ban to get DBB's accuracy without a false positive rate that's orders of magnitude higher.
I said we run this in production - how well does that work? We run it on one of of CloudLab clusters that already had a firewall - subscribing to popular blocklists and running something very much like fail2ban. It's catching four-fifths of the attacks that were not already getting caught by these measures, and so far it hasn't caused a single false positive.
savez vous comment configurer #fail2ban pour qu'il ne banisse pas les IP des utilisateurs définis dans /etc/passwd
Autrement dit : ne pas bannir le utilisateurs locaux.
Ou est-ce intrinsèque ?
Irgendwie wäre es noch toll wenn man das an alle möglichen Dienste z.B. #Mastodon anbinden könnte, so dass Account's die von gemeldeten IP's erstellt werden, manuell geprüft werden müssen.
Project #fail2ban implemented on my cloud server! I currently have 101 IP addresses on that ban list and I have the list set to expire in 6 months for any of the bots that try to log in as root, admin, or administrator. That'll keep 'em out.
la configuration est très versatile : la doc ne parle que d’iptables mais j’ai utilisé firewalld sans aucune difficulté, c’est vraiment très, très simple.
Tunneled #Jellyfin through my ISP's CGNAT, and did some #Traefik magic on the other end to give it a URL. Then sprinkled some #Fail2Ban in there for good measure.
If anyone else wonders how to use #fail2ban on #Debian 12 without #rsyslog logging:
(Since rsyslog is not installed anymore - journalctl provides that part now):
Create a jail.local file under /etc/fail2ban/ and make sure to add "backend = systemd" to make #fail2ban use journalctl.
Was wondering about how to make #Fail2Ban (a tool that peruses web server, SSH and other log files and adds offenders to firewall blocks) increase the block time for frequent offenders.
Found a hint that it's actually implemented, and found example settings in the jail.conf file as comments. Now I've got it doubling time (plus a randomiser) for repeated break-in attempts!
NOTICE [sshd] Ban 185.209.161.107
NOTICE [sshd] Increase Ban 185.209.161.107 (2 # 2h 13m -> 2023-10-26 17:29:42)
Just setup #fail2ban, seems I forgot to do that on my website server...
Status for the jail: sshd
|- Filter
| |- Currently failed: 2
| |- Total failed: 117
| - Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd - Actions
|- Currently banned: 49
|- Total banned: 49
People REALLY want to get in. Thank goodness for key-only based auth!
Disabled the default "Allow ping" rule in #OpenWRT on my router. Let's see if that reduces the amount of people that go poking around and get banned by #Fail2Ban.
DDoS attacks appear to have become increasingly more commonplace, and the other day the miscreants decided to target my Internet forum. I don't use CloudFlare or other third-party DDoS protection services since I try to avoid dependency on external services as much as possible. However, even an old CentOS7 server will have various tools available to protect against or mitigate such attacks. You just need to know how to use them! Here is the true story of how I fought and won!
@dbdemon Thanks for the blog - useful to know that nginx has some rate limiting capabilities (well, it is modular), and that they spit out log entries, and #Fail2Ban can trigger off that.
I'll probably study them frantically the next time there's a DDOS attack on my server. :-)