oldbrad wrote on Sep 13^th, 2013 at 11:28am:

As you have found out blocking IP with the .htaccess is not always successful. Personally I feel a blocked IP is only good for about 3 months after that its a good chance a legit person might be using it.

Using .htaccess to prevent bots crawling your site helps reduce bandwidth, you just need to remember not all bots are bad so you need to be careful which ones you block.

Derek suggests StopForumSpam, I use it and I fully recommend it. Another great one is Spamfruits. Between these two mods all spammer registrations are blocked at the forum level. Sadly they don't reduce bandwidth.

IMO gnn's anti-spam is only useful if you allow guests posting.

Derek Barnstorm wrote on Sep 13^th, 2013 at 12:30pm:

The problem with blocks at the .htaccess level, is just that it' s at that level. Bandwidth is still being consumed.

I'm already doing iptables blocking. This works at the TCP level. Once blocked the server no longer exists to the requester.

There is no more effective mechanism!

The point of the post is that ultimately, in my case within 6 months, iptables blocking will effectively result in a denial of service.

In the event that the spam bot attempts increase at the apparent current rate, then sooner or later any kind of blocking has to consume all server resources! Given that iptables blocks happen at the lowest possible level. Doing anything at a higher level, logging, modifying .htaccess or whatever after the request reaches the server must amount to a denial of service eventually.

My calculations show iptables blocks WILL FAIL in 6 months. I've no idea how long it takes to generate an .htaccess files that become so big that the server spends all its time scanning to determine which IPs to block?

What we need is to slow down the number of fake requests. By definition, that cannot be accomplished from inside the firewall.

The immediate solution then appears to be reject all HTTP at the incoming router. Game over, sysadmins nil spam bots won

Hello Admins,

The ever increasing volume of automated registration attempts has finally forced me to disable on-line guest registration. ie 'Select the registration scheme for this forum' set to 'Only admin can register new members'.

This works fine for our low volume forum, but: In a attempt to kill the resulting torrent of error messages I have installed and configured fail2ban. This is a log analyser which is configurable to make iptables block entries for the offending IPs for some specified time before unblocking.

Initially I effected a 1 week block. With a net result of NO decrease in the number of attempts! I now permanently block the offending IPs with the result NO DECREASE in the number of attempts! This means the bots, or whatever, are either increasing in number exponentially or switching IP address.

Today I'm seeing of the order of 150 IP addresses per day being permanently blocked! While writing this post I have 10 more.

The received wisdom is that when the number of iptables rules exceeds, of the order of, 25000 records iptables will be destabilised.

If something doesn't change, a simple calculation says that in under 6 months I will effectively have a 'denial of service' situation!

I can of-course disable fail2ban and just ignore "Error: The registration feature has been disabled on this forum." and/or turn off error logging, even disable apache logging. (Not a good idea.)

The point is that if permanent IP blocks at the firewall are NOT effective then NOTHING is. Sooner or later the volume of automated hits results in  DENIAL OF SERVICE.

Therefore, something else has to be done to stop, or at least reduce, the source of these attempts.

Ideas?