How to filter web traffic with blocklists

Photo of author
Written By Jan Reilink

Windows Server system administrator & enthusiast.

Block and filter unwanted web HTTP traffic with blocklists, on both IIS and Apache. Protect your website easily with this PHP blocklist class. Let’s create our own HTTP web blocklist filter.

Update: renamed “blacklist” to “blocklist” as much as possible without changing code.

My need for an HTTP blocklist and why you need one too

As a systems administrator for web hosting company Vevida, I deal with unwanted HTTP traffic on a daily basis. Customer websites are hacked, comment forms, blogs and forums are flooded with spam, and so on.

There are numerous ways of blocking unwanted visitors, like forum or comment spammers, spam or hack bots, from your web site. One of those methods is to filter traffic using existing blocklists. There are quite a few blocklists available, but for as far as I know most are intended for SMTP traffic, not HTTP.

And I needed a HTTP blocklist.

Two of such HTTP blocklists that do exist, are Project Honey Pot and StopForumSpam. A lot of “offending” IP addresses are listed in these blocklists, so why not use them? Both blocklists offer Application Programming Interfaces or API’s, to query their databases. Also, both have lists of written implementations, or plugins to use: here and here. Contributed by the community.

One thing I missed (or couldn’t find) is an implementation that utilizes both blocklist databases, and offers support for the creation of a local blocklist. This local blocklist can be used to reduce remote look-ups and decreases the load on both external databases.

So I decided to-try to create such an implementation.

The web blocklist filter can be used with either Windows Server IIS IP Address and Domain Restrictions module, stored in a web.config file, or stored in a flat text file for use with .htaccess files. Both Helicon Ape and Apache mod_rewrite support a RewriteMap directive.

This makes the blocklist, code and usage cross platform!

As said, we’ll be using Project Honey Pot and StopForumSpam.

About Project Honey Pot

Project Honey Pot is the first and only distributed system for identifying spammers and the spambots they use to scrape addresses from your website. Using the Project Honey Pot system you can install addresses that are custom-tagged to the time and IP address of a visitor to your site. If one of these addresses begins receiving email we not only can tell that the messages are spam, but also the exact moment when the address was harvested and the IP address that gathered it.

Project Honey Pot was created by Unspam Technologies, Inc – an anti-spam company with the singular mission of helping design and enforce effective anti-spam laws.

Project Honey Pot’s Http:Bl has been implemented on a number of different web servers, content management systems, blogging platforms, and forums. These systems query the http:BL servers for visitors to your site and restrict their access if they are found to be malicious. See the available implementations for WordPress, Drupal, Typo3, ASP.NET / IIS, phpBB and so forth.

About StopForumSpam

We provide lists of spammers that persist in abusing forums and blogs with their scams, ripoffs, exploits and other annoyances. We provide these lists so that you don’t have to endure the never ending job of having to moderate, filter and delete their rubbish.

We provide a “free for use” site where you can check registrations and posts against our database. We list known forum and blog spammers, including IP and email addresses, usernames, how busy they are and, in some cases, evidence of their spam.

Do not use StopForumSpam as a firewall to your website!

How to: Create a local HTTP web blocklist in PHP

Use Project Honey Pot and Stop Forum Spam to block spammers on your website. Here we’ll create a local web blocklist of IP addresses we don’t want to hit our website. The code is PHP, but you can port it to ASP.NET (C#, VB.NET), classic ASP, Perl or whatever you want.

Since Apache’s mod_rewrite offers the same functionality as Helicon Ape, the usage of this blocklist is cross platform, for both Windows Server IIS and Linux Apache.

The logic behind all is as follows:

  1. a visitor visits your website
  2. The IP address of the visitor is looked up in the local blacklist.txt file and (if set up) web.config file
    • IP address found? Block the visitor and log the visit attempt
    • IP address not found? Look up the IP address in StopForumSpam and Project Honey Pot databases
  3. If the IP adress is found in either one of the databases:
    • write the IP address in the local blacklist.txt file and (if set up) web.config file
    • log the visit attempt.
    • block the visitor

When the blacklist.txt file is in use, there is nothing more you have to do. A visitor will receive a HTTP 403 Forbidden message automatically by IIS.

Let’s block us some visitors!

PHP web blocklist filter: <?php echo “code and documentation”; ?>

The code below is commented. The downloadable code is not. I’ve also created a readme documentation (PDF, opens in a new screen). Beware, I’m not a die hard PHP programmer, the code might be (is…) sloppy.

<?php setlocale( LC_ALL, 'nld_nld' ); Class httpBL { public $configfile = "/path/to/your/config.ini"; public $ini_array; public function _read_config_file() { return $this->ini_array = parse_ini_file( $this->configfile, true ); } public function _check_required_params() { $this->_read_config_file() ; if( ( $this->ini_array["PHPaccesskey"] != "" ) && ( $this->ini_array["SFSaccesskey"] != "" ) && ( $this->ini_array["blfileloc"] != "" ) ) { return TRUE; } else { trigger_error( "not all required variables are filled out." ); exit; } } public function _retrieve_remote_IP_address() { $ip = ( isset( $_SERVER['HTTP_X_FORWARDED_FOR'] ) ? $_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR'] ); return $ip; } public function validate_IP_address( $ip ) { if( filter_var( $ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4 | FILTER_FLAG_NO_PRIV_RANGE ) !== FALSE ) { return TRUE; } else { return FALSE; } } public function _reverse_octet_format( $ip ) { // another, old, method: return preg_replace( "/( d{1,3} ).( d{1,3} ).( d{1,3} ).( d{1,3} )$/",'${4}.${3}.${2}.${1}',$ip ); return implode( ".", array_reverse( explode( '.', $ip ) ) ); } public function _retrieve_IP_address_status_PHP( $ip ) { if( $this->_check_required_params() == TRUE ) { if( !$this->validate_IP_address( $ip ) ) { //trigger_error( "not a valid IP address" ); return FALSE; } if( !$this->check_is_current_listed( $ip,'webconfig' ) && ( !$this->check_is_current_listed( $ip,'htaccess' ) ) ) { $lookup = $this->ini_array["PHPaccesskey"] .".".implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".dnsbl.httpbl.org"; $result = explode( ".", gethostbyname( $lookup ) ); if( !empty( $result ) && ( $result["0"] == "127" ) ) { if( ( $result["1"] >= $this->ini_array["minDayinBl"] ) && ( $result["2"] >= $this->ini_array["minThreatLevel"] ) ) { $this->save_positive_result( $ip ); $this->save_any_listing_file( $ip ); return TRUE; } } } else { $this->save_any_listing_file( $ip ); return TRUE; } } else { trigger_error( "not all required variables are filled out." ); exit; } } public function _retrieve_IP_address_status_SFP( $ip ) { if( $this->_check_required_params() == 1 ) { if( !$this->validate_IP_address( $ip ) ) { return FALSE; } if( !$this->check_is_current_listed( $ip,'webconfig' ) && ( !$this->check_is_current_listed( $ip,'htaccess' ) ) ) { if( $this->http_GET( $ip, 'serial' ) == TRUE ) { $this->save_positive_result( $ip ); $this->save_any_listing_file( $ip ); return TRUE; } } else { $this->save_positive_result( $ip ); $this->save_any_listing_file( $ip ); return TRUE; } } else { trigger_error( "not all required variables are filled out." ); exit; } } public function http_GET( $ip, $format='serial' ) { $ch = curl_init() ; $url = "http://www.stopforumspam.com/api?ip=".$ip."&f=".$format.""; curl_setopt( $ch, CURLOPT_URL, $url ); curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 ); curl_setopt( $ch, CURLOPT_HEADER, 0 ); $output = curl_exec( $ch ); curl_close( $ch ); $result = unserialize( $output ); if( $result["ip"]["appears"] == 1 ) { return TRUE; } } public function save_positive_result( $ip ) { if( file_exists( $this->ini_array["blfileloc"] ) && is_writable( $this->ini_array["blfileloc"] ) ) { if( $this->check_is_current_listed( $ip, 'htaccess' ) != TRUE ) { $f = fopen( $this->ini_array["blfileloc"], "ab" ); fwrite( $f, $ip ." -\r\n" ); fclose( $f ); clearstatcache() ; } } else { trigger_error( $this->ini_array["blfileloc"] ." not writable!" ); } if( file_exists( $this->ini_array["webconfigFile"] ) && is_writable( $this->ini_array["webconfigFile"] ) ) { if( !$this->check_is_current_listed( $ip, 'webconfig' ) ) { $this->_save_web_dot_config_positive_result( $ip ); } } else { trigger_error( $this->ini_array["webconfigFile"] ." not writable!" ); } } public function check_is_current_listed( $ip, $file='' ) { if( $file == '' || $file == 'htaccess' ) { if( file_exists( $this->ini_array["blfileloc"] ) ) { $f = file_get_contents( $this->ini_array["blfileloc"] ); if( strstr( $f, $ip ." -" ) != FALSE ) { return TRUE; } } } if( $file == 'webconfig' ) { if( ( $this->ini_array["use_webconfigFile"] == "1" ) && ( file_exists( $this->ini_array["webconfigFile"] ) ) ) { $f = file_get_contents( $this->ini_array["webconfigFile"] ); $sstring = "ipAddress="".$ip."" allowed="false""; if( strstr( $f,$sstring ) == TRUE ) { return TRUE; } } } } public function save_any_listing_file( $ip ) { if( file_exists( $this->ini_array["bllogfileloc"] ) && is_writable( $this->ini_array["bllogfileloc"] ) ) { $f = fopen( $this->ini_array["bllogfileloc"],"ab" ); fwrite( $f, date( "Y.m.d.G:i" ) ." - " . $ip ."\r\n" ); fclose( $f ); clearstatcache() ; } else { trigger_error( "could not find logfile " .$this->ini_array["bllogfileloc"] ); } } public function _save_web_dot_config_positive_result( $ip ) { $formatxml = PHP_EOL; $formatxml .= " <add ipAddress="".$ip."" allowed="false" />"; $doc = new DOMDocument() ; if( $doc->load( $this->ini_array["webconfigFile"] ) === false ) { return false; } $xpath = new DOMXPath( $doc ); $iprestrictions_nodes = $xpath->query( '/configuration/system.webServer/security/ipSecurity[starts-with( @add,'ipAddress' )]' ); if( $iprestrictions_nodes->length > 0 ) { return true; } $xmlnodes = $xpath->query( '/configuration/system.webServer/security/ipSecurity' ); if ( $xmlnodes->length > 0 ) { $ipsecurity_node = $xmlnodes->item( 0 ); } else { $ipsecurity_node = $doc->createElement( 'ipSecurity' ); $xmlnodes = $xpath->query( '/configuration/system.webServer/security' ); if( $xmlnodes->length > 0 ) { $security_node = $xmlnodes->item( 0 ); $security_node->appendChild( $ipsecurity_node ); } else { $security_node = $doc->createElement( 'security' ); $security_node->appendChild( $ipsecurity_node ); $xmlnodes = $xpath->query( '/configuration/system.webServer' ); if( $xmlnodes->length > 0 ) { $system_webServer_node = $xmlnodes->item( 0 ); $system_webServer_node->appendChild( $security_node ); } else { $system_webServer_node = $doc->createElement( 'system.webServer' ); $system_webServer_node->appendChild( $security_node ); $xmlnodes = $xpath->query( '/configuration' ); if ( $xmlnodes->length > 0 ) { $config_node = $xmlnodes->item( 0 ); $config_node->appendChild( $system_webServer_node ); } else { $config_node = $doc->createElement( 'configuration' ); $doc->appendChild( $config_node ); $config_node->appendChild( $system_webServer_node ); } } } } $rule_fragment = $doc->createDocumentFragment() ; $rule_fragment->appendXML( $formatxml ); $ipsecurity_node->appendChild( $rule_fragment ); $doc->encoding = "UTF-8"; $doc->formatOutput = true; $this->saveDomDocument( $doc, $this->ini_array["webconfigFile"] ); return true; } function saveDomDocument( $doc, $filename ) { $config = $doc->saveXML() ; $config = preg_replace( "/( [^r] )\n/", "$1\r\n", $config ); $fp = fopen( $filename, 'wb' ); fwrite( $fp, $config ); fclose( $fp ); } } ?>
Code language: PHP (php)

Download full source package, with all files and directory structure here. Text version of the class is here.

The code is sloppy, might be faulty, is provided AS-IS, is probably not secured and should not be used in production without testing! But it does work. Use at your own risk.

I hope you find this PHP class helpful in protecting your website from unwanted visitors. If you do, then please donate me a beer or cup of coffee!

Frequently Asked Questions

I do not give support on how to use this class. The code is highly documented with comments and tested in production. Because I am not a PHP programmer, there is much to improve in code. Please do so.

If you have an urgent question, please post it as a comment to this post. I’ll try to answer it, and visitors are also free to respond of course.

Check an IP address against arbitrary blocklists, using PHP – Bonus

You can easily shorten the code above to use PHP to check an IP address against one or two arbitrary blocklists, for example dnsbl.sorbs.net and b.barracudacentra.org, or zen.spamhaus.org.

If the IP is listed in either: deny website access. Here’s an example class, you’ll recognize some of the function names from above.

<?php Class httpBL { public function _retrieve_remote_IP_address() { $ip = ( isset( $_SERVER['HTTP_X_FORWARDED_FOR'] ) ? $_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR'] ); return $ip; } public function validate_IP_address( $ip ) { if( filter_var( $ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4 | FILTER_FLAG_NO_PRIV_RANGE ) !== false ) { return true; } else { return false; } } public function _retrieve_IP_address_status_PHP( $ip ) { if( !$this->validate_IP_address( $ip ) ) { return false; } else { $lookup1 = implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".dnsbl.sorbs.net"; $lookup2 = implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".b.barracudacentral.org"; $result1 = explode( ".", gethostbyname( $lookup1 ) ); $result2 = explode( ".", gethostbyname( $lookup2 ) ); if( !empty( $result1 ) && ( $result1["0"] === "127" ) ) { error_log( $ip ." is listed in dnsbl.sorbs.net" ); return true; } if( !empty( $result2 ) && ( $result2["0"] === "127" ) ) { error_log( $ip ." is listed in b.barracudacentral.org" ); return true; } } } } $mybl = new httpBL(); if( $mybl->_retrieve_IP_address_status_PHP( $mybl->_retrieve_remote_IP_address() ) ) { $output = "<html><head><title>Unauthorised access</title></head>"; $output .= "<body><h1>Unauthorised access</h1><h2>Access from"; $output .= "your IP address to this website is prohibited!</h2>"; $output .= "<span>Your IP address ". $mybl->_retrieve_remote_IP_address() ." is listen in either dnsbl.sorbs.net, zen.spamhaus.org or b.barracudacentral.org. Contact the webmaster if you believe this"; $output .= " is an error. This is only a temporary test.</span></body></html>"; header( "HTTP/1.0 403.6 Forbidden" ); die( $output ); } ?>
Code language: PHP (php)

Hope this helps! :)


Did you like: How to filter web traffic with blocklists

Then please, take a second to support Sysadmins of the North and donate!

Your generosity helps pay for the ongoing costs associated with running this website like coffee, hosting services, library mirrors, domain renewals, time for article research, and coffee, just to name a few.



3 thoughts on “How to filter web traffic with blocklists”

    • Hi Alexander, thank you for your comment and my apologies! I moved my blog from a case-insensitive web server (Windows Server IIS) to a case-sensitive one (Linux/Apache).

      I’ve updated the download link accordingly.

      Reply

Hi! Join the discussion, leave a reply!

%d bloggers like this: