Skip to main content

How to filter web traffic with blacklists

Block and filter unwanted web HTTP traffic with blacklists, on both IIS and Apache. Protect your website easily with this PHP blacklist class. Let’s create our own HTTP web blacklist filter.

My need for an HTTP Blacklist and why you need one too

As a systems administrator for web hosting company Vevida, I deal with unwanted HTTP traffic on a daily basis. Customer websites are hacked, comment forms, blogs and forums are flooded with spam, and so on.

There are numerous ways of blocking unwanted visitors, like forum or comment spammers, spam or hack bots, from your web site. One of those methods is to filter traffic using existing blacklists. There are quite a few blacklists available, but for as far as I know most are intended for SMTP traffic, not HTTP.

And I needed a HTTP blacklist.

Two of such HTTP blacklists that do exist, are Project Honey Pot and StopForumSpam. A lot of “offending” IP addresses are listed in these blacklists, so why not use them? Both blacklists offer Application Programming Interfaces or API’s, to query their databases. Also, both have lists of written implementations, or plugins to use: here and here. Contributed by the community.

One thing I missed (or couldn’t find) is an implementation that utilizes both databases, and offers support for the creation of a local blacklist. This local blacklist can be used to reduce remote lookups and decreases the load on both external databases.

So I decided to -try to- create such an implementation.

The web blacklist filter can be used with either IIS IP Address and Domain Restrictions module, stored in a web.config file, or stored in a flat text file for use with .htaccess files. Both Helicon Ape and Apache mod_rewrite support a RewriteMap directive.

This makes the blacklist, code and usage cross platform!

As said, we’ll be using Project Honey Pot and StopForumSpam.

About Project Honey Pot

Project Honey Pot is the first and only distributed system for identifying spammers and the spambots they use to scrape addresses from your website. Using the Project Honey Pot system you can install addresses that are custom-tagged to the time and IP address of a visitor to your site. If one of these addresses begins receiving email we not only can tell that the messages are spam, but also the exact moment when the address was harvested and the IP address that gathered it.

Project Honey Pot was created by Unspam Technologies, Inc – an anti-spam company with the singular mission of helping design and enforce effective anti-spam laws.

Project Honey Pot’s Http:Bl has been implemented on a number of different web servers, content management systems, blogging platforms, and forums. These systems query the http:BL servers for visitors to your site and restrict their access if they are found to be malicious. See the available implementations for WordPress, Drupal, Typo3, ASP.NET / IIS, phpBB and so forth.

About StopForumSpam

We provide lists of spammers that persist in abusing forums and blogs with their scams, ripoffs, exploits and other annoyances. We provide these lists so that you don’t have to endure the never ending job of having to moderate, filter and delete their rubbish.

We provide a “free for use” site where you can check registrations and posts against our database. We list known forum and blog spammers, including IP and email addresses, usernames, how busy they are and, in some cases, evidence of their spam.

Note: Do not use StopForumSpam as a firewall to your website!

I thought you might find this interesting:   Time to disable SSLv3, or what?

How to: Create a local HTTP web blacklist in PHP

Use Project Honey Pot and Stop Forum Spam to block spammers on your website. Here we’ll create a local web blacklist of IP addresses we don’t want to hit our website. The code is PHP, but should be easily portable to ASP.NET (C#, VB.NET), classic ASP, Perl or whatever you want.

Since Apache’s mod_rewrite offers the same functionality as Helicon Ape, the usage of this blacklist is cross platform, for both Windows Server IIS and Linux Apache.

The logic behind all is as follows:

  1. a visitor visits your website
  2. The IP address of the visitor is looked up in the local blacklist.txt file and (if set up) web.config file
    • IP address found? Block the visitor and log the visit attempt
    • IP address not found? Look up the IP address in StopForumSpam and Project Honey Pot databases
  3. If the IP adress is found in either one of the databases:
    • write the IP address in the local blacklist.txt file and (if set up) web.config file
    • log the visit attempt.
    • block the visitor

When the blacklist.txt file is in use, there is nothing more you have to do. A visitor will receive a HTTP 403 Forbidden message automatically by IIS.

Let’s block us some visitors!

PHP web blacklist filter: <?php echo “code and documentation”; ?>

The code below is commented. The downloadable code is not. I’ve also created a readme documentation (PDF, opens in a new screen). Beware, I’m not a die hard PHP programmer, the code might be (is…) sloppy.

<?php
setlocale( LC_ALL, 'nld_nld' );

/*
 * ( c ) 2012 - Jan Reilink - jan@saotn.nl
 *  follow me on Twitter: @HertogJanR
 * 
 * Yep, I host my PHP sites on Windows Server IIS : )
 * @ Vevida ( https://vevida.com )
 * 
 * PHP class for quering IP adresses agains the Project Honey Pot and 
 * Stop Forum Spam databases.
 * 
 * - An Access Key is mandatory!
 * http://www.projecthoneypot.org/create_account.php
 * - Project Honey Pot appreciates a donation
 * http://www.projecthoneypot.org/donate.php
 * API information
 * http://www.projecthoneypot.org/httpbl_api.php
 * 
 * About Project Honey Pot ( PHP )
 * http://www.projecthoneypot.org/about_us.php
 * 
 * The same goes for Stop Forum Spam ( SFS )
 * http://www.stopforumspam.com/usage
 * http://www.stopforumspam.com/donate
 * http://www.stopforumspam.com/keys
 * 
 * 
 * Files used:
 * database\config.ini : with configuration settings
 * database\bl_log.txt : to log blocked requests
 * database\blacklist.txt : the blacklist with which Helicon Ape ( or
 *   mod_rewrite ) RewiteMap directive works
 * 
 * www\httpbl.class.php : the PHP class which does all the magic
 * www\web.config : IIS 7.0 / 7.5 configuration file for use with IIS 
 *   IP Address and Domain Restrictions. Needs to be writable
 * www\.htaccess : configuration file for IIS Helicon Ape or Apache 
 *   mod_rewrite. Needs to be writable
 * 
 * Example usage:
 * 
 * require_once( "httpbl.class.php" );
 * $mybl = new httpBL() ;
 * if( $mybl->_retrieve_IP_address_status_PHP( 
 *   $mybl->_retrieve_remote_IP_address()  ) )
 * {
 *      $output = "<html><head><title>Unauthorised access</title></head>";
 *      $output .= "<body><h1>Unauthorised access</h1><h2>Access from"; 
 *      $output .= "your IP address to this website is prohibited!</h2>";
 *      $output .= "<span>Contact the webmaster if you believe this";
 *      $output .= " is an error.</span></body></html>";
 *      header( "HTTP/1.0 403 Forbidden" );
 *      die( $output );
 *  }
 * 
 */

Class httpBL {
  // CHANGE THE PATH ON THE LINE BELOW!
  public $configfile = "/path/to/your/config.ini";
  public $ini_array;
  
  public function _read_config_file() {
    return $this->ini_array = parse_ini_file( $this->configfile, true );
  }

  public function _check_required_params() {
    $this->_read_config_file() ;
    
    if( ( $this->ini_array["PHPaccesskey"] != "" ) &&
      ( $this->ini_array["SFSaccesskey"] != "" ) &&
      ( $this->ini_array["blfileloc"] != "" ) ) {
      return TRUE;
    } else {
      trigger_error( "not all required variables are filled out." );
      exit;
    }
  }
  
  public function _retrieve_remote_IP_address() {
    $ip = ( isset( $_SERVER['HTTP_X_FORWARDED_FOR'] ) ? $_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR'] );
    return $ip;
  }

  /**
   * Only for public IP v4 address space
   * http://www.php.net/manual/en/filter.filters.validate.php
   **/
  public function validate_IP_address( $ip ) {
    if( filter_var( $ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4 | FILTER_FLAG_NO_PRIV_RANGE ) !== FALSE ) {
      // IPV4 === TRUE
      return TRUE;
    } else {
      return FALSE;
    }
  }

  /**
   * Note that the IP address being queried should be sent in the reversed 
   * octet format. In other words, "127.1.1.7" should become "7.1.1.127" 
   * for all DNS queries. For more detailed information, please see the 
   * http:BL API ( http://www.projecthoneypot.org/httpbl_api.php ).
   **/
  public function _reverse_octet_format( $ip )
  {
    // another, old, method: return preg_replace( "/( d{1,3} ).( d{1,3} ).( d{1,3} ).( d{1,3} )$/",'${4}.${3}.${2}.${1}',$ip );
    return implode( ".", array_reverse( explode( '.', $ip ) ) );
  }

  /**
   * Look up IP address using DNS to retrieve its status with
   * Project Honey Pot ( PHP )
   * We can also validate IP addresses against different blacklists:
   *   dns_get_record( $revip.".cbl.abuseat.org", DNS_A ); or
   *   zen.spamhaus.org, but that's beyond the scope of this Class
   **/
  public function _retrieve_IP_address_status_PHP( $ip ) {
    if( $this->_check_required_params()  == TRUE ) {
      if( !$this->validate_IP_address( $ip ) ) {
        //trigger_error( "not a valid IP address" );
        return FALSE;
      }
      if( !$this->check_is_current_listed( $ip,'webconfig' ) && ( !$this->check_is_current_listed( $ip,'htaccess' ) ) ) {
        $lookup = $this->ini_array["PHPaccesskey"] .".".implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".dnsbl.httpbl.org";
        $result = explode( ".", gethostbyname( $lookup ) );
        if( !empty( $result ) && ( $result["0"] == "127" ) ) {
          // minDayinBl ( 2 ) and minThreatLevel ( 5 )
          if( ( $result["1"] >= $this->ini_array["minDayinBl"] ) &&
            ( $result["2"] >= $this->ini_array["minThreatLevel"] ) ) {
            $this->save_positive_result( $ip );
            $this->save_any_listing_file( $ip );
            return TRUE;
          }
        }
      } else {
        $this->save_any_listing_file( $ip );
        return TRUE;
      }
    }
    else {
      trigger_error( "not all required variables are filled out." );
      exit;
    }
  }

  /**
   * function to retrieve the IP status from Stop Forum Spam
   * $ip : IP address in *normal* octet format - ( variable )
   * result is either: listed in Stop Forum Spam and now added to our 
   * own blacklist, or already listed in our own little blacklist
   **/
  public function _retrieve_IP_address_status_SFP( $ip ) {
    if( $this->_check_required_params()  == 1 ) {
      if( !$this->validate_IP_address( $ip ) ) {
        return FALSE;
      }
      if( !$this->check_is_current_listed( $ip,'webconfig' ) && ( !$this->check_is_current_listed( $ip,'htaccess' ) ) ) {
        if( $this->http_GET( $ip, 'serial' ) == TRUE ) {
          $this->save_positive_result( $ip );
          $this->save_any_listing_file( $ip );
          return TRUE;
        }
      } else {
        $this->save_positive_result( $ip );
        $this->save_any_listing_file( $ip );
        return TRUE;
      }
    } else {
      trigger_error( "not all required variables are filled out." );
      exit;
    }
  }
  
  /**
   * $url    : HTTP URL to Stop Forum Spam - ( fixed )
   * $ip    : IP address in *normal* octet format - ( variable )
   * $format  : serialized ( serialize()  or JSON ) - ( variable, 
   *         serialize()  is used as default )
   **/
  public function http_GET( $ip, $format='serial' ) {
    $ch = curl_init() ;
    $url = "http://www.stopforumspam.com/api?ip=".$ip."&f=".$format."";
    curl_setopt( $ch, CURLOPT_URL, $url );
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
    curl_setopt( $ch, CURLOPT_HEADER, 0 );
    $output = curl_exec( $ch );
    curl_close( $ch );
    $result = unserialize( $output );
    if( $result["ip"]["appears"] == 1 ) {
      return TRUE;
    }
  }
  
  /**
   * function to save a positive listing result in our own blacklist
   * $ip : IP address in *normal* octet format - ( variable )
   * 
   * Helicon Ape uses .htaccess, in combination with 
   * a blacklist.txt file ( RewriteMap ):
   * 
   * 108.59.10.145 -
   * 
   * More information:
   * http://www.saotn.org/htaccess-as-web-application-firewall-waf/
   * http://helicontech.blogspot.com/2009/02/isapirewrite-faq.html
   * 
   * Should also work for Apache's mod_rewrite RewriteMap
   * http://httpd.apache.org/docs/2.4/mod/mod_rewrite.html#rewritemap
   * http://httpd.apache.org/docs/2.4/rewrite/access.html#host-deny
   **/
  public function save_positive_result( $ip ) {
    if( file_exists( $this->ini_array["blfileloc"] ) &&
      is_writable( $this->ini_array["blfileloc"] ) ) {
      if( $this->check_is_current_listed( $ip, 'htaccess' ) != TRUE ) {
        $f = fopen( $this->ini_array["blfileloc"], "ab" );
        fwrite( $f, $ip ." -\r\n" );
        fclose( $f );
        clearstatcache() ;
      }
    } else {
      trigger_error( $this->ini_array["blfileloc"] ." not writable!" );
    }

    if( file_exists( $this->ini_array["webconfigFile"] ) &&
      is_writable( $this->ini_array["webconfigFile"] ) ) {
      if( !$this->check_is_current_listed( $ip, 'webconfig' ) ) {
        $this->_save_web_dot_config_positive_result( $ip );
      }
    } else {
      trigger_error( $this->ini_array["webconfigFile"] ." not writable!" );
    }
  }
  
  /**
   * function to see if it is already listed
   * $ip : IP address in *normal* octet format - ( variable )
   * result is either TRUE ( listed in own blacklist ) or FALSE ( not listed )
   **/
  public function check_is_current_listed( $ip, $file='' ) {
    if( $file == '' || $file == 'htaccess' ) {
      if( file_exists( $this->ini_array["blfileloc"] ) ) {
        $f = file_get_contents( $this->ini_array["blfileloc"] );
        if( strstr( $f, $ip ." -" ) != FALSE ) {
          return TRUE;
        }
      }
    }

    if( $file == 'webconfig' ) {
      if( ( $this->ini_array["use_webconfigFile"] == "1" ) && ( file_exists( $this->ini_array["webconfigFile"] ) ) ) {
        $f = file_get_contents( $this->ini_array["webconfigFile"] );
        $sstring = "ipAddress="".$ip."" allowed="false"";
        if( strstr( $f,$sstring ) == TRUE ) {
          return TRUE;
        }
      }
    }
  }

  /**
   * saves any hit of blocked IP addresses to a log file
   **/
  public function save_any_listing_file( $ip ) {
    if( file_exists( $this->ini_array["bllogfileloc"] ) &&
      is_writable( $this->ini_array["bllogfileloc"] ) ) {
      $f = fopen( $this->ini_array["bllogfileloc"],"ab" );
      fwrite( $f, date( "Y.m.d.G:i" ) ." - " . $ip ."\r\n" );
      fclose( $f );
      clearstatcache() ;
    } else {
      trigger_error( "could not find logfile "
        .$this->ini_array["bllogfileloc"] );
    }
  }

  /**
   * IIS 7 / 7.5 IP Address and Domain Restrictions format
   * uses web.config for configuration, see my WordPress web.config
   * for more information: https://www.saotn.org/my-wordpress-web-config/
   * 
   * <system.webServer>
   *   <security>
   *     <ipSecurity>
   *       <add ipAddress="aa.bbb.ccc.dd" allowed="false" />
   *     </ipSecurity>
   *   </security>
   * </system.webServer>
   **/
  public function _save_web_dot_config_positive_result( $ip ) {
    $formatxml = PHP_EOL;
    $formatxml .= "        <add ipAddress="".$ip."" allowed="false" />";

    $doc = new DOMDocument() ;
    if( $doc->load( $this->ini_array["webconfigFile"] ) === false  ) {
      return false;
    }
    $xpath = new DOMXPath( $doc );
    $iprestrictions_nodes = $xpath->query( '/configuration/system.webServer/security/ipSecurity[starts-with( @add,'ipAddress' )]' );
    if( $iprestrictions_nodes->length > 0 ) {
      return true;
    }

    $xmlnodes = $xpath->query( '/configuration/system.webServer/security/ipSecurity' );
    if ( $xmlnodes->length > 0 ) {
      $ipsecurity_node = $xmlnodes->item( 0 );
    } else {
      $ipsecurity_node = $doc->createElement( 'ipSecurity' );
    
      $xmlnodes = $xpath->query( '/configuration/system.webServer/security' );
      if( $xmlnodes->length > 0 ) {
        $security_node = $xmlnodes->item( 0 );
        $security_node->appendChild( $ipsecurity_node );
      } else {
        $security_node = $doc->createElement( 'security' );
        $security_node->appendChild( $ipsecurity_node );
        
        $xmlnodes = $xpath->query( '/configuration/system.webServer' );
        if( $xmlnodes->length > 0 )
        {
          $system_webServer_node = $xmlnodes->item( 0 );
          $system_webServer_node->appendChild( $security_node );
        } else {
          $system_webServer_node = $doc->createElement( 'system.webServer' );
          $system_webServer_node->appendChild( $security_node );

          $xmlnodes = $xpath->query( '/configuration' );
          if ( $xmlnodes->length > 0 ) {
            $config_node = $xmlnodes->item( 0 );
            $config_node->appendChild( $system_webServer_node );
          } else {
            $config_node = $doc->createElement( 'configuration' );
            $doc->appendChild( $config_node );
            $config_node->appendChild( $system_webServer_node );
          }
        }
      }
    }

    $rule_fragment = $doc->createDocumentFragment() ;
    $rule_fragment->appendXML( $formatxml );
    $ipsecurity_node->appendChild( $rule_fragment );

    $doc->encoding = "UTF-8";
    $doc->formatOutput = true;
    $this->saveDomDocument( $doc, $this->ini_array["webconfigFile"] );

    return true;
  }

  function saveDomDocument( $doc, $filename ) {
    $config = $doc->saveXML() ;
    $config = preg_replace( "/( [^r] )\n/", "$1\r\n", $config );
    $fp = fopen( $filename, 'wb' );
    fwrite( $fp, $config );
    fclose( $fp );
  }
}
?>

Note: The code is sloppy, might be faulty, is provided AS-IS, is probably not secured and should not be used production without testing! But it does work. Use at your own risk.

Download full source package, with all files and directory structure here. Text version of the class is here.

I hope you find this PHP class helpful in protecting your website from unwanted visitors.

Frequently Asked Questions

I do not give support on how to use this class. The code is highly documented with comments and tested in production. Because I am not a PHP programmer, there is much to improve in code. Please do so.

If you have an urgent question, please post it as a comment to this post. I’ll try to answer it, and visitors are also free to respond of course.

PHP function to check an IP address against arbitrary blacklists

You can easily shorten the code above to check an IP address against one or two arbitrary blacklists, for example dnsbl.sorbs.net and b.barracudacentra.org. If the IP is listed in either one; deny website access.

Here’s an example class, you’ll notice some of the function names from above.

<?php
Class httpBL {
	public function _retrieve_remote_IP_address()  {
		$ip = ( isset( $_SERVER['HTTP_X_FORWARDED_FOR'] ) ? 
			$_SERVER['HTTP_X_FORWARDED_FOR'] : $_SERVER['REMOTE_ADDR'] );
		return $ip;
	}

	public function validate_IP_address( $ip ) {
		if( filter_var( $ip, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4 | FILTER_FLAG_NO_PRIV_RANGE ) !== false ) {
			return true;
		} else {
			return false;
		}
	}

	public function _retrieve_IP_address_status_PHP( $ip ) {
		if( !$this->validate_IP_address( $ip ) ) {
			return false;
		} else {
			$lookup1 = implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".dnsbl.sorbs.net";
			$lookup2 = implode( ".", array_reverse( explode ( ".", $ip ) ) ) .".b.barracudacentral.org";
			$result1 = explode( ".", gethostbyname( $lookup1 ) );
			$result2 = explode( ".", gethostbyname( $lookup2 ) );
			if( !empty( $result1 ) && ( $result1["0"] === "127" ) ) {
				error_log( $ip ." is listed in dnsbl.sorbs.net" );
				return true;
			}
			if( !empty( $result2 ) && ( $result2["0"] === "127" ) ) {
				error_log( $ip ." is listed in b.barracudacentral.org" );
				return true;
			}
		}
	}
}

$mybl = new httpBL();
if( $mybl->_retrieve_IP_address_status_PHP( $mybl->_retrieve_remote_IP_address() ) ) {
	$output = "<html><head><title>Unauthorised access</title></head>";
	$output .= "<body><h1>Unauthorised access</h1><h2>Access from"; 
	$output .= "your IP address to this website is prohibited!</h2>";
	$output .= "<span>Your IP address ". $mybl->_retrieve_remote_IP_address() ." is listen in either dnsbl.sorbs.net, zen.spamhaus.org or b.barracudacentral.org. Contact the webmaster if you believe this";
	$output .= " is an error. This is only a temporary test.</span></body></html>";
	header( "HTTP/1.0 403.6 Forbidden" );
	die( $output );
}
?>

Jan Reilink

My name is Jan. I am not a hacker, coder, developer, programmer or guru. I am merely a system administrator, doing my daily thing at Vevida in the Netherlands. With over 15 years of experience, my specialties include Windows Server, IIS, Linux (CentOS, Debian), security, PHP, websites & optimization.

One thought to “How to filter web traffic with blacklists”

Hi! Join the discussion, leave a reply!