Zabbix logo

Getting more out your Windows Performance Counters monitoring for web applications – part 3

Jan Reilink Windows Server

This is part 3 about Zabbix monitoring for your websites and ASP.NET applications in IIS. This time I’ll show you how to get data from Win32_PerfRawData_PerfProc_Process counter, fast, for everey application pool This counter is notorious for its slowness, but you can get data a bit faster. IIS AppPool Insights in Zabbix – because there is always more than one way.

Table of Contents show

In part 1 I showed you how to create an IIS website autodiscovery item and application discovery, and in part 2 you monitor the performance of your ASP.NET web application utilizing Windows performance counters in Zabbix. Now we take it a step further, to monitor Win32_PerfRawData_perfProc_Process data for all currently running application pools (autodiscovered).

I query PerfRawData, because using Win32_PerfFormattedData_perfProc_Process costs a small performance hit, and results may be rounded to 0 and not 0,1.

Autodiscover running IIS application pools

In this part you’ll query Win32_PerfRawData_PerfProc_Process performance counters using WMI/CIM in PowerShell. The Win32_PerfRawData_PerfProc_Process class contains a lot of usefull properties for your IIS website application pools.

The gathered data needs to be transformed into a JSON blob so Zabbix can use it, but you also need to somehow match your application pool’s (w3wp.exe) process ID to the result of your WMI query (called “WQL”, or SQL for WMI. See Querying with WQL, WQL (SQL for WMI) for more information). This is done in PowerShell.

Let’s begin.

Create a file called w3wp-discovery.ps1 and add the following into it:

$allpools = (Get-CimInstance -EA SilentlyContinue -ClassName applicationpool -Namespace root\webadministration).Name

This outputs simple all applications pools. Now you’ll need to add a Zabbix MACRO {#APPPOOLNAME} object.

@{  
	"data" = $allpools | foreach { @{
		"{#APPPOOLNAME}" = $_
	}}
} | ConvertTo-JsonCode language: PHP (php)

This’ll give you an object $mypools with all discovered IIS application pools and a macro {#APPPOOLNAME}. It looks like:

{
	"data":  [
				 {
					 "{#APPPOOLNAME}":  "DefaultAppPool"
				 },
				 {
					 "{#APPPOOLNAME}":  ".NET v4.5 Classic"
				 },
				 {
					 "{#APPPOOLNAME}":  ".NET v4.5"
				 },
				 {
					 "{#APPPOOLNAME}":  "example.com"
				 },
				 {
					 "{#APPPOOLNAME}":  "example.org"
				 },
....Code language: JSON / JSON with Comments (json)

The macro {#APPPOOLNAME} is used to reference application pools in your autodiscover template. We need this information later on.

Zabbix conf

For Zabbix, create a configuration file w3wp.conf in your zabbix_agentd.confd folder, and add the following UserParameter to it:

UserParameter=w3wp.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File C:\path\to\w3wp-discovery.ps1

This creates the w3wp.discovery key.

Counter query Powershell script for dependent items

To query Win32_PerfRawData_perfProc_Process for performance counter values, we need PowerShell. You can use Get-WmiObject of Get-CimInstance cmdlets, and the data I want to monitor for application pools is (you are free to choose your own of course).

  • HandleCount
  • IDProcess
  • IOReadBytesPerSec
  • IOReadOperationsPerSec
  • IOWriteBytesPerSec
  • IOWriteOperationsPerSec
  • PercentProcessorTime
  • PrivateBytes
  • ThreadCount
  • WorkingSet
  • WorkingSetPeak

This is where the magic happens and you get all that info:

$processinfo = Get-CimInstance -EA SilentlyContinue -Query "select * from Win32_PerfRawData_PerfProc_Process where Name like 'w3wp_%'" -Namespace root\cimv2 | Select HandleCount,IDProcess,IOReadBytesPerSec,IOReadOperationsPerSec,IOWriteBytesPerSec,IOWriteOperationsPerSec,PercentProcessorTime,PrivateBytes,ThreadCount,WorkingSet,WorkingSetPeak
Code language: PHP (php)

The WQL queries Win32_PerfRawData_PerfProc_Process for all data, and then I use PowerShell Select to filter the output. Why? Because this is way faster than running a query per application pool like "select * from Win32_PerfRawData_PerfProc_Process where IDProcess='w3wp.exe PID'". Especially if you need to this 300 or 500 times on a server. WMI is really slow, so let PowerShell do the filtering of data.

Increase WMI memory to support large volume of queries

The output may look as follows:

HandleCount             : 2537
IDProcess               : 7652
IOReadBytesPerSec       : 123547753
IOReadOperationsPerSec  : 112674
IOWriteBytesPerSec      : 78788008
IOWriteOperationsPerSec : 88060
PercentProcessorTime    : 4461250000
PrivateBytes            : 278888448
ThreadCount             : 159
WorkingSet              : 283172864
WorkingSetPeak          : 288657408

HandleCount             : 3038
IDProcess               : 5156
IOReadBytesPerSec       : 27230185458
IOReadOperationsPerSec  : 3777431
IOWriteBytesPerSec      : 87141280
IOWriteOperationsPerSec : 89040
PercentProcessorTime    : 248751718750
PrivateBytes            : 1265516544
ThreadCount             : 183
WorkingSet              : 1267384320
WorkingSetPeak          : 1358626816

It doesn’t look like something you can work with in a Zabbix template, but fortunately you can use PowerShell to transform this in a nicely formatted JSON. This is why we need the IDProcess property; it get’s matched to an application pool’s .ProcessId property.

The following funcion get’s called in a ForEach loop to select the correct data for each ProcessId / IDProcess.

function get-WmiProcess($wpname) {
	Try {
		$procid = ($allprocids | Where-Object AppPoolName -eq "${wpname}").ProcessId
		$data = ($processinfo | Where-Object IDProcess -eq $procid)
		return @{
			IOReadOperationsPerSec = $data.IOReadOperationsPerSec
			IOWriteOperationsPerSec = $data.IOWriteOperationsPerSec
			IOReadBytesPerSec = $data.IOReadBytesPerSec
			WorkingSetPeak = $data.WorkingSetPeak
			HandleCount = $data.HandleCount
			ThreadCount = $data.ThreadCount
			IOWriteBytesPerSec = $data.IOWriteBytesPerSec
			PercentProcessorTime = $data.PercentProcessorTime
			WorkingSet = $data.WorkingSet
			WorkingSetPrivate = $data.WorkingSetPrivate
			PrivateBytes = $data.PrivateBytes
		}
	}
	Catch {
	}
}Code language: PHP (php)

And the ForEach loop to get the data and eventually output the JSON:

$processinfo = (Get-CimInstance -EA SilentlyContinue -ClassName Win32_PerfRawData_PerfProc_Process -Filter "Name like 'w3wp%'") | select HandleCount,IDProcess,IOReadBytesPerSec,IOReadOperationsPerSec,IOWriteBytesPerSec,IOWriteOperationsPerSec,PercentProcessorTime,PrivateBytes,ThreadCount,WorkingSet,WorkingSetPeak,WorkingSetPrivate
$procinfo = @{}
$allpools | foreach {
	$procinfo[$_] = get-WmiProcess $_
}
$procinfo | ConvertTo-Json
Code language: PHP (php)

When ran, you can expect the following output to use in your Zabbix template:

"example.com":  {
	"IOReadOperationsPerSec":  124196,
	"IOWriteOperationsPerSec":  121311,
	"IOReadBytesPerSec":  6556014,
	"WorkingSetPrivate":  109973504,
	"WorkingSetPeak":  187043840,
	"HandleCount":  1849,
	"ThreadCount":  107,
	"IOWriteBytesPerSec":  68971258,
	"PercentProcessorTime":  204531250,
	"WorkingSet":  180256768,
	"PrivateBytes":  151367680
},Code language: JavaScript (javascript)

Zabbix template for WMI Win32_PerfRawData_PerfProc_Process counters

In your autodiscover template, you can create an autodiscover item “Fetch”. But first you have to add to your w3wp.conf file the following UserParameter:

UserParameter=Fetch,powershell -NoProfile -ExecutionPolicy Bypass -File C:\path\to\zabbix\scripts\w3wp-wmi.ps1

Now you can use Fetch in your template:

			<items>
				<item>
					<name>NewFetcher</name>
					<key>Fetch</key>
					<history>0</history>
					<trends>0</trends>
					<value_type>TEXT</value_type>
				</item>
	...Code language: HTML, XML (xml)

And a discovery_rule:

<discovery_rules>
	<discovery_rule>
		<!-- Yes, I created this for Vevida.com
			rock solid Windows Server hosting for your websites and 
			applications
		-->
		<name>Vevida IIS AppPool Insights</name>
		<key>w3wp.pooldiscovery</key>
		<delay>1m</delay>
		<item_prototypes>
			<item_prototype>
				<name>appPool: {#APPPOOLNAME} IOReadOperationsPerSec</name>
				<type>DEPENDENT</type>
				<key>iis.apppool.IOReadOperationsPerSec[{#APPPOOLNAME}]</key>
				<delay>0</delay>
				<history>7d</history>
				<value_type>FLOAT</value_type>
				<description>The rate at which the process is writing bytes to I/O operations. This counter counts all I/O activity generated by the process to include file, network and device I/Os.</description>
				<application_prototypes>
					<application_prototype>
						<name>appPool: {#APPPOOLNAME}</name>
					</application_prototype>
				</application_prototypes>
				<preprocessing>
					<step>
						<type>JSONPATH</type>
						<params>$.IOReadOperationsPerSec</params>
					</step>
					<step>
						<type>CHANGE_PER_SECOND</type>
						<params/>
					</step>
				</preprocessing>
				<master_item>
					<key>newfetch[{#APPPOOLNAME}]</key>
				</master_item>
			</item_prototype>
...
			<item_prototype>
				<name>NewFetch for {#APPPOOLNAME}</name>
				<type>DEPENDENT</type>
				<key>newfetch[{#APPPOOLNAME}]</key>
				<delay>0</delay>
				<history>0</history>
				<trends>0</trends>
				<value_type>TEXT</value_type>
				<application_prototypes>
					<application_prototype>
						<name>appPool: {#APPPOOLNAME}</name>
					</application_prototype>
				</application_prototypes>
				<preprocessing>
					<step>
						<type>JSONPATH</type>
						<params>$['{#APPPOOLNAME}']</params>
					</step>
				</preprocessing>
				<master_item>
					<key>Fetch</key>
				</master_item>
			</item_prototype>
		</item_prototypes>
	</discovery_rule>
</discovery_rules>Code language: HTML, XML (xml)

Continue creating an item_prototype for your counter data, and you’re all set. Keep in mind that PercentProcessorTime is measured by the number of CPU seconds a process uses, per second. Meaning 1 is 100%. You can add a multiplier of 100, so 100% is 1 CPU core completely used. For readability, in my template I have two multiplier preprocessing steps: 0.0000001 and 100 (yes, one 0.00001 was also enough). And don’t forget to set CHANGE_PER_SECOND.

Eliminate and discard unsupported monitoring items

Now that you have your template for IIS application pool monitoring complete, what happens to an item if the script doesn’t return a value for -for example- WorkingSetPeak? The item becomes “unsupported”, and you don’t want that. Eliminate Zabbix unsupported items by adding an error_handler to your items as a preprocessing step:

<preprocessing>
	<step>
		<type>JSONPATH</type>
		<params>$.WorkingSetPeak</params>
		<error_handler>DISCARD_VALUE</error_handler>
	</step>
	<step>
		<type>DISCARD_UNCHANGED_HEARTBEAT</type>
		<params>10m</params>
	</step>
</preprocessing>
Code language: HTML, XML (xml)

The preprocessing documentation reads:

If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value or set a specified error message.

Are you looking for rock solid, eco-friendly, .NET hosting? Look no further! UmbHost offers powerful hosting services for websites and businesses of all sizes, and is powered by 100% renewable energy!

Related Posts

Share this post

Share on LinkedIn Share on Reddit

Jan Reilink

Hi, my name is Jan. I am not a hacker, coder, developer or guru. I am merely an application manager / systems administrator, doing my daily thing at Embrace - The Human Cloud. In the past I worked for clidn and Vevida. With over 20 years of experience, my specialties include Windows Server, IIS, Linux (CentOS, Debian), security, PHP, websites & optimization. I blog at https://www.saotn.org.