Monitor .NET CLR Garbage Collected heap from your web application

Are you worried about your .NET webapp running out of memory? In order to let your .NET (web) application run smooth over a longer period of time, it is important to monitor the .NET CLR Garbage Collector (GC) and collection. The what? The .NET Common Language Runtime Garbage Collector. Here is how you can monitor this in a Zabbix template using Powershell WMI / CIM and Win32_PerfRawData_NETFramework_NETCLRMemory Windows Performance Counters.

Microsoft explains the fundamentals of garbage collection, memory allocation and release. As either a .NET developer or systems administrator, we’re all too familiar with – as I call it – memory pollution. Suddenly your .NET web application throws an “OutOfMemoryException”, and you have no idea why. Remember The Dangers of the Large Object Heap? Monitor your .NET web application’s memory usage in Zabbix to understand where memory goes.

Getting more out your Windows Performance Counters monitoring for web applications – part 3

Here is how to create Zabbix monitoring for Win32_PerfRawData_NETFramework_NETCLRMemory in your template. For example, monitor the “Large Object Heap Size” performance counter to gain insight in how many bytes are allocated to the large object heap. The same for Gen0PromotedBytesPerSec and Gen1PromotedBytesPerSec performance counters. These counters display the bytes per second that are promoted from generation 0 (youngest) to generation 1, and generation 1 to generation 2 (oldest). Memory is promoted when it survives a garbage collection. You can collect all this information using WMI and PowerShell.

Many of the steps to follow are the same as they are in part 3 of this series, see the link above. Therefore I’ll provide you with the relevants to query Windows Win32_PerfRawData_NETFramework_NETCLRMemory Performance Counters and you can repeat the earlier steps.

Get all application pools by querying WMI root\webadministration namespace

This queries Windows Management Instrumentation (WMI) root\webadministration namespace for all application pools and saves into an object $allpools.

$allpools = (
  Get-CimInstance
    -EA SilentlyContinue
    -ClassName applicationpool
    -Namespace root\webadministration
).NameCode language: PowerShell (powershell)

You do need to put this in one line.

Query WMI for all application pool’s process id’s

You also need to have the process id belonging to an application pool, query the same root\webadministration namespace:

$allprocids = (
  Get-CimInstance
    -EA SilentlyContinue
    -ClassName WorkerProcess
    -Namespace root\webadministration
) | Select AppPoolName,ProcessId
Code language: PowerShell (powershell)

Process queried Win32_PerfRawData_NETFramework_NETCLRMemory properties

This is (part 1 of) where the real magic happens. I’ll post the code for you first:

function get-WmiNetClrMem($wpname) {
	Try {
		$procid = ($allprocids | Where-Object AppPoolName -eq "${wpname}").ProcessId
 		$data = ($netclrmemory | Where-Object ProcessID -eq $procid)
		$hashtable = @{
			AllocatedBytesPersec = $data.AllocatedBytesPersec
			Gen0heapsize = $data.Gen0heapsize
			Gen0PromotedBytesPerSec = $data.Gen0PromotedBytesPerSec
			Gen1heapsize = $data.Gen1heapsize
			Gen1PromotedBytesPerSec = $data.Gen1PromotedBytesPerSec
			Gen2heapsize = $data.Gen2heapsize
			LargeObjectHeapsize = $data.LargeObjectHeapsize
			NumberTotalcommittedBytes = $data.NumberTotalcommittedBytes
			NumberTotalreservedBytes = $data.NumberTotalreservedBytes
			PercentTimeinGC = $data.PercentTimeinGC
		}
		@($hashtable.keys) | % { 
 		   if (-not $hashtable[$_]) { $hashtable.Remove($_) } 
		}
		return $hashtable
	}
	Catch {
	}
}Code language: PowerShell (powershell)

A lot is going on here:

• The ProcessId property for the application pool currently queried is selected from all Process ID’s:
  • $procid = ($allprocids | Where-Object AppPoolName -eq "${wpname}").ProcessId
• That $procid property then is used to match a ProcessID in the object $netclrmemory (we’ll come to that one later), and is saved in object $data:
  • $data = ($netclrmemory | Where-Object ProcessID -eq $procid)
• All WMI properties we want are filtered out the $data object and saved into a hashtable
• The hashtable is stripped from empty values; ConvertTo-Json returns the JSON representation of null when the input object is $null. Zabbix then sees this as an string, which fails when you’re expecting an integer.
• The hashtable is returned, having valid values for
  • AllocatedBytesPersec
  • Gen0heapsize
  • Gen0PromotedBytesPerSec
  • Gen1heapsize
  • Gen1PromotedBytesPerSec
  • Gen2heapsize
  • LargeObjectHeapsize
  • NumberTotalcommittedBytes
  • NumberTotalreservedBytes
  • PercentTimeinGC

Or less if a property didn’t have a value.

Now for part 2 of the magic.

Query WMI for all required Win32_PerfRawData_NETFramework_NETCLRMemory properties

This is where you actually query Windows Management Instrumentation (WMI) / CIM Win32_PerfRawData_NetFramework_NETCLRMemory for properties you want to monitor in Zabbix. Be creative.

$netclrmemory = (Get-CimInstance -EA SilentlyContinue -ClassName Win32_PerfRawData_NETFramework_NETCLRMemory -Filter "Name like 'w3wp%'") | select AllocatedBytesPersec,Gen0heapsize,Gen1heapsize,Gen2heapsize,LargeObjectHeapsize,NumberTotalcommittedBytes,NumberTotalreservedBytes,PercentTimeinGC,ProcessID,Gen0PromotedBytesPerSec,Gen1PromotedBytesPerSec
$netclrinfo = @{}
$allpools | foreach {			
	$netclrinfo[$_] = get-WmiNetClrMem $_
}
$netclrinfo | ConvertTo-JsonCode language: PowerShell (powershell)

Using Get-CimInstance, you query the aforementioned WMI Class, and filter on the process name like 'w3wp%', since you are only interested in IIS application pools – or worker processes. Output is saved in an object $netclrmemory, the one you noticed above.

A new hashtable object $netclrinfo is created, and you loop through $allpools in a foreach, executing the function get-WmiNetClrMem($wpname). The end result in hashtable $netclrinfo is then converted into an JSON string for Zabbix.

Here is the JSON output:

"reilink.nl(domain)(4.0)(pool)":  {
	"LargeObjectHeapsize":  74864,
	"Gen0heapsize":  1660248,
	"Gen2heapsize":  2805364,
	"Gen1heapsize":  341388,
	"NumberTotalreservedBytes":  33546240,
	"NumberTotalcommittedBytes":  7282688,
	"PercentTimeinGC":  48,
	"Gen0PromotedBytesPerSec":  114060,
	"Gen1PromotedBytesPerSec":  0,
	"AllocatedBytesPersec":  80405312
},Code language: JSON / JSON with Comments (json)

Here you can see 114060 bytes per second is promoted from Generation 0 garbage collection to Generation 1 (“Gen0PromotedBytesPerSec”). As Microsoft explains: Objects that are not reclaimed in a garbage collection are known as survivors and are promoted to the next generation.

Add the following XML to your template file:

<item_prototype>
	<name>.NET CLR Memory: Gen0heapsize</name>
	<type>DEPENDENT</type>
	<key>iis.apppool.Gen0heapsize[{#APPPOOLNAME}]</key>
	<delay>0</delay>
	<history>7d</history>
	<description>This counter displays the maximum bytes that can be allocated in generation 0 (Gen 0); its does not indicate the current number of bytes allocated in Gen 0. A Gen 0 GC is triggered when the allocations since the last GC exceed this size. The Gen 0 size is tuned by the Garbage Collector and can change during the execution of the application. At the end of a Gen 0 collection the size of the Gen 0 heap is infact 0 bytes; this counter displays the size (in bytes) of allocations that would trigger the next Gen 0 GC. This counter is updated at the end of a GC; its not updated on every allocation.</description>
	<units>B</units>
	<application_prototypes>
		<application_prototype>
			<name>{#APPPOOLNAME}</name>
		</application_prototype>
	</application_prototypes>
	<preprocessing>
		<step>
			<type>JSONPATH</type>
			<params>$.Gen0heapsize</params>
			<error_handler>DISCARD_VALUE</error_handler>
		</step>
		<step>
			<type>DISCARD_UNCHANGED_HEARTBEAT</type>
			<params>10m</params>
		</step>
	</preprocessing>
	<master_item>
		<key>netclrmemfetch[{#APPPOOLNAME}]</key>
	</master_item>
</item_prototype>Code language: HTML, XML (xml)

Repeat for other metrics, and you’re all done!

Here are three graphs as an example:

Are you looking for rock solid, eco-friendly, .NET hosting? Look no further! UmbHost offers powerful hosting services for websites and businesses of all sizes, and is powered by 100% renewable energy!

Windows Server and IIS monitoring with Zabbix, the series

This is part of a series of posts about monitoring Windows Server and IIS (web sites, application pools incluis) with Zabbix. Other editions in this series are:

• Monitor IIS application pools in Zabbix, part 1 – aka how to create an IIS website autodiscovery item and application pool discovery
• ASP.NET web application monitoring in Zabbix, part 2 – aka monitor the performance of your ASP.NET web application in your IIS web server utilizing Windows performance counters in Zabbix
• Part 3 shows you how to get more out your Windows Performance Counters to monitor auto discovered web application pools for Win32_PerfRawData_PerfProc_Process properties.
• In part 4 I show you how to monitor .NET CLR Garbage Collected heap from your web application.

I hope you like it.