Monitor website performance in IIS with Zabbix

Monitor your website performance on IIS with Zabbix, Performance Counters, PowerShell and WMI, because it is important to keep an eye on website and webserver performance. For this, Zabbix is a great tool and knowing how to collect metrics using Performance Counters, PowerShell and WMI is readily in your toolbox.

I've written about Zabbix monitoring before: Use Zabbix to monitor your .NET CLR Garbage Collected heap, and Getting more out your Windows Performance Counters monitoring for web applications - for example. This time you're using almost the same techniques to monitor a website hosted in IIS. Sweet, right? For this you use Win32_PerfRawData_W3SVC_WebService class.

The Web Service object includes counters specific to the World Wide Web Publishing Service.

There are roughly 95 properties in the Win32_PerfRawData_W3SVC_WebService class. A lot are interesting, and a lot are only interesting if you're using them. For instance, I wouldn't want to know PutRequestsPersec and TotalPutRequests if I'm not using the HTTP PUT verb on my site or server. In one situation, these are my preferred items to monitor (metrics to collect):

• Name
• AnonymousUsersPersec
• BytesReceivedPersec
• BytesSentPersec
• BytesTotalPersec
• ConnectionAttemptsPersec
• CurrentAnonymousUsers
• CurrentConnections
• DeleteRequestsPersec
• FilesPersec
• FilesReceivedPersec
• FilesSentPersec
• GetRequestsPersec
• LockedErrorsPersec
• LockRequestsPersec
• LogonAttemptsPersec
• MaximumConnections
• NonAnonymousUsersPersec
• NotFoundErrorsPersec
• OptionsRequestsPersec
• OtherRequestMethodsPersec
• PostRequestsPersec
• PropfindRequestsPersec
• PutRequestsPersec
• ServiceUptime
• TotalAnonymousUsers
• TotalBytesReceived
• TotalBytesSent
• TotalBytesTransferred
• TotalDeleteRequests
• TotalFilesReceived
• TotalFilesSent
• TotalFilesTransferred
• TotalGetRequests
• TotalLockedErrors
• TotalLockRequests
• TotalLogonAttempts
• TotalMethodRequests
• TotalMethodRequestsPersec
• TotalNonAnonymousUsers
• TotalNotFoundErrors
• TotalOptionsRequests
• TotalPostRequests
• TotalPropfindRequests
• TotalPutRequests
• TotalUnlockRequests
• UnlockRequestsPersec

Yes this is a lot, but you can always remove items if they don't prove valuable.

Psst, here is how to get the number of current connections in IIS using PowerShell:

Use PowerShell, Performance Counters and WMI to get the current number of active connections to IIS websites. Perfect for monitoring IIS webservers in Zabbix.

Get-WebsiteMetrics.ps1 - Get all Website related performance metrics with PowerShell

The following PowerShell script is what puts everything together and returns the info in a JSON. You run this through Zabbix Agent (2) as a UserParameter.

<#
.SYNOPSIS
    Feeds IIS website metrics to Zabbix: low level discovery of the sites, and a
    JSON document with the performance counters per site.

.DESCRIPTION
    Reads Win32_PerfRawData_W3SVC_WebService, the IIS counters at site level.
    That is a class at W3SVC level rather than CLR level, so it keeps working for
    sites on .NET Framework and on .NET Core alike.

    The script makes a single WMI call and builds both answers from it.

    The values are raw, ever-increasing counters. Add a "Change per second"
    preprocessing step in Zabbix to every item whose name ends in Persec; without
    it you get a monotonically rising line instead of a rate. The Total* fields
    are meant to be cumulative and can stay as they are.

.PARAMETER Action
    discovery  returns the LLD list with {#WEBSITE}
    metrics    returns the measurement document per site

    The value getwebsiteinfo keeps working as an alias for metrics, so an
    existing UserParameter does not break silently.

.PARAMETER IncludeTotal
    Includes the _Total instance. Off by default: it is not a website, and
    discovery would create a site in Zabbix that does not exist in IIS.

.EXAMPLE
    PS> .\Get-WebsiteMetrics.ps1
    The LLD list, as JSON.

.EXAMPLE
    PS> .\Get-WebsiteMetrics.ps1 -Action metrics
    Every counter per site, as JSON.

.EXAMPLE
    PS> .\Get-WebsiteMetrics.ps1 -Action metrics -IncludeTotal
    The same, with the _Total instance included.

.NOTES
    Zabbix agent, without a PowerShell profile and with a fixed execution policy:

        UserParameter=iis.website.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action discovery
        UserParameter=iis.website.metrics,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action metrics

    Make iis.website.metrics a master item and hang the individual counters off
    it as dependent items with a JSONPATH step. That way one PowerShell process
    runs per interval instead of one per counter.
#>

#Requires -Version 5.1

[CmdletBinding()]
param(
    [ValidateSet('discovery', 'metrics', 'getwebsiteinfo')]
    [string] $Action = 'discovery',

    [switch] $IncludeTotal
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'

# The counters the document carries. This list doubles as the column list for
# the WQL query, so the query and the output cannot drift apart.
#
# Projecting columns barely saves any time on a Win32_PerfRawData_* class: the
# provider materialises the object regardless. Measured on an IIS host with 74
# sites, "select *" took 151-204 ms and a projected query 169-198 ms. It is here
# for readability and for that single source of truth, not as an optimisation.
$COUNTERS = @(
    'AnonymousUsersPersec'
    'BytesReceivedPersec'
    'BytesSentPersec'
    'BytesTotalPersec'
    'ConnectionAttemptsPersec'
    'CurrentAnonymousUsers'
    'CurrentConnections'
    'DeleteRequestsPersec'
    'FilesPersec'
    'FilesReceivedPersec'
    'FilesSentPersec'
    'GetRequestsPersec'
    'LockedErrorsPersec'
    'LockRequestsPersec'
    'LogonAttemptsPersec'
    'MaximumConnections'
    'NonAnonymousUsersPersec'
    'NotFoundErrorsPersec'
    'OptionsRequestsPersec'
    'OtherRequestMethodsPersec'
    'PostRequestsPersec'
    'PropfindRequestsPersec'
    'PutRequestsPersec'
    'ServiceUptime'
    'TotalAnonymousUsers'
    'TotalBytesReceived'
    'TotalBytesSent'
    'TotalBytesTransferred'
    'TotalDeleteRequests'
    'TotalFilesReceived'
    'TotalFilesSent'
    'TotalFilesTransferred'
    'TotalGetRequests'
    'TotalLockedErrors'
    'TotalLockRequests'
    'TotalLogonAttempts'
    'TotalMethodRequests'
    'TotalMethodRequestsPersec'
    'TotalNonAnonymousUsers'
    'TotalNotFoundErrors'
    'TotalOptionsRequests'
    'TotalPostRequests'
    'TotalPropfindRequests'
    'TotalPutRequests'
    'TotalUnlockRequests'
    'UnlockRequestsPersec'
)

function Get-WebServiceInstance {
    <#
        Fetches the counters for every site in one call.

        No -OperationTimeoutSec here. A cap only helps where a stall is the
        failure mode, such as the .NET performance classes that hang for roughly
        95 seconds when their category is empty. This class is neither slow nor
        prone to hanging, and an expired timeout would hand back an empty
        document that looks perfectly valid.
    #>
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)] [string[]] $Property
    )

    $columns = (@('Name') + $Property) -join ', '
    $query = "SELECT $columns FROM Win32_PerfRawData_W3SVC_WebService"

    # Deliberately no -ErrorAction SilentlyContinue. If WMI fails this script
    # should stop with a message on stderr and a non-zero exit code, so the item
    # in Zabbix goes NOT SUPPORTED. Staying silent produces a valid-looking
    # document with half the data missing, and nobody ever sees that.
    return @(Get-CimInstance -Namespace 'root\cimv2' -Query $query)
}

function ConvertTo-MetricMap {
    <#
        Turns the WMI instances into a map of site name to counters.

        Keys are compared ordinally. A plain PowerShell hashtable is
        case-insensitive, so two IIS sites differing only in casing would
        silently overwrite each other.
    #>
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)] [AllowEmptyCollection()] [object[]] $Instance,
        [Parameter(Mandatory)] [string[]] $Property
    )

    $map = [System.Collections.Specialized.OrderedDictionary]::new([StringComparer]::Ordinal)

    foreach ($i in $Instance) {
        $metrics = [ordered]@{}
        $metrics['Name'] = $i.Name

        foreach ($p in $Property) {
            # No falsy filter. A counter reading zero is a measurement, not a
            # missing value. The original dropped every zero, so a site without
            # 404s lost its NotFoundErrorsPersec key entirely and "no errors"
            # became indistinguishable from "not collected".
            $metrics[$p] = $i.$p
        }

        $map[$i.Name] = $metrics
    }

    return $map
}

# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

try {
    $instances = @(Get-WebServiceInstance -Property $COUNTERS)
}
catch {
    Write-Error "Cannot read Win32_PerfRawData_W3SVC_WebService: $($_.Exception.Message)"
    exit 1
}

# Order matters here. Zero instances in the raw result means the class cannot be
# read or W3SVC is not running, and that is an error. Zero instances AFTER
# filtering out _Total only means there are no sites, which is a perfectly valid
# state: IIS runs fine without any. Conflating the two gives an item that goes
# NOT SUPPORTED on a healthy host.
if ($instances.Count -eq 0) {
    Write-Error 'No instances in Win32_PerfRawData_W3SVC_WebService. Is the W3SVC service running?'
    exit 1
}

if (-not $IncludeTotal) {
    # _Total is the sum across all sites, not a website. Left in, discovery
    # creates a site in Zabbix that does not exist in IIS.
    $instances = @($instances | Where-Object { $_.Name -ne '_Total' })
}

# Keeps the alias from the original UserParameter working.
if ($Action -eq 'getwebsiteinfo') { $Action = 'metrics' }

switch ($Action) {
    'discovery' {
        # Zabbix has accepted a bare array since 4.2, but the data wrapper works
        # in every version and breaks no existing LLD rule.
        $lld = @{
            data = @($instances | ForEach-Object { @{ '{#WEBSITE}' = $_.Name } })
        }
        $lld | ConvertTo-Json -Depth 4 -Compress
    }

    'metrics' {
        $map = ConvertTo-MetricMap -Instance $instances -Property $COUNTERS

        # -Depth set explicitly: the default is 2 and this document is exactly
        # two levels deep. That works today, but one extra level would land in
        # the JSON silently as "System.Collections.Hashtable".
        $map | ConvertTo-Json -Depth 4 -Compress
    }
}

When ran, you can expect the following JSON output to use in your Zabbix template:

 "example.com":  {
    "TotalNotFoundErrors":  23,
    "MaximumConnections":  12,
    "LogonAttemptsPersec":  221,
    "TotalNonAnonymousUsers":  120,
    "TotalMethodRequests":  222,
    "TotalFilesTransferred":  27,
    "BytesReceivedPersec":  196478,
    "ServiceUptime":  2993992,
    "TotalLogonAttempts":  221,
    "BytesTotalPersec":  1825488,
    "PostRequestsPersec":  4,
    "TotalGetRequests":  212,
    "FilesPersec":  27,
    "BytesSentPersec":  1629010,
    "TotalPostRequests":  4,
    "NonAnonymousUsersPersec":  120,
    "NotFoundErrorsPersec":  23,
    "Name":  "example.com",
    "TotalBytesTransferred":  1825488,
    "TotalMethodRequestsPersec":  222,
    "TotalFilesSent":  27,
    "AnonymousUsersPersec":  94,
    "FilesSentPersec":  27,
    "GetRequestsPersec":  212,
    "TotalBytesSent":  1629010,
    "TotalBytesReceived":  196478,
    "TotalAnonymousUsers":  94,
    "ConnectionAttemptsPersec":  219
  },

  "example.org":  {
    "TotalNotFoundErrors":  25,
    "MaximumConnections":  8,
    "OtherRequestMethodsPersec":  1,
    "LogonAttemptsPersec":  375,
    "TotalNonAnonymousUsers":  58,
    "TotalMethodRequests":  378,
    "TotalFilesTransferred":  33,
    "BytesReceivedPersec":  133698,
    "ServiceUptime":  2993992,
    "TotalLogonAttempts":  375,
    "BytesTotalPersec":  1339416,
    "PostRequestsPersec":  28,
    "TotalGetRequests":  339,
    "FilesPersec":  33,
    "BytesSentPersec":  1205718,
    "TotalPostRequests":  28,
    "NonAnonymousUsersPersec":  58,
    "NotFoundErrorsPersec":  25,
    "Name":  "example.org",
    "TotalBytesTransferred":  1339416,
    "TotalMethodRequestsPersec":  378,
    "TotalFilesSent":  33,
    "AnonymousUsersPersec":  192,
    "FilesSentPersec":  33,
    "GetRequestsPersec":  339,
    "TotalBytesSent":  1205718,
    "TotalBytesReceived":  133698,
    "TotalAnonymousUsers":  192,
    "ConnectionAttemptsPersec":  358
  }

Save the above code into a file (Get-WebsiteMetrics.ps1 for example) and configure it in your Zabbix configuration as a UserParameter, along with a parent fetcher:

UserParameter=iis.website.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action discovery
UserParameter=iis.website.metrics,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action metrics

This way you can create an autodiscover template with dependent items:

<?xml version="1.0" encoding="UTF-8"?>
<zabbix_export>
    <version>7.4</version>
    <template_groups>
        <template_group>
            <uuid>3f96a87e1cbe41099adf8a7e1d0905b2</uuid>
            <name>Templates/Applications</name>
        </template_group>
    </template_groups>
    <templates>
        <template>
            <uuid>3ad00360d13d40d9b9985d18491cc03d</uuid>
            <template>IIS website metrics by Zabbix agent</template>
            <name>IIS website metrics by Zabbix agent</name>
            <description>IIS website metrics from Win32_PerfRawData_W3SVC_WebService, collected by Get-WebsiteMetrics.ps1
through the Zabbix agent. Link this to Windows hosts running IIS.

Two agent items carry everything. iis.website.discovery returns the list of sites, and
iis.website.metrics returns one JSON document with every counter for every site. All 46 metrics are
dependent items on that one document, so a single PowerShell process runs per interval instead of
one per counter.

The counters are raw. Every item whose name ends in Persec gets a Change per second step; the
Total* fields are cumulative by design and are stored as they come in.

Required in zabbix_agentd.conf:

    UserParameter=iis.website.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action discovery
    UserParameter=iis.website.metrics,powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\Get-WebsiteMetrics.ps1 -Action metrics</description>
            <vendor>
                <name>saotn.org</name>
                <version>7.4-1</version>
            </vendor>
            <groups>
                <group>
                    <name>Templates/Applications</name>
                </group>
            </groups>
            <items>
                <item>
                    <uuid>1da4489e390144019d64a42f2781b7a2</uuid>
                    <name>IIS website metrics</name>
                    <type>ZABBIX_ACTIVE</type>
                    <key>iis.website.metrics</key>
                    <delay>{$IIS.WEBSITE.INTERVAL}</delay>
                    <history>0</history>
                    <value_type>TEXT</value_type>
                    <description>The only polled item for the metrics. One JSON document per interval, keyed by website name, with all 46 counters inside each entry. 
                    history is 0 on purpose: every value is reachable through the dependent items, and keeping the raw document as well would store the same data a second time. Note that this makes lastvalue and lastclock stay empty in the API even when the item works perfectly.</description>
                    <tags>
                        <tag>
                            <tag>Application</tag>
                            <value>IIS website</value>
                        </tag>
                    </tags>
                </item>
            </items>
            <discovery_rules>
                <discovery_rule>
                    <uuid>c9866d1d8f974611b2af386274ed6ec2</uuid>
                    <name>IIS websites</name>
                    <type>ZABBIX_ACTIVE</type>
                    <key>iis.website.discovery</key>
                    <delay>{$IIS.WEBSITE.DISCOVERY.INTERVAL}</delay>
                    <filter>
                        <evaltype>AND</evaltype>
                        <conditions>
                            <condition>
                                <macro>{#WEBSITE}</macro>
                                <value>{$IIS.WEBSITE.MATCHES}</value>
                            </condition>
                            <condition>
                                <macro>{#WEBSITE}</macro>
                                <value>{$IIS.WEBSITE.NOT_MATCHES}</value>
                                <operator>NOT_MATCHES_REGEX</operator>
                            </condition>
                        </conditions>
                    </filter>
                    <lifetime>7d</lifetime>
                    <enabled_lifetime_type>DISABLE_NEVER</enabled_lifetime_type>
                    <description>Discovers the websites from Win32_PerfRawData_W3SVC_WebService. The _Total instance is filtered out
by the script, because it is the sum across all sites and not a website.

Lost resources are kept enabled. A site that is stopped disappears from the counters, and disabling
its items would leave a data gap plus triggers that switch themselves off. They are deleted after
the lifetime instead, so a site that is really gone does not linger.</description>
                    <item_prototypes>
                        <item_prototype>
                            <uuid>c96c14cd272240f8bc724cb5a83213b4</uuid>
                            <name>Anonymous users on {#WEBSITE}</name>
                            <type>DEPENDENT</type>
                            <key>iis.website.AnonymousUsersPersec[&quot;{#WEBSITE}&quot;]</key>
                            <history>7d</history>
                            <trends>90d</trends>
                            <value_type>FLOAT</value_type>
                            <description>Raw counter from Win32_PerfRawData_W3SVC_WebService, converted to a rate by the Change per second
step. The raw value only ever increases, so without that step this graph is a straight line going up.

A restart of W3SVC resets the counter, which makes one interval go negative. That interval is
discarded instead of turning the item unsupported.</description>
                            <preprocessing>
                                <step>
                                    <type>JSONPATH</type>
                                    <parameters>
                                        <parameter>$[&quot;{#WEBSITE}&quot;].AnonymousUsersPersec</parameter>
                                    </parameters>
                                    <error_handler>DISCARD_VALUE</error_handler>
                                </step>
                                <step>
                                    <type>MATCHES_REGEX</type>
                                    <parameters>
                                        <parameter>^[0-9]+$</parameter>
                                    </parameters>
                                    <error_handler>DISCARD_VALUE</error_handler>
                                </step>
                                <step>
                                    <type>CHANGE_PER_SECOND</type>
                                    <parameters>
                                        <parameter/>
                                    </parameters>
                                    <error_handler>DISCARD_VALUE</error_handler>
                                </step>
                            </preprocessing>
                            <master_item>
                                <key>iis.website.metrics</key>
                            </master_item>
                            <tags>
                                <tag>
                                    <tag>Application</tag>
                                    <value>IIS website</value>
                                </tag>
                                <tag>
                                    <tag>website</tag>
                                    <value>{#WEBSITE}</value>
                                </tag>
                            </tags>
                        </item_prototype>

      <!-- ... -->

  </discovery_rule>
</discovery_rules>

<!-- ... -->

Robustness

Every preprocessing step uses Discard value as its error handler, so a single bad reading never takes an item down.

Each item starts with a JSONPath step to pull its counter out of the shared document, followed by a ^[0-9]+$ Matches regular expression guard. That guard catches the case where a counter is missing from the output: the JSONPath step then yields nothing rather than an error, and without the guard that empty value would be stored as a zero.

The 22 rate items add a third step, Change per second. Discarding on error matters most here. A restart of W3SVC resets the underlying counter to zero, which makes one interval come out negative. Zabbix treats a negative rate as an error, so without the handler that single interval would flip the item to NOT SUPPORTED and keep it there until you notice. With it, one interval is dropped and collection carries on.

The 24 gauges and Total* fields have no third step: they are stored exactly as the counter reports them.