Sending Windows Container Metrics to Prometheus and Grafana

Last Updated On September 18, 2018
You are here:

This section describes how to use Sonar for monitoring Docker containers on Nano Server without cloud services or using any cloud vendor. Sonar is available under BSD license.

Introduction

TL;DR – This article explain how to monitor Windows containers or hosts on-premise with Sonar, Prometheus and Grafana. The rest of this article is organized as follows:

  • Prerequisites
  • Architecture
  • Results
  • Steps to achieve the above results

Prerequisites

  • Docker or Kubernetes
  • Grafana
  • Prometheus

Architecture

To build proof of concept, ABCDEF decides to deploy containers using Nano Server and use only additional containers needed for monitoring datacenter. They decide to deploy one Sonar container per Nano Server host to use it as WMI scraper to collect all the metrics from other containers and host itself.

 

Deployment

The below diagram shows simple environment in private datacenter, which is completely autonomous and not connected to internet:

 

The figure above shows deployment of Unix containers for monitoring only:

  • Grafana container – for data visualization and simple alerts.

In some cases additional Prometheus instance is recommended to collect metrics from each host where its is deployed and use re-labeling to drill down from master to a host specific instance.

The same figure shows deployment of Windows containers for enterprise applications:

  • Windows container(s) – for custom applications. These applications only leverage Docker and do not use Hyper-V isolation.
  • Sonar container – for monitoring all containers within a host as well as host metrics.
  • Prometheus container – for scraping metrics using Sonar or other containers endpoints configured in its job schedule. Alerts can be configured using alert manager, included in Prometheus.
  • Windows host – physical or virtual machine external to docker with additional enterprise applications (for example, BizTalk Server or SQL Server).

Results

TL;DR – the below metrics are collected for Nano server host and its containers by Sonar, stored in Prometheus and shown in Grafana:

 

The figure shown above depicts metrics for Nano-Server host, Sonar and application containers. The Sonar container is depicted as “localhost” simply because it is monitoring itself. As you can see, these containers share the same system resources with the Nano Server that hosts them using Docker. Below are few observations:

  • Host metrics are accessed via internal container network – the default Docker host address is 172.24.224.1
  • Containers use Docker process isolation without Hyper-V – they share metrics for same disk size, do not list physical network adapters and therefore only process metrics are different.
  • Host metrics include docker daemon – shown in working set graph in the above dashboard.

Log Aggregation

Using Prometheus is not recommended for log aggregation. Thus, we recommend to use InfluxDb for this purpose by configuring event log queries in Sonar with that specific type of output.

Steps

The following steps were performed to achieve the above results. Note that steps for configuring and deploying InfluxDb are omitted from this article. The UDP configuration for InfluxDb ( in our case, to use sonar database on UDP port 8092) are well documented on InfluxData web site.

Step 1: Prepare Container Deployment on Nano Server

This step is required to deploy sonar container on Nano Server. For this scenario, ABCDEF decided to deploy Sonar configuration file using shared volume on the host instead of Docker container volume. Download sample configuration file from our repository on GitHub. To accomplish this, create new directory with the following path using remote Powershell on the Nano Server:

PowerShell

Next, using connection to Docker daemon running on Nano Server, the below command was executed to deploy Sonar container and test container to monitor:

Command Shell

Note that the above commands used unsecured connection from remote machine to Nano Server for demonstration purposes.

Step 2: Configure Windows Containers

The below steps demonstrate enabling WSMan for Nano Server and Windows containers to use Basic authentication over HTTP. Please note that this is not secure configuration and should not be used in production. For monitoring Windows containers within isolated Docker network , additional scenarios may be considered. More information about security configurations supported by Sonar is available here.

To enable Sonar container to monitor itself execute the following command in its PowerShell remote session:

PowerShell
>Set-WSManInstance WinRM/Config/Service -ValueSet @{AllowUnencrypted=”true”}

To enable access to WinRm on “nano-test” Windows container, create additional user account with necessary permissions. The below example show how to enable basic authentication using HTTP protocol:

PowerShell
>Set-WSManQuickConfig
>Set-WSManInstance WinRM/Config/Service/Auth -ValueSet @{Basic=”true”}
>Set-WSManInstance WinRM/Config/Service -ValueSet @{AllowUnencrypted=”true”}

Step 3: Configure Sonar container

This step is documented a second in sequence only for illustration purposes. To accomplish this step, modify Sonar configuration file deployed on container host (Nano Server)to reflect host names for containers and hosts, including InfluxDB time series database.

Sonar.config
<Sonar>
<Runtime scrapeIntervalSeconds= “5” skipSSLCheck=”true” threads=”4″ />
<Servers>
<add name=”host” url=”http://172.24.224.1:5985/wsman” username=”my-username”password=”my-password” timeoutMilliseconds=”5000″ forceBasic=”true”/>
<add name=”nano-test” url=”http://172.24.224.72:5985/wsman” username=”my-username”password=”my-password” timeoutMilliseconds=”5000″/>
<add name=”infra-sonar” url=”http://localhost:5985/wsman” username=”my-username”password=”my-password” timeoutMilliseconds=”5000″ authType=”Negotiate”/>
<add name=”win7″ url=”https://10.0.0.212:5986/wsman” username=”my-username” password=”my-password” timeoutMilliseconds=”10000″ forceBasic=”true”/>
</Servers>

The above example shows how to configure:

  • Sending collected metrics to Prometheus, which is default when attribute named “output” is not specified.
  • Monitoring Windows containers and hosts – Sonar Servers section specifies: Nano Server host, test container (“nano”), Sonar container to monitor itself(“sonar”) as well as Windows host configured to use SSL certificate for WinRm to access its metrics remotely.
  • Monitoring windows host – specified for “win7” host, which is configured to only allow HTTPS and self-signed certificate.
  • The attribute named “threads” specifies how many worker threads should be pre-allocated per server listed in “Servers” element.

The above example also illustrates “forceBasic” setting, which directs Sonar to send user credentials for Basic authentication to remote WsMan endpoint.

Step 4: Configure Sonar Query

Modify Sonar.config file by adding schedule to execute WMI query on target host:

Sonar.config
<Queries>
<add name=”PerfOS_Memory”
filter=”SELECT Name, AvailableBytes, PageFaultsPersec, PageReadsPersec, PagesInputPersec FROM Win32_PerfFormattedData_PerfOS_Memory”
resource=”http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/*”>
<Tags>
<add name=”Name” value = “Name”/>
</Tags>
<Values>
<add name=”AvailableBytes” value=”CimType.UInt64″/>
<add name=”PageFaultsPersec”value=”CimType.UInt32″/>
<add name=”PageReadsPersec”value=”CimType.UInt32″/>
<add name=”PagesInputPersec”value=”CimType.UInt32″/>
</Values>
</add>

When the output attribute is omitted, Sonar will expose result of this schedule to it’s metrics endpoint, with default port 5000
After this steps are completed, simply restart Sonar container to refresh its configuration settings.

Step 5: Configure Sonar Schedule

Modify Sonar.config file by adding schedule to execute the above query every 10 seconds:

Sonar.config
<Schedules>
<add name=”myquery” query=”PerfOS_Memory” server=”nano-test” intervalSeconds=”10″/>
</Schedules>

When the output attribute is omitted, Sonar will expose result of this schedule to it’s metrics endpoint, with default port 5000; After this step is completed, restart Sonar container to refresh its configuration settings.

Step 6: Configure Prometheus

This step is required to configure Prometheus to scrap metrics from Sonar container. To accomplish this modify configuration :

Sonar.config
global:
scrape_interval: 10s
external_labels:
monitor: ‘local-monitor’
# A configuration containing what to scrape:
scrape_configs:
– job_name: ‘WindowsDockerHost’
static_configs:
– targets: [‘172.24.224.71:5000’]
As you can see, the IP address for Prometheus target should be set to Sonar container IP, which is depicted above as “localhost” and in this example has physical address 172.24.224.71 with scrape endpoint on port 5000;

Once this step is completed, start Prometheus container to refresh its configuration settings.

Summary

The following benefits and liabilities should be considered before using Sonar:

Benefits

  • Cost effectiveness – using Sonar, Prometheus standalone and Grafana does not require upfront investment.
  • Ability to choose a subset of metrics to monitor Windows containers in datacenter or any cloud.
  • No dependency on type of cloud or specific vendor.
  • Flexible deployment options – Sonar can be deployed as process or container (Windows service, Docker container or Kubernetes pod).
  • Minimal performance impact for monitoring critical metrics and logs.
  • Ability to scale – Sonar container is deployed per host.
  • Enabling anomaly detection – storing metrics in InfluxDb enables many options for analysis and alerting ( using Grafana, Kapacitor or any other software).
  • Easy to configure – no coding is required for adding new settings or changing existing ones.
  • Open to broad scenarios – monitoring WMI metrics from any software product, including BizTalk Server and SQL Server.

Liabilities

  • Properly securing WSMan endpoints for containers and hosts for access by Sonar.
  • Avoid exposing Sonar to external networks.
  • InfluxDb time series database is required by Sonar for storing collected metrics.

References

Tags: