Sending Windows Container Metrics to InfluxDb and Grafana

Last Updated On December 08, 2017
You are here:
  • KB Home
  • Sonar
  • Sending Windows Container Metrics to InfluxDb and Grafana

This section describes how to use Sonar for monitoring Docker containers on Nano Server without cloud services or using any cloud vendor. Sonar is available under BSD license.

Introduction

TL;DR –  This article explain how to monitor Windows containers or hosts on-premise with Sonar, InfluxDb and Grafana. The rest of this article is organized as follows:

  • Prerequisites
  • Architecture
  • Results
  • Steps to achieve the above results

Prerequisites

  • Docker or Kubernetes
  • Grafana
  • InfluxDb

Architecture

To build proof of concept, ABCDEF decides to deploy containers using Nano Server and use only additional containers needed for monitoring datacenter. They decide to deploy one Sonar container per Nano Server host to use it as WMI scraper to collect all the metrics from other containers and host itself using cloud native approach (on the left):

Deployment

The below diagram shows simple environment in private datacenter, which is completely autonomous and not connected to internet:

The figure above shows deployment of Unix containers for monitoring only:

  • InfluxDB time series database – for collecting metrics using UDP protocol from Nano Server hosts.
  • Telegraf container – for measuring performance of monitoring environment deployed as Unix containers.
  • Grafana container – for data visualization and simple alerts.

The same figure shows deployment of Windows containers for enterprise applications:

  • Windows container(s) – for custom applications. These applications only leverage Docker and do not use Hyper-V isolation.
  • Sonar container – for monitoring all containers within a host as well as host metrics.
  • Windows host – physical or virtual machine with additional enterprise applications (for example, BizTalk Server or SQL Server).

Results

TL;DR – the below metrics are collected for Nano server host and its containers by Sonar, stored in InfluxDb and shown in Grafana:

The figure shown above depicts metrics for Nano-Server host, Sonar and application containers. The Sonar container is depicted as “localhost” simply because it is monitoring itself. As you can see, these containers share the same system resources with the Nano Server that hosts them using Docker. Below are few observations:

  • Host metrics are accessed via internal container network – the default Docker host address is 172.24.224.1
  • Containers use Docker process isolation without Hyper-V – they share metrics for same disk size, do not list physical network adapters and therefore only process metrics are different.
  • Host metrics include docker daemon – shown in working set graph in the above dashboard.

The below picture shows consolidated event logs from Nano Server and its containers:

As you can see, event log properties corresponds to the name of the container or physical name and IP address used to gather runtime metrics.

Steps

The following steps were performed to achieve the above results. Note that steps for configuring and deploying InfluxDb are omitted from this article. The UDP configuration for InfluxDb ( in our case, to use sonar database on UDP port 8092) are well documented on InfluxData web site.

Step 1: Prepare Container Deployment on Nano Server

This step is required to deploy sonar container on Nano Server. For this scenario, ABCDEF decided to deploy Sonar configuration file using shared volume on the host instead of Docker container volume. Download sample configuration file from our repository on GitHub. To accomplish this, create new directory with the following path using remote Powershell on the Nano Server:

shell

Next, using connection to Docker daemon running on Nano Server, the below command was executed to deploy Sonar container and test container to monitor:

shell

Note that the above commands used unsecured connection from remote machine to Nano Server for demonstration purposes.

Step 2: Configure Windows Containers

The below steps demonstrate enabling WSMan for Nano Server and Windows containers to use Basic authentication over HTTP. Please note that this is not secure configuration and should not be used in production. For monitoring Windows containers within isolated Docker network , additional scenarios may be considered. More information about security configurations supported by Sonar is available here.

To enable Sonar container to monitor itself execute the following command in its PowerShell remote session or add it to Dockerfile:

PowerShell

To enable access to WinRm on “nano-test” Windows container, create additional user account with necessary permissions. The below example show how to enable basic authentication using HTTP protocol:

PowerShell
The repository with samples for Sonar on GitHub does include examples for Negotiate, NTLM authentication as well.

Step 3: Configure Sonar container

This step is need to configure Sonar container. To accomplish this step, modify Sonar configuration file deployed on container host (Nano Server)to reflect host names for containers and hosts, including InfluxDB time series database.

Sonar.config
The above example shows how to configure:

  • Sending collected metrics to InfluxDb – specified as connection string above with host address and name of time series database.
  • Monitoring Windows containers and hosts –  Sonar  Servers section specifies: Nano Server host, test container (“nano”), Sonar container to monitor itself(“sonar”) as well as Windows host configured to use SSL certificate for WinRm to access its metrics remotely.
  • Monitoring windows host – specified for “win7” host, which is configured to only allow HTTPS and self-signed certificate.

The above example also illustrates “forceBasic” setting, which directs Sonar to send user credentials for Basic authentication to remote WsMan endpoint. Once this step is completed, simply restart Sonar container to refresh its configuration settings. Note that metrics can be sent as UDP broadcast by setting IP address to 255.255.255.255 However, one or more InfluxDb instances should be deployed on the same local subnet for being able to receive metrics.

Step 4: Configure Sonar Query

Modify Sonar.config file by adding schedule to execute WMI query on target host:

Sonar.config
When the output attribute is omitted, Sonar will expose result of this schedule to it’s metrics endpoint, with default port 5000
After this steps are completed, simply restart Sonar container to refresh its configuration settings.

Step 5: Configure Sonar Schedule

Modify Sonar.config file by adding schedule to execute the above query every 10 seconds:

Sonar.config
When the output attribute is omitted, Sonar will expose result of this schedule to it’s metrics endpoint, with default port 5000; After this step is completed, restart Sonar container to refresh its configuration settings.

Step 6: Event Log Aggregation

This is optional step to enable aggregation for event logs from Windows containers and hosts using Sonar. The below example shows how to configure collecting only errors from Windows “nano-test” container.

First, modify Sonar.config file by adding new query element for polling errors from applications event log on Windows:

Sonar.config
Next, add new schedule for the host to use the above query for checking errors every 15 seconds with time shift 30sec at the time it runs:
Sonar.config
Finally, restart Sonar container (in our case, infra-sonar) to reload configuration changes.  The second figure included in this article shows end result in Grafana dashboard.

Summary

The following benefits and liabilities should be considered before using Sonar:

Benefits

  • Cost effectiveness – using Sonar, InfluxDb standalone and Grafana does not require upfront investment.
  • Ability to choose a subset of metrics to monitor Windows containers in datacenter or any cloud.
  • No dependency on type of cloud or specific vendor.
  • Flexible deployment options – Sonar can be deployed as process or container (Windows service, Docker container or Kubernetes pod).
  • Minimal performance impact for monitoring critical metrics and logs.
  • Ability to scale – Sonar container is deployed per host.
  • Enabling anomaly detection – storing metrics in InfluxDb enables many options for analysis and alerting ( using Grafana, Kapacitor or any other software).
  • Easy to configure – no coding is required for adding new settings or changing existing ones.
  • Open to broad scenarios – monitoring WMI metrics from any software product, including BizTalk Server and  SQL Server.

Liabilities

  • Properly securing WSMan endpoints for containers and hosts for access by Sonar.
  • Avoid exposing Sonar to external networks.
  • InfluxDb time series database is required by Sonar for storing collected metrics.

References