External Metrics Discovery on Kubernetes

You are here:

Problem

How to use Prometheus on Kubernetes as central monitoring system in hybrid enterprise, with containers or hosts deployed externally on Windows?

Solution

Use Prometheus metrics discovery to locate targets deployed as exporters external to Kubernetes cluster.

Considerations

The following benefits and liabilities should be considered:

Benefits

  • Use common monitoring system in hybrid enterprise scenarios with gradual adoption of Kubernetes.
  • Adoption of vendor neutral monitoring technologies like Prometheus available from Kubernetes ecosystem without cost of proprietary monitoring software.

Prerequisites

  • Prometheus deployed on Kubernetes cluster.
  • Windows Server virtual machine with Sonar monitoring agent or other exporter exposing metrics in Prometheus format via HTTP protocol.

Steps

The below steps show how to configure Prometheus for discovering external metrics on Kubernetes. In this example, Windows Server 2019 virtual machine with IP 10.0.0.145 has been deployed with Sonar monitoring agent, to expose metrics on port 9500, to produce the following result:

Shell

Step 1: Create Kubernetes Service

The above example defines service and endpoint with external IP address and port exposed by monitoring agent (as Prometheus exporter) within Kubernetes cluster. Note that service has annotations, required for dynamic discovery by Prometheus. This step is required to define service for pointing to metrics agent deployed externally outside Kubernetes cluster:

Plain Text
external-svc.yaml

Step 2: Test Kubernetes Service

This step is optional to verify that external metrics can be retrieved within Kubernetes cluster by running the following commands:

Information

In this example, busyboxpod and nano-service are in default Kubernetes namespace.

Shell

After curl pod has been started on cluster, use below command from the pod’s shell:

Shell

Step 3: Validate Prometheus Targets

This step is need to confirm that external service has been discovered by Prometheus deployed on Kubernetes. To complete this step, review list of targets and verify that external metrics endpoint is being present:

External metrics endpoint listed in Prometheus on k8s cluster.
Last Updated On February 24, 2019