How to use Prometheus on Kubernetes as central monitoring system in hybrid enterprise, with containers or hosts deployed externally on Windows?
Use Prometheus metrics discovery to locate targets deployed as exporters external to Kubernetes cluster.
The following benefits and liabilities should be considered:
- Use common monitoring system in hybrid enterprise scenarios with gradual adoption of Kubernetes.
- Adoption of vendor neutral monitoring technologies like Prometheus available from Kubernetes ecosystem without cost of proprietary monitoring software.
- Prometheus deployed on Kubernetes cluster.
- Windows Server virtual machine with Sonar monitoring agent or other exporter exposing metrics in Prometheus format via HTTP protocol.
The below steps show how to configure Prometheus for discovering external metrics on Kubernetes. In this example, Windows Server 2019 virtual machine with IP 10.0.0.145 has been deployed with Sonar monitoring agent, to expose metrics on port 9500, to produce the following result:
Step 1: Create Kubernetes Service
The above example defines service and endpoint with external IP address and port exposed by monitoring agent (as Prometheus exporter) within Kubernetes cluster. Note that service has annotations, required for dynamic discovery by Prometheus. This step is required to define service for pointing to metrics agent deployed externally outside Kubernetes cluster:
Step 2: Test Kubernetes Service
This step is optional to verify that external metrics can be retrieved within Kubernetes cluster by running the following commands:
In this example, busyboxpod and nano-service are in default Kubernetes namespace.
After curl pod has been started on cluster, use below command from the pod’s shell:
Step 3: Validate Prometheus Targets
This step is need to confirm that external service has been discovered by Prometheus deployed on Kubernetes. To complete this step, review list of targets and verify that external metrics endpoint is being present: