Prometheus Monitoring K8S cluster

4 min readMay 12, 2020

For this post, I will be using cAdvisor and kube-state-metrics to monitor K8S cluster itself as well as the pods and containers running on the cluster.

What is cAdvisor?

For more details, please refer github and prometheus.

What is kube-state-metrics?

For more details, please refer github.

Before we start, you should have a K8S cluster ready. Things we will be doing

Deploy Prometheus with Configmap
Deploy Grafana, setup and import dashboards
Deploy cAdvisor
Deploy kube-state-metrics
Expose Prometheus internally and Grafana to internet

Let’s get started.

Prometheus

First create a Configmap yaml file with content below.

Next, create a deployment file with content below.

Now deploy Configmap and Prometheus with the yaml files.

Prometheus is up and running now.

Grafana

For this demo, I am using the simplest way I know to deploy Grafana.

kubectl create deploy grafana --image grafana/grafana

cAdvisor

I am using helm to install cAdvisor. We add a repository first, then use helm to install cAdvisor.

helm repo add code-chris https://code-chris.github.io/helm-charts

kube-state-metrics

Install it from github project so first clone it to your computer.

git clone https://github.com/kubernetes/kube-state-metrics.git

Change all the namespace variables under /examples/standard/ from kube-system to default, then deploy it.

kubeclt apply -f kube-state-metrics/examples/standard/kubectl get all -l app.kubernetes.io/name=kube-state-metrics -n kube-system

Expose Prometheus and Grafana

Expose Prometheus to ClusterIP. Since my K8S is built on AWS, therefore I will expose Grafana through CLB.

kubectl expose deploy prometheus --type ClusterIP --port 9090
kubectl expose deploy grafana --type LoadBalancer --port 3000

Grafana Configuration

Access to CLB url at port 3000. Default credential is admin / admin.

Change default password once login.

First add Prometheus as data source. After “Add data source”, select Prometheus.

Just enter Prometheus deployment location then scroll down to the bottom and save & test.

Next we import Grafana dashboard for cAdvisor and kube-state-metrics.

Input 9706 and click the cursor away so that it will load the information automatically.

Choose Prometheus as data source and click import.

Once redirect to kube-state-metrics dashboard, change time range and refresh interval. Wait for a few minutes for Prometheus to collect datas and Grafana will show it a while later.