Prometheus Monitoring K8S cluster
For this post, I will be using cAdvisor and kube-state-metrics to monitor K8S cluster itself as well as the pods and containers running on the cluster.
What is cAdvisor?
What is kube-state-metrics?
For more details, please refer github.
Before we start, you should have a K8S cluster ready. Things we will be doing
- Deploy Prometheus with Configmap
- Deploy Grafana, setup and import dashboards
- Deploy cAdvisor
- Deploy kube-state-metrics
- Expose Prometheus internally and Grafana to internet
Let’s get started.
First create a Configmap yaml file with content below.
Next, create a deployment file with content below.
Now deploy Configmap and Prometheus with the yaml files.
Prometheus is up and running now.
For this demo, I am using the simplest way I know to deploy Grafana.
kubectl create deploy grafana --image grafana/grafana
I am using helm to install cAdvisor. We add a repository first, then use helm to install cAdvisor.
helm repo add code-chris https://code-chris.github.io/helm-charts
Install it from github project so first clone it to your computer.
Change all the namespace variables under /examples/standard/ from kube-system to default, then deploy it.
kubeclt apply -f kube-state-metrics/examples/standard/kubectl get all -l app.kubernetes.io/name=kube-state-metrics -n kube-system
Expose Prometheus and Grafana
Expose Prometheus to ClusterIP. Since my K8S is built on AWS, therefore I will expose Grafana through CLB.
kubectl expose deploy prometheus --type ClusterIP --port 9090
kubectl expose deploy grafana --type LoadBalancer --port 3000
Access to CLB url at port 3000. Default credential is admin / admin.
Change default password once login.
First add Prometheus as data source. After “Add data source”, select Prometheus.
Just enter Prometheus deployment location then scroll down to the bottom and save & test.
Next we import Grafana dashboard for cAdvisor and kube-state-metrics.
Input 9706 and click the cursor away so that it will load the information automatically.
Choose Prometheus as data source and click import.
Once redirect to kube-state-metrics dashboard, change time range and refresh interval. Wait for a few minutes for Prometheus to collect datas and Grafana will show it a while later.
Repeat same step to add dashboard for cAdvisor. The ID is 3119.