A while ago I implemented Prometheus and Grafana on K8S. Everything works fine but as we know that Pod is stateless and could be recycle anytime, therefore any settings configured on Prometheus and Grafana would be gone when Pod is replaced.
Therefore, to solve this problem, we need to use Persistent Volumes(PV), PersistentVolumeClaims(PVC) and ConfigMap. (There should be other ways but these are ways I know for now)
PV, as named, the provisioned storage space in cluster to store data. Data is stored in PV rather than Pod itself, so data is retain when Pod is replaced.
PVC, as named, is used to claim the provisioned storage space for use.
ConfigMap, is used to store the configurations of the application running in Pod. For example, the httpd.conf for httpd. You could modify the httpd.conf file in Pod but everything will be gone once Pod is replace. So we can write the desired configurations in ConfigMap and tell Pod to use such configuration whenever Pod is replaced. Therefore Pod is always running with how we wanted it to be.
In this post, I will show you, as well as recored for my future reference, how to use ConfigMap for Prometheus, PV, PVC for Prometheus and Grafana for storing persistent data.
Prometheus ConfigMap
Let’s start by creating ConfigMap for Prometheus. In the ConfigMap, we can put whatever settings we want Prometheus to be. For example, I will use the default Prometheus configuration and add one client to monitor.
Create a ConfigMap and edit it.
Everything ABOVE redline is generated automatically. Everything BELOW redline is input by user. Pay attention to the blue cycle, |-
For some reason, if | or |- is not input, the validation of yaml file will failed. For this example, I added an client to be monitored at the last three lines.
Prometheus PVC
Next, let’s create PV for Prometheus. Since my cluster is running on AWS using EC2s, PV will be using EBS.
For some cases, such as provisioning NFS as PV, seem like PV needs to be provision first. But for my case, I just need to provision PVC and PV is created automatically. I guess this is the advantage of using other management tools such as kops on cloud services.
Create PVC with yaml content below, and PV is created automatically. All monitored statistics will be stored in PV so that historical data is available even if Prometheus is replaced.
Though PV is created automatically in my case, it is worth mentioning StorageClass. StorageClass is a way for K8S administrator to describe the classes/types of PV they create. Below is an StorageClass example for EBS.
Prometheus Deployment
Next, let’s create Prometheus deployment with yaml content below.
Point 1 + 2 + 3 is saying that I want to use the ConfigMap created previously for Prometheus container which is located at /etc/prometheus.
Point 4 + 5 + 6 + 7 is saying that I want to mount the created PV on the location /prometheus inside container.
Once created, you would see the resources with red underline is created.
Expose Prometheus
In order for Prometheus to be accessible, we need to expose it. There are four types of ServiceTypes, I am not going into details and since my K8S is running AWS, I will use LoadBalancer ServiceType which will be ELB.
Wait for a few minutes for ELB to pass health check. Once passed, verify by accessing ELB’s URL with port 9090.
If everything is setup correctly, including setting up a client server with node exporter installed, you can now access Prometheus and query all kinds of information. Let’s verify by query some data of the client server, which was added in the ConfigMap of Prometheus, as shown below.
Till this stage, we have done setting up Prometheus. Let’s move to Grafana.
Grafana
Grafana will be used for analysis and presentation layer of statistics scraped by Prometheus, which means that rather than allowing Prometheus to be accessible by internet, we only need it to be accessible by Grafana INTERNALLY. For this, we simply change LoadBalancer ServiceType to ClusterIP.
Edit Prometheus Service at line 23, remove NodePort and its value. Edit line 27, change value from LoadBalancer to ClusterIP. Save and exit.
Once you exited, K8S will pick the change and update the Service right away.
K8S DNS
It is worth mentioning that K8S has its own DNS service, kube-dns, within the cluster. For Service with ClusterIP ServiceType, all Pods in the same namespace can lookup and communicate with each other through DNS name.
In my case, for example, Grafana Pod can communicate to Prometheus Pod by accessing Prometheus Service, which is named Prometheus.
Grafana PVC
Process of building Grafana is pretty similar to deploying Prometheus. PV is created by provisioning PVC which is used to store Grafana customizations such as extra plugins, templates etc.
Contain of yaml file:
Grafana Deployment
Next we need to provision Grafana deployment using the PVC. The yaml file:
Pay attention that:
- containers.volumeMounts.name needs to match volumes.name.
- volumes.persistentVolumeClaim.claimName is the metadata.name in PVC yaml file.
- mountPath is the location in container where you want to mount the PV.
4. I had privilege issue when mounting PV, adding securityContext.runAsUser: 0 solve the problem.
SSH into Grafana Container to confirm that PV is mounted as expected.
Expose Grafana
Lastly, we need to expose Grafana to outside world. Since my cluster is in AWS, I will use ELB to expose it.
Wait for a few minutes for ELB to pass health check. Once passed, verify everything by accessing ELB URL with port 3000.
If everything is setup correctly, you will see the login page of Grafana. That’s it! You have successfully deploy Prometheus and Grafana on K8S!
Verification
The default credential is admin/admin. You can change the default password at first login.
Once login, the first step is to add data source, which will be Prometheus Server, which is the container running in Prometheus Pod, exposing to the cluster through Service named Prometheus. The idea of DNS concept kicks in.
To me, I think it is very import to know how K8S networking/DNS works so you would know how to config applications running inside Pods.
Key in your Service name for Prometheus at the URL field in tag 1. This is where DNS will kicks in to work in the background.
Click Save & Test at the bottom and if very is setup correctly, a good to go message will show up.
Now scroll back up and click tag 2, import default dashboards to virtualize the statistics!
Now click Prometheus 2.0 Stats dashboard under Dashboard page.
Replace Pods
The purpose of this post is to verify that data is still available when Pods are replaced. Let’s verify it by deleting the Prometheus and Grafana Pods.
Once deleted, K8S will quick re-generates new ones, then go back to Grafana page and reload. And again, if everything is setup correctly, Prometheus statistics and Grafana settings will remain available!
