Steps & output to set up a metric dashboard for a flask application using Grafana, Prometheus and Jaeger Tracing.

Setup

Run $ vagrant up in the home directory of the repo where the Vagrantfile is located.

This will create up a opensuse box with the name 'dashboard' in you VB environment.

The Vagrantfile has some scripts pre written which will do the following:

Install basic some dependencies.
Install k3s
Install Helm
Install Grafana and Promethous using Helm
Install Jaeger CRD, service account, role, role binding and operator
Configure Jaeger operator to be cluster wide.
Verify if all the installations.

Verify Setup

SSH into the vagrant box and use the kubectl command to check if all the deployments, service, pods are running properly or not.

$ vagrant ssh
$ kubectl get all --all-namespaces

Configure Vagrant Box

Here we have to ssh into the vagrant box to write kubectl command but we can abstract that on our locally installed kubectl by configuring our local kubeconfig file.

$ vagrant ssh
$ cat /etc/rancher/k3s/k3s.yaml
(copy the content show by above command and paste it in "~/.kube/config" file using below command)
$ vim ~/.kube/config

Steps to expose grafana dashboard

Search for grafana service by kubectl get svc -n monitoring.
We are using vagrant to virtualise our environment so we have occupied port 3000 of our local system in our vagrantfile and connected it to port 3000 of our vagrant environment. We will port forward our svc/prometheus-grafana service using the belong command after we ssh into the vagrant.
$ vagrant ssh
$ kubectl port-forward -n monitoring --address 0.0.0.0 svc/prometheus-grafana 3000:80
Now go to url "http:localhost:3000" in your browser. It will redirect you to grafana login page. Put the following credentials:
Email or username: admin
Password: prom-operator
You can now see the Grafana Dashboard.

Create Jaeger Instance

In order to use Jaeger we have to create a Jaeger Instance. Follow the steps below.

$ kubectl apply -f jaeger-tracing/jaeger.yaml
(We have to enable sidecar injector for our instance located in a particular namespace to allow jeager instance to check for traces in application in a different namespace.)
$ kubectl apply -f jaeger-tracing/injector-enable.yaml

Steps to expose Jaeger UI

Check the namespace where your jaeger instance is been created. let it be ${namespace}
We are using vagrant to virtualise our environment so we have occupied port 16686 of our local system in our vagrantfile and connected it to port 16686 of our vagrant environment. We will port forward our svc/my-traces-query (jaeger instance service) using the below command after we ssh into the vagrant.
$ vagrant ssh
$ kubectl port-forward -n ${namespace} --address 0.0.0.0 svc/$(kubectl get svc -n ${namespace} -l app=jaeger -o jsonpath={.items[2].metadata.name}) 16686:16686
You can now see the Jaeger UI at "http:localhost:16686" in your local browser.

Run our flask application.

Go to manifest and deploy our application to kubernetes. Following is the command to do it in one go. $ kubectl apply -f manifests/app/.

Expose Frontend of our boiler code.

Our frontend is deployed in default namespace with a service name svc/frontend.
We are using vagrant to virtualise our environment so we have occupied port 8080 of our local system in our vagrantfile and connected it to port 8080 of our vagrant environment. We will port forward our svc/frontend (jaeger instance service) using the below command after we ssh into the vagrant.
$ vagrant ssh
$ kubectl port-forward --address 0.0.0.0 svc/frontend 8080:8080
You can now see the out frontend UI at "http:localhost:8080" in your local browser.

Open/Create a dashboard

Now that Grafana is up and Tracing is enabled and prometheus is set as one of the data source. We can create a dashboard to metrics over observation. I have already created a dashboard and have exported/saved it's corresponing json file in dashboards/dashboard.json. One can upload this json back to grafana to see the metrices dashboard.

Below are the images showing outputs.

Setup the Jaeger and Prometheus source

Exposing Grafana to the internet and then setting Prometheus as a data source. Screenshot of the home page after logging into Grafana.

Describe SLO/SLI for our project

What the SLIs are, based on an SLO of monthly uptime and request response time.

(SLI) monthly uptime : monthly uptime percentage of website/network application/physical server/etc. is 99.8%. This is the ratio of percentage to the time during which website/network application/physical server/etc. is able to serve requests or perform programmed operations.

We can set our (SLO) goal for the future - monthly uptime percentage of website/network application/physical server/etc. to be 99.9%.

(SLI) request response time- request response time of specified website/network application is 20 ms. This is the time during which the response is being handled by a remote website/network application.

We can set our (SLO): request response time of website to 10 ms.

Creating SLI metrics.

The 4 Golden Signals we already know and the 5th one is also popular metric - Uptime:

Latency - response time during which the requests are served, usually measured in milliseconds;
Traffic - the value of load on the target, usually measured in requests per second, or megabits/kilobits per second;
Errors - the amount of failed requests, usually measured in number of requests with return HTTP-code 500 in a second;
Saturation/Utilization - indicates the percentage of system resources used (CPU usage %, Memory usage %/GB/MB);
Uptime - the overall availability metric, could be shown as time from the last reboot/fail or more often as a percentage, indicating the ratio of time the monitored target was available for service to the time it wasn't (eg. 99.9% of uptime, so in contrast - 0.1% of overall time it was in downtime or in maintenance, it didn't serve its designated work).

Create a Dashboard to measure our SLIs

Creating a dashboard to measure the uptime of the frontend and backend services We will also want to measure to measure 40x and 50x errors.

Tracing our Flask App

We will create a Jaeger span to measure the processes on the backend.

Jaeger in Dashboards

Now that the trace is running, let's add the metric to our current Grafana dashboard.

Creating SLIs and SLOs

We want to create an SLO guaranteeing that our application has a 99.95% uptime per month. Name four SLIs that you would use to measure the success of this SLO.

As our application I would consider 'backend' application:

Uptime. SLI: Uptime of the application availability per month. SLO: Uptime of the application - 99.95%.
Latency:
- SLI: Average response time per 30 seconds periods per month. SLO: Average response time per 30 sec periods per month less than 100ms.
- SLI: Percentage of request count which complete in less than 100ms. SLO: 99% of request count will complete in less than 100ms.
Errors. SLI: HTTP 500 errors % rate per 1 minute ranges. SLO: HTTP 500 errors % rate per 1 minute is less than 1%.
Traffic. SLI: Total requests per minute. SLO: Total requests per minute is less than 1800.

Final Dashboard

Creating a Final Dashboard containing graphs that capture all the metrics of your KPIs and adequately representing your SLIs and SLOs. Screenshot of the dashboard is below, a description of what graphs are represented in the dashboard.

Dashboard: General Dashboards Project:

Row 'Prometheus':

CPU Usage: CPU Usage fractions of 2 application deployments - backend-app and frontend-app;
Memory Usage: Memory usage in MiB of 2 application deployments - backend-app and frontend-app;
Uptime ('frontend'): frontend application uptime graph;
Uptime ('backend'): backend application uptime graph;
HTTP 4xx Errors: the number of HTTP requests with response code 4xx, over 30 sec intervals, shown per application and path;
HTTP 5xx Errors: the number of HTTP requests with response code 5xx, over 30 sec intervals, shown per application and path;
HTTP requests per second: the number of HTTP requests per second;
Total requests per minute: the amount of all HTTP requests measured over one minute intervals;
Average response time [30s]: the average response time for HTTP requests over 30 sec intervals, shown per application and path;
Requests under 100ms: the percentage of requests which were finished within 100ms, shown per application and path;
HTTP 5xx Errors % rate per minute: the percentage rate of HTTP 5xx errors per 1 minute intervals;
HTTP 4xx Errors % rate per minute: the percentage rate of HTTP 4xx errors per 1 minute intervals;

Row 'Jaeger':

Min latency ( backend '/api' ): minimal latency in milliseconds of the requests which were served by the '/api' endpoint of backend app, within the specified time range;
Avg latency ( backend '/api' ): average latency in milliseconds of the requests which were served by the '/api' endpoint of backend app, within the specified time range;
Max latency ( backend '/api' ): maximum latency in milliseconds of the requests which were served by the '/api' endpoint of backend app, within the specified time range;
Requests Count by Latency ( backend '/api' ): the amount of total requests served by the backend 'api'-endpoint distributed by the serving duration, within the specified time range;
Backend HTTP 500 Errors Count Indicator: the amount of requests with response code HTTP 500, served by the backend 'api'-endpoint distributed by the serving duration, within the specified time range.

Navigation Menu

Repository files navigation