Overview
Having relevant information about the state of the infrastructure and other systems in a microservices architecture is crucial for ensuring stability, thus keeping the end-user satisfied. Metrics concerning health and performance of the deployments let the team react faster while having a negligible effect on the system.
Systems applying the microservice architecture pattern are harder to monitor, as Kubernetes pods are mortal, therefore a fixed scraping config is off the mark. That's where the service discovery comes in.
This tutorial walks you through building a microservice monitoring solution from scratch with dynamic service discovery inside a Kubernetes cluster using Prometheus and Grafana.
Most monitoring platforms utilize time-series databases, which are intended for storing time-based information, such as application metrics. Some products only manage the database part (e.g. InfluxDB), while Prometheus offers visualization alongside the metrics management. However, it's common to extend its visualization capabilities with Grafana.
So without further ado, let's get started!
Prerequisite
The tutorial assumes that you have a Kubernetes cluster set up and ready-to-go.
Also, it's handy to have a Spring Boot microservice laying around to monitor.
Publishing metrics from Spring Boot applications
Spring Boot's Actuator provides a production-ready solution for shipping sensible default application metrics without any hassle.
Dependencies
Our first step should be adding the required dependencies.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Configuring Actuator
Actuator exposes some additional information out of the box. To disable these endpoints and include those which are needed in our monitoring setup, add this to your application.yml
:
management:
endpoints:
enabled-by-default: false
web:
base-path: /status
exposure:
include: prometheus, health
In a production environment, it is advised to disable these endpoints, as they might expose details about the system's underlying implementation via various HTTP endpoints. Remapping the base path is optional.
(Optional) Publish custom application name
Some Grafana dashboards provide grouping capabilities if you publish the application name alongside the default metrics.
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class MeterRegistryCustomizerConfiguration {
@Value("${spring.application.name}")
private String applicationName;
@Bean
MeterRegistryCustomizer<MeterRegistry> metricsCommonTags(){
return registry -> registry.config().commonTags("application", applicationName);
}
}
Your application now exposes two more additional endpoints: /status/health
and /status/prometheus
to let you monitor your application.
Collecting the information
Prometheus is an open-source system for monitoring and alerting. We'll use it to collect application metrics.
Prometheus configuration
Prometheus is designed to be configured via command-line flags and a configuration file. Providing this configuration file as a ConfigMap is a simple but reliable solution for setting up Prometheus according to our needs. The previously mentioned service discovery's essence comes from relabeling the existing pod scrape endpoints to the ones exposed by Spring Boot Actuator. This is done by simple regex expressions.
kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-config
namespace: my-namespace
labels:
k8s-app: metrics-service
data:
prometheus.yml: |-
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
Note the following annotations: prometheus.io/scrape
, prometheus.io/path
, prometheus.io/port
. We'll wire them in later.
Setting up a service account for Prometheus
The service discovery relies on metrics provided by Kubernetes. To access these, we'll need a service account.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus-kube
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus-kube
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-kube
subjects:
- kind: ServiceAccount
name: default
namespace: my-namespace
An example Prometheus deployment
Prometheus should be accessible inside the cluster in order to use it as Grafana's datasource. There is no need to expose it outside the cluster though, therefore a ClusterIP is enough.
kind: Service
apiVersion: v1
metadata:
name: prometheus-service
namespace: my-namespace
labels:
k8s-app: prometheus-service
spec:
ports:
- protocol: TCP
port: 9090
targetPort: 9090
selector:
k8s-app: prometheus-service
type: ClusterIP
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: prometheus-service
namespace: my-namespace
labels:
k8s-app: prometheus-service
spec:
replicas: 1
selector:
matchLabels:
k8s-app: prometheus-service
template:
metadata:
name: prometheus-service
labels:
k8s-app: prometheus-service
spec:
volumes:
- name: prometheus-config-volume
configMap:
name: prometheus-config
containers:
- name: prometheus
image: prom/prometheus
imagePullPolicy: Always
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus
args: ["--config.file=/etc/prometheus/prometheus.yml"]
restartPolicy: Always
dnsPolicy: ClusterFirst
It is advised to set a data retention policy, with the --storage.tsdb.retention=32d
container flag.
Updating your deployments
The expressions showed previously utilize the annotations defined in your Kubernetes deployments. The regex based relabeling allows the actual pod scrape endpoint to be configured via the following annotations:
prometheus.io/scrape
: Set the value totrue
to enable scraping of pods belonging to the deploymentprometheus.io/path
: If the metrics path is not/metrics
override this. In our example, it's/status/prometheus
.prometheus.io/port
: Scrape the pod on the indicated port instead of the default of9102
.
spec:
template:
metadata:
annotations:
prometheus.io/path: '/status/prometheus'
prometheus.io/port: '8080'
prometheus.io/scrape: 'true'
Adding Grafana to the cluster
To be able to access your Grafana dashboard outside the cluster, you should assign a NodePort to it.
kind: Service
apiVersion: v1
metadata:
name: grafana-service
namespace: my-namespace
labels:
k8s-app: grafana-service
spec:
ports:
- protocol: TCP
port: 3000
targetPort: 3000
nodePort: 3000
selector:
k8s-app: grafana-service
type: NodePort
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: grafana-service
namespace: my-namespace
labels:
k8s-app: grafana-service
spec:
replicas: 1
selector:
matchLabels:
k8s-app: grafana-service
template:
metadata:
name: grafana-service
labels:
k8s-app: grafana-service
spec:
volumes:
containers:
- name: grafana
image: grafana/grafana
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: GF_SECURITY_ADMIN_PASSWORD
value: D4vidH4ss3lh0ff1sC00L
- name: GF_SECURITY_ADMIN_USER
value: admin
- name: GF_USERS_ALLOW_SIGN_UP
value: "false"
restartPolicy: Always
dnsPolicy: ClusterFirst
Connecting Grafana to Prometheus
After you've applied the previous deployments, Grafana should be accessible at http://your-node.example.com:3000/login with the credentials we defined previously.
The next step is to configure Grafana to use Prometheus as a datasource. In the configuration wizard add the following datasource:
Picking a Grafana dashboard from the store
Grafana has a store, where premade dashboards are available. We recommend this one for Spring Boot applications.
Conclusion
In this tutorial, we explored how to scrape service discovery based application metrics from Spring Boot using Prometheus and Grafana inside a Kubernetes cluster. Congratulations! You have just set up a microservice monitoring infrastructure!
Follow us on social media