http-crawler-spring-boot

Welcome to the Http Crawler Spring boot repository.

This repository holds a Java Springboot program that makes it easy to crawl two HTTP endpoints at the moment. It offers easy way to build and deploy the Springboot program on Kubernetes and monitor Prometheus format metrics (emitted by Springboot application) for application scaling and observability purposes.

What can I do with http-crawler-spring-boot repository?

You can use this repository as a sample design to run a Spring application on Kubernetes Container based environment with advanced usescases and best practices. Below are few features this repository offers,

• Simple application written in Springboot that queries 2 URL's (https://httpstat.us/503 & https://httpstat.us/200) every one second.
• Checks if external URL's are up (based on http status code 200) and returns response code.
• Integrated with micrometer to emit Prometheus format metrics on path /actuator/prometheus using Prometheus Java library.

Getting Started

prerequisites

• Docker
• Make
• Kubernetes Cluster
• kubectl
• Helm version 3
• Java 1.8 (only if Docker is not installed)

Run http-crawler-spring-boot on local machine

make docker-run

# OR use below command if Docker is not installed, Java is configured

make run

Deploy http-crawler-spring-boot to Kubernetes Cluster

• Configure kubectl configuration pointing to a Kubernetes Cluster

• Deploy http-crawler-spring-boot using make

  make k8s-deploy
  

Setup Infrastructure

• Create a Kubernetes Cluster

• Install Prometheus

  kubectl create namespace prometheus
  
  helm install prometheus prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2" \
  --set server.persistentVolume.storageClass="gp2"
  
  kubectl port-forward -n prometheus deploy/prometheus-server 8080:9090
  

• Install Grafana

  mkdir ${HOME}/environment/grafana
  
  cat << EoF > ${HOME}/environment/grafana/grafana.yaml
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.prometheus.svc.cluster.local
        access: proxy
        isDefault: true
  EoF
  
  kubectl create namespace grafana
  
  helm install grafana grafana/grafana \
      --namespace grafana \
      --set persistence.storageClassName="gp2" \
      --set persistence.enabled=true \
      --set adminPassword='EKS!sAWSome' \
      --values ${HOME}/environment/grafana/grafana.yaml \
      --set service.type=LoadBalancer
  
  echo "Sleep 180 seconds and wait for Load Balancer URL"
  sleep 180
  export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
  
  echo "http://$ELB"
  
  kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
  

• Deploy http-crawler-spring-boot on Kubernetes

  kubectl apply -f k8s-deployment.yml
  
  #OR
  
  make k8s-deploy
  

---

Questions or Something not working?

Please feel free to raise a GitHub issue