mq-to-db (message queue to database)

This is a Golang (go) program to read a specific JSON Payload message from a Message Queue System and Store into Database using concurrency

This is a close image of how it works:

Consumers supported

• RabbitMQ

Storage supported

• PostgreSQL

Characteristics

• The number of queue consumers could be different from the numbers of storage workers, see config-sample.yaml
• The process (job) of consuming one message from the queue and store into the database is synchronous because every message needs to be acknowledged (confirmed as stored).
• Golang pprof enabled via --profile command line when starting the service
• Prometheus metrics for consumers, storage workers, go statistics, and database
• Grafana dashboard for Prometheus.io metrics
• Dockerfile multi-stage build
• Makefile to facilitate the project builds
• docker-compose file and configuration to testing all elements
• docker images at docker-hub and Github Packages
• CI/CD Github Action pipeline workflow

How to execute

There are many ways to do it, but always is necessary PostgreSQL and RabbitMQ dependencies, the easy way to see how it works is using containers.

docker-compose

The program and all dependencies and visibility systems at once

Up

docker-compose up --build

Down

docker-compose down -v

Available links

After docker-compose start all the services, you have the following links ready to be curious, I prepared a simple grafana dashboard to show you part of the prometheus metrics implemented

• mq-to-db-01 home page
• mq-to-db-02 home page
• Prometheus Dashboard
• Grafana Dashboard
• RabbitMQ Dashboard

Profiling

# terminal 1, for mq-to-db-01 inside the docker-compose-file
go tool pprof http://127.0.0.1:8080/debug/pprof/goroutine

# terminal 2, for mq-to-db-02 inside the docker-compose-file
go tool pprof http://127.0.0.1:8081/debug/pprof/goroutine

# once you are into tool pprof, execute the command web
(pprof) web

See the Logs

docker-compose logs mq-to-db-01
docker-compose logs mq-to-db-02

Manually

First install dependencies

RabbitMQ

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

NOTES:

• RabbitMQ web console: http://localhost:15672
• Username: guest
• Password: guest

PostgreSQL

docker run --rm  --name postgresql -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres

# logs
docker logs postgresql -f

# remember to  stop and remove (--rm in docker run do it for you)
docker stop postgresql

Using source code

git clone https://github.com/christiangda/mq-to-db.git
cd mq-to-db/
go run -race ./cmd/mq-to-db/main.go --help

# and then
go run -race ./cmd/mq-to-db/main.go --configFile config-sample.yaml

NOTE: the parameter -raceis to check race conditions because we are using Go Concurrency

How to build

compiling for your ARCH and OS

# make executable first
make

# check the available options
./build/mq-to-db --help

# execute
./build/mq-to-db --configFile config-sample.yaml

Cross-compiling

make build-dist

# cross-compiling files
ls -l ./dist/

NOTES related to make

• 1. This create a cross-compiling binaries and also Docker Image (linux 64bits)
• 2. Check the Makefile to see The make available targets options
• 3. Remember to start dependencies first

Using docker image

Here I use latest tag, but you can see all releases here

# pull the image first
docker pull christiangda/mq-to-db:latest

# see available option
docker run --rm --name mq-to-db christiangda/mq-to-db:latest - --help

# run with a config file mapped and with profile option
docker run --rm -v <path to config file>:/etc/mq-to-db/config.yaml --name mq-to-db christiangda/mq-to-db:latest - --profile

NOTES:

• Remember to start dependencies first

Available endpoints

The application expose different endpoints via http server

• http://localhost:8080/
• http://localhost:8080/metrics
• http://localhost:8080/health
• http://localhost:8080/debug/pprof

References

Internals

• Configuration
• Messages Type

Externals

• references

License

This module is released under the GNU General Public License Version 3:

• http://www.gnu.org/licenses/gpl-3.0-standalone.html

Author Information

• Christian González Di Antonio