Documentation - SQS Segment Mapping Calculator (Consumer + Relation Processing)

[tentse]

Introduction

The SQS Segment Mapping Calculator is the core processing service in the pipeline. It consumes messages from the first SQS queue, computes related segment mappings using Neo4j (including graph traversal logic), stores results in PostgreSQL, updates job progress, and publishes completion messages to a second SQS queue.

This service performs the main computation step and supports asynchronous, scalable processing.

Task

• Service overview — role in pipeline and end-to-end processing responsibility
• SQS consumption flow — consumer setup, message parsing, and task execution flow
• Relation calculation logic — Neo4j traversal/BFS approach and segment mapping generation
• Database operations — storing segment_mapping, upsert/idempotency strategy, job status/progress updates
• Queue — message schema, field descriptions, and sample JSON
• Reliability concerns — retries, visibility timeout, multi-worker/concurrency notes
• Error handling, logging, config & libraries — operational safeguards and dependencies used

[karmatai]

@tentse needs to write more documentation segment mapping part.

[karmatai]

The task are done .@lobsam will be review the card.

[tentse]

Wiki documentation: https://github.com/OpenPecha/sqs-related-segment-calculator