Financial Move Webhook Foundation

Overview

This project provides a serverless foundation on AWS for ingesting, validating, and routing webhook events from various financial partners, starting with Digital Guru.

The core architecture utilizes:

• API Gateway: Provides secure HTTP endpoints for webhook ingestion with custom domain support.
• AWS Lambda: Handles webhook requests, performs authentication, validates payloads, and publishes events.
• AWS Secrets Manager: Securely stores API tokens required for webhook authentication.
• AWS EventBridge: Uses a custom event bus (FMEvents) for domain-specific event routing based on event content.
• AWS CDK: Defines the infrastructure as code (IaC) for repeatable deployments.
• AWS Lambda Powertools for Python: Used for best practices in logging, metrics, and tracing within Lambda functions.

Features

• Scalable Webhook Ingestion: Handles incoming webhooks via API Gateway and Lambda.
• Secure Authentication: Validates incoming requests using API tokens stored securely in AWS Secrets Manager.
• Token Caching: Improves performance and reduces Secrets Manager costs by caching API tokens in memory.
• Event-Driven Routing: Publishes validated events to a custom EventBridge bus (FMEvents).
• Domain-Specific Rules: Configures EventBridge rules to route events based on source and type (e.g., Contact, Subscription, Transaction) to appropriate downstream targets (initially CloudWatch Logs).
• Infrastructure as Code: Easily deploy and manage the infrastructure using AWS CDK.
• Monitoring & Logging: Leverages Lambda Powertools and CloudWatch for observability.
• Extensible: Designed to easily add support for new webhook providers and event types.

Architecture

graph TD
    subgraph "Webhook Providers"
        DG[Digital Guru]
    end

    subgraph "AWS Cloud"
        APIGW[API Gateway<br>(Custom Domain)] --> LambdaWH{Webhook Lambda}
        LambdaWH -- Get Token --> SM[Secrets Manager]
        LambdaWH -- Publish Event --> EB[EventBridge Custom Bus<br>(FMEvents)]
        
        EB -- Route Rule --> TargetContact[Target: Contact]
        EB -- Route Rule --> TargetSub[Target: Subscription]
        EB -- Route Rule --> TargetTxn[Target: Transaction]
        EB -- Route Rule --> TargetProd[Target: Product]
        
        subgraph "Example Targets (Initially Logs)"
            TargetContact --> CWLogs[(CloudWatch Logs)]
            TargetSub --> CWLogs
            TargetTxn --> CWLogs
            TargetProd --> CWLogs
        end

        LambdaWH -- Logs/Metrics --> CW[CloudWatch]
        APIGW -- Logs/Metrics --> CW
        EB -- Logs --> CW
    end

    DG --> APIGW

Loading

1. Webhook Provider (Digital Guru): Sends HTTP POST requests to the API Gateway endpoint.
2. API Gateway: Receives the request, handles custom domain mapping, and forwards it to the Webhook Lambda.
3. Webhook Lambda:
   • Receives the request from API Gateway.
   • Extracts the API token from the X-API-Key header.
   • Retrieves the expected token from AWS Secrets Manager (using caching).
   • Validates the incoming token against the stored token.
   • If valid, parses the request body to determine the event type.
   • Constructs a domain-specific event.
   • Publishes the event to the custom EventBridge Bus (FMEvents).
4. EventBridge (FMEvents Bus):
   • Receives the event from the Lambda function.
   • Uses pre-configured rules based on source and detail-type to match the event.
   • Routes the event to the target(s) associated with the matching rule(s) (e.g., CloudWatch Logs, other Lambdas, SQS queues - initially CloudWatch Logs).
5. CloudWatch: Collects logs and metrics from API Gateway, Lambda, and EventBridge for monitoring and debugging.

Project Structure

fm-webhook-foundation/
├── cdk/                    # AWS CDK infrastructure code
│   ├── stacks/             # CDK Stack definitions (Webhook, Nested)
│   └── ...
├── src/                    # Lambda function source code
│   ├── domains/            # Domain-specific logic
│   │   └── webhook_ingestion/ # Webhook ingestion handlers
│   │       └── handlers/
│   ├── shared/             # Shared utilities and models
│   │   ├── auth.py         # Authentication logic (Secrets Manager)
│   │   └── models/         # Pydantic models for events
│   └── utils/            # General utilities
├── tests/                  # Automated tests
│   ├── unit/               # Unit tests (mocked dependencies)
│   └── integration/        # Integration tests (require deployed resources)
├── docs/                   # Project documentation
├── scripts/                # Helper scripts
├── lambda-layers/          # Pre-built Lambda layers (e.g., Powertools)
├── .env.example            # Example environment variables
├── app.py                  # CDK App entry point
├── cdk.json                # CDK configuration
├── requirements.txt        # Python dependencies for CDK
├── requirements-dev.txt    # Development/test dependencies
└── README.md               # This file

Getting Started

Prerequisites

• AWS Account and configured credentials
• Node.js and npm (for AWS CDK)
• Python 3.13+
• AWS CDK Toolkit (npm install -g aws-cdk)
• Docker (optional, for building Lambda layers locally)

Setup

1. Clone the repository:

   git clone <repository-url>
   cd fm-webhook-foundation

2. Set up Python virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt -r requirements-dev.txt 

3. Install Node.js dependencies (for CDK):

   npm install

4. Configure Environment Variables: Copy .env.example to .env and fill in the required values:

   • FM_DOMAIN: The domain name for the API Gateway (e.g., webhooks.dev.yourdomain.com).
   • API_TOKEN_SECRET_NAME: The name (or ARN) of the secret in AWS Secrets Manager containing the API token (e.g., fm-webhook-dev/api-token). The secret should be a JSON object like {"token": "your-secure-api-token"}.
   • AWS_REGION, AWS_ACCOUNT_ID (or configure via AWS CLI profiles).
   • Optional: TEST_API_URL, VALID_TEST_API_TOKEN for running integration tests.

5. Create API Token Secret: Ensure the secret specified by API_TOKEN_SECRET_NAME exists in AWS Secrets Manager in your target region. It must contain a JSON structure with a token key:

   {
     "token": "your-super-secret-api-key-here"
   }

Deployment

1. Bootstrap CDK (if first time in the region/account):

   cdk bootstrap aws://<ACCOUNT_ID>/<REGION>

2. Synthesize the CDK Stack:

   cdk synth -c env=<your_env_name> -c domain=<your_fm_domain>
   # Example: cdk synth -c env=dev -c domain=webhooks.dev.example.com

3. Deploy the Stack:

   cdk deploy -c env=<your_env_name> -c domain=<your_fm_domain> --require-approval never
   # Example: cdk deploy -c env=dev -c domain=webhooks.dev.example.com --require-approval never 

   • Replace <your_env_name> (e.g., dev, staging, prod) and <your_fm_domain>.
   • The -c flag passes context variables to the CDK app.

Testing

• Unit Tests:

  pytest tests/unit

• Integration Tests: (Require deployed stack and environment variables like TEST_API_URL, VALID_TEST_API_TOKEN)

  pytest tests/integration -m integration 

  • Make sure TEST_API_URL points to the deployed API Gateway URL (from cdk deploy output).
  • Make sure VALID_TEST_API_TOKEN matches the token stored in the deployed secret.

Monitoring

• CloudWatch Logs: Check logs for the API Gateway stage, Webhook Lambda function, and the FMEvents EventBridge log group (/aws/events/fmevents-<env_name>).
• CloudWatch Metrics: Monitor API Gateway metrics (latency, errors), Lambda metrics (invocations, errors, duration), and EventBridge metrics (invocation counts, failed invocations) under the FinancialMove/WebhookFoundation namespace (for Lambda) and AWS default namespaces.

Contributing

Please refer to CONTRIBUTING.md.

License

This project is licensed under the MIT License - see the LICENSE file for details.