Add rds multitenant module by iakov-gan · Pull Request #381 · aws-solutions-library-samples/cloud-intelligence-dashboards-data-collection

iakov-gan · 2025-08-08T06:01:19Z

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

- Update Athena tables to match actual S3 data partitioning structure - Add payer_id and region partition keys to support multi-account data - Configure Parquet SerDe for proper data format handling - Remove predefined Glue tables, let crawlers auto-discover schema - Add hourly_ prefix to separate crawler table creation - Update pi_data_view to reference hourly_rds_multitenant table - Update deployment templates to support RDS multitenant module

- Add engine detection to skip database dimension for Oracle/SQL Server - Collect db.load metrics by both user and database dimensions - Add dimension_type field to distinguish between user and database metrics - Fix Parquet field names to use underscores instead of dots for valid column names - Update Athena view to handle both dimension types with proper CASE statements - Fix view JOIN order to prevent row duplication - Maintain backward compatibility with existing data structure

- Add engine field to collected metrics data - Update process_metrics function to accept engine parameter - Include engine column in Athena view for filtering and analysis - Enable cost allocation analysis by database engine type

update docs

iakov-gan · 2025-08-08T06:06:31Z

+
+
+
+  GlueServiceRole:


We have a common glue role. Please use that.

iakov-gan · 2025-08-08T06:07:19Z

+                  - s3:PutObject
+                Resource: !Sub "${DestinationBucketARN}/*"
+
+  PerformanceInsightsRDSCrawler:


the patttern is to run crawler in step function that we have already. please check other modules

iakov-gan · 2025-08-08T06:09:05Z

+                  - !Sub "${DestinationBucketARN}"
+                  - !Sub "${DestinationBucketARN}/*"
+
+  CreateAthenaViewsLambda:


This make sense to create this on higher level so that all modules could run athena queries as custom resources.

iakov-gan · 2025-08-08T06:09:28Z

+                  'Data': response_data
+              }
+
+              http = urllib3.PoolManager()


use cfnrequest lib

iakov-gan · 2025-08-08T06:10:17Z

+
+
+
+  GlueDatabaseName:


do we need that?

iakov-gan · 2025-08-08T06:12:36Z

+                      # Convert to Arrow table
+                      table = pa.Table.from_pylist(grouped_metrics)
+
+                      # Write to Parquet format


do we need to write to parquert format? Why not just json? Surely not as performant but no dependency needed

iakov-gan · 2025-08-08T06:19:41Z

+            except Exception as e:
+                logger.error(f"Error processing DB instance {db_instance['DBInstanceIdentifier']}: {str(e)}")
+
+        # Handle pagination


you can make it simplier with simple paginators

for db_instance in rds_client.get_paginator('describe_db_instances').paginate().search('DBInstances'):

iakov-gan · 2025-08-08T06:28:00Z

+        'body': 'Athena views created successfully'
+    }
+
+def create_workgroup_if_not_exists(athena_client, workgroup_name):


i do not think we should create resources that are supported by CFN in lambda.

- Add CloudWatch Database Insights for multi-tenant cost allocation to main README - Update data-collection README with comprehensive RDS multitenant module documentation - Use correct CloudWatch Database Insights terminology (replacing deprecated Performance Insights naming) - Include module details, supported engines, key features, and prerequisites

- Add docstrings to all Lambda functions explaining their purpose - Document multi-tenant cost allocation logic and dimension types - Explain engine compatibility limitations for Oracle/SQL Server - Add CloudFormation template comments for key resources - Document S3 partitioning strategy and Parquet storage optimization - Improve code maintainability and understanding

- Update Lambda function to use REGIONS_IN_SCOPE environment variable - Fallback to all regions if parameter is empty - Improves performance by limiting data collection to specified regions

- Remove else branch that queries all AWS regions when RegionsInScope is empty - Function now processes empty list when no regions specified - Simplifies logic and prevents unintended region scanning

- Align version with original repository

- Add Glue table definition for hourly_rds_multitenant with correct schema - Add glue:GetDatabases permission to AthenaViewsLambdaRole - Add conditional view creation to handle missing table gracefully - Update table references from rds_multitenant to hourly_rds_multitenant - Include all required columns: engine, dimension_type, db_user_name, db_database_name

…module Enhances the existing RDS Multi-Tenant module with configurable Performance Insights dimensions and API pagination support for improved data collection flexibility and completeness. Key Enhancements: - Added configurable dimensions via DimensionsToTrack parameter (user, database, host, sql, waits) - Implemented pagination for Performance Insights API to handle >25 results per dimension - Uses tokenized SQL queries (db.sql_tokenized) for better query pattern aggregation - Engine-specific dimension filtering (e.g., database not supported on Oracle/SQL Server) - Default tracks all dimensions: user,database,host,sql,waits Changes: - data-collection/deploy/module-rds-multitenant.yaml: * Added DimensionsToTrack parameter with validation pattern * Implemented DIMENSION_CONFIG dictionary for dimension management * Added pagination loop in get_performance_metrics() to fetch all results * Updated dimension detection to support db.sql_tokenized fields * Added is_dimension_supported() function for engine compatibility checks - data-collection/deploy/deploy-data-collection.yaml: * Updated RDS multitenant module template URL reference - data-exports/README.md: * Added comprehensive beta deployment documentation * Step-by-step deployment guide with manual instructions * Data schema and cost allocation query examples * Engine compatibility matrix * Known limitations and troubleshooting guide - .gitignore: * Added patterns for test scripts and temporary files Technical Details: - Pagination ensures all dimension values are captured (not limited to 25) - Tokenized SQL groups similar queries for better analysis - Engine compatibility automatically enforced based on AWS documentation - Backward compatible: existing deployments continue to work

Moved the RDS Multi-Tenant module beta deployment documentation from data-exports/README.md to data-collection/README.md where it belongs, as it's a data collection module.

Restored data-exports/README.md to the original version from main branch, removing the RDS Multi-Tenant documentation that was incorrectly added there.

davidecoccia added 6 commits July 18, 2025 09:59

Add RDS multitenant module for cost allocation

9afe7cb

fix: Restore version to v3.12.1 in all deployment templates

16e324c

Merge pull request #1 from aws-solutions-library-samples/main

ba61e54

update docs

iakov-gan commented Aug 8, 2025

View reviewed changes

Comment thread data-collection/deploy/module-rds-multitenant.yaml

GlueDatabaseName:

iakov-gan Aug 8, 2025

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need that?

iakov-gan commented Aug 8, 2025

View reviewed changes

davidecoccia added 12 commits August 11, 2025 12:11

docs: Update data-collection README with additional changes

a7c3697

feat: respect RegionsInScope parameter in RDS multitenant module

05a3f46

- Update Lambda function to use REGIONS_IN_SCOPE environment variable - Fallback to all regions if parameter is empty - Improves performance by limiting data collection to specified regions

refactor: remove redundant fallback to all regions

7ebe561

- Remove else branch that queries all AWS regions when RegionsInScope is empty - Function now processes empty list when no regions specified - Simplifies logic and prevents unintended region scanning

chore: bump version to 3.12.1

0605780

- Align version with original repository

docs: Move RDS Multi-Tenant documentation to data-collection README

4ee0ac6

Moved the RDS Multi-Tenant module beta deployment documentation from data-exports/README.md to data-collection/README.md where it belongs, as it's a data collection module.

docs: Restore data-exports/README.md to main branch version

244e7c6

Restored data-exports/README.md to the original version from main branch, removing the RDS Multi-Tenant documentation that was incorrectly added there.

docs: Clarify deployment instructions and fix typos

007ae28

chore: updated readme

6671c85

Navigation Menu

Uh oh!

Add rds multitenant module - #381

Add rds multitenant module#381
iakov-gan wants to merge 18 commits into
aws-solutions-library-samples:mainfrom
davidecoccia:add-rds-multitenant-module

iakov-gan commented Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

iakov-gan Aug 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

iakov-gan commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants