Merge pull request #562 from aws-solutions/release/v2.2.0

update to v2.2.0

Author: abewub

.gitignore

@@ -52,6 +52,7 @@ source/frontend/cypress/videos
 **/.nyc_output
 
 # build
+source/backend/functions/account-import-templates-api/src/global-resources.template
 source/backend/functions/lambda-layers/aws_sdk/python
 source/backend/functions/lambda-layers/cr_helper/python
 source/backend/functions/lambda-layers/decorators/python/*

CHANGELOG.md

@@ -5,11 +5,50 @@ All notable changes to this project are documented in this file.
 Based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2024-11-20
+
+### Added
+- Support for custom identity providers (SAML and OIDC) [61](https://github.com/aws-solutions/workload-discovery-on-aws/issues/61) [510](https://github.com/aws-solutions/workload-discovery-on-aws/issues/510)
+- Export of diagrams to myApplications
+- Export of diagrams to SVG
+- Application Insights dashboard to monitor health of solution
+- Number of supported resource types is over 450, newly added types include, but are not limited to:
+  - `AWS::AppSync::DataSource`
+  - `AWS::AppSync::Resolver`
+  - `AWS::MediaConnect::Flow`
+  - `AWS::MediaConnect::FlowEntitlement`
+  - `AWS::MediaConnect::FlowSource`
+  - `AWS::MediaConnect::FlowVpcInterface`
+  - `AWS::MediaPackage::PackagingConfiguration`
+  - `AWS::MediaPackage::PackagingGroup`
+  - `AWS::ServiceCatalogAppRegistry::Application`
+- Increase in number of relationships not captured by AWS Config, including, but not limited to:
+  - `AWS::AppSync::DataSource` -> `AWS::Lambda::Function`/`AWS::DynamoDB::Table`/`AWS::Events::EventBus`/`AWS::RDS::DBCluster`/`AWS::OpenSearchService::Domain`
+  - `AWS::AppSync::Resolver` -> `AWS::AppSync::GraphQLApi`/`AWS::AppSync::DataSource`
+  - `AWS::S3::Bucket` -> `AWS::Lambda::Function`/`AWS::SQS::Queue`/`AWS::SNS::Topic`
+  - `AWS::IAM::InstanceProfile` -> `AWS::IAM::Role`
+  - `AWS::MediaConnect::FlowEntitlement` -> `AWS::MediaConnect::Flow`
+  - `AWS::MediaConnect::FlowSource` -> `AWS::MediaConnect::Flow`/`AWS::MediaConnect::FlowEntitlement`/`AWS::MediaConnect::FlowVpcInterface`/`AWS::IAM::Role`/`AWS::SecretsManager::Secret`
+  - `AWS::MediaConnect::FlowVpcInterface` -> `AWS::EC2::VPC`/`AWS::EC2::Subnet`/`AWS::EC2::SecurityGroup`/`AWS::EC2::NetworkInterface`
+  - `AWS::MediaPackage::PackagingConfiguration` -> `AWS::MediaPackage::PackagingGroup`/`AWS::IAM::Role`
+
+### Changed
+- Multiple resources can now be selected in the search bar on the Diagram page
+- Migrate AppRegistry integration to myApplications for monitoring solution costs and usage.
+- Migrate JavaScript lambda functions and the discovery process from CommonJS to ESM
+- Update Neptune Engine version to `1.3.1.0`.
+- Update OpenSearch version to `2.11.0`.
+
+### Fixed
+- `AWS::EC2::NetworkInterface` -> `AWS::OpenSearchService::Domain` relationships are not discovered
+- Failed writes to OpenSearch should stop the corresponding write being made to Neptune
+- Metrics lambda times out when hundreds of accounts have been imported
+
 ## [2.1.15] - 2024-9-30
 
 ### Fixed
 
-- Cron expression for running discovery process every 24 hours. [546](https://github.com/aws-solutions/workload-discovery-on-aws/discussions/546)
+- Cron expression for running discovery process every 24 hours. [546](https://github.com/aws-solutions/workload-discovery-on-aws/issues/546)
 - Security [vulnerability](https://github.com/advisories/GHSA-gcx4-mw62-g8wm) in `rollup`.
 - Security [vulnerability](https://github.com/advisories/GHSA-9cwx-2883-4wfx) in `vite`.
 - Security [vulnerability](https://github.com/advisories/GHSA-64vr-g452-qvp3) in `vite`.
@@ -18,9 +57,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- Cron expression for running discovery process every 24 hours. [546](https://github.com/aws-solutions/workload-discovery-on-aws/discussions/546)
-- Intermittent failures in `cleanup-bucket` custom resource. [545](https://github.com/aws-solutions/workload-discovery-on-aws/discussions/545)
-- SCP error relating to `putConfigAggregator` when adding accounts in an AWS organisation using Control Tower. [544](https://github.com/aws-solutions/workload-discovery-on-aws/discussions/544)
+- Cron expression for running discovery process every 24 hours. [546](https://github.com/aws-solutions/workload-discovery-on-aws/issues/546)
+- Intermittent failures in `cleanup-bucket` custom resource. [545](https://github.com/aws-solutions/workload-discovery-on-aws/issues/545)
+- SCP error relating to `putConfigAggregator` when adding accounts in an AWS organisation using Control Tower. [544](https://github.com/aws-solutions/workload-discovery-on-aws/issues/544)
 - Security [vulnerability](https://github.com/advisories/GHSA-952p-6rrq-rcjv) in `micromatch`.
 - Security [vulnerability](https://github.com/advisories/GHSA-9wv6-86v2-598j) in `path-to-regexp`.
 - Security [vulnerability](https://github.com/advisories/GHSA-m6fv-jmcg-4jfg) in `send`.

NOTICE.txt

@@ -28,6 +28,8 @@ license-expression
 py-serializable
 @aws-amplify/ui-react
 @aws-sdk/client-api-gateway
+@aws-sdk/client-appsync
+@aws-sdk/client-athena
 @aws-sdk/client-cognito-identity-provider
 @aws-sdk/client-config-service
 @aws-sdk/client-dynamodb
@@ -39,9 +41,12 @@ py-serializable
 @aws-sdk/client-elastic-load-balancing-v2
 @aws-sdk/client-iam
 @aws-sdk/client-lambda
+@aws-sdk/client-mediaconnect
 @aws-sdk/client-opensearch
 @aws-sdk/client-organizations
+@aws-sdk/client-resource-groups-tagging-api
 @aws-sdk/client-s3
+@aws-sdk/client-service-catalog-appregistry
 @aws-sdk/client-sns
 @aws-sdk/client-ssm
 @aws-sdk/client-sts
@@ -58,7 +63,10 @@ py-serializable
 **********************
 MIT License
 **********************
+@aws-lambda-powertools/logger
+@uiw/react-json-view
 add
+ajv
 pip
 cfn-nag
 csv-parse
@@ -108,6 +116,8 @@ mocha
 rewire
 socks5-https-client
 moment-timezone
+memoize
+quick-lru
 unzipper
 mime-types
 node-fetch
@@ -116,7 +126,7 @@ undici
 acorn
 msw
 aws-opensearch-connector
-athena-express
+athena-express-plus
 most
 @tuplo/dynoexpr
 aws-sdk-client-mock
@@ -163,6 +173,7 @@ pkgutil_resolve_name
 pyrsistent
 @supercharge/promise-pool
 @most/core
+zod
 
 **********************
 BSD-2-Clause

README.md

@@ -1,4 +1,4 @@
-# Workload Discovery on AWS (v2.1.15)
+# Workload Discovery on AWS (v2.2.0)
 
 Workload Discovery on AWS is a tool that quickly visualizes AWS Cloud workloads as architecture diagrams. 
 You can use the solution to build, customize, and share detailed workload visualizations based on live data from AWS. 
@@ -70,7 +70,7 @@ The cost component processes [AWS Cost and Usage Reports](https://docs.aws.amazo
 (AWS CUR) to make cost data available in Workload Discovery. To use this feature, you must [create a report in AWS CUR](https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html) 
 to deliver the reports to the ```CostAndUsageReportBucket``` Amazon S3 bucket. When an AWS CUR is delivered, it 
 triggers an [AWS Lambda](http://aws.amazon.com/lambda) function to trigger a AWS Glue Crawler that will update a 
-table ready for Amazon Athena to query. You can query these AWS CURs via the Perspectie UI. You can bring in cost 
+table ready for Amazon Athena to query. You can query these AWS CURs via the Workload Discovery UI. You can bring in cost 
 data from other accounts discoverable to Workload Discovery by setting up a AWS CUR and setting up replication between 
 the S3 bucket in the discoverable account and the ```CostAndUsageReportBucket```
 

deployment/build-open-source-dist.sh

@@ -43,9 +43,8 @@ gitzip -d $dist_dir/$1.zip \
       -x "codescan-*.sh" \
       -x "buildspec.yml" \
       -x ".viperlight*" \
-      -x "source/infrastructure" \
-      -x "docs" \
+      -x "internal" \
       -x "sonar-project.properties" \
       -x "solution-manifest.yaml" \
-      -x ".nightswatch/*" \
+      -x ".nightswatch" \
       -x "Config"

deployment/build-s3-dist.sh

@@ -31,7 +31,7 @@ source_dir="$template_dir/../source"
 nested_stack_template_dir="$source_dir/cfn/templates"
 
 auditDeps () {
-   npm_config_yes=true npx better-npm-audit audit --production -l high
+   npm_config_yes=true npx better-npm-audit audit --production
    OUTPUT=$?
    if [[ "$OUTPUT" -eq 0 ]];
    then
@@ -62,6 +62,7 @@ echo "--------------------------------------------------------------------------
 cp "$nested_stack_template_dir"/*.template "$build_dist_dir"
 cd "$build_dist_dir"
 sedi "s|<BUCKET_NAME>|${1}|g; s|<SOLUTION_NAME>|${2}|g; s|<VERSION>|${3}|g; s|<IMAGE_VERSION>|${4}|g" main.template
+sedi "s|<VERSION>|${3}|g;" org-global-resources.template
 
 echo "------------------------------------------------------------------------------"
 echo "[Packing] Main Distribution Template"
@@ -118,6 +119,14 @@ rm -rf dist && mkdir dist
 zip -q -r9 dist/cleanup-ecr.zip cleanup_ecr.py
 cp ./dist/cleanup-ecr.zip "${build_dist_dir}/cleanup-ecr.zip"
 
+echo "------------------------------------------------------------------------------"
+echo "[Rebuild] Identity Provider Custom Resource"
+echo "------------------------------------------------------------------------------"
+cd "${source_dir}/backend/functions/identity-provider"
+rm -rf dist && mkdir dist
+zip -q -r9 dist/identity-provider.zip identity_provider.py
+cp ./dist/identity-provider.zip "${build_dist_dir}/identity-provider.zip"
+
 echo "------------------------------------------------------------------------------"
 echo "[Rebuild] Drawio Lambda"
 echo "------------------------------------------------------------------------------"
@@ -191,6 +200,15 @@ auditDeps
 npm run build
 cp ./dist/cur-setup.zip "${build_dist_dir}/cur-setup.zip"
 
+
+echo "------------------------------------------------------------------------------"
+echo "[Rebuild] Metrics Uuid Custom Resource"
+echo "------------------------------------------------------------------------------"
+cd "${source_dir}/backend/functions/metrics-uuid"
+rm -rf dist && mkdir dist
+zip -q -r9 dist/metrics_uuid.zip metrics_uuid.py
+cp ./dist/metrics_uuid.zip "${build_dist_dir}/metrics_uuid.zip"
+
 echo "------------------------------------------------------------------------------"
 echo "[Rebuild] Metrics"
 echo "------------------------------------------------------------------------------"
@@ -199,6 +217,22 @@ auditDeps
 npm run build
 cp ./dist/metrics.zip "${build_dist_dir}/metrics.zip"
 
+echo "------------------------------------------------------------------------------"
+echo "[Rebuild] Metrics Subscription Filter"
+echo "------------------------------------------------------------------------------"
+cd "${source_dir}/backend/functions/metrics-subscription-filter"
+auditDeps
+npm run build
+cp ./dist/metrics-subscription-filter.zip "${build_dist_dir}/metrics-subscription-filter.zip"
+
+echo "------------------------------------------------------------------------------"
+echo "[Rebuild] Export to myApplication"
+echo "------------------------------------------------------------------------------"
+cd "${source_dir}/backend/functions/myapplications"
+auditDeps
+npm run build
+cp ./dist/myapplications.zip "${build_dist_dir}/myapplications.zip"
+
 echo "------------------------------------------------------------------------------"
 echo "[Rebuild] Discovery"
 echo "------------------------------------------------------------------------------"

deployment/run-unit-tests.sh

@@ -32,6 +32,14 @@ pipenv install -q --dev
 pipenv run pytest --cov-report xml --cov .
 echo "$(awk '{gsub(/<source>.*\/source\//, "<source>source/")}1' coverage.xml)" > coverage.xml
 
+echo "------------------------------------------------------------------------------"
+echo "[Test] Identity Provider Custom Resource"
+echo "------------------------------------------------------------------------------"
+cd $source_dir/backend/functions/identity-provider
+pipenv install --dev
+pipenv run pytest --cov-report xml --cov .
+echo "$(awk '{gsub(/<source>.*\/source\//, "<source>source/")}1' coverage.xml)" > coverage.xml
+
 echo "------------------------------------------------------------------------------"
 echo "[Test] Account Import Templates"
 echo "------------------------------------------------------------------------------"
@@ -50,13 +58,34 @@ echo "--------------------------------------------------------------------------
 cd $source_dir/backend/functions/metrics
 npm run test:ci
 
+echo "------------------------------------------------------------------------------"
 echo "[Test] Bucket cleanup Custom Resource"
 echo "------------------------------------------------------------------------------"
 cd $source_dir/backend/functions/cleanup-bucket
 pipenv install -q --dev
 pipenv run pytest --cov-report xml --cov .
 echo "$(awk '{gsub(/<source>.*\/source\//, "<source>source/")}1' coverage.xml)" > coverage.xml
 
+echo "------------------------------------------------------------------------------"
+echo "[Test] Metrics UUID Custom Resource"
+echo "------------------------------------------------------------------------------"
+cd $source_dir/backend/functions/metrics-uuid
+pipenv install -q --dev
+pipenv run pytest --cov-report xml --cov .
+echo "$(awk '{gsub(/<source>.*\/source\//, "<source>source/")}1' coverage.xml)" > coverage.xml
+
+echo "------------------------------------------------------------------------------"
+echo "[Test] Metrics Subscription Filter"
+echo "------------------------------------------------------------------------------"
+cd $source_dir/backend/functions/metrics-subscription-filter
+npm run test:ci
+
+echo "------------------------------------------------------------------------------"
+echo "[Test] myApplications Resolver"
+echo "------------------------------------------------------------------------------"
+cd $source_dir/backend/functions/myapplications
+npm run test:ci
+
 echo "------------------------------------------------------------------------------"
 echo "[Test] Discovery"
 echo "------------------------------------------------------------------------------"

source/backend/discovery/Dockerfile

@@ -1,4 +1,4 @@
-FROM public.ecr.aws/amazonlinux/amazonlinux:2023
+FROM public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
 
 RUN dnf install -y shadow-utils
 
@@ -18,4 +18,4 @@ RUN npm ci --omit=dev
 
 COPY src/ src/
 
-CMD ["node", "src/index.js"]
+CMD ["node", "src/index.mjs"]