Add AWS Lambda for upcoming data integration (ARCH-356) #72

tschaffter · 2024-12-03T22:27:46Z

Closes ARCH-356
Depends on #73

Changelog

Add AWS Lambda using image built in the monorepo.
Add role to execute the lambda.
Add EventBridge Scheduler that triggers the lambda every 5 minutes.

Preview

Test the lambda from the AWS Console:

Notes

The GH hosted runners should already have access to the docker command, which is now required to run cdk ....

README.md

tschaffter · 2024-12-03T22:33:41Z

Needs Docker

The current implementation needs Docker to be accessible inside the devcontainer (and the CI/CD workflow). I'm still looking for a way to avoid that requirements.

$ cdk deploy --all
openchallenges-dev-data-integration:  fail: Unable to execute 'docker' in order to build a container asset. Please install 'docker' and try again.

dpulls · 2024-12-04T23:31:18Z

🎉 All dependencies have been resolved !

.devcontainer/devcontainer.json

app.py

openchallenges/data_integration_lambda.py

.devcontainer/devcontainer.json

openchallenges/data_integration_stack.py

zaro0508

A few additional suggestions if you want do further refinements. Nice work converting to L2 construct, looks much nicer.

requirements.txt

openchallenges/data_integration_props.py

openchallenges/data_integration_lambda.py

openchallenges/data_integration_props.py

openchallenges/data_integration_stack.py

BryanFauble

Changes look good! I added a few comment around that can improve some areas, but nothing too significant.

* [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 * [pre-commit.ci] pre-commit autoupdate (#56) updates: - [github.com/awslabs/cfn-python-lint: v1.16.0 → v1.16.1](aws-cloudformation/cfn-lint@v1.16.0...v1.16.1) - [github.com/sirosen/check-jsonschema: 0.29.3 → 0.29.4](python-jsonschema/check-jsonschema@0.29.3...0.29.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#59) updates: - [github.com/awslabs/cfn-python-lint: v1.16.1 → v1.18.1](aws-cloudformation/cfn-lint@v1.16.1...v1.18.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#60) updates: - [github.com/awslabs/cfn-python-lint: v1.18.1 → v1.18.2](aws-cloudformation/cfn-lint@v1.18.1...v1.18.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Set concurrency to 5 (best results) (#61) * [pre-commit.ci] pre-commit autoupdate (#62) updates: - [github.com/awslabs/cfn-python-lint: v1.18.2 → v1.18.4](aws-cloudformation/cfn-lint@v1.18.2...v1.18.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update image tag, app version and data release date (#63) * [pre-commit.ci] pre-commit autoupdate (#66) updates: - [github.com/awslabs/cfn-python-lint: v1.18.4 → v1.19.0](aws-cloudformation/cfn-lint@v1.18.4...v1.19.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Refactor mounting volumes (#67) The current implementation to mount volumes was very specific to one container. We are replacing it with an implementation that is much more generic to make it easy to mount volumes in other containers. * [IT-4003] Auto-update pre-commit hook versions monthly Change the frequency that PRs to update pre-commit hook versions are auto-generated from weekly (the default) to monthly. * Update to OC v1.1.1 (#69) * Update to v1.1.1 * update data updated on * [pre-commit.ci] pre-commit autoupdate (#70) updates: - [github.com/awslabs/cfn-python-lint: v1.19.0 → v1.20.1](aws-cloudformation/cfn-lint@v1.19.0...v1.20.1) - [github.com/sirosen/check-jsonschema: 0.29.4 → 0.30.0](python-jsonschema/check-jsonschema@0.29.4...0.30.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3951] Fix guardduty container (#71) We enable guardduty security monitoring for ECS in every account. For that to work we need to give Fragate tasks access to do ECS stuff with the service-role/AmazonECSTaskExecutionRolePolicy[1]. [1] https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs * remove source.bat (#74) * Add Docker in Docker to the dev container (#73) * Add Docker in Docker to the devcontainer * add docs about docker * forward local environment variables to the devcontainer * remove containerEnv * Add AWS Lambda for upcoming data integration (ARCH-356) (#72) * update docs on setup tools * define lambda role and function * update path to Dockerfile * update README * trigger the lambda every 5 minutes * use plural form of the unit * Remove lambda fct architecture * Migrate data integration code to L2 constructs * Add @DataClass to DataIntegrationProps * Add docstrings * Replace `_lambda` by `lambda_` * Add docstrings * Add docstrings * Externalize the description of the schedule (#75) * define lambda role and function * update path to Dockerfile * Externalize the description of the schedule --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> Co-authored-by: Joni Harker <joni.harker@sagebase.org> Co-authored-by: Joni Harker <506966+ConsoleCatzirl@users.noreply.github.com>

* Update stage environment (#57) * [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> * Merge dev to stage (#76) * [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 * [pre-commit.ci] pre-commit autoupdate (#56) updates: - [github.com/awslabs/cfn-python-lint: v1.16.0 → v1.16.1](aws-cloudformation/cfn-lint@v1.16.0...v1.16.1) - [github.com/sirosen/check-jsonschema: 0.29.3 → 0.29.4](python-jsonschema/check-jsonschema@0.29.3...0.29.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#59) updates: - [github.com/awslabs/cfn-python-lint: v1.16.1 → v1.18.1](aws-cloudformation/cfn-lint@v1.16.1...v1.18.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#60) updates: - [github.com/awslabs/cfn-python-lint: v1.18.1 → v1.18.2](aws-cloudformation/cfn-lint@v1.18.1...v1.18.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Set concurrency to 5 (best results) (#61) * [pre-commit.ci] pre-commit autoupdate (#62) updates: - [github.com/awslabs/cfn-python-lint: v1.18.2 → v1.18.4](aws-cloudformation/cfn-lint@v1.18.2...v1.18.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update image tag, app version and data release date (#63) * [pre-commit.ci] pre-commit autoupdate (#66) updates: - [github.com/awslabs/cfn-python-lint: v1.18.4 → v1.19.0](aws-cloudformation/cfn-lint@v1.18.4...v1.19.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Refactor mounting volumes (#67) The current implementation to mount volumes was very specific to one container. We are replacing it with an implementation that is much more generic to make it easy to mount volumes in other containers. * [IT-4003] Auto-update pre-commit hook versions monthly Change the frequency that PRs to update pre-commit hook versions are auto-generated from weekly (the default) to monthly. * Update to OC v1.1.1 (#69) * Update to v1.1.1 * update data updated on * [pre-commit.ci] pre-commit autoupdate (#70) updates: - [github.com/awslabs/cfn-python-lint: v1.19.0 → v1.20.1](aws-cloudformation/cfn-lint@v1.19.0...v1.20.1) - [github.com/sirosen/check-jsonschema: 0.29.4 → 0.30.0](python-jsonschema/check-jsonschema@0.29.4...0.30.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3951] Fix guardduty container (#71) We enable guardduty security monitoring for ECS in every account. For that to work we need to give Fragate tasks access to do ECS stuff with the service-role/AmazonECSTaskExecutionRolePolicy[1]. [1] https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs * remove source.bat (#74) * Add Docker in Docker to the dev container (#73) * Add Docker in Docker to the devcontainer * add docs about docker * forward local environment variables to the devcontainer * remove containerEnv * Add AWS Lambda for upcoming data integration (ARCH-356) (#72) * update docs on setup tools * define lambda role and function * update path to Dockerfile * update README * trigger the lambda every 5 minutes * use plural form of the unit * Remove lambda fct architecture * Migrate data integration code to L2 constructs * Add @DataClass to DataIntegrationProps * Add docstrings * Replace `_lambda` by `lambda_` * Add docstrings * Add docstrings * Externalize the description of the schedule (#75) * define lambda role and function * update path to Dockerfile * Externalize the description of the schedule --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> Co-authored-by: Joni Harker <joni.harker@sagebase.org> Co-authored-by: Joni Harker <506966+ConsoleCatzirl@users.noreply.github.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> Co-authored-by: Joni Harker <joni.harker@sagebase.org> Co-authored-by: Joni Harker <506966+ConsoleCatzirl@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 * Update stage environment (#57) * [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#56) updates: - [github.com/awslabs/cfn-python-lint: v1.16.0 → v1.16.1](aws-cloudformation/cfn-lint@v1.16.0...v1.16.1) - [github.com/sirosen/check-jsonschema: 0.29.3 → 0.29.4](python-jsonschema/check-jsonschema@0.29.3...0.29.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#59) updates: - [github.com/awslabs/cfn-python-lint: v1.16.1 → v1.18.1](aws-cloudformation/cfn-lint@v1.16.1...v1.18.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#60) updates: - [github.com/awslabs/cfn-python-lint: v1.18.1 → v1.18.2](aws-cloudformation/cfn-lint@v1.18.1...v1.18.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Set concurrency to 5 (best results) (#61) * [pre-commit.ci] pre-commit autoupdate (#62) updates: - [github.com/awslabs/cfn-python-lint: v1.18.2 → v1.18.4](aws-cloudformation/cfn-lint@v1.18.2...v1.18.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update image tag, app version and data release date (#63) * [pre-commit.ci] pre-commit autoupdate (#66) updates: - [github.com/awslabs/cfn-python-lint: v1.18.4 → v1.19.0](aws-cloudformation/cfn-lint@v1.18.4...v1.19.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Refactor mounting volumes (#67) The current implementation to mount volumes was very specific to one container. We are replacing it with an implementation that is much more generic to make it easy to mount volumes in other containers. * [IT-4003] Auto-update pre-commit hook versions monthly Change the frequency that PRs to update pre-commit hook versions are auto-generated from weekly (the default) to monthly. * Update to OC v1.1.1 (#69) * Update to v1.1.1 * update data updated on * [pre-commit.ci] pre-commit autoupdate (#70) updates: - [github.com/awslabs/cfn-python-lint: v1.19.0 → v1.20.1](aws-cloudformation/cfn-lint@v1.19.0...v1.20.1) - [github.com/sirosen/check-jsonschema: 0.29.4 → 0.30.0](python-jsonschema/check-jsonschema@0.29.4...0.30.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3951] Fix guardduty container (#71) We enable guardduty security monitoring for ECS in every account. For that to work we need to give Fragate tasks access to do ECS stuff with the service-role/AmazonECSTaskExecutionRolePolicy[1]. [1] https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs * remove source.bat (#74) * Add Docker in Docker to the dev container (#73) * Add Docker in Docker to the devcontainer * add docs about docker * forward local environment variables to the devcontainer * remove containerEnv * Add AWS Lambda for upcoming data integration (ARCH-356) (#72) * update docs on setup tools * define lambda role and function * update path to Dockerfile * update README * trigger the lambda every 5 minutes * use plural form of the unit * Remove lambda fct architecture * Migrate data integration code to L2 constructs * Add @DataClass to DataIntegrationProps * Add docstrings * Replace `_lambda` by `lambda_` * Add docstrings * Add docstrings * Externalize the description of the schedule (#75) * define lambda role and function * update path to Dockerfile * Externalize the description of the schedule * Merge dev to stage (#76) * [pre-commit.ci] pre-commit autoupdate (#48) updates: - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) - [github.com/awslabs/cfn-python-lint: v1.9.0 → v1.15.0](aws-cloudformation/cfn-lint@v1.9.0...v1.15.0) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/sirosen/check-jsonschema: 0.29.1 → 0.29.2](python-jsonschema/check-jsonschema@0.29.1...0.29.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3918] Fix the image URLs returned by the image service (#51) * update the config of the image service * use `{fully_qualified_domain_name}` * use an f-string * [pre-commit.ci] pre-commit autoupdate (#52) updates: - [github.com/awslabs/cfn-python-lint: v1.15.0 → v1.15.2](aws-cloudformation/cfn-lint@v1.15.0...v1.15.2) - [github.com/sirosen/check-jsonschema: 0.29.2 → 0.29.3](python-jsonschema/check-jsonschema@0.29.2...0.29.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#53) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/awslabs/cfn-python-lint: v1.15.2 → v1.16.0](aws-cloudformation/cfn-lint@v1.15.2...v1.16.0) - [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Parametrize stack version and update app config (#54) * parametrize stack version * update data update date * set Google tag manager ID * rename `stack_version` to `image_version` * Increase GH workflow timeout (#55) A change[1] was made to update all containers at the same time which takes longer to deploy so we need to increase the deployment timeout. [1] #54 * [pre-commit.ci] pre-commit autoupdate (#56) updates: - [github.com/awslabs/cfn-python-lint: v1.16.0 → v1.16.1](aws-cloudformation/cfn-lint@v1.16.0...v1.16.1) - [github.com/sirosen/check-jsonschema: 0.29.3 → 0.29.4](python-jsonschema/check-jsonschema@0.29.3...0.29.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#59) updates: - [github.com/awslabs/cfn-python-lint: v1.16.1 → v1.18.1](aws-cloudformation/cfn-lint@v1.16.1...v1.18.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#60) updates: - [github.com/awslabs/cfn-python-lint: v1.18.1 → v1.18.2](aws-cloudformation/cfn-lint@v1.18.1...v1.18.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Set concurrency to 5 (best results) (#61) * [pre-commit.ci] pre-commit autoupdate (#62) updates: - [github.com/awslabs/cfn-python-lint: v1.18.2 → v1.18.4](aws-cloudformation/cfn-lint@v1.18.2...v1.18.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update image tag, app version and data release date (#63) * [pre-commit.ci] pre-commit autoupdate (#66) updates: - [github.com/awslabs/cfn-python-lint: v1.18.4 → v1.19.0](aws-cloudformation/cfn-lint@v1.18.4...v1.19.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Refactor mounting volumes (#67) The current implementation to mount volumes was very specific to one container. We are replacing it with an implementation that is much more generic to make it easy to mount volumes in other containers. * [IT-4003] Auto-update pre-commit hook versions monthly Change the frequency that PRs to update pre-commit hook versions are auto-generated from weekly (the default) to monthly. * Update to OC v1.1.1 (#69) * Update to v1.1.1 * update data updated on * [pre-commit.ci] pre-commit autoupdate (#70) updates: - [github.com/awslabs/cfn-python-lint: v1.19.0 → v1.20.1](aws-cloudformation/cfn-lint@v1.19.0...v1.20.1) - [github.com/sirosen/check-jsonschema: 0.29.4 → 0.30.0](python-jsonschema/check-jsonschema@0.29.4...0.30.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [IT-3951] Fix guardduty container (#71) We enable guardduty security monitoring for ECS in every account. For that to work we need to give Fragate tasks access to do ECS stuff with the service-role/AmazonECSTaskExecutionRolePolicy[1]. [1] https://docs.aws.amazon.com/guardduty/latest/ug/prereq-runtime-monitoring-ecs-support.html#before-enable-runtime-monitoring-ecs * remove source.bat (#74) * Add Docker in Docker to the dev container (#73) * Add Docker in Docker to the devcontainer * add docs about docker * forward local environment variables to the devcontainer * remove containerEnv * Add AWS Lambda for upcoming data integration (ARCH-356) (#72) * update docs on setup tools * define lambda role and function * update path to Dockerfile * update README * trigger the lambda every 5 minutes * use plural form of the unit * Remove lambda fct architecture * Migrate data integration code to L2 constructs * Add @DataClass to DataIntegrationProps * Add docstrings * Replace `_lambda` by `lambda_` * Add docstrings * Add docstrings * Externalize the description of the schedule (#75) * define lambda role and function * update path to Dockerfile * Externalize the description of the schedule --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> Co-authored-by: Joni Harker <joni.harker@sagebase.org> Co-authored-by: Joni Harker <506966+ConsoleCatzirl@users.noreply.github.com> * update dev (#78) * Increase role duration to 90 minutes (#79) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Khai Do <3697686+zaro0508@users.noreply.github.com> Co-authored-by: Joni Harker <joni.harker@sagebase.org> Co-authored-by: Joni Harker <506966+ConsoleCatzirl@users.noreply.github.com>

tschaffter self-assigned this Dec 3, 2024

tschaffter commented Dec 3, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

tschaffter changed the title ~~Add AWS Lambda for upcoming data integration~~ Add AWS Lambda for upcoming data integration (ARCH-356) Dec 3, 2024

tschaffter mentioned this pull request Dec 4, 2024

Add Docker in Docker to the dev container #73

Merged

tschaffter marked this pull request as ready for review December 4, 2024 23:37

tschaffter requested review from a team as code owners December 4, 2024 23:37

tschaffter commented Dec 4, 2024

View reviewed changes

.devcontainer/devcontainer.json Show resolved Hide resolved

zaro0508 requested changes Dec 5, 2024

View reviewed changes

tschaffter added 7 commits December 5, 2024 23:26

update docs on setup tools

de3e55a

define lambda role and function

87ea463

update path to Dockerfile

1db404d

update README

cc19dc4

trigger the lambda every 5 minutes

687ffc8

use plural form of the unit

4aa9b84

Remove lambda fct architecture

7d2765f

tschaffter force-pushed the create-lambda branch from abbf22d to 7d2765f Compare December 5, 2024 23:26

Migrate data integration code to L2 constructs

635444f

tschaffter marked this pull request as draft December 6, 2024 00:35

tschaffter marked this pull request as ready for review December 6, 2024 00:39

tschaffter requested a review from zaro0508 December 6, 2024 00:39

zaro0508 approved these changes Dec 6, 2024

View reviewed changes

requirements.txt Show resolved Hide resolved

openchallenges/data_integration_props.py Show resolved Hide resolved

Add @DataClass to DataIntegrationProps

a951e91