Commit 731baa1
authored
feat: [PAYMCLOUD-541] Update ClouDO module, add runbooks, and improve AKS integrations (#3635)
* Add `.terraform.lock.hcl` file to version control to lock provider versions.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Initial setup for `ClouDO` infrastructure, including Terraform configurations, environment variables, and required modules.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Update Terraform module source and Azure provider version
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add new runbooks, update Terraform resources, and include encrypted secrets for ClouDO
- Added multiple runbook scripts for AKS scaling, rollouts, system checks, and App Gateway metrics.
- Updated Terraform modules and configurations, including new data sources and parameters.
- Included encrypted `cloudo-slack-token` and `opsgenie_token` secrets.
- Modified schemas and updated Docker image tags.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Update `aks-increate-max-keda-pod-scaling.sh` to remove unused arguments and simplify usage.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Adjust `aks-increate-max-keda-pod-scaling.sh` to conditionally decrement `maxReplicaCount` when `MONITOR_CONDITION` is resolved.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Ensure `maxReplicaCount` does not decrease below 1 when `MONITOR_CONDITION` is resolved.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Fix missing `fi` statement in `aks-increate-max-keda-pod-scaling.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Update Azure provider to 4.50.0, add new AKS scaling runbook, and enhance node pool scaling logic
- Upgraded `azurerm` provider in `.terraform.lock.hcl` to version `4.50.0`.
- Introduced `scale-pagopa-d-aks-user01-nodepool` runbook for managing DEV environment node pools.
- Enhanced `aks-scale-node-pool.sh` to handle autoscaling node pools.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Log current node pool mode in `aks-scale-node-pool.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Refactor `aks-scale-node-pool.sh` to enhance scaling logic and accommodate autoscaling adjustments.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Simplify `aks-scale-node-pool.sh` by removing
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Align variable names in `aks-scale-node-pool
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Fix typo in `--min-count` parameter for `aks-scale-node-pool.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add log to display min and max node pool values during scaling
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Enable `--update-cluster-autoscaler` flag in `aks-scale-node-pool.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add log message to indicate node pool scaling operation in `aks-scale-node-pool.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add missing log messages to indicate node pool scaling operations in `aks-scale-node-pool.sh`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add `aks-info.py` runbook to retrieve AKS namespace details and update DEV schemas configuration
- Introduced a new runbook `aks-info.py` for fetching namespace details, resource quotas, pod counts, and deployment status in AKS.
- Updated `schemas.json.tpl` to include the `aks-info-dev` runbook entry.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Refactor `aks-info.py` to improve command execution with `run_kubectl` helper and enhance error handling and logging.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Remove unnecessary print statements and unused namespace information parsing in `aks-info.py`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Remove unused resource quota handling in `aks-info.py`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Add return statement in `aks-info.py` to indicate successful execution.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* wip
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Update `aks-deployments-rollout.sh` and DEV schemas
- Add fallback for deployment name and monitor condition check in `aks-deployments-rollout.sh`.
- Introduce `restart-pod` runbook entry in DEV `schemas.json.tpl`.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* Introduce ClouDO container configuration variables
- Add `cloudo_orchestrator` and `cloudo_worker` variables for container image and registry configuration.
- Replace hardcoded values with variable references in Terraform definitions.
- Update `terraform.tfvars`, `01_cloudo.tf`, and `99_variables.tf` accordingly.
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update schemas and dependencies for ClouDO deployment adjustments.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* wip
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update ClouDO configuration with UI parameters, network data sources, and Terraform dependencies adjustments.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Remove unused App Service configuration parameters in ClouDO module.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update ClouDO module reference, remove unused UI tier, and add approval runbook configuration.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Add AKS pod crash analysis runbook, enable ClouDO UI, and update network data sources.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Add AKS runbooks for event retrieval, node status checks, pod logs, pod restarts, and enhance existing scripts with error handling.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Add AKS runbook to check pod CPU and Memory usage with threshold validations.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Enhance AKS resource usage runbook with fallback for missing metrics-server, detailed resource requests/limits, and improved validations.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Add runbooks for Azure Application Gateway health, Storage Account checks, and VPN Gateway connection status
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update Azure Application Gateway health runbook to support positional parameters for resource group and gateway name
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Replace AKS namespace info script with FDR-specific health check runbook to consolidate Pod status, log checks, and mitigation via PostgreSQL cache restart.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Enhance AKS runbooks with CPU/memory unit parsing functions for consistent resource conversion and fix node status output formatting.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Refactor AKS resource usage runbook to replace `bc` dependency with `awk` for CPU/memory unit conversion, improving portability and efficiency.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Enhance AKS resource usage runbook to support filtering by specific pod, improve error messaging, and refine CPU/memory usage handling.
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update ClouDO module references, refine Slack channel naming, add Google SSO integration, and improve schema structure/clarity. Upgrade dependencies (Terraform provider, SOPs).
Signed-off-by: ffppa <fabio.felici@pagopa.it>
* Update ClouDO module reference to latest commit hash
Signed-off-by: ffppa <fabio.felici@pagopa.it>
---------
Signed-off-by: Fabio Felici <fabio.felici@pagopa.it>
Signed-off-by: ffppa <fabio.felici@pagopa.it>1 parent 6c65d67 commit 731baa1
File tree
35 files changed
+1558
-25
lines changed- src
- cloudo
- env/dev
- runbooks
- aks
- azure
- fdr
- next-core-secrets/secret
- dev
- prod
- uat
35 files changed
+1558
-25
lines changedSome generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
Whitespace-only changes.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
0 commit comments