Skip to content

Commit 97c3dce

Browse files
bonzofenixgithub-advanced-security[bot]github-actions[bot]
authored
(feature): Enable operator in mtar deployment (#3717)
* Refactor configureDb function to accept a pointer to the database config map in metricsforwarder config * Refactor database configuration in autoscaler API and metrics forwarder • Centralize database configuration logic into db.ConfigureDb and db.ConfigureStoredProcedureDb • Remove individual configurePolicyDb and configureBindingDb functions • Move DatabaseConfig struct to a new models.go file in the db package • Update calls to database configuration in both API and metrics forwarder to use the new centralized functions * Refactor database configuration in autoscaler API and metrics forwarder • Change db.ConfigureDb calls to vcapReader.ConfigureDb for PolicyDb and BindingDb. • Introduce vcapReader.ConfigureStoredProcedureDb for handling stored procedure database configuration. • Remove ConfigureStoredProcedureDb and ConfigureDb from db/helper.go. • Update tests to reflect changes in database configuration methods. * Remove debug print statements and enable ConfigureStoredProcedureDb test in autoscaler config utils * Add time import and DatabaseConfig struct to db.go; remove models.go - Import the "time" package in db.go for time.Duration usage. - Introduce DatabaseConfig struct in db.go with connection parameters. - Delete models.go, moving DatabaseConfig to db.go. * Add stored procedure support to autoscaler API config • Implement configuration loading for stored procedure database in config.go • Enable tests for stored procedure database when cred_helper_impl is set to default in config_test.go * Potential fix for code scanning alert no. 431: Potentially unsafe quoting Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Fix lints * Increase instances for publicapiserver and metricsforwarder, add eventgenerator module with configuration and routes, and set default instances to 0 for new modules in autoscaler app. * Implement LoadConfig function with VCAP support and error handling in eventgenerator • Refactor LoadConfig to read from file and VCAP services • Add error variables for configuration loading issues • Introduce loadYamlFile and loadVcapConfig helper functions • Update tests with new configuration loading logic and VCAP reader mocks • Remove unused imports and adjust defaultConfig function signature * Rename ConfigureDb to ConfigureDbInMap * WIP * Refactor tests and update stored procedure DB handling - Correct grammar in test descriptions for database configuration calls. - Change cred_helper_impl to 'stored_procedure' for storedProcedure_db service. - Remove redundant storedProcedureConfig and read directly from VCAP services. - Remove outdated tests related to default cred_helper_impl for storedProcedure_db. * Implement LoadConfig function with VCAP support and error handling in eventgenerator • Refactor LoadConfig to read from file and VCAP services • Add error variables for configuration loading issues • Introduce loadYamlFile and loadVcapConfig helper functions • Update tests with new configuration loading logic and VCAP reader mocks • Remove unused imports and adjust defaultConfig function signature * Add ConfigureDb method to VCAPConfiguration and integrate with eventgenerator config • Implement ConfigureDb method in VCAPConfiguration to set database URL • Use ConfigureDb to configure PolicyDb and AppMetricsDb in eventgenerator • Add AppMetricsDb constant to db package • Update tests to reflect changes in database configuration methods * Refactor DBConfig and DatabaseConfig references to pointer types in eventgenerator • Update DBConfig and DatabaseConfig struct references to use pointers in eventgenerator_suite_test.go and main.go. • Change struct field names from DB to Db and PolicyDB to PolicyDb, AppMetricDB to AppMetricDb to align with Go naming conventions. * Initialize default database configurations for PolicyDb and AppMetricDb in eventgenerator * fix tests * Fix tests * Fix lints * Add configutil and new vendor dependencies to eventgenerator spec * Update environment variable names to uppercase in mta.tpl.yaml for autoscaler module * Refactor eventgenerator configuration and tests for CF environment - Update eventgenerator configuration to use pointers for optional structs and add JSON tags - Modify eventgenerator test suite to handle CF environment variables and dynamically set config - Adjust eventgenerator main.go to load and validate config with VCAP services support - Implement ToJSON method in config.go for marshaling configuration to JSON - Change environment variable keys in mta.tpl.yaml to uppercase and use placeholders for Go version * Add eventgenerator health password and logcache-client TLS config • Introduce eventgenerator_health_password in build-extension-file.sh for event generator health checks. • Add eventgenerator-config resource with basic auth password in mta.tpl.yaml. • Implement configureMetricsCollectorTLS function in eventgenerator/config/config.go to load TLS configuration for the metrics collector from logcache-client. • Update eventgenerator unit tests to reflect new TLS configuration loading. • Remove scaling engine and metric collector TLS configs from default_config.json, streamline cf_server and db configurations. • Add logcache-client as a user-provided service in mta.tpl.yaml. * Refactor event generator configuration to use node count instead of node addresses * Remove metricsforwarder and eventgenerator from instance groups in use-cf-services.yml * Fix mtar configuration * Add comments to NodeCount and NodeIndex in eventgenerator config * Enable parallel deployments in autoscaler MTA configuration * Add pool configuration and refactor eventgenerator instance management - Introduce pool configuration to manage eventgenerator node count and index - Remove commented-out XFCC and route registration configurations - Implement instance index and space/org GUID retrieval in configutil - Adjust eventgenerator main.go to use new pool configuration - Update tests and default JSON config to reflect changes * Fix integration * Fix linting * Refactor eventgenerator tests to use testhelpers for client creation and XFCC header setting * Remove eventgenerator configuration from cf-services * Fix metric poller test * Fix test * Add metric_collector_url configuration to eventgenerator-config in autoscaler extension build script * Change metric collector URL to use non-HTTPS scheme in default_config.json * Updated metricCollector configuration to include metric_collector_url in autoscaler extension build script. * Fix tests * Potential fix for code scanning alert no. 432: Potentially unsafe quoting Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * WIP * Fix eventgenerator deployments * Reduce instance count for publicapiserver module from 2 to 1 in autoscaler build script * Remove syslog-client and logcache-client dependencies from autoscaler components • Eliminate syslog-client requirement from metricsforwarder module and related resources • Remove logcache-client dependency from eventgenerator and metricsforwarder modules • Delete configuration code for Metrics Collector TLS setup in eventgenerator * Fix build * Refactor test helper functions for VCAP services in autoscaler • Consolidate GetDbVcapServices functions to accept a slice of database names. • Rename serviceName parameter to databaseName in GetStoredProcedureDbVcapServices for clarity. • Remove unused GetDbVcapServices overload with serviceName and dbType parameters. * Refactor EventGenerator config to use map for DB configs in components_test * Fix merge typos * Add syslog-client dependency to metricsforwarder module and define new syslog-client resource in mta.tpl.yaml * Refactor db structure for eventgenerator config * Remove empty line from ExpectConfigureDatabasesCalledOnce in testhelpers/db.go * Update CircuitBreakerConfig YAML tag and add GinkgoHelper to event generator in tests * Standardize 'dbUri' parameter to 'dbURI' in autoscaler integration tests configs * Refactor postgresDB service URI handling in configutil tests • Simplify postgresDB service URI test cases • Remove mysqlDB related tests • Consolidate AfterEach cleanup for temporary certificate files * Disable CF server tests and add TODO for log-cache UAA client connection in eventgenerator tests * Reduce instances for eventgenerator and metricsforwarder to 1 * Update event generator host configuration and routes in autoscaler build script * Update autoscaling API endpoint in cf-login script to use DEPLOYMENT_NAME * Rename VCAPServer to CFServer in Autoscaler API configuration and references * Refactor health check endpoint constants in broker server and routes * Refactor event generator routing and disable a test for Postgres config loading - Change `CreateEventGeneratorRoutes` to `CreateEventGeneratorSubrouter` for better clarity. - Disable the test case for loading the database configuration from VCAP_SERVICES for Postgres. - Rename `httpServer` to `eventgeneratorServer` to reflect its specific use. - Update `Server` struct to include `autoscalerRouter`. - Modify `createEventGeneratorRouter` to `createEventGeneratorRoutes` and adjust its implementation. - Fix `config_test.go` to correctly assign empty `DatabaseConfig` instead of clearing URL. - Remove redundant `createEventGeneratorRouter` and integrate health server setup into `CreateCFServer` and `CreateMtlsServer`. * Add support for MySQL in autoscaler Makefile and make DB type configurable * Update HttpClientTimeout default value and YAML tag in autoscaler config * Fix test * Remove tests for wrong method on aggregated metrics history endpoint * Remove empty line from CreateMtlsServer function in eventgenerator server. * Improve output for cf loging * Refactor deployment naming and add ENABLE_MTAR flag handling in vars.source.sh * Refactor Makefile for acceptance tests to use hardcoded config path • Removed ACCEPTANCE_CONFIG_PATH variable definition • Updated acceptance-tests-config target to use a hardcoded config path directly in the script call * Remove unnecessary ACCEPTANCE_CONFIG_PATH dependency from acceptance-tests-config target in Makefile * Update default DEPLOYMENT_NAME in vars.source.sh to use 'autoscaler-mta' instead of 'autoscaler-mtar' * Refactor health endpoint and update server routing in eventgenerator - Remove health endpoint from eventgenerator routes - Add health and v1 prefix handling to autoscalerRouter in CFServer creation - Update server tests to reflect routing changes and add health endpoint authentication tests * Adds uaac login command * Update eventgenerator config to separate CFServer port from Server port * Update eventgenerator and metricsforwarder UAA client config - Remove metricsforwarder UAA client ID and secret - Add eventgenerator log cache UAA client ID and secret - Adjust resource UAA client config to use new eventgenerator credentials * Add environment-based TLS configuration for Scaling Engine in eventgenerator * Refactor GetOrgGuid function to streamline JSON unmarshalling and error handling * Replace custom contains function with slices.Contains from the standard library in cf.go * Remove Klauspost compression dependencies from eventgenerator and metricsforwarder specs * Refactor TLS certificate configuration for autoscaler components • Remove direct os package usage for TLS certificate configuration in autoscaler components. • Centralize TLS certificate retrieval using GetInstanceTLSCerts method from VCAPConfigurationReader. • Update tests to reflect changes in TLS certificate configuration and remove redundant environment variable settings. * Remove VCAP_SERVICES logcache credentials loading test from eventgenerator config tests * Add cpuinfo internal package to metricsforwarder autoscaler dependencies * WIP: Refactor config test * Improve logging message for ignored scaling when instance count is correct * Add default instance configuration for EVENTGENERATOR and align instance count with EVENTGENERATOR_INSTANCES variable T * Refactor test" * Add GinkgoHelper call to RunningInstances function in acceptance tests * Set default server port to 0 and enable plain text logging in VCAP config - Initialize Server.Port to 0 in both API and EventGenerator VCAP configurations - Enable plain text logging for EventGenerator when running on CF - Update tests to reflect new default values and logging configuration * Refactor configureStoredProcedureDb to private method and update tests • Change ConfigureStoredProcedureDb to unexported configureStoredProcedureDb. • Remove tests for ConfigureStoredProcedureDb as the method is now private. • Adjust ConfigureDatabases tests to account for changes in stored procedure configuration. • Remove outdated tests related to storedProcedure_db service when cred_helper_impl is stored_procedure or default. • Update test helpers to use any type alias for JSON unmarshalling. * Remove redundant ServerConfig type from eventgenerator config * Remove ConfigureStoredProcedureDb method from VCAPConfigurationReader interface in cf.go * Fix flaky test * Removes unecessary logout * Rename node_count and node_index to total_instances and instance_index * Refactor: simplify NewAppManager signature * Remove unused fixture * Remove instances from mta tpl * Remove unused method * Set BOSH server port to 0 to avoid conflicts and enable plain text logging in autoscaler configs * Makes ConfigureDb a a private method * Fix issue in eventgenerator main cmd * Remove deployment name from github actions to resolve through ENABLE_MTAR flag * Updated bash shebang in common.sh and modified package versions in devbox.lock * Update devbox.lock Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix deployment issue with github actions * Update acceptance test workflows to conditionally deploy apps and require deploy_apps parameter • Add deploy_apps input with default false to acceptance_tests_broker.yaml • Make deploy_apps input required in acceptance_tests_reusable.yaml • Use conditional logic to set ENABLE_MTAR environment variable based on deploy_apps input • Include environment variables output before running deploy-register-cf command in acceptance_tests_reusable.yaml * Adds debugging to github actions * Update GitHub Actions to dynamically set deployment names based on PR number - Add `deployment_name` input to acceptance_tests_reusable workflow for dynamic naming - Set `DEPLOYMENT_NAME` env variable in acceptance_tests_broker, acceptance_tests_broker_close, acceptance_tests_mta, and acceptance_tests_mta_close workflows using PR number - Remove commented code and unused `DEBUG` env variable in acceptance_tests_reusable workflow - Refactor `ENABLE_MTAR` to be set based on `deploy_apps` input in acceptance_tests_reusable workflow * Rename appManager files and refactor test policy string in tests - Rename appManager.go to app_manager.go - Rename appManager_test.go to app_manager_test.go - Replace inline policy string with const testPolicyStr in tests - Introduce newAppMetric helper function to simplify metric creation in tests * Refactor app_manager_test.go by moving "Save and query metrics" tests - Relocate "Save and query metrics" context block to a different section within app_manager_test.go - Maintain logical test structure and ensure all tests remain unchanged * Refactor variable names and improve health check authentication in autoscaler - Rename `cfg` to `conf` for consistency across test suites - Update HTTP client variable names for clarity in `api_test.go` - Add `CheckHealthAuth` helper function to streamline health endpoint authentication tests - Utilize `CheckHealthAuth` in `eventgenerator_test.go` for cleaner authentication checks - Remove redundant `BeforeEach` blocks and direct `runner.Start()` calls in favor of `JustBeforeEach` - Adjust health metric string checks to use more generic substrings * Refactor app_manager_test.go to use Gomega's HaveExactKeys matcher and When blocks - Replace Context blocks with When for better readability - Utilize HaveExactKeys custom matcher to simplify policy retrieval assertions - Remove redundant clock increment and policy existence checks * Remove autoscaling API setup from os-infrastructure-login script * Add deferred clt push osing of response body in health endpoint test helper * Remove autoscaling-api setup from cf_login in common.sh * Remove eventgenerator-config and database dependencies from metricsforwarder module in build-extension-file.sh * Add nixpkgs-unstable source and update plugin versions for mysql and python * Disable shellcheck SC2154 warning in common.sh for autoscaler scripts * Optimize TLS certificate retrieval in loadVcapConfig function * Bump module instances * Set default instances parameter to 0 in autoscaler module parameters, to be overridden in build script * Refactor ToJSON function from config to configutil module • Remove ToJSON method from Config structs in api and eventgenerator modules • Add generic ToJSON function in configutil module to handle JSON marshaling • Update tests and usages to call configutil.ToJSON instead of Config's ToJSON • Remove redundant tests for ToJSON in api/config after refactoring * Refactor eventgenerator config loading and update VCAP_SERVICES structure • Replace interface{} with any in VCAP_SERVICES map • Remove redundant logcache-client TODO comment • Move YAML file loading to helpers.LoadYamlFile • Remove unused os import from config.go * Removed eventgenerator cmd test when running in CF We should reframe from adding complexity/tests to the cmd and relay on integration and acceptance instead. main.go should have as less logic as possible. * Ensure proper handling of script arguments by quoting "$@" * Added sourcing of vars.source.sh and updated bosh_login call with BBL_STATE_PATH in uaa-login.sh * Remove unused vcapPort variable from eventgenerator tests * Refactor error handling to use shared ErrReadYaml from helpers in eventgenerator config. * Remove unnecessary newline in LoadConfig function of eventgenerator config * Wip * Wip * Add configutil and new vendor dependencies, refactor operator config • Introduce autoscaler/configutil/* for configuration utilities. • Add new vendor dependencies: github.com/cloud-gov/go-cfenv and github.com/mitchellh/mapstructure. • Refactor operator configuration to use a map for database configs, simplifying AppMetricsDB, ScalingEngineDB, and DBLock structures. • Update WriteYmlConfig function parameter type to any. * Renamed app_metrics_db to appmetrics_db and scaling_engine_db to scalingengine_db in various configuration files and tests. Updated operator.yml.erb and eventgenerator.yml.erb templates to reflect these changes. Adjusted operator config and test files to align with the new database configuration keys. * Refactor TLS client certificate assignment in autoscaler operator config loading * Remove exampleconfig/example.yml from autoscaler operator directory * Fix lint * Fix rspec * Remove unused code * Remove username and password from Postgres URLs in config files * Remove code duplication * Refactor tests * Remove health server basic auth tests from operator_test.go * Refactor operator test configuration variable name and update health check tests - Rename `cfg` to `conf` in operator test files for consistency - Replace direct health check HTTP requests with `testhelpers.CheckHealthAuth` utility function - Adjust `loadVcapConfig` function to move `tlsCerts` declaration closer to usage * Refactor policy DB initialization and improve error handling in operator main • Refactor policy DB initialization to handle errors and improve logging upon connection failure. • Update operator tests to use table-driven approach for testing DB connection failures. * Refactor test descriptions from Context to When in config_test.go * Remove testhelpers dependency and update health check functions in scalingengine tests * Fix broken test --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent 3673a92 commit 97c3dce

File tree

32 files changed

+945
-811
lines changed

32 files changed

+945
-811
lines changed

devbox.lock

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -583,7 +583,38 @@
583583
"outputs": [
584584
{
585585
"name": "out",
586-
"path": "/nix/store/6k49f7lzfvbfwx995xfz1n7nhjnzwdx2-ginkgo-2.23.4",
586+
"path": "/nix/store/daixhlyywv99fwb7vj566s4hfn0741ds-ginkgo-2.23.4",
587+
"default": true
588+
}
589+
],
590+
"store_path": "/nix/store/daixhlyywv99fwb7vj566s4hfn0741ds-ginkgo-2.23.4"
591+
},
592+
"x86_64-linux": {
593+
"outputs": [
594+
{
595+
"name": "out",
596+
"path": "/nix/store/rkl11pk4dgrdf1j7r13qlmlhfmxv556l-ginkgo-2.23.4",
597+
"default": true
598+
}
599+
],
600+
"store_path": "/nix/store/rkl11pk4dgrdf1j7r13qlmlhfmxv556l-ginkgo-2.23.4"
601+
}
602+
}
603+
},
604+
"github:NixOS/nixpkgs/nixpkgs-unstable": {
605+
"resolved": "github:NixOS/nixpkgs/b3582c75c7f21ce0b429898980eddbbf05c68e55?lastModified=1746576598&narHash=sha256-FshoQvr6Aor5SnORVvh%2FZdJ1Sa2U4ZrIMwKBX5k2wu0%3D"
606+
},
607+
"glibcLocales@latest": {
608+
"last_modified": "2025-05-06T08:06:31Z",
609+
"resolved": "github:NixOS/nixpkgs/1cb1c02a6b1b7cf67e3d7731cbbf327a53da9679#glibcLocales",
610+
"source": "devbox-search",
611+
"version": "2.40-66",
612+
"systems": {
613+
"aarch64-linux": {
614+
"outputs": [
615+
{
616+
"name": "out",
617+
"path": "/nix/store/7af0gzp82zy3g8gn9khxwj60npcsmbvh-glibc-locales-2.40-66",
587618
"default": true
588619
}
589620
],
@@ -1259,7 +1290,7 @@
12591290
},
12601291
"mysql@10.6.12": {
12611292
"last_modified": "2023-02-24T09:01:09Z",
1262-
"plugin_version": "0.0.3",
1293+
"plugin_version": "0.0.4",
12631294
"resolved": "github:NixOS/nixpkgs/7d0ed7f2e5aea07ab22ccb338d27fbe347ed2f11#mysql",
12641295
"source": "devbox-search",
12651296
"version": "10.6.12"
@@ -1503,7 +1534,7 @@
15031534
},
15041535
"python@3.13.3": {
15051536
"last_modified": "2025-05-16T20:19:48Z",
1506-
"plugin_version": "0.0.3",
1537+
"plugin_version": "0.0.4",
15071538
"resolved": "github:NixOS/nixpkgs/12a55407652e04dcf2309436eb06fef0d3713ef3#python313",
15081539
"source": "devbox-search",
15091540
"version": "3.13.3",

jobs/eventgenerator/templates/eventgenerator.yml.erb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ db:
9292
max_open_connections: <%= p("autoscaler.policy_db_connection_config.max_open_connections") %>
9393
max_idle_connections: <%= p("autoscaler.policy_db_connection_config.max_idle_connections") %>
9494
connection_max_lifetime: <%= p("autoscaler.policy_db_connection_config.connection_max_lifetime") %>
95-
app_metrics_db:
95+
appmetrics_db:
9696
url: <%= app_metrics_db_url %>
9797
max_open_connections: <%= p("autoscaler.appmetrics_db_connection_config.max_open_connections") %>
9898
max_idle_connections: <%= p("autoscaler.appmetrics_db_connection_config.max_idle_connections") %>

jobs/operator/templates/operator.yml.erb

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,6 @@ cf:
5353
idle_connection_timeout_ms: <%= p("autoscaler.cf.idle_connection_timeout_ms") %>
5454
max_idle_conns_per_host_ms: <%= p("autoscaler.cf.max_idle_conns_per_host_ms") %>
5555

56-
57-
5856
logging:
5957
level: <%= p("autoscaler.operator.logging.level") %>
6058
health:
@@ -72,21 +70,33 @@ health:
7270

7371
http_client_timeout: <%= p("autoscaler.operator.http_client_timeout") %>
7472

75-
app_metrics_db:
76-
db:
73+
db:
74+
appmetrics_db:
7775
url: <%= app_metrics_db_url %>
7876
max_open_connections: <%= p("autoscaler.appmetrics_db_connection_config.max_open_connections") %>
7977
max_idle_connections: <%= p("autoscaler.appmetrics_db_connection_config.max_idle_connections") %>
8078
connection_max_lifetime: <%= p("autoscaler.appmetrics_db_connection_config.connection_max_lifetime") %>
81-
refresh_interval: <%= p("autoscaler.operator.app_metrics_db.refresh_interval") %>
82-
cutoff_duration: <%= p("autoscaler.operator.app_metrics_db.cutoff_duration") %>
83-
84-
scaling_engine_db:
85-
db:
79+
scalingengine_db:
8680
url: <%= scaling_engine_db_url %>
8781
max_open_connections: <%= p("autoscaler.scalingengine_db_connection_config.max_open_connections") %>
8882
max_idle_connections: <%= p("autoscaler.scalingengine_db_connection_config.max_idle_connections") %>
8983
connection_max_lifetime: <%= p("autoscaler.scalingengine_db_connection_config.connection_max_lifetime") %>
84+
policy_db:
85+
url: <%= policy_db_url %>
86+
max_open_connections: <%= p("autoscaler.policy_db_connection_config.max_open_connections") %>
87+
max_idle_connections: <%= p("autoscaler.policy_db_connection_config.max_idle_connections") %>
88+
connection_max_lifetime: <%= p("autoscaler.policy_db_connection_config.connection_max_lifetime") %>
89+
lock_db:
90+
url: <%= lock_db_url %>
91+
max_open_connections: <%= p("autoscaler.lock_db_connection_config.max_open_connections") %>
92+
max_idle_connections: <%= p("autoscaler.lock_db_connection_config.max_idle_connections") %>
93+
connection_max_lifetime: <%= p("autoscaler.lock_db_connection_config.connection_max_lifetime") %>
94+
95+
app_metrics_db:
96+
refresh_interval: <%= p("autoscaler.operator.app_metrics_db.refresh_interval") %>
97+
cutoff_duration: <%= p("autoscaler.operator.app_metrics_db.cutoff_duration") %>
98+
99+
scaling_engine_db:
90100
refresh_interval: <%= p("autoscaler.operator.scaling_engine_db.refresh_interval") %>
91101
cutoff_duration: <%= p("autoscaler.operator.scaling_engine_db.cutoff_duration") %>
92102

@@ -107,18 +117,8 @@ scheduler:
107117
ca_file: /var/vcap/jobs/operator/config/certs/scheduler/ca.crt
108118

109119
app_syncer:
110-
db:
111-
url: <%= policy_db_url %>
112-
max_open_connections: <%= p("autoscaler.policy_db_connection_config.max_open_connections") %>
113-
max_idle_connections: <%= p("autoscaler.policy_db_connection_config.max_idle_connections") %>
114-
connection_max_lifetime: <%= p("autoscaler.policy_db_connection_config.connection_max_lifetime") %>
115120
sync_interval: <%= p("autoscaler.operator.app_sync_interval") %>
116121

117122
db_lock:
118123
ttl: <%= p("autoscaler.operator.db_lock.ttl") %>
119124
retry_interval: <%= p("autoscaler.operator.db_lock.retry_interval") %>
120-
db:
121-
url: <%= lock_db_url %>
122-
max_open_connections: <%= p("autoscaler.lock_db_connection_config.max_open_connections") %>
123-
max_idle_connections: <%= p("autoscaler.lock_db_connection_config.max_idle_connections") %>
124-
connection_max_lifetime: <%= p("autoscaler.lock_db_connection_config.connection_max_lifetime") %>

operations/configure-cf-services.yml

Lines changed: 0 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -46,29 +46,3 @@
4646
uris:
4747
- ((deployment_name))-cf-scalingengine.((system_domain))
4848

49-
50-
## EVENTGENERATOR - Enable cf Server to receive calls from api running on cf --
51-
52-
- type: replace
53-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_org_guid?
54-
value: ((!autoscaler_cf_server_xfcc_valid_org_guid))
55-
56-
- type: replace
57-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/xfcc?/valid_space_guid?
58-
value: ((!autoscaler_cf_server_xfcc_valid_space_guid))
59-
60-
61-
- type: replace
62-
path: /instance_groups/name=eventgenerator/jobs/name=eventgenerator/properties/autoscaler/eventgenerator/cf_server?/port?
63-
value: 8080
64-
65-
- type: replace
66-
path: /instance_groups/name=eventgenerator/jobs/name=route_registrar/properties/route_registrar/routes/-
67-
value:
68-
name: ((deployment_name))-cf-eventgenerator
69-
registration_interval: 20s
70-
port: 8080
71-
tags:
72-
component: autoscaler_cf_eventgenerator
73-
uris:
74-
- ((deployment_name))-cf-eventgenerator.((system_domain))

operations/use-cf-services.yml

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,12 +51,17 @@
5151
- type: remove
5252
path: /instance_groups/name=apiserver
5353

54-
5554
## METRICSFORWARDER - remove metricsforwarder
5655
- type: remove
5756
path: /instance_groups/name=metricsforwarder
5857

59-
## METRICSFORWARDER - remove metricsforwarder
58+
## EVENTGENERATOR - remove eventgenerator
6059
- type: remove
6160
path: /instance_groups/name=eventgenerator
6261

62+
## OPERATOR - remove operator
63+
- type: remove
64+
path: /instance_groups/name=operator
65+
66+
67+

packages/operator/spec

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ files:
99
- autoscaler/vendor/*
1010
- autoscaler/operator/db/*
1111
- autoscaler/cf/* # gosub
12+
- autoscaler/configutil/* # gosub
1213
- autoscaler/db/* # gosub
1314
- autoscaler/db/sqldb/* # gosub
1415
- autoscaler/healthendpoint/* # gosub
@@ -29,6 +30,7 @@ files:
2930
- autoscaler/vendor/filippo.io/edwards25519/field/* # gosub
3031
- autoscaler/vendor/github.com/beorn7/perks/quantile/* # gosub
3132
- autoscaler/vendor/github.com/cespare/xxhash/v2/* # gosub
33+
- autoscaler/vendor/github.com/cloud-gov/go-cfenv/* # gosub
3234
- autoscaler/vendor/github.com/go-logr/logr/* # gosub
3335
- autoscaler/vendor/github.com/go-logr/logr/funcr/* # gosub
3436
- autoscaler/vendor/github.com/go-logr/stdr/* # gosub
@@ -55,6 +57,7 @@ files:
5557
- autoscaler/vendor/github.com/jackc/puddle/v2/internal/genstack/* # gosub
5658
- autoscaler/vendor/github.com/jmoiron/sqlx/* # gosub
5759
- autoscaler/vendor/github.com/jmoiron/sqlx/reflectx/* # gosub
60+
- autoscaler/vendor/github.com/mitchellh/mapstructure/* # gosub
5861
- autoscaler/vendor/github.com/munnerz/goautoneg/* # gosub
5962
- autoscaler/vendor/github.com/openzipkin/zipkin-go/idgenerator/* # gosub
6063
- autoscaler/vendor/github.com/openzipkin/zipkin-go/model/* # gosub

spec/jobs/eventgenerator/eventgenerator_spec.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,14 +119,14 @@
119119

120120
context "appmetrics_db" do
121121
it "includes the ca, cert and key in url when configured" do
122-
rendered_template["db"]["app_metrics_db"]["url"].tap do |url|
122+
rendered_template["db"]["appmetrics_db"]["url"].tap do |url|
123123
check_if_certs_in_url(url, "appmetrics_db")
124124
end
125125
end
126126

127127
it "does not include the ca, cert and key in url when not configured" do
128128
properties["autoscaler"]["appmetrics_db"]["tls"] = nil
129-
rendered_template["db"]["app_metrics_db"]["url"].tap do |url|
129+
rendered_template["db"]["appmetrics_db"]["url"].tap do |url|
130130
check_if_certs_not_in_url(url, "appmetrics_db")
131131
end
132132
end

spec/jobs/operator/operator_spec.rb

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -40,59 +40,59 @@
4040
context "uses tls" do
4141
context "policy_db" do
4242
it "includes the ca, cert and key in url when configured" do
43-
rendered_template["app_syncer"]["db"]["url"].tap do |url|
43+
rendered_template["db"]["policy_db"]["url"].tap do |url|
4444
check_if_certs_in_url(url, "policy_db")
4545
end
4646
end
4747

4848
it "does not include the ca, cert and key in url when not configured" do
4949
properties["autoscaler"]["policy_db"]["tls"] = nil
50-
rendered_template["app_syncer"]["db"]["url"].tap do |url|
50+
rendered_template["db"]["policy_db"]["url"].tap do |url|
5151
check_if_certs_not_in_url(url, "policy_db")
5252
end
5353
end
5454
end
5555

5656
context "app_metrics_db" do
5757
it "includes the ca, cert and key in url when configured" do
58-
rendered_template["app_metrics_db"]["db"]["url"].tap do |url|
58+
rendered_template["db"]["appmetrics_db"]["url"].tap do |url|
5959
check_if_certs_in_url(url, "appmetrics_db")
6060
end
6161
end
6262

6363
it "does not include the ca, cert and key in url when not configured" do
6464
properties["autoscaler"]["appmetrics_db"]["tls"] = nil
65-
rendered_template["app_metrics_db"]["db"]["url"].tap do |url|
65+
rendered_template["db"]["appmetrics_db"]["url"].tap do |url|
6666
check_if_certs_not_in_url(url, "appmetrics_db")
6767
end
6868
end
6969
end
7070

7171
context "scaling_engine_db" do
7272
it "includes the ca, cert and key in url when configured" do
73-
rendered_template["scaling_engine_db"]["db"]["url"].tap do |url|
73+
rendered_template["db"]["scalingengine_db"]["url"].tap do |url|
7474
check_if_certs_in_url(url, "scalingengine_db")
7575
end
7676
end
7777

7878
it "does not include the ca, cert and key in url when not configured" do
7979
properties["autoscaler"]["scalingengine_db"]["tls"] = nil
80-
rendered_template["scaling_engine_db"]["db"]["url"].tap do |url|
80+
rendered_template["db"]["scalingengine_db"]["url"].tap do |url|
8181
check_if_certs_not_in_url(url, "scalingengine_db")
8282
end
8383
end
8484
end
8585

8686
context "db_lock" do
8787
it "includes the ca, cert and key in url when configured" do
88-
rendered_template["db_lock"]["db"]["url"].tap do |url|
88+
rendered_template["db"]["lock_db"]["url"].tap do |url|
8989
check_if_certs_in_url(url, "lock_db")
9090
end
9191
end
9292

9393
it "does not include the ca, cert and key in url when not configured" do
9494
properties["autoscaler"]["lock_db"]["tls"] = nil
95-
rendered_template["db_lock"]["db"]["url"].tap do |url|
95+
rendered_template["db"]["lock_db"]["url"].tap do |url|
9696
check_if_certs_not_in_url(url, "lock_db")
9797
end
9898
end

src/autoscaler/Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@ package-dbtasks:
168168
vendor-changelogs:
169169
cp $(MAKEFILE_DIR)/api/db/* $(MAKEFILE_DIR)/dbtasks/src/main/resources/.
170170
cp $(MAKEFILE_DIR)/eventgenerator/db/* $(MAKEFILE_DIR)/dbtasks/src/main/resources/.
171+
cp $(MAKEFILE_DIR)/operator/db/* $(MAKEFILE_DIR)/dbtasks/src/main/resources/.
171172

172173
.PHONY: clean
173174
clean:

src/autoscaler/api/cmd/api/api_test.go

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,10 @@ var _ = Describe("Api", func() {
2727
runner *ApiRunner
2828
rsp *http.Response
2929

30-
brokerHttpClient *http.Client
31-
httpClientForHealth *http.Client
32-
apiHttpClient *http.Client
33-
cfServerHttpClient *http.Client
30+
brokerHttpClient *http.Client
31+
healthHttpClient *http.Client
32+
apiHttpClient *http.Client
33+
cfServerHttpClient *http.Client
3434

3535
serverURL *url.URL
3636
brokerURL *url.URL
@@ -47,7 +47,7 @@ var _ = Describe("Api", func() {
4747
vcapPort = 8080 + GinkgoParallelProcess()
4848

4949
brokerHttpClient = testhelpers.NewServiceBrokerClient()
50-
httpClientForHealth = &http.Client{}
50+
healthHttpClient = &http.Client{}
5151
apiHttpClient = testhelpers.NewPublicApiClient()
5252
cfServerHttpClient = &http.Client{}
5353

@@ -207,20 +207,13 @@ var _ = Describe("Api", func() {
207207
runner.Interrupt()
208208
Eventually(runner.Session, 5).Should(Exit(0))
209209
})
210+
210211
When("a request to query health comes", func() {
211212
It("returns with a 200", func() {
212-
rsp, err := httpClientForHealth.Get(healthURL.String())
213-
Expect(err).NotTo(HaveOccurred())
214-
Expect(rsp.StatusCode).To(Equal(http.StatusOK))
215-
raw, _ := io.ReadAll(rsp.Body)
216-
healthData := string(raw)
217-
Expect(healthData).To(ContainSubstring("_concurrent_http_request"))
218-
Expect(healthData).To(ContainSubstring("_policyDB"))
219-
Expect(healthData).To(ContainSubstring("_bindingDB"))
220-
Expect(healthData).To(ContainSubstring("go_goroutines"))
221-
Expect(healthData).To(ContainSubstring("go_memstats_alloc_bytes"))
222-
rsp.Body.Close()
223-
213+
testhelpers.CheckHealthResponse(healthHttpClient, healthURL.String(), []string{
214+
"autoscaler_golangapiserver_concurrent_http_request", "autoscaler_golangapiserver_policyDB",
215+
"autoscaler_golangapiserver_bindingDB", "go_goroutines", "go_memstats_alloc_bytes",
216+
})
224217
})
225218
})
226219
})
@@ -234,13 +227,13 @@ var _ = Describe("Api", func() {
234227
When("Health server is ready to serve RESTful API with basic Auth", func() {
235228
When("username and password are incorrect for basic authentication during health check", func() {
236229
It("should return 401", func() {
237-
testhelpers.CheckHealthAuth(GinkgoT(), httpClientForHealth, healthURL.String(), "wrongusername", "wrongpassword", http.StatusUnauthorized)
230+
testhelpers.CheckHealthAuth(GinkgoT(), healthHttpClient, healthURL.String(), "wrongusername", "wrongpassword", http.StatusUnauthorized)
238231
})
239232
})
240233

241234
When("username and password are correct for basic authentication during health check", func() {
242235
It("should return 200", func() {
243-
testhelpers.CheckHealthAuth(GinkgoT(), httpClientForHealth, healthURL.String(), conf.Health.BasicAuth.Username, conf.Health.BasicAuth.Password, http.StatusOK)
236+
testhelpers.CheckHealthAuth(GinkgoT(), healthHttpClient, healthURL.String(), conf.Health.BasicAuth.Username, conf.Health.BasicAuth.Password, http.StatusOK)
244237
})
245238
})
246239
})

0 commit comments

Comments
 (0)