Fix ConfigurationVO load exception after schema change #10485

abh1sar · 2025-02-28T18:11:40Z

Description

This PR fixes #10480

The configuration table schema was changed in PR #10300
But it causes problem if the ConfigurationVO class structure was cached with the old fields.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-02-28T18:19:48Z

Codecov Report

Attention: Patch coverage is 0% with 14 lines in your changes missing coverage. Please review.

Project coverage is 16.57%. Comparing base (8e4fe1c) to head (04dbc7e).

Files with missing lines	Patch %	Lines
...ain/java/com/cloud/utils/db/TransactionLegacy.java	0.00%	6 Missing ⚠️
...src/main/java/com/cloud/upgrade/dao/DbUpgrade.java	0.00%	3 Missing ⚠️
...ava/com/cloud/upgrade/dao/Upgrade42010to42100.java	0.00%	3 Missing ⚠️
...java/com/cloud/upgrade/DatabaseUpgradeChecker.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10485      +/-   ##
============================================
- Coverage     16.57%   16.57%   -0.01%     
  Complexity    13988    13988              
============================================
  Files          5745     5746       +1     
  Lines        510847   510862      +15     
  Branches      62140    62142       +2     
============================================
- Hits          84696    84695       -1     
- Misses       416677   416692      +15     
- Partials       9474     9475       +1

Flag	Coverage Δ
uitests	`3.91% <ø> (ø)`
unittests	`17.47% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

abh1sar · 2025-03-03T08:30:02Z

@blueorangutan package

blueorangutan · 2025-03-03T08:32:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-03T08:33:34Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12634

abh1sar · 2025-03-03T11:28:17Z

@blueorangutan package

blueorangutan · 2025-03-03T11:30:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-03T12:31:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12636

abh1sar · 2025-03-03T12:52:05Z

@blueorangutan test

blueorangutan · 2025-03-03T12:54:04Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-03-03T13:50:24Z

[SF] Trillian Build Failed (tid-12539)

rohityadavcloud · 2025-03-04T06:09:08Z

@blueorangutan test

blueorangutan · 2025-03-04T06:10:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-03-04T07:12:28Z

[SF] Trillian Build Failed (tid-12546)

abh1sar · 2025-03-04T08:22:36Z

@blueorangutan package

blueorangutan · 2025-03-04T08:24:03Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-04T08:24:05Z

[SF] Trillian Build Failed (tid-12553)

blueorangutan · 2025-03-04T09:27:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12648

blueorangutan · 2025-03-04T10:29:47Z

[SF] Trillian Build Failed (tid-12555)

JoaoJandre · 2025-03-04T15:05:24Z

Instead of changing how we get the value, shouldn't we normalize the database data so that it works with the current way of getting the values?

Otherwise, if someone in the future creates a method to get the value the old way and only tests on a new install, it might introduce a bug for people running old installs.

Signed-off-by: Abhishek Kumar <[email protected]>

…StringValueInternal()

into upgrade-config

abh1sar · 2025-03-11T13:15:24Z

@JoaoJandre We identified the issue with how BackupDaoImpl class caches the columns of the table. Even though both configurations table and ConfigurationVO code has the new schema, the ConfigurationsDao._allColumns field still had the older schema from before upgrade. That's why after management server restart ConfigurationsDaoImpl_allColumns was getting regenerated with the correct fields.
I have reverted the older commit and added the commit to regenerate ConfigurationsDaoImpl._allColumns when the Configurations table schema is changed.

abh1sar · 2025-05-19T04:02:27Z

@abh1sar based on your testing and discussion this would need changes.

I was able to not see the error by evicting old connections when HIkariCP is used for pooling.

Thanks @shwstppr I was able to verify the suggested changed.

I'm not sure about DBCP. I've not reproduced the issue there yet and based on my little research it may need closing the datasource. Also, we need to remove that markForColumnsRefresh logic as that doesn't seem to be the problem.

I don't see the issue with DBCP, have tried it multiple times. I am not sure but the caching behaviour is a bit different from HikariCP. So, I have not added any code to handle DBCP.

Sorry, it took a while to come back. I was busy with some other deliverables.
Please check @shwstppr @JoaoJandre .

shwstppr

code LGTM.

abh1sar · 2025-05-19T09:35:50Z

@blueorangutan package

blueorangutan · 2025-05-19T09:38:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-19T10:49:43Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13441

engine/schema/src/main/java/com/cloud/upgrade/dao/DbUpgrade.java

JoaoJandre · 2025-05-19T16:36:31Z

Could you post the test steps you took to reproduce the issue and verify that this PR fixes it?

Co-authored-by: João Jandre <[email protected]>

abh1sar · 2025-05-20T04:36:58Z

Could you post the test steps you took to reproduce the issue and verify that this PR fixes it?

@JoaoJandre The first time management server is started after running the cloudstack-setup-database commands will cause lots of exceptions like below in the management-server.log

I tested the same with the changes with both HikariCP and Apache DBCP. I observed no Exceptions and management-server was coming up as expected.

2025-02-27 13:26:26,070 ERROR [o.a.c.f.c.d.ConfigurationDaoImpl] (VMSchedulerPollTask:[ctx-f804ce4a]) (logid:30a4a0a1) DB Exception on: HikariProxyPreparedStatement@1499732297 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[782]: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated, configuration.group_id, configuration.subgroup_id, configuration.parent, configuration.display_text, configuration.kind, configuration.options FROM configuration WHERE configuration.name = x'6472732e706c616e2e6578706972652e696e74657276616c' java.sql.SQLDataException: Cannot determine value type from string ''
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:115)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:98)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:90)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:64)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:74)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:96)
	at com.mysql.cj.jdbc.result.ResultSetImpl.getObject(ResultSetImpl.java:1431)
	at com.mysql.cj.jdbc.result.ResultSetImpl.getInt(ResultSetImpl.java:830)
	at com.zaxxer.hikari.pool.HikariProxyResultSet.getInt(HikariProxyResultSet.java)
	at com.cloud.utils.db.GenericDaoBase.setField(GenericDaoBase.java:594)
	at com.cloud.utils.db.GenericDaoBase.setField(GenericDaoBase.java:2059)
	at com.cloud.utils.db.GenericDaoBase.toEntityBean(GenericDaoBase.java:1919)
	at com.cloud.utils.db.GenericDaoBase.toEntityBean(GenericDaoBase.java:1880)
	at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1075)
	at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:1050)
	at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:996)
	at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)

abh1sar · 2025-05-21T04:01:26Z

@blueorangutan package

blueorangutan · 2025-05-21T04:02:03Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-21T05:22:40Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13470

abh1sar · 2025-05-21T12:50:59Z

@blueorangutan test

blueorangutan · 2025-05-21T12:52:05Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-05-22T03:09:43Z

[SF] Trillian test result (tid-13371)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48522 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10485-t13371-kvm-ol8.zip
Smoke tests completed. 133 look OK, 1 have errors, 7 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestIpv6Vpc>:setup	`Error`	0.00	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

shwstppr · 2025-07-09T07:29:01Z

@abh1sar @sureshanaparti I think we should try to get this in 4.21

@blueorangutan package

blueorangutan · 2025-07-09T07:30:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland

clgtm

blueorangutan · 2025-07-09T09:36:33Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14115

DaanHoogland · 2025-07-09T10:13:34Z

@blueorangutan test

blueorangutan · 2025-07-09T10:14:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-07-10T03:33:41Z

[SF] Trillian test result (tid-13737)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 59320 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10485-t13737-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2025-07-10T07:47:54Z

@abh1sar et al, please advice on testing. It seems to me an upgrade test is needed at least.

abh1sar · 2025-07-10T07:55:30Z

@DaanHoogland
The exceptions are seen every time the management server is started after running the cloudstack-setup-database command. So, the fix can be verified that way.
Both HikariCP and Apache DBCP should be tested.

Upgrade should also be tested for any regressions, and logs reviewed after the upgrade to check that the exceptions are not there.

Look up only selected fields in getConfigStringValueInternal()

af6d9f1

boring-cyborg bot added the component:vpc label Feb 28, 2025

abh1sar changed the title ~~Fix ConfigurationVO load exception on fresh install~~ Fix ConfigurationVO load exception after schema change Feb 28, 2025

DaanHoogland added this to the 4.21.0 milestone Mar 3, 2025

Merge branch 'main' into upgrade-config

8ab6142

shwstppr and others added 3 commits March 11, 2025 18:29

fix column cache issue

adf411a

Signed-off-by: Abhishek Kumar <[email protected]>

Revert commit for af6d9f1 : Look up only selected fields in getConfig…

b6873ea

…StringValueInternal()

Merge branch 'upgrade-config' of https://github.com/shapeblue/cloudstack

e56109a

into upgrade-config

boring-cyborg bot added the component:database label Mar 11, 2025

abh1sar self-assigned this Mar 11, 2025

shwstppr approved these changes May 19, 2025

View reviewed changes

JoaoJandre reviewed May 19, 2025

View reviewed changes

engine/schema/src/main/java/com/cloud/upgrade/dao/DbUpgrade.java Outdated Show resolved Hide resolved

abh1sar and others added 2 commits May 20, 2025 09:58

Update engine/schema/src/main/java/com/cloud/upgrade/dao/DbUpgrade.java

9629f7d

Co-authored-by: João Jandre <[email protected]>

fix pre-commit failure in DbUpgrade.java

db52b84

JoaoJandre approved these changes May 20, 2025

View reviewed changes

sureshanaparti added this to Apache CloudStack 4.21.0 Jun 5, 2025

sureshanaparti moved this to In Progress in Apache CloudStack 4.21.0 Jun 5, 2025

Merge branch 'main' into upgrade-config

04dbc7e

DaanHoogland approved these changes Jul 9, 2025

View reviewed changes

DaanHoogland added the status:needs-testing label Jul 10, 2025

Fix ConfigurationVO load exception after schema change #10485

Are you sure you want to change the base?

Fix ConfigurationVO load exception after schema change #10485

Uh oh!

Conversation

abh1sar commented Feb 28, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

abh1sar commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

abh1sar commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

abh1sar commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

blueorangutan commented Mar 3, 2025

Uh oh!

rohityadavcloud commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

abh1sar commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

blueorangutan commented Mar 4, 2025

Uh oh!

JoaoJandre commented Mar 4, 2025

Uh oh!

abh1sar commented Mar 11, 2025

Uh oh!

abh1sar commented May 19, 2025

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented May 19, 2025

Uh oh!

blueorangutan commented May 19, 2025

Uh oh!

blueorangutan commented May 19, 2025

Uh oh!

Uh oh!

JoaoJandre commented May 19, 2025

Uh oh!

abh1sar commented May 20, 2025

Uh oh!

abh1sar commented May 21, 2025

Uh oh!

blueorangutan commented May 21, 2025

Uh oh!

blueorangutan commented May 21, 2025

Uh oh!

abh1sar commented May 21, 2025

codecov bot commented Feb 28, 2025 •

edited

Loading