Shared mini test suite #5536

dlmarion · 2025-05-06T14:26:47Z

Created SimpleSharedMacTestSuiteIT which starts Mini, runs all of the tests with
the SIMPLE_MINI_CLUSTER_SUITE tag, then shuts down Mini. Annotated most of
the tests that extended SharedMiniClusterBase and called startMiniCluster with
no configuration callback with the SIMPLE_MINI_CLUSTER_SUITE tag. These test
with that did not use a configuration callback where most likely to not do
destructive things with Mini. ManagerApiIT was one exception and I did not put
the tag on this class.

Modified SharedMiniClusterBase so that stopMiniCluster would not stop the
cluster if STOP_DISABLED was set to true, which it is in the new test suite class.
This enables a single test class to be run and behave normally, but also allows
the class to be part of the test suite and not shut down Mini after the tests have
run. Modified stopMiniCluster to instead delete all user tables when STOP_DISABLED
is set to true to clean up tables from the test class. Leaving the tables around
in the instance worked for most tests, but not for tests that counted tables
and expected a certain number (CreateTableIT) or for tests that kicked off a compaction
and waited for it to finish (ComprehensiveIT.testCompaction) as the single
Compactor may have been busy compacting other tables.

Modified the pom to have two failsafe executions, one for the ITs without the
SIMPLE_MINI_CLUSTER_SUITE and one for the test suite.

Closes #5529

All ITs that extend SharedMiniClusterBase, but start it without the callback parameter can likely be run together. This commit creates a new class, SimpleSharedMacTestSuite, which will set up the shared MiniAccumuloCluster before running all tests that have the SIMPLE_MINI_CLUSTER_ONLY tag, and then shut the cluster down.

dlmarion · 2025-05-07T20:54:55Z

Testing locally using mvn clean verify with 0d19981 and it runs the new test suite first. Saw two tests fail in the test suite:

[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.85 s <<< FAILURE! -- in org.apache.accumulo.test.fate.user.UserFateStoreFateIT
[ERROR] org.apache.accumulo.test.fate.user.UserFateStoreFateIT.testCreateWithKeyInProgress -- Time elapsed: 0.630 s <<< ERROR!
java.lang.IllegalStateException: Attempted to reserve a tx that does not exist: FATE:USER:1036c868-7b72-3aa2-8e55-a807d1518a24
	at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
	at org.apache.accumulo.core.fate.AbstractFateStore.reserve(AbstractFateStore.java:148)
	at org.apache.accumulo.test.fate.FateStoreIT.testCreateWithKeyInProgress(FateStoreIT.java:396)
	at org.apache.accumulo.test.fate.user.UserFateStoreFateIT.executeTest(UserFateStoreFateIT.java:63)
	at org.apache.accumulo.test.fate.FateTestRunner.executeTest(FateTestRunner.java:33)
	at org.apache.accumulo.test.fate.FateStoreIT.testCreateWithKeyInProgress(FateStoreIT.java:364)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

and

[ERROR] org.apache.accumulo.test.CloneIT.testClonedMarker(Range, Range, Range)[2] -- Time elapsed: 2.014 s <<< ERROR!
java.lang.IllegalStateException: No prev endrow seen.  tableId: 0 endrow: m
	at org.apache.accumulo.core.metadata.schema.TabletMetadata.getPrevEndRow(TabletMetadata.java:488)
	at org.apache.accumulo.core.metadata.schema.TabletMetadata.lambda$new$1(TabletMetadata.java:190)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:198)
	at org.apache.accumulo.core.metadata.schema.TabletMetadata.getExtent(TabletMetadata.java:478)
	at org.apache.accumulo.core.metadata.schema.LinkingIterator.goodTransition(LinkingIterator.java:85)
	at org.apache.accumulo.core.metadata.schema.LinkingIterator.next(LinkingIterator.java:174)
	at org.apache.accumulo.core.metadata.schema.LinkingIterator.next(LinkingIterator.java:50)
	at org.apache.accumulo.server.util.MetadataTableUtil.checkClone(MetadataTableUtil.java:297)
	at org.apache.accumulo.test.CloneIT.testClonedMarker(CloneIT.java:386)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

It looks like the ITs that are run as part of the suite are being run again in the second IT run that is meant to exclude them. Need to continue debugging that.

dlmarion · 2025-05-08T12:45:24Z

@keith-turner @kevinrr888 - the tests in the comment above are consistently failing for me locally with the same error. The change here is that these tests are being run in a MiniAccumuloCluster instance that is much longer lived with other tests happening before the tests that are failing.

dlmarion · 2025-05-08T12:46:40Z

The change in 2efdef7 seems to resolve the tests being run twice. I think I have this where I want it now. I'm going to kick off a full build to see if it sped things up.

Removing the IT suffix should prevent the suite from getting picked up in the second failsafe execution. The excludedGroup tag was not working for some reason.

dlmarion · 2025-05-12T11:42:17Z

Full IT build successful. Compared the most recent 4.0 full build:

both had the same number of ITs run (1085).
the IT build for this PR took 7h39m with 95 minutes wasted in timed out ITs
the 4.0 build took 6h36m with 40 minutes wasted in timed out ITs

It's interesting that ScanServerUpgrade11to12TestIT eventually succeeded in the PR build. It's been failing consistently in the 4.0 build for a while now as noted in #5531.

test/src/main/java/org/apache/accumulo/harness/SharedMiniClusterBase.java

kevinrr888

So long as Christopher is good with the build changes, I think this is good to go!

I am working on getting this to work for the ITs that set a custom config. Was originally considering just putting that PR up against this branch, but now thinking it will be better suited as a standalone PR after this is merged in (easier for others to review and won't hold up this PR).

ctubbsii

I don't like the all-or-nothing approach to the tests in the suite, but if it saves a lot of time in the normal case, it's probably worth it. What happens when you run an individual test in the suite in the IDE? Does it still work correctly? Or do you have to run the suite test and see the output of all the tests in the suite?

Also, I'm not a fan of the fact that it seems to run every time, even when specifying -Dit.test= for a specific test. That can be addressed in a profile, as I mentioned in the below comment.

ctubbsii · 2025-05-15T16:52:29Z

pom.xml

+          <execution>
+            <id>run-integration-test-suite</id>
+            <goals>
+              <goal>integration-test</goal>
+              <goal>verify</goal>
+            </goals>
+            <configuration>
+              <test>org.apache.accumulo.suites.SimpleSharedMacTestSuite</test>
+            </configuration>
+          </execution>


It looks like this overwrites the user's -Dit.test= selection, and will always run, regardless of which tests the user specifies. I'm not sure a good workaround for that, except maybe put this execution in a profile that is activated when that test variable is not provided, as in <property>!it.test</property>. In that case, it would also be skipped when it.test is used to specify one of the tests in the suite itself, so it's not a perfect solution, but it's probably better to suppress the execution when the user wants to run an explicit test.

ctubbsii · 2025-05-15T16:55:10Z

test/pom.xml

+    <dependency>
+      <groupId>org.junit.platform</groupId>
+      <artifactId>junit-platform-suite-api</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.junit.platform</groupId>
+      <artifactId>junit-platform-suite-engine</artifactId>
+    </dependency>


Are these compile time dependencies or runtime dependencies? It seems like maybe the engine is a runtime dependency. It probably should have a runtime scope.

dlmarion · 2025-05-15T17:06:59Z

I don't like the all-or-nothing approach to the tests in the suite, but if it saves a lot of time in the normal case, it's probably worth it. What happens when you run an individual test in the suite in the IDE?

Running a single test that is part of the suite on it's own should still work.

ctubbsii · 2025-05-15T19:08:52Z

Running a single test that is part of the suite on it's own should still work.

Okay, good to know!

- Noticed several SharedMiniClusterBase ITs were not calling `stopMiniCluster()`. All tests which start a mini cluster now stop the mini cluster as well. - Added some javadoc changes to SharedMiniClusterBase and SimpleSharedMacTestSuite - Misc. other minor changes

* Minor follow on to #5536 - Noticed several SharedMiniClusterBase ITs were not calling `stopMiniCluster()`. All tests which start a mini cluster now stop the mini cluster as well. - Added some javadoc changes to SharedMiniClusterBase and SimpleSharedMacTestSuite - Small typo fixes - Changed junit-platform-suite-engine dependency in test/pom.xml from being compile scope to being test scope

dlmarion added 5 commits May 1, 2025 20:41

Got suite to work, need to fix failing tests

7ea03b8

Got everything working, had to fix some tests

8a9a27b

Fix dependency check issue

f3558d5

Merge branch 'main' into shared-mini-test-suite

66b8fbe

dlmarion added this to the 4.0.0 milestone May 6, 2025

dlmarion requested review from keith-turner, ctubbsii and kevinrr888 May 6, 2025 14:26

dlmarion self-assigned this May 6, 2025

dlmarion added 4 commits May 6, 2025 14:50

Updated script to allow platform.suite

cfc156d

Fix import sort

3682808

Trying to get Maven build working

1381f67

Got suite to run in Maven, but suite tests running twice

0d19981

dlmarion added 2 commits May 8, 2025 11:43

Merge branch 'main' into shared-mini-test-suite

534d333

Fix excludedGroups configuration parameter

2efdef7

dlmarion marked this pull request as ready for review May 8, 2025 12:43

Renamed SimpleSharedMacTestSuiteIT to remove IT suffix

af0ce90

Removing the IT suffix should prevent the suite from getting picked up in the second failsafe execution. The excludedGroup tag was not working for some reason.

keith-turner approved these changes May 13, 2025

View reviewed changes

test/src/main/java/org/apache/accumulo/harness/SharedMiniClusterBase.java Show resolved Hide resolved

kevinrr888 approved these changes May 14, 2025

View reviewed changes

dlmarion added 3 commits May 14, 2025 19:20

Merge branch 'main' into shared-mini-test-suite

336671c

Delete namespaces when cleaning up

cbad765

Merge branch 'main' into shared-mini-test-suite

6a1d09e

dlmarion merged commit 41792a4 into apache:main May 15, 2025
8 checks passed

dlmarion deleted the shared-mini-test-suite branch May 15, 2025 11:47

ctubbsii reviewed May 15, 2025

View reviewed changes

kevinrr888 mentioned this pull request May 16, 2025

Shared mini test suite minor follow on #5555

Merged

dlmarion mentioned this pull request May 19, 2025

CloneIT failures #5557

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Shared mini test suite #5536

Shared mini test suite #5536

Uh oh!

dlmarion commented May 6, 2025

Uh oh!

dlmarion commented May 7, 2025

Uh oh!

dlmarion commented May 8, 2025

Uh oh!

dlmarion commented May 8, 2025

Uh oh!

dlmarion commented May 12, 2025

Uh oh!

Uh oh!

kevinrr888 left a comment

Uh oh!

Uh oh!

ctubbsii left a comment

Uh oh!

ctubbsii May 15, 2025

Uh oh!

ctubbsii May 15, 2025

Uh oh!

dlmarion commented May 15, 2025

Uh oh!

ctubbsii commented May 15, 2025

Uh oh!

Uh oh!

Shared mini test suite #5536

Shared mini test suite #5536

Uh oh!

Conversation

dlmarion commented May 6, 2025

Uh oh!

dlmarion commented May 7, 2025

Uh oh!

dlmarion commented May 8, 2025

Uh oh!

dlmarion commented May 8, 2025

Uh oh!

dlmarion commented May 12, 2025

Uh oh!

Uh oh!

kevinrr888 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ctubbsii left a comment

Choose a reason for hiding this comment

Uh oh!

ctubbsii May 15, 2025

Choose a reason for hiding this comment

Uh oh!

ctubbsii May 15, 2025

Choose a reason for hiding this comment

Uh oh!

dlmarion commented May 15, 2025

Uh oh!

ctubbsii commented May 15, 2025

Uh oh!

Uh oh!