Dremio merge 2025 05 14 / Adopt config keys (#65)

Last merged commit f2d4b55

The NoSQL-parts contains changes to align the nosql-specific configuration keys + values as follows. No immediate action needed for Dremio, as `org.apache.polaris.persistence.nosql.quarkus.config.ConfigBackwardsCompat` takes care of relocating old config keys + values.
```
polaris.persistence.type=persistence
polaris.backend.name=...
```
to
```
polaris.persistence.type=noqsl
polaris.persistence.backend.type=...
```


* main: Update dependency io.micrometer:micrometer-bom to v1.14.7 (apache#1570)

* JDBC: Simplify JDBC entity conversion (apache#1564)

* fix(Catalog): Add List PolarisStorageAction for all metadata read operations (apache#1391)

* fix(Catalog): Add List PolarisStorageAction for all metadata read operations

* Site : Update cloud providers quickstart to use  (apache#1554)

* [JDBC] Add retries with delay (apache#1517)

[JDBC] Add retries with delay

This change adds retries in the JDBC persistence layer, these retries are with jitter and are tunable in the following ways :
a. max_retries : Total number of retries we expect the persistence to do on Connection Reset exception and serializable error exceptions, before giving up.
b. max_duaration_in_ms : Time in ms since the first attempt this retries should be done. For ex on configured 500 ms the total time spent in retrying should not exceed 500ms (optimistically)
c. initial_delay_in_ms : initial delay before the first attempt

* [Docs] Add JDBC retry properties (apache#1550)

* Use env var in spark container (apache#1522)

* added

Signed-off-by: owenowenisme <[email protected]>

* fix

Signed-off-by: owenowenisme <[email protected]>

* add export

Signed-off-by: owenowenisme <[email protected]>

* update docs using .env

Signed-off-by: owenowenisme <[email protected]>

* update docs

Signed-off-by: owenowenisme <[email protected]>

* change back from using .env to export

Signed-off-by: owenowenisme <[email protected]>

* Apply suggestions from code review

Co-authored-by: Adnan Hemani <[email protected]>

---------

Signed-off-by: owenowenisme <[email protected]>
Co-authored-by: Adnan Hemani <[email protected]>

* Migrate catalog configs to the new reserved prefix (apache#1557)

* rewrite

* rewrite

* stable

* changes per comments

* Remove unused javadoc parameter in BasePersistence (apache#1580)

* Site: Publish table maintenance policies (apache#1581)

* Add schema symlinks to static site directory
Co-authored-by: Yufei Gu <yufei.apache.org>

* Remove `defaults` / `overrides` from feature configurations (apache#1572)

* double WithParentName

* autolint

* Revert some

* autolint

* add to BCconfigs

* autolint

* yank

* copy yuns test

* autolint

* remove defaults

* repair test

* autolint

* stablize test

* stable

* autolint

* configmap change

* copypaste

* regen helm docs

* autolint

* no dots in props

* remove accidental file

* small changes per review

* clean out defaults

* BCC fix

* autolint

* typefix

* autolint

* main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.7 (apache#1578)

* main: Update dependency io.micrometer:micrometer-bom to v1.15.0 (apache#1575)

* main: Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.6 (apache#1577)

* main: Update dependency boto3 to v1.38.15 (apache#1574)

* NoSQL: BREAKING: Change NoSQL configuration options

Requires changing the Quarkus configuration(s) from
```
polaris.persistence.type=persistence
polaris.backend.name=...
```
to
```
polaris.persistence.type=noqsl
polaris.persistence.backend.type=...
```

* DREMIO: Use 'nosql' persistence by default

* DREMIO: backwards-compatible configuations

* Revert "DREMIO: Use 'nosql' persistence by default"

This reverts commit ccba4976b5398511c1c987a242ac11517278c700.

Causes test failures :(

* damn defaults refactoring 😡

* try fix helm

---------

Signed-off-by: owenowenisme <[email protected]>
Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: Yufei Gu <[email protected]>
Co-authored-by: fivetran-ashokborra <[email protected]>
Co-authored-by: Prashant Singh <[email protected]>
Co-authored-by: Owen Lin (You-Cheng Lin) <[email protected]>
Co-authored-by: Adnan Hemani <[email protected]>
Co-authored-by: Eric Maynard <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>

Author: 9 people

bom/build.gradle.kts

@@ -89,6 +89,7 @@ dependencies {
     api(project(":polaris-jpa-model"))
 
     api(project(":polaris-quarkus-admin"))
+    api(project(":polaris-quarkus-common"))
     api(project(":polaris-quarkus-test-commons"))
     api(project(":polaris-quarkus-defaults"))
     api(project(":polaris-quarkus-server"))

client/python/pyproject.toml

@@ -34,7 +34,7 @@ dependencies = [
     "python-dateutil>=2.8.2",
     "pydantic>=2.0.0",
     "typing-extensions>=4.7.1",
-    "boto3==1.38.13",
+    "boto3==1.38.15",
 ]
 
 [project.urls]

extension/persistence/nosql/persistence/api/src/main/java/org/apache/polaris/persistence/nosql/api/backend/Backend.java

@@ -35,7 +35,7 @@
 public interface Backend extends AutoCloseable {
   /** Name of this backend. This value serves as an identifier to select the correct backend. */
   @Nonnull
-  String name();
+  String type();
 
   /**
    * Called to set up the database schema.

extension/persistence/nosql/persistence/api/src/main/java/org/apache/polaris/persistence/nosql/api/backend/BackendConfiguration.java

@@ -25,12 +25,12 @@
 import org.apache.polaris.immutables.PolarisImmutable;
 
 /** Polaris persistence backend configuration. */
-@ConfigMapping(prefix = "polaris.backend")
+@ConfigMapping(prefix = "polaris.persistence.backend")
 @JsonSerialize(as = ImmutableBuildableBackendConfiguration.class)
 @JsonDeserialize(as = ImmutableBuildableBackendConfiguration.class)
 public interface BackendConfiguration {
   /** Name of the persistence backend to use. */
-  Optional<String> name();
+  Optional<String> type();
 
   @PolarisImmutable
   interface BuildableBackendConfiguration extends BackendConfiguration {
@@ -39,6 +39,6 @@ static ImmutableBuildableBackendConfiguration.Builder builder() {
     }
 
     @Override
-    Optional<String> name();
+    Optional<String> type();
   }
 }

extension/persistence/nosql/persistence/benchmark/NOTES.md

@@ -28,14 +28,14 @@ podman run --rm -ti \
 
 ```bash
 ./gradlew :polaris-persistence-nosql-benchmark:jmhJar && java \
-  -Dpolaris.backend.name=InMemory \
+  -Dpolaris.persistence.backend.type=InMemory \
   -jar persistence/benchmark/build/libs/polaris-persistence-nosql-benchmark-1.0.0-incubating-SNAPSHOT-jmh.jar
 ```
 
 ```bash
 ./gradlew :polaris-persistence-nosql-benchmark:jmhJar && java \
-  -Dpolaris.backend.name=MongoDb \
-  -Dpolaris.backend.mongodb.connection-string=mongodb://localhost:27017/ \
-  -Dpolaris.backend.mongodb.database-name=test \
+  -Dpolaris.persistence.backend.type=MongoDb \
+  -Dpolaris.persistence.backend.mongodb.connection-string=mongodb://localhost:27017/ \
+  -Dpolaris.persistence.backend.mongodb.database-name=test \
   -jar persistence/benchmark/build/libs/polaris-persistence-nosql-benchmark-1.0.0-incubating-SNAPSHOT-jmh.jar
 ```

extension/persistence/nosql/persistence/benchmark/src/jmh/java/org/apache/polaris/persistence/nosql/benchmark/BaseParam.java

@@ -41,7 +41,7 @@ void setupPersistence() {
     this.clock = MonotonicClockImpl.newDefaultInstance();
     this.backend = configurer.buildBackendFromConfiguration(factory);
     var info = backend.setupSchema().orElse("");
-    System.out.printf("Opened new persistence backend '%s' %s%n", backend.name(), info);
+    System.out.printf("Opened new persistence backend '%s' %s%n", backend.type(), info);
 
     var idGenerator =
         new SnowflakeIdGeneratorFactory()

extension/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/BackendProvider.java

@@ -42,17 +42,17 @@ Backend backend(
 
     var backendName =
         backendConfiguration
-            .name()
+            .type()
             .orElseThrow(
                 () ->
                     new IllegalStateException(
-                        "Mandatory configuration option polaris.backend.name is missing!"));
+                        "Mandatory configuration option polaris.persistence.backend.type is missing!"));
 
     var backendBuilder = backendBuilders.select(BackendType.Literal.of(backendName));
     if (!backendBuilder.isResolvable()) {
       throw new IllegalStateException(
           format(
-              "Backend '%s' provided in configuration polaris.backend.name is not available. Available backends: %s",
+              "Backend '%s' provided in configuration polaris.persistence.backend.type is not available. Available backends: %s",
               backendName,
               backendBuilders
                   .handlesStream()
@@ -70,7 +70,7 @@ Backend backend(
     if (backendBuilder.isAmbiguous()) {
       throw new IllegalStateException(
           format(
-              "Multiple implementations match the backend name '%s' provided in configuration polaris.backend.name is not available. All available backends: %s",
+              "Multiple implementations match the backend name '%s' provided in configuration polaris.persistence.backend.type is not available. All available backends: %s",
               backendName,
               backendBuilders
                   .handlesStream()
@@ -90,7 +90,7 @@ Backend backend(
     var backend = builder.buildBackend();
     try {
       var setupSchemaResult = backend.setupSchema().orElse("");
-      LOGGER.info("Opened new persistence backend '{}' {}", backend.name(), setupSchemaResult);
+      LOGGER.info("Opened new persistence backend '{}' {}", backend.type(), setupSchemaResult);
 
       return builder.buildBackend();
     } catch (Exception e) {

extension/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/ObservingBackend.java

@@ -64,8 +64,8 @@ public ObservingBackend(@NotObserved Backend backend) {
 
   @Nonnull
   @Override
-  public String name() {
-    return backend.name();
+  public String type() {
+    return backend.type();
   }
 
   @Nonnull

extension/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/QuarkusJdbcConfig.java

@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.persistence.nosql.quarkus.config;
+
+import io.smallrye.config.ConfigSourceInterceptorContext;
+import io.smallrye.config.ConfigValue;
+import io.smallrye.config.RelocateConfigSourceInterceptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ConfigBackwardsCompat extends RelocateConfigSourceInterceptor {
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConfigBackwardsCompat.class);
+
+  public ConfigBackwardsCompat() {
+    super(
+        name -> {
+          if ("polaris.backend.name".equals(name)) {
+            name = newName(name, "polaris.persistence.backend.type");
+          } else if (name.startsWith("polaris.persistence.cache-invalidations.")) {
+            name =
+                newName(
+                    name,
+                    "polaris.persistence.distributed-cache-invalidations."
+                        + name.substring("polaris.persistence.cache-invalidations.".length()));
+          }
+          return name;
+        });
+  }
+
+  private static String newName(String name, String newName) {
+    LOGGER.warn("Using legacy configuration key '{}', migrate to '{}'", name, newName);
+    return newName;
+  }
+
+  @Override
+  public ConfigValue getValue(ConfigSourceInterceptorContext context, String name) {
+    var value = super.getValue(context, name);
+    if ("polaris.persistence.type".equals(name)) {
+      if ("persistence".equals(value.getValue())) {
+        LOGGER.warn(
+            "Using legacy value for 'polaris.persistence.type=persistence', migrate to 'polaris.persistence.type=nosql'");
+        value = value.withValue("nosql");
+      }
+    }
+    return value;
+  }
+}