Skip to content

CHANGELOG.md

Latest commit

 

History

History
323 lines (251 loc) · 14.2 KB

CHANGELOG.md

File metadata and controls

323 lines (251 loc) · 14.2 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

8.2.4 - 2026-07-17

Added

  • apiary-gluesync-listener: per-event observability via a new glue_listener_event Micrometer counter, tagged with operation (e.g. create_table), result (success, failure, ignored), and outcome (e.g. created, updated, deleted, not_found, renamed, exception class name). Covers all 8 HMS event handlers. A glue_listener_table_rename_duration timer is also recorded on every table rename.
  • kafka-metastore-receiver: Kafka consumer metrics via Micrometer KafkaClientMetrics. Bind a registry at build time with KafkaMessageReaderBuilder.withMeterRegistry(registry).

Fixed

  • apiary-gluesync-listener: fallback Glue operations (update-after-AlreadyExistsException, create-after-EntityNotFoundException) that previously escaped their catch blocks unhandled are now consistently caught, logged, and metered by the outer exception handler.

Changed

  • Bump Micrometer from 1.9.9 to 1.14.14.

Reverted

  • Revert Change kafka listener hashing for ALTER_TABLE events (#133). ALTER_TABLE events now hash on the current table name again. The reverted change caused ordering discrepancies in the event flow.

8.1.18 - 2026-06-09

Fixed

  • apiary-gluesync-listener: glue sync metrics were silently dropped in HMS deployments; counters are now exported via JMX. In framework deployments (e.g. Dronefly/Spring Boot), metrics continue to flow to the existing registry (e.g. Prometheus) unchanged.

8.1.16 - 2026-05-14

Added

  • apiary-gluesync-listener: new GLUE_SKIP_ARCHIVE environment variable to set the default value of SkipArchive on AWS Glue UpdateTable requests (triggered by Hive ALTER TABLE events). Accepts true or false (case-insensitive). When unset, the existing built-in default (true) is used. Any other value fails at startup. The per-table Hive property apiary.gluesync.skipArchive still takes precedence.

8.1.15 - 2025-11-28

Changed

  • When sending the events we hash based on db and table name. By default we are taking the newest table name. We realised this could potentially break the order of operations because it will send the event to a different kafka partition.

8.1.14 - 2025-11-25

Changed

  • Remove metric relocations for apiary-gluesync-listener because Spring doesn't find those in bean initialisation.

8.1.13 - 2025-11-13

Changed

  • Upgrade maven shade plugin to 3.6.1 (was 3.1.1).
  • Upgrade AWS Java SDK in apiary-gluesync-listener to 1.12.792 (was 1.12.276)
  • Send view original texts in apiary-gluesync-listener.
  • Add possibility to only sync views in apiary-gluesync-listener CLI/
  • Shade micrometer libraries into glue-sync CLI.

8.1.12 - 2025-09-15

Added

  • GlueSyncCli with fat jar for on-demand syncing.

8.1.11 - 2025-08-25

Changed

  • Add classification support for parquet, avro and orc table formats in glue-event-listener.
  • Fixed Glue Classification NPE exception when SerDes library are not found in Hive table properties.
  • Table and partition locations s3 prefixes are changed to s3:// because platforms like Databricks and AWS Cleanrooms, do not support other prefixes.

8.1.10 - 2025-08-14

Changed

  • Update Hive version for hive-hooks to 3.1.3 (was 2.3.7)
  • Update Hadoop version for hive-hooks to 3.1.0 (was 2.7.1)
  • Updated parent pom to deploy to sonatype central.

8.1.9 - 2025-07-07

Changed

  • Update exceptions thrown from JsonMetaStoreEventSerDe to include the cause.

8.1.8 - 2025-05-22

Changed

  • Update metrics to use micrometer instead of Codahale in glue-event-listener.

8.1.7 - 2025-05-21

Changed

  • Upgrading kafka-clients dependency to 3.9.0 which is the latest 3.x version. Also upgraded kafka-junit4 to latest version, dependency which we use for Kafka integration testing.

8.1.6 - 2025-05-16

Changed

  • Glue sync failures are now handled silently.
  • Metrics added to glue-sync-listener.

8.1.5 - 2025-05-13

Changed

  • Improved performance when cleaning column comments in glue listener.

8.1.4 - 2025-05-05

Added

  • Since Glue support Iceberg renames, it has been excluded from the ApiaryGluePreEventListener (tested using Spark client).
  • Add env variable ENABLE_HIVE_TO_GLUE_RENAME_OPERATION to enable Hive rename workaround in Glue.
  • Remove non-unicode characters when syncing into Glue.

8.1.3 - 2025-04-22

Added

  • New functionality to allow renames in Glue Event Listener. The table will be crated with the new name, all existing partitions from the old table will be copied to the new table, and finally the old table will be dropped.

8.1.2 - 2025-04-08

Fix

  • Hive version validation when multiple listeners are used.

8.1.1 - Release Action Failed, no release made.

8.1.0 - 2025-03-28

Added

  • Support for MSK Kafka clusters.

8.0.2 - 2024-08-29

Changed

  • ApiaryGlueSync override skipArchive to be true by default. Backward incompatible behavior that turns off table archiving by default. Allows for per table overrides if needed. If you don't rely on Glue table version it is safe to upgrade to this version of the glue listener without making changes.

7.3.11 - 2024-06-19

Changed

  • Upgrade aws version from 1.11.520 to 1.12.276 in apiary-receiver-sqs.

7.3.10 - 2024-06-18

Added

  • Add aws-java-sdk-sts to enable IRSA authentication in apiary-receiver-sqs.

7.3.9 - 2024-06-03

Added

  • Add aws-java-sdk-sts to enable IRSA authentication in apiary-metastore-listener.

7.3.8 - 2023-08-02

Added

7.3.7 - 2023-07-19

Added

7.3.6 - 2022-11-14

Fixed

  • apiary-gluesync-listener when getting null SortOrder in Hive & Iceberg tables.

7.3.5 - 2022-11-11

Fixed

  • apiary-gluesync-listener when getting last access time in Hive & Iceberg tables.

7.3.4 - 2022-11-02

Changed

  • Add aws-java-sdk-sts to enable IRSA authentication in apiary-gluesync-listener.

7.3.3 - 2022-10-03

Changed

  • aws-java-sdk version to 1.12.276 (was 1.11.333) so we can switch from KIAM to IRSA.

7.3.2 - 2022-05-31

Changed

  • hive-hooks - Logging table name when no conversion can take place.

7.3.1 - 2022-05-27

Added

  • hive-hooks - Fixing potential NullPointerException in ApiaryMetastoreFilter and PathConverter.

7.3.0 - 2022-05-24

Added

  • hive-event-listeners/apiary-gluesync-listener - Added implementation for Glue database sync.

7.2.5 - 2022-03-01

Fixed

  • hive-event-listeners/apiary-gluesync-listener added fix for NullPointerException on unpartitioned tables.

Added

  • log4j managed dependencies to overwrite transitive dependencies with log4j security issues being pulled in. These dependencies come from Hive and are provided but they break our tests.

7.2.4 - 2021-10-12

  • Same as 7.2.2. Released while configuring GitHub actions workflow.

7.2.3 - 2021-10-12

  • Same as 7.2.2. Released while configuring GitHub actions workflow.

7.2.2 - 2021-05-27

Fixed

  • hive-hooks component to convert Storage Descriptor Info 'path' parameter (instead of Storage Descriptor).

7.2.1 - 2021-05-25

Changed

  • hive-hooks component to convert Storage Descriptor 'path' parameter.
  • hive-hooks component to convert table parameter 'avro.schema.url'.

Removed

  • Excluded org.pentaho:pentaho-aggdesigner-algorithm from kafka-metastore-events, apiary-gluesync-listener, apiary-metastore-auth and apiary-ranger-metastore-plugin.

7.2.0 - 2021-03-16

Added

  • Added hive-hooks component for generic path and table location modification.

7.1.0 - 2021-02-18

Changed

  • Relocated Kafka libraries in kafka-metastore-listener under com.expediagroup.apiary.extensions.shaded.org.apache.kafka to avoid clashes when moving other components to a newer version of Kafka.
  • eg-oss-parent version updated to 2.3.2 (was 1.2.0).
  • jdk.tools excluded as a transitive dependency everywhere to allow compilation on Java 11.
  • A few transitive dependencies made explicit to allow compilation on Java 11.

7.0.0 - 2020-04-29

Changed

  • Renamed hive-metastore-events submodule to hive-event-listeners.

6.0.2 - 2020-04-24

Changed

  • Upgraded version of hive.version to 2.3.7 (was 2.3.3). Allows apiary-extensions to be used on JDK>=9.

6.0.1 - 2020-04-02

Fixed

  • Bug in privileges-grantor-lambda where Apiary Events were not being deserialized properly.

6.0.0 - 2020-03-12

Changed

  • client.id is now required to configure Listener.
  • topic Kafka configuration parameter is now called topic.name.
  • No longer sending MetaStore uris with Kafka message.
  • Kafka Receiver is now configured using a builder.

5.0.2 - 2020-02-04

Changed

  • eg-oss-parent version updated to 1.2.0 (was 1.1.0).

Added

  • sns-metastore-event SNS message attribute: qualifiedTableName.

5.0.1 - 2020-01-10

Changed

  • Upgraded Solr version to 7.7.1 (was 5.5.4) to fix issue where Ranger plugin jar (version 2.0.0) could not instantiate Solr client correctly.

5.0.0 - 2019-12-09

Changed

  • Moved apiary-gluesync-listener, apiary-metastore-auth and apiary-ranger-metastore-plugin (all previously top-level modules) to be submodules under a new top-level module named hive-metastore-events.
  • Grouped apiary-metastore-consumers, apiary-metastore-listener and apiary-receivers (all previously submodules in apiary-metastore-events) in a new module named sns-metastore-events, itself a submodule of apiary-metastore-events.
  • Updated apiary-ranger-metastore-plugin to use Ranger 2.0.0 (was 1.1.0).
  • Changed parent to com.expediagroup:eg-oss-parent:1.1.0 (was com.hotels:hotels-oss-parent:4.0.1).

4.2.0 - 2019-08-02

Added

  • apiary-ranger-metastore-plugin module has an ApiaryRangerAuthAllAccessPolicyProvider class that can be configured in ranger-hive-security.xml to allow for an audit-only Ranger pre-event listener. Intent is to configure it on the Apiary read-only HMS instances.

4.1.0 - 2019-06-27

Added

  • privileges-grantor-lambda supports granting access to tables that were renamed.
  • apiary-metastore-consumers sub-module of apiary-metastore-events.
  • metastore-consumer-common module that contains common metastore consumer classes.
  • privileges-grantor-core module that grants "SELECT" privileges on a table.
  • privileges-grantor-lambda module that triggers privileges-grantor-core logic from an AWS Lambda.
  • DbName and TableName as additional message attributes for SNS.

[4.0.0] - 2019-05-08

Added

  • apiary-metastore-events module and moved apiary-metastore-listener and apiary-receivers into this.

Changed

  • apiary-metastore-listener and apiary-receivers package paths renamed from com.expediagroup.apiary.extensions to com.expediagroup.apiary.extensions.events (if you refer to these classes by fully qualified name downstream you will need to update these references before upgrading to this version).

[3.1.0] - 2019-05-03

Fixed

  • apiary-metastore-listener now serializes metastore table properties as a Map instead of a String.

[3.0.0] - 2019-04-12

Changed

  • Maven group ID is now com.expediagroup.apiary (was com.expedia.apiary).
  • All Java classes moved to com.expediagroup.apiary (was com.expedia.apiary).
  • hotels-oss-parent version updated to 4.0.1 (was 2.3.5).

[2.0.0] - 2019-04-08

Changed

  • MessageReader.read() now returns MessageEvent, wrapping ListenerEvent and MessageProperty.

Added

  • SqsMessageReader.read() no longer deletes messages. Instead, SqsMessageReader.delete() has been added to give more control over inflight messages.

Removed

  • aws-java-sdk dependency from apiary-receiver-sqs.

[1.4.0] - 2019-03-27

Added

  • apiary-receivers parent module and apiary-receiver-sqs sub-module, a library to poll Apiary SQS messaging infrastructure for Hive events.

[1.3.2] - 2019-03-22

Fixed

  • Metastore events not published to SNS due to the Message Attribute type being undefined.

Added

  • Filtering of the SNS messages produced in ApiarySNSListener using the eventType Message Attribute.

[1.3.0] - 2019-03-14 [[YANKED]]

Added

  • Filtering of the SNS messages produced in ApiarySNSListener using the eventType Message Attribute.

[1.2.0] - 2019-03-13

Added

  • Old locations for Table and Partitions to ApiarySNSListener events.
  • Table parameters with a configurable regular expression to control which table parameters will be added to the ApiarySNSListener message. Default will be none.

[1.1.0] - 2019-02-15

Added

  • Apiary ReadOnly Auth Event Listener.

Changed

  • Upgraded hotels-oss-parent version to 2.3.5 (was 2.3.3).

Removed

  • Transitive dependency on org.apache.hbase hbase-client. See #18.

[1.0.0] - 2018-10-31

Added

  • Partition Keys with Data types added to the JSON Events from ApiarySNSListener.

Changed

  • partition is renamed to partitionValues and oldPartition to oldPartitionValues in the JSON Events from ApiarySNSListener.

Fixed

  • JSON Events from ApiarySNSListener now marshals list as a JSONArray.

[0.2.0] - 2018-10-03

Added

  • apiary-gluesync-listener sub-module.
  • apiary-metastore-metrics sub-module.
  • apiary-ranger-metastore-plugin sub-module.

[0.1.0] - 2018-09-11

Changed

  • Apiary SNS listener now configured using standard AWS configuration mechanisms. See #4.

Added

  • A protocolVersion field to the SNS messages generated by ApiarySnsListener. See #2.
  • Support for INSERT metastore events. See #6.

[0.0.1] - 2018-08-23

Added

  • Initial version to test that releases to Maven Central work.