Skip to content

Iceberg staging hub #425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: oss-main
Choose a base branch
from
Open

Iceberg staging hub #425

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7880f89
refactor: Move toValues from InPredicate.cpp to Filter.h
yingsu00 Mar 15, 2025
5dfa21f
feat(connector): Support reading Iceberg split with equality deletes
yingsu00 May 1, 2024
29cffa3
Support insert data into iceberg table.
PingLiuPing Oct 3, 2025
d7d5e26
Add iceberg partition transforms.
PingLiuPing Jul 1, 2025
a949d0c
Add NaN statistics to parquet writer.
PingLiuPing Sep 4, 2025
496e924
Collect Iceberg data file statistics in dwio.
PingLiuPing Jul 28, 2025
def7691
Fix incorrect min max stats when the column value are infinity or -in…
PingLiuPing Aug 26, 2025
b0e5587
Integrate Iceberg data file statistics and adding unit test.
PingLiuPing Sep 1, 2025
500d361
Support write field_id to parquet metadata SchemaElement.
PingLiuPing Sep 5, 2025
8826076
Implement iceberg sort order
PingLiuPing May 30, 2025
2499c2e
Add clustered Iceberg writer mode.
PingLiuPing Sep 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions velox/connectors/hive/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@
velox_add_library(velox_hive_config OBJECT HiveConfig.cpp)
velox_link_libraries(velox_hive_config velox_core velox_exception)

add_subdirectory(iceberg)

velox_add_library(
velox_hive_connector
OBJECT
Expand All @@ -33,9 +31,11 @@ velox_add_library(
TableHandle.cpp
)

add_subdirectory(iceberg)

velox_link_libraries(
velox_hive_connector
PUBLIC velox_hive_iceberg_splitreader
PRIVATE velox_hive_iceberg_splitreader
PRIVATE velox_common_io velox_connector velox_dwio_catalog_fbhive velox_hive_partition_function
)

Expand Down
5 changes: 5 additions & 0 deletions velox/connectors/hive/HiveConfig.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,4 +273,9 @@ std::string HiveConfig::schema(const config::ConfigBase* session) const {
kSchema, config_->get<std::string>(kSchema, ""));
}

bool HiveConfig::fanoutEnabled(const config::ConfigBase* session) const {
return session->get<bool>(
kFanoutEnabledSession, config_->get<bool>(kFanoutEnabled, true));
}

} // namespace facebook::velox::connector::hive
9 changes: 9 additions & 0 deletions velox/connectors/hive/HiveConfig.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,12 @@ class HiveConfig {
static constexpr const char* kSource = "source";
static constexpr const char* kSchema = "schema";

/// Controls the writer mode, whether the fanout mode writer is enabled,
/// default value is true, setting to false means clustered mode.
/// Currently applies only to the Iceberg writer.
static constexpr const char* kFanoutEnabled = "fanout-enabled";
static constexpr const char* kFanoutEnabledSession = "fanout_enabled";

InsertExistingPartitionsBehavior insertExistingPartitionsBehavior(
const config::ConfigBase* session) const;

Expand Down Expand Up @@ -305,6 +311,9 @@ class HiveConfig {
/// Schema of the query. Used for storage logging.
std::string schema(const config::ConfigBase* session) const;

/// Return if fanout writer mode is enabled.
bool fanoutEnabled(const config::ConfigBase* session) const;

HiveConfig(std::shared_ptr<const config::ConfigBase> config) {
VELOX_CHECK_NOT_NULL(
config, "Config is null for HiveConfig initialization");
Expand Down
9 changes: 9 additions & 0 deletions velox/connectors/hive/HiveConnectorUtil.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@
#include "velox/expression/ExprConstants.h"
#include "velox/expression/ExprToSubfieldFilter.h"

#include <boost/lexical_cast.hpp>
#include <boost/uuid/uuid_generators.hpp>
#include <boost/uuid/uuid_io.hpp>

namespace facebook::velox::connector::hive {
namespace {

Expand Down Expand Up @@ -946,4 +950,9 @@ core::TypedExprPtr extractFiltersFromRemainingFilter(
}
return expr;
}

std::string makeUuid() {
return boost::lexical_cast<std::string>(boost::uuids::random_generator()());
}

} // namespace facebook::velox::connector::hive
2 changes: 2 additions & 0 deletions velox/connectors/hive/HiveConnectorUtil.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,4 +116,6 @@ core::TypedExprPtr extractFiltersFromRemainingFilter(
common::SubfieldFilters& filters,
double& sampleRate);

std::string makeUuid();

} // namespace facebook::velox::connector::hive
Loading