Skip to content

Should ConnectorMetadata and Optimizer require a registered Connector #947

@hdikeman

Description

@hdikeman

In order to use the Optimizer or generate a TableLayout, the existence and registration of a Velox Connector object is required

ConnectorMetadata

For the TableLayout, a non-owning pointer to the connector is an explicit parameter:

https://github.com/facebookincubator/axiom/blob/main/axiom/connectors/ConnectorMetadata.h#L331-L338

This is not exposed except to access the Connector ID and through a direct getter:

  /// The Connector to use for generating ColumnHandles and TableHandles for
  /// operations against this layout.
  velox::connector::Connector* connector() const {
    return connector_;
  }

This is explicitly null checked by the TableLayout constructor:

https://github.com/facebookincubator/axiom/blob/main/axiom/connectors/ConnectorMetadata.cpp#L154

Optimizer

In the Optimizer, it is only used to access the Connector ID to seed the ConnectorSession:

https://github.com/facebookincubator/axiom/blob/main/axiom/optimizer/ToVelox.cpp#L169-L171

And in one place to lookup the Metadata:

https://github.com/facebookincubator/axiom/blob/main/axiom/optimizer/ToVelox.cpp#L1467-L1469

Rationale for Removal

Given that Velox is the execution library, and Axiom is the query frontend, it may not make sense to require the setup of the query execution layer in order to plan a query; you can imagine a scenario where Connector initialization is undesirable in the query frontend layer (for example, if the Connector eagerly allocates some significant resources)

Some metadata implementations (TpchConnectorMetadata, HiveConnectorMetadata, TestConnectorMetadata) accept the Connector as a parameter, but these contracts tightly couple Axiom and Velox initialization. Open to other thoughts though

cc @mbasmanova

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions