[WIP] [Kernel] Catalog Commits (CCv2) Prototype #4128
Conversation
| //////////////////////// | ||
|
|
||
| class InMemoryCCv2Client extends CCv2Client { | ||
| override def resolveTable(tableName: String): ResolvedMetadata = { |
There was a problem hiding this comment.
// UC_REST_CLIENT
uc_catalog.delta.protocol.$field1
// GLUE_REST_CLIENT
glue_catalog.delta.protocol.$field1
Should the UC_CCV2_CLIENT and the GLUE_CCV2_CLIENT then translate to KERNEL?
kernel.delta.protocol
Q: if we give them a bag of properties, will they add a prefix? are they responsible for stripping the prefix when they give the bag of properties back to us?
Q: is the goal that the CCv2 clients should not have to do a bunch of translation / deep understanding?
Intuition: Kernel should be able to commit a bag of properties and then get back that exact same bag of properties. Maybe the CCv2 client needs to add/strip some prefixes when it goes back and forth to the catalog.
There was a problem hiding this comment.
the interface Kernel <> CCv2Client should be the bag of properties
| //////////////////////// | ||
|
|
||
| class InMemoryCCv2Client extends CCv2Client { | ||
| override def resolveTable(tableName: String): ResolvedMetadata = { |
There was a problem hiding this comment.
resolveTableAsBagOfProperties(tableName: String) -> Map<String, String>
scottsand-db
force-pushed
the
kernel_ccv2_prototype_a
branch
from
bcf2d04 to
3b32f86
Compare
| val txnState = txn.getTransactionState(defaultEngine) | ||
| val stagedFiles = stageData(txnState, Map.empty, data.toList) | ||
| val stagedActionsIterable = CloseableIterable.inMemoryIterable(stagedFiles) | ||
| txn.commit(defaultEngine, stagedActionsIterable) |
There was a problem hiding this comment.
val fileActions = ...
val allActions = txn.finalizeActions(fileActions, ...)
ResolvedMetadata::commit(allActions, ...)
...