-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: implement catalog table functions #504
Conversation
@@ -82,12 +82,12 @@ make test | |||
Runs all unit tests. | |||
|
|||
> [!NOTE] | |||
> To run an individual unit test where `my_deltacat_test` exists in either the test file, class, | |||
> To run an individual unit test where `my_deltacat_test` exists in either the test file, class, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
linter errors from previous commit.
@@ -899,7 +901,7 @@ def update_table( | |||
**kwargs, | |||
) | |||
if not old_table: | |||
raise ValueError(f"Table `{namespace}.{table_name}` does not exist.") | |||
raise TableNotFoundError(f"Table `{namespace}.{table_name}` does not exist.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As pointed out by @pdames update the rest of the storage implementation to use Deltacat exceptions instead of throwing ValueErrors.
|
||
TODO: Honor purge once garbage collection is implemented. | ||
""" | ||
table: Optional[Table] = get_table( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned by @pdames
Can we remove this get_namespace call and just catch/reraise a more clear ValueError from the commit itself?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will try to create an in memory Table/TableLocator instead of calling get_table
.
Drops the given table namespace and all its contents. Raises an error if the | ||
given namespace does not exist. | ||
""" | ||
namespace: Optional[Namespace] = get_namespace( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as previous get_table
call. Try to use an in memory solution.
name: str, | ||
*args, | ||
namespace: Optional[str] = None, | ||
table_version: Optional[str] = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As suggested by @mcember
Should we also optionally accept a version?
Should namespace here go after *args? It feels weird to have positional arguments in the order [table, namespace], since people will think of it as the hierarchy from left to right [namespace, table]. At least in create table I thought to demote it from positional argument into keyword only argument for this reason
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added an optional TableVersion parameter. Also decided to ensure consistency with param ordering.
positional params,
*args,
optional params,
**kwargs
*args, | ||
namespace: Optional[str] = None, | ||
table_version: Optional[str] = None, | ||
stream_format: StreamFormat = StreamFormat.DELTACAT, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As suggested by @mattcember
In the future when we have multiple streams, how are users going to specify which stream they want in get / list / etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added an explicit stream_format parameter to specify the stream. It defaults to the Deltacat stream.
Summary
This PR continues work on the catalog interface layer for Deltacat. It primarily focuses on implementing table related functions.
Functions Implemented:
Also includes minor bugfixes and better exception handling in the storage interface layer.
Testing
make test
Checklist
Unit tests covering the changes have been added
E2E testing has been performed
Additional Notes
There are some follow up tasks that have been added as TODO's.