feat: implement catalog table functions #504
Conversation
| @@ -82,12 +82,12 @@ make test | |||
| Runs all unit tests. | |||
|
|
|||
| > [!NOTE] | |||
| > To run an individual unit test where `my_deltacat_test` exists in either the test file, class, | |||
| > To run an individual unit test where `my_deltacat_test` exists in either the test file, class, | |||
There was a problem hiding this comment.
linter errors from previous commit.
| @@ -899,7 +901,7 @@ def update_table( | |||
| **kwargs, | |||
| ) | |||
| if not old_table: | |||
| raise ValueError(f"Table `{namespace}.{table_name}` does not exist.") | |||
| raise TableNotFoundError(f"Table `{namespace}.{table_name}` does not exist.") | |||
There was a problem hiding this comment.
As pointed out by @pdames update the rest of the storage implementation to use Deltacat exceptions instead of throwing ValueErrors.
|
|
||
| TODO: Honor purge once garbage collection is implemented. | ||
| """ | ||
| table: Optional[Table] = get_table( |
There was a problem hiding this comment.
As mentioned by @pdames
Can we remove this get_namespace call and just catch/reraise a more clear ValueError from the commit itself?
There was a problem hiding this comment.
Will try to create an in memory Table/TableLocator instead of calling get_table.
| Drops the given table namespace and all its contents. Raises an error if the | ||
| given namespace does not exist. | ||
| """ | ||
| namespace: Optional[Namespace] = get_namespace( |
There was a problem hiding this comment.
Same as previous get_table call. Try to use an in memory solution.
| name: str, | ||
| *args, | ||
| namespace: Optional[str] = None, | ||
| table_version: Optional[str] = None, |
There was a problem hiding this comment.
As suggested by @mcember
Should we also optionally accept a version?
Should namespace here go after *args? It feels weird to have positional arguments in the order [table, namespace], since people will think of it as the hierarchy from left to right [namespace, table]. At least in create table I thought to demote it from positional argument into keyword only argument for this reason
There was a problem hiding this comment.
Added an optional TableVersion parameter. Also decided to ensure consistency with param ordering.
positional params,
*args,
optional params,
**kwargs
| *args, | ||
| namespace: Optional[str] = None, | ||
| table_version: Optional[str] = None, | ||
| stream_format: StreamFormat = StreamFormat.DELTACAT, |
There was a problem hiding this comment.
As suggested by @mattcember
In the future when we have multiple streams, how are users going to specify which stream they want in get / list / etc?
There was a problem hiding this comment.
Added an explicit stream_format parameter to specify the stream. It defaults to the Deltacat stream.
Summary
This PR continues work on the catalog interface layer for Deltacat. It primarily focuses on implementing table related functions.
Functions Implemented:
Also includes minor bugfixes and better exception handling in the storage interface layer.
Testing
make testChecklist
Unit tests covering the changes have been added
E2E testing has been performed
Additional Notes
There are some follow up tasks that have been added as TODO's.