Skip to content

Fix timeout property data type in ClientConfig for Deltalake. #28912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 16, 2025

Conversation

edsoncezar16
Copy link
Contributor

@edsoncezar16 edsoncezar16 commented Apr 1, 2025

Summary & Motivation

The current implementation of ClientConfig expects the timeout parameter to be an int.

However, this leads to errors when using the io manager, for instance, to load an input:

dagster._core.errors.DagsterExecutionLoadInputError: Error occurred while loading input "xxx" of step "xxx": 

...

The above exception was caused by the following exception:OSError: Generic Config error: failed to parse "120" as Duration 

According to this discussion in the deltalake community, the expected format for this parameter is a string like "120s".

How I Tested These Changes

Running locally with a modification to the DeltalakeDbClient to make the proposed modification effective I was able to use the parameter without the aforementioned error:

class CustomDeltaLakeDbClient(DbClient):
    @staticmethod
    @contextmanager
    def connect(context, table_slice: TableSlice) -> Iterator[TableConnection]:
        
        ...

        storage_options = {
            **{k: str(v) for k, v in storage_options.items() if v is not None},
            **{k: str(v) for k, v in client_options.items() if v is not None},
        }
        # fix timeout setup
        if "timeout" in storage_options:
            storage_options["timeout"] += "s"

        ...        

Changelog

[dagster-deltalake] Fix timeout property data type in ClientConfig to be str instead of int

@garethbrickman
Copy link
Contributor

cc @ion-elgreco for sanity check

@ion-elgreco
Copy link
Contributor

ion-elgreco commented Apr 1, 2025

cc @ion-elgreco for sanity check

I don't maintain the dagster-deltalake, only dagster-delta (https://github.com/ASML-Labs/dagster-delta)

@garethbrickman
Copy link
Contributor

garethbrickman commented May 14, 2025

@edsoncezar16 Line 82 of test_delta_table_resource.py causes pydantic/pyright errors now. Could you fix that?

/workdir/python_modules/libraries/dagster-deltalake/dagster_deltalake_tests/test_delta_table_resource.py:
82:29: Argument of type "Literal[1]" cannot be assigned to parameter "timeout" of type "str \| None" in function "__init__"
Type "Literal[1]" is not assignable to type "str \| None"
"Literal[1]" is not assignable to "str"
"Literal[1]" is not assignable to "None" (reportArgumentTyp

@garethbrickman garethbrickman self-requested a review May 16, 2025 13:37
@garethbrickman garethbrickman merged commit e08bf38 into dagster-io:master May 16, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants