Enable running local python code/models

[ismailsimsek]

Currently dbt allows running python inside destination system/database(snowpar, pyspark). But for some cases it is useful to run local python scripts

for example web-api->local_dbt_python_model(reads data from api)->converts to pandas.dataframe->save_to_destination_table

for this probably submit_local_python_job method could be created. similar to submit_python_job

https://github.com/dbt-labs/dbt-adapters/blob/2a99e36a7479ec23f40456e89377419aae7388e7/dbt/adapters/base/impl.py#L1423

[ismailsimsek]

will be great to add this to core

[ismailsimsek]

this is now possible with opendbt project.

explained here

[candalfigomoro]

The ability to run local Python models (without Spark) would be especially important since dbt-fal (https://github.com/fal-ai/dbt-fal) has been discontinued.

[ismailsimsek]

@candalfigomoro this is implemented now with opendbt project.

Documented here: https://github.com/memiiso/opendbt/blob/main/docs/EXAMPLES.md#execute-python-model-locally
Unit test (Example): https://github.com/memiiso/opendbt/blob/0da7344b1eecb4d91582d33c5aab78831cb59959/tests/test_executepython_materialization.py#L11

[candalfigomoro]

@ismailsimsek Thanks, I looked at the examples but it seems cumbersome compared to having a simple native implementation in dbt.

[ismailsimsek]

Related dbt-labs/dbt-adapters#261

[mrcool4]

We are also interested in this feature