Managing Dependency

[ritab-snow]

When we look at the execution order of dbt models, the python script always runs at the end of all the models.
Order of model execution can be handled using {{ref()}}. Is there a way that we can control the order of execution for the python script as well ? I want to run my models in the below order.

├── models
│   ├── aggregate
│   │   └── aggr_distance_travelled.sql.  [SEQ5]
│   ├── staging
│   │   ├── base_view_trips.sql.     [SEQ1]
│   │   ├── schema.yml
│   │   └── sources.yml
│   └── transform
│       ├── tfm_trip_aggr.sql	     [SEQ3]
│       └── tfm_trip_distance.sql.   [SEQ2]
├── python_transform
│   └── tfm_getzipcode.py            [SEQ4]

How can I make this possible ?

[chamini2]

I guess this probably means tfm_getzipcode write to the data warehouse? In that case, I would recommend you move to Python Data Models, a new way of writing fal scripts to be represented as a dbt model.

[ritab-snow]

Thanks you for your response. I am aware that we can write back to the data warehouse from python scripts. It's a wonderful feature. In my project I wanted to perform some heavy weight aggregation using a .sql file in the data warehouse itself after the python script is run. If this is limitation now then I will have to add the aggregation step within the python script as well.
Can I conclude that as of this version of fal, the python script will always be the last step in the workflow ?

[chamini2]

No, Python can be in the middle of your workflow. You can achieve that with Python Data Models. Can you check out those docs and see if it fits your setup?

You can also reach out in Discord for faster feedback if you want.

[ritab-snow]

Thanks again. I checked the document but then I did not find an answer if a python script can be run somewhere in the middle of a dbt model. If you can share some quick links that will really help.

[ritab-snow]

Many Thanks. I tried this. It works !! Regards, Ritab

…

On Thu, Jun 23, 2022 at 4:23 PM Meder Kamalov ***@***.***> wrote: @ritab-snow <https://github.com/ritab-snow> thank you for using fal! We're about to announce a special type of a fal python script called "post-hook". In your schema.yml you can have something like: - name: tfm_trip_aggr meta: fal: post-hook: - tfm_getzipcode.py and run fal flow run --experimental-models, then my_post_hook.py will run after tfm_trip_aggr and before aggr_distance_travelled. Beware that there are two functions that don't work with post hooks: write_to_model and write_to_source. If you would like your Python scripts to write to a data warehouse using write_to_model, we recommend trying a Python model as @chamini2 <https://github.com/chamini2> suggested. In this case, you can put tfm_getzipcode.py in your models directory and do a ref to it in aggr_distance_travelled (even just in comments) and then tfm_getzipcode.py will run in required order. — Reply to this email directly, view it on GitHub <https://github.com/fal-ai/fal/discussions/414#discussioncomment-3009434>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZDCNAVUH7ADPFLJLVO3BDDVQQ62HANCNFSM5ZNKX36A> . You are receiving this because you were mentioned.Message ID: ***@***.***>