Best practice for project layout

[nagyj2]

Im looking to start using prefect and am trying to find best practices for how to structure my git project but have been unable to find many references in the documentation. I did find a discussion on flow storage which had the intention of delivering guidance but I couldnt find anything that came out of it. It also doesnt cover on how other aspects of the project are stored.

From what Ive found, tasks and flows should be kept together at the project root level along with any scripts to deploy the flows to the prefect server (though maybe they should go in a dedicated scripts folder?) Other project standard, like tests and docs also are at root. The root should contain the prefect.yaml output of prefect init. I am currently using a docker compose to set up my prefect server but I am unsure if the docker fields in the prefect.yaml are meaning the prefect server or some docker image that represents my output application. Currently the compose file is ina docker directory.

Below is the structure I have been thinking about

.
├── docker
├── docs
├── flows
│   ├── flow.py
│   └── deploy.sh
├──  tests
└── prefect.yaml

I dont have any experience with prefect or airflow so any help would be greatly appreciated.

[bdalpe]

Hi @nagyj2,

I think it would be good if we put our thoughts into the official docs, though it's hard to give a definitive answer because Prefect is pretty agnostic about repo layouts.

Basically the hard requirements are:

• your flow code is importable by Python
• each deployment can point at a flow “entrypoint” like path/to/file.py:flow_fn
• your deployment config lives in prefect.yaml (by default at repo root) and tells Prefect how to prepare code at run time via pull (and optionally build and push)

A personally-opinionated suggestion for a single project:

.
├── prefect.yaml
├── .prefectignore
├── pyproject.toml            # or requirements.txt
├── src/
│   └── my_project/
│       ├── flows/
│       │   └── flow.py
│       ├── tasks/
│       └── common/           # helpers, clients, config loaders, etc
├── scripts/
│   └── deploy.sh             # optional wrapper around `prefect deploy`
├── infra/                    # or docker/
│   └── docker-compose.yml    # Prefect Server (and maybe a worker) for local dev
├── tests/
└── docs/

By breaking pieces of your code apart and separating them into different modules like flows, tasks, common, etc., it makes it easier to perform unit tests on (if that's something you're interested in doing).

I'd also look at using a .prefectignore file if you're using a Docker-based deployment, otherwise your entire project structure gets copied into the Docker container. The .prefectignore file has the same syntax and operates exactly like a .gitignore file does.