Partition Docs

Test Plan: unit

Reviewers: prha

Reviewed By: prha

Differential Revision: https://dagster.phacility.com/D2010

Author: helloworld

docs/index.rst

@@ -1,7 +1,7 @@
 .. title:: Home
 
 .. toctree::
-  :maxdepth: 3
+  :maxdepth: 5
   :includehidden:
   :name: Documentation
 

docs/sections/api/apidocs/partitions.rst

@@ -0,0 +1,13 @@
+.. currentmodule:: dagster
+
+Partitions
+==========
+
+.. autoclass:: Partition
+.. autoclass:: PartitionSetDefinition
+
+.. autodecorator:: repository_partitions
+
+.. currentmodule:: dagster.utils.partitions
+
+.. autofunction:: date_partition_range

docs/sections/api/apidocs/schedules.rst

@@ -4,6 +4,7 @@ Schedules
 =========
 
 .. autodecorator:: schedules
+.. autodecorator:: monthly_schedule
 .. autodecorator:: hourly_schedule
 .. autodecorator:: daily_schedule
 

docs/sections/api/index.rst

@@ -14,6 +14,7 @@ API Docs
   apidocs/errors
   apidocs/utilities
   apidocs/schedules
+  apidocs/partitions
   apidocs/cli
   apidocs/dagstermill
 

docs/sections/learn/guides/index.rst

@@ -2,10 +2,12 @@ Guides
 =======================
 
 .. toctree::
-  :maxdepth: 1
+  :maxdepth: 2
+
 
   dagster_types
   solid/solid
+  partitioning/index
   testing/testing
   testing/expectations
   logging/logging

docs/sections/learn/guides/partitioning/backfill.png

@@ -0,0 +1,22 @@
+Peforming a backfill
+^^^^^^^^^^^^^^^^^^^^
+
+Now that we have a partition set defined for a pipeline, we can perform a backfill over all the partitions.
+
+To do backfills, make sure you've set up a Run Launcher. In a separate terminal window, run ``dagit`` for our current repository. Now let's run ``dagster pipeline backfill`` and follow the prompts
+
+.. code-block:: console
+
+    $ dagster pipeline backfill compute_total_stock_volume
+        Pipeline: compute_total_stock_volume
+    Partition set: stock_ticker_partition_sets
+    Partitions:
+                    AAPL            GOOG            MSFT            TSLA
+
+    Do you want to proceed with the backfill (4 partitions)? [y/N]: t
+    Launching runs...
+    Launched backfill job `uufbdyqg`
+
+The backfill script just launched 4 runs configured individually for each partition. If we load dagit and filter for the tag ``dagster/backfill=uufbdyqg`` we'll be able to see our four runs:
+
+.. thumbnail:: backfill.png

docs/sections/learn/guides/partitioning/backfill.rst

@@ -0,0 +1,44 @@
+Creating a Date Partition Set
+===============================
+
+The most common partitioning scheme is date based partitions. We'll now create a :py:class:`PartitionSetDefinition <dagster.PartitionSetDefinition>` for a set of dates.
+
+First, let's modify our solid to take a date range as config:
+
+
+.. literalinclude:: ../../../../../examples/dagster_examples/stocks/repository.py
+   :linenos:
+   :lines: 4-19
+   :language: python
+   :caption: repository.py
+
+
+Just as before, we write a function that returns a list of partitions, but this time we return a list of ``datetime`` objects.
+
+.. literalinclude:: ../../../../../examples/dagster_examples/stocks/date_partitions.py
+   :linenos:
+   :lines: 9-15
+   :language: python
+   :caption: repository.py
+
+
+Next, we define a function that takes a date :py:class:`Partition <dagster.Partition>` and returns config. Here, we calculate the first and last days of the previous month for each date partition to pass to the solid config.
+
+.. literalinclude:: ../../../../../examples/dagster_examples/stocks/date_partitions.py
+   :linenos:
+   :lines: 18-34
+   :language: python
+   :caption: repository.py
+
+Since it's not practical to hardcode all the possible date partitions, and we usually want date partitions between two date ranges, Dagster provides a utility to generate a list of partitions given a date range and time interval: :py:func:`date_partition_range <dagster.utils.partitions.date_partition_range>`. We'll use this function instead of the one we wrote above.
+
+.. literalinclude:: ../../../../../examples/dagster_examples/stocks/date_partitions.py
+   :linenos:
+   :lines: 37-46
+   :language: python
+   :caption: repository.py
+
+
+Now, let's load dagit again and head to the Playground tab. This time, we'll see our date partitions in the partition selector.
+
+.. thumbnail:: playground_date_partitions.png

docs/sections/learn/guides/partitioning/config_1.png

@@ -0,0 +1,45 @@
+Partitioning and Scheduling Guide
+=================================
+
+Tutorial
+^^^^^^^^
+
+.. toctree::
+  :maxdepth: 2
+
+  partition_pipeline
+  partition_set
+  backfill
+  date_partition_set
+  schedule_partitions
+
+You can find all of the code for this guide checked into the dagster repository at ``dagster/examples/dagster_examples/stocks``
+
+In this guide, we'll learn about why we partition data and how to write Dagster pipelines that can easily work with partitioned data. We'll also explore how to schedule partitioned work using Dagster's built-in scheduler functionalities.
+
+Partitioning and Scheduling in Dagster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Many data engineering pipelines need to be written so they operate on partitioned segments of data. These pipelines are also often schedules so that they execute partitions of data at some interval.
+
+For example, lets say we have a metrics pipeline that partitions data based on 24 hour time intervals. We will probably want to schedule this pipeline to run every day at midnight on the previous 24 hours' metrics data. With this setup, we'll have data continuously processed for every 24 hour segment.
+
+Traditional workflow systems tightly couple partitioning and scheduling and provide solutions to date partitioned schedules like this metrics pipeline. However, as we've seen, there are many other ways to partition data, and these two concepts do not need to be completely intertwined. Being able to partition data on axis other than time is especially useful when conducting backfills.
+
+Dagster aims to provides a flexible and generalized solution to partitioning, backfills, and scheduling that can extend beyond datetime partitioning, while also providing a set of easy to use tools for the common datetime partitioning use case.
+
+In the next part of the tutorial, we'll build a partitioned schedule.
+
+What is scheduling?
+^^^^^^^^^^^^^^^^^^^
+
+The term "scheduling" is often overloaded in the context of workflow engines like Dagster. In this case, scheduling pipeline runs specifically refers to triggering pipeline runs to happen at some interval. For example, I might want to schedule an email marketing pipeline once a day at 9AM.
+
+What is data partitioning?
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When working with small datasets, we can easily make queries or run calculations over the entire dataset at once. Even after updating the base dataset, it's not expensive to redo queries and calculations to get new answers.
+
+When datasets get extremely large it becomes difficult to work with the entire dataset in once piece. Queries become slower and inefficient and even fast data processing tools can take a long time to process extremely large quantities of data.
+
+When we partition data, we break it up into discrete chunks using some partitioning scheme. For example, if we're working with a large metrics dataset, we might want to partition the data by country or other customer segments. It's often easier to reason about data when segmented into such partitions. Also, with smaller units of the data, we can process the partitions with less effort and, if there is no dependence between partitions, we can process the partitions in parallel.