connect steps

Author: rtso

apps/nextra/next.config.mjs

@@ -423,6 +423,12 @@ export default withBundleAnalyzer(
         destination: "/en/build/indexer/indexer-sdk/documentation/parsing-txns",
         permanent: true,
       },
+      {
+        source: "/indexer/indexer-sdk/documentation/connect-steps",
+        destination:
+          "/en/build/indexer/indexer-sdk/documentation/connect-steps",
+        permanent: true,
+      },
       {
         source: "/indexer/txn-stream/labs-hosted",
         destination: "/en/build/indexer/api/labs-hosted",

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/_meta.tsx

@@ -9,4 +9,7 @@ export default {
   "parsing-txns": {
     title: "Parsing Transactions",
   },
+  "connect-steps": {
+    title: "Connecting Steps",
+  },
 };

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/connect-steps.mdx

@@ -0,0 +1,50 @@
+---
+title: "Connecting Steps"
+---
+
+# Connecting Steps
+
+## Pre-requisite
+At this point, you'd have already followed the [Creating a Processor](./create-processor.mdx) and [Creating a Step](./steps.mdx) guides.
+Our next goal is to put those two pieces together and connect steps within the processor.
+
+## How to connect steps 
+
+Now that you have created a step, you can connect it to other steps.
+To do so, we use a builder class called `ProcessorBuilder` to specifiy a sequence of steps that make up a processor.
+
+1. After you've instantiated your steps, you need to convert them into [`RunnableStep`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/traits/runnable_step.rs#L6).
+`RunnableStep` is a trait that wraps around a step.
+It provides the necessary input and output channels that feed into the step and allows the step to be spawned in a task.
+The SDK provides a helper function [`.into_runnable_step`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/traits/into_runnable_step.rs#L13) to convert a step into a `RunnableStep`.
+2. Setup your first step with [`ProcessorBuilder::new_with_inputless_first_step`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/builder/processor_builder.rs#L222). 
+In almost all cases, the first step should be a `TransactionStreamStep`. {/* <!-- Add link --> */} 
+3. Connect the previous step to the next step using [`.connect_to`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/builder/processor_builder.rs#L303). 
+ `connect_to` uses trait bounds to ensure at compile time that the output type of the previous step matches the input type of the next step.
+ When calling `.connect_to`, a channel gets created with size `channel_size` and connects the previous and next steps. 
+ It also handles spawning the previous step in a task.
+4. Repeat step 3 for each step in your processor.
+5. To close off the `ProcessorBuilder`, use [`.end_and_return_output_receiver`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/builder/processor_builder.rs#L400). 
+This return an [`InstrumentedAsyncReceiver`](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/instrumented-channel/src/lib.rs#L88) which you can use to process the output of the last step in the graph. 
+
+Here's a simple example of connecting two steps:
+```rust
+let (processor_builder, buffer_receiver) = ProcessorBuilder::new_with_inputless_first_step(
+      transaction_stream_step.into_runnable_step(),
+  )
+  .connect_to(extractor_step.into_runnable_step(), 10)
+  .end_and_return_output_receiver(10);
+```
+
+Here's a [full example](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/processors/events/events_processor.rs#L75) from `aptos-indexer-processor-example`. 
+
+{/* <!-- Add link to common SDK steps --> */} 
+
+## Visualizing the processor
+As you connect steps, `ProcessorBuilder` in the background is constructing a graphical representation of the steps in your processor using [`petgraph`](https://docs.rs/petgraph/latest/petgraph/).
+You can see the visual representation of the graph by calling
+```rust
+let dot = processor_builder.graph.dot();
+println!("{}", dot);
+```
+This will output a graph in the [DOT language](https://graphviz.gitlab.io/_pages/doc/info/lang.html) that you can visualize using tools like [Graphviz](https://graphviz.org/).

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/create-processor.mdx

@@ -101,7 +101,7 @@ loop {
 }
 ```
 
-You can see a full example of a processor that indexes raw Aptos events in [`aptos-indexer-processor-example`](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/processors/events_processor.rs). 
+You can see a full example of a processor that indexes raw Aptos events in [`aptos-indexer-processor-example`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/processors/events/events_processor.rs). 
 As a reference, you can also see all of the processors that make up the [Indexer API](../../aptos-hosted.mdx) in [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors/tree/main/rust/sdk-processor/src/processors).
 
 ## How to define `main.rs`

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/steps.mdx

@@ -87,30 +87,6 @@ To create a step with the SDK, follow these instructions:
         }
         ```
 
-## How to connect steps
-
-Now that you have created a step, you can connect it to other steps in the processor. 
-`ProcessorBuilder` is used to connect a graph of steps to construct a processor. 
-It uses trait bounds to ensure that the output type of each step matches the input type of its connected step. 
-
-### How to use `ProcessorBuilder`
-
-1. Initialize the processor with the first step using `ProcessorBuilder::new_with_inputless_first_step`. 
-2. Connect the next step using `.connect_to(second_step.into_runnable_step(), channel_size)`. 
-- `.into_runnable_step()` converts your step into a `RunnableStep`, which enables it to store the step's input and output channels and allows the step to be spawned in a task. 
-- When you use `.connect_to`, a channel gets created with size `channel_size` and connected to the previous and current steps, and the previous step is spawned in a task. 
-3. To close off the `ProcessorBuilder`, use `.end_and_return_output_receiver(channel_size)`. This returns a channel receiver that can be used to receive the final output of the processor.
-
-### Example
-```rust
-let (pb, buffer_receiver) = ProcessorBuilder::new_with_inputless_first_step(
-      first_step.into_runnable_step(),
-  )
-  .connect_to(second_step.into_runnable_step(), channel_size)
-  .connect_to(third_step.into_runnable_step(), channel_size)
-  .end_and_return_output_receiver(channel_size);
-```
-
 ## Common steps
 
 The SDK provides several common steps that you can use in your processor.