Skip to content
This repository was archived by the owner on Jul 30, 2025. It is now read-only.

Commit 8bc51ba

Browse files
authored
update to processor v2 repo (#905)
### Description ### Checklist <!-- Read the Nextra Contributor's Guide here: https://aptos.dev/en/developer-platforms/contribute --> - If any existing pages were renamed or removed: - [ ] Were redirects added to [next.config.mjs](../apps/nextra/next.config.mjs)? - [ ] Did you update any relative links that pointed to the renamed / removed pages? - Do all Lints pass? - [ ] Have you ran `pnpm fmt`? - [ ] Have you ran `pnpm lint`?
1 parent cdb0d7c commit 8bc51ba

File tree

7 files changed

+32
-45
lines changed

7 files changed

+32
-45
lines changed

apps/nextra/components/beta-notice/IndexerBetaNotice.tsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ export function IndexerBetaNotice() {
77
currently in beta. Please report any problems you encounter by creating an
88
issue in the{" "}
99
<a
10-
href="https://github.com/aptos-labs/aptos-indexer-processors/issues/new/choose"
10+
href="https://github.com/aptos-labs/aptos-indexer-processors-v2/issues/new/choose"
1111
className="_text-primary-600 _underline _decoration-from-font"
1212
style={{ textUnderlinePosition: "from-font" }}
1313
>

apps/nextra/pages/en/build/apis/data-providers.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Indexer layer on top of this will provide a [GRPC transaction stream](../indexer
1616

1717
On top of this transaction stream, we've built out some product logic tables that can be queried through [GraphQL](../indexer/)
1818

19-
Since the logic to parse out transaction is [public](https://github.com/aptos-labs/aptos-indexer-processors), some vendors have implemented similar parsing logic to create a subset of tables and made them available to query.
19+
Since the logic to parse out transaction is [public](https://github.com/aptos-labs/aptos-indexer-processors-v2), some vendors have implemented similar parsing logic to create a subset of tables and made them available to query.
2020

2121
## SQL Tables
2222

apps/nextra/pages/en/build/indexer/indexer-api/self-hosted.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ From here you will likely want to change the values of some of these fields. Let
4444
A single instance of the service only runs a single processor. If you want to run multiple processors, you must run multiple instances of the service. In this case, it is up to you whether to use the same database or not.
4545
</Callout>
4646

47-
This is the processor you want to run. You can see what processors are available [here](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/processors/mod.rs#L23). Some examples:
47+
This is the processor you want to run. You can see what processors are available [here](https://github.com/aptos-labs/aptos-indexer-processors-v2/tree/main/processor/src/processors). Some examples:
4848

4949
- `coin_processor`
5050
- `ans_processor`
@@ -72,10 +72,10 @@ Clone the repo:
7272

7373
```bash filename="Terminal"
7474
# SSH
75-
git clone git@github.com:aptos-labs/aptos-indexer-processors.git
75+
git clone git@github.com:aptos-labs/aptos-indexer-processors-v2.git
7676
7777
# HTTPS
78-
git clone https://github.com/aptos-labs/aptos-indexer-processors.git
78+
git clone https://github.com/aptos-labs/aptos-indexer-processors-v2.git
7979
```
8080

8181
Navigate to the directory for the service:

apps/nextra/pages/en/build/indexer/indexer-sdk.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ Learn how to use the Indexer SDK through guides and documentation.
2828

2929

3030
## Example Processors
31-
As a reference, you can see all Aptos-Hosted processors that comprise the Indexer API [here](https://github.com/aptos-labs/aptos-indexer-processors).
31+
As a reference, you can see all Aptos-Hosted processors that comprise the Indexer API [here](https://github.com/aptos-labs/aptos-indexer-processors-v2).

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/create-processor.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ For basic cases, you can copy the [`IndexerProcessorConfig` from the `aptos-inde
3737
It's used by `IndexerProcessorConfig.run()` to map the processor name to the right `ProcessorConfig`.
3838

3939
You can see a basic example of a `ProcessorConfig` [here](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/config/processor_config.rs).
40-
An example of a more complex setup that includes multiple processors and configurations is [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/config/processor_config.rs#L84).
40+
An example of a more complex setup that includes multiple processors and configurations is [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/config/processor_config.rs#L84).
4141

4242
## How to create a processor
4343
Now that you've got the configuration pieces set up, the next step is to create the processor.
@@ -102,7 +102,7 @@ loop {
102102
```
103103

104104
You can see a full example of a processor that indexes raw Aptos events in [`aptos-indexer-processor-example`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/processors/events/events_processor.rs).
105-
As a reference, you can also see all of the processors that make up the [Indexer API](../../indexer-api.mdx) in [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors/tree/main/rust/sdk-processor/src/processors).
105+
As a reference, you can also see all of the processors that make up the [Indexer API](../../indexer-api.mdx) in [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors-v2/tree/main/processor/src/processors).
106106

107107
## How to define `main.rs`
108108
You may copy the [`main.rs`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/main.rs) file from the `aptos-indexer-processor-example`.

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/define-schema.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ By using the indexer, a single indexer query can often replace multiple queries
1717
## Aptos Core Processors
1818
All data exposed by the [Indexer API](../../indexer-api.mdx) is initially indexed using custom processors.
1919
Each core processor indexes a specific type of data.
20-
You can explore the [full list of processors](https://github.com/aptos-labs/aptos-indexer-processors/tree/main/rust/sdk-processor/src/processors) and the [complete indexer data schema](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/db/postgres/schema.rs).
20+
You can explore the [full list of processors](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/db/schema.rs).
2121

2222
The Aptos core processors and the [Quickstart Guide](../quickstart.mdx) use [PostgreSQL](https://www.postgresql.org/) as the database and [Diesel](https://diesel.rs/) as the ORM.
2323
If you'd also like to use PostgreSQL and Diesel, you can follow the instructions in [PostgreSQL Installation](../quickstart.mdx#postgresql-installation).

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/version-tracking.mdx

Lines changed: 23 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ title: "Version Tracking"
55
# Version Tracking, Restart Behavior, and Backfilling
66

77
## Version Tracking
8-
The `VersionTrackerStep` is a [common step in the SDK](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L29C12-L29C30) that checkpoints indexer progress. The struct requires implementation of the [`ProcessorStatusSaver` trait](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L16) and can be used with the same [`.connect_to()`](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/processors/events_processor.rs#L117) method as other steps. Upon a successfully processed batch, the `VersionTrackerStep` will [call](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L57) the trait implementation of `save_processor_status()`.
8+
The `VersionTrackerStep` is a [common step in the SDK](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/events/events_processor.rs#L125) method as other steps. Upon a successfully processed batch, the `VersionTrackerStep` will [call](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L57) the trait implementation of `save_processor_status()`.
99

1010
### ProcessorStatusSaver
1111
The `ProcessorStatusSaver` trait requires the implementation of the method `save_processor_status` with the following signature:
@@ -16,11 +16,17 @@ async fn save_processor_status(
1616
last_success_batch: &TransactionContext<()>,
1717
) -> Result<(), ProcessorError>;
1818
```
19-
This method is where checkpointing should be written. Examples can be found [here](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L23). It is possible to checkpoint progress in different ways by using enums. [This example](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L131) inserts into Postgres using a simple [`processor_status` model](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/db/postgres/models/processor_status.rs).
19+
This method is where checkpointing should be written.
20+
If you're writing to Postgres, you can use the SDK's Postgres implementation [here](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/utils/checkpoint.rs#L66).
21+
It is possible to checkpoint progress in different ways by using enums.
22+
The SDK's Postgres implementation inserts using a simple [`processor_status` model](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/models/processor_status.rs).
2023

2124

2225
## Restart Behavior
23-
Now that the processor successfully writes to the chosen store for version tracking, upon restarting it needs to retrieve the latest successful version from that store. [Here is an example](https://github.com/aptos-labs/aptos-indexer-processors/blob/29f4e9c2266db11cbec2177c17fc210d4c687d32/rust/sdk-processor/src/utils/starting_version.rs#L26) of a `get_starting_version()` method that returns the latest processed version saved. This `starting_version: u64` can then be used as below. If there is no checkpoint, the processor will start from the beginning of the chain.
26+
Now that the processor successfully writes to the chosen store for version tracking, upon restarting it needs to retrieve the latest successful version from that store.
27+
[Here is an example](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/utils/checkpoint.rs#L118) of a `get_starting_version()` method that returns the latest processed version saved.
28+
This `starting_version: u64` can then be used as below.
29+
If there is no checkpoint, the processor will start from the beginning of the chain.
2430

2531
```rust
2632
let transaction_stream = TransactionStreamStep::new(TransactionStreamConfig {
@@ -32,47 +38,28 @@ Now that the processor successfully writes to the chosen store for version track
3238

3339

3440
## Backfilling
35-
To enable backfilling, `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` need some updates. Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor.
41+
The SDK does not provide an implementation of `ProcessorStatusSaver` that will save backfill progress.
42+
To enable saving backfill progress, `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` need some updates.
43+
Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor.
3644

3745
### Updates to Config
38-
[Add an additional field](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/config/indexer_processor_config.rs#L20) on your `IndexerProcessorConfig` for a `BackfillConfig` which contains a single field, `backfill_alias`. The introduction of this field will be used to determine whether the processor is a backfill processor or regular one.
39-
40-
```rust
41-
#[derive(Clone, Debug, Deserialize, Serialize)]
42-
#[serde(deny_unknown_fields)]
43-
pub struct IndexerProcessorConfig {
44-
pub processor_config: ProcessorConfig,
45-
pub transaction_stream_config: TransactionStreamConfig,
46-
pub db_config: DbConfig,
47-
pub backfill_config: Option<BackfillConfig>,
48-
}
49-
50-
#[derive(Debug, Clone, Serialize, Deserialize)]
51-
pub struct BackfillConfig {
52-
pub backfill_alias: String,
53-
}
54-
```
46+
[Add an additional field](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/config/processor_mode.rs#L29) on your `IndexerProcessorConfig` for a `BackfillConfig`.
47+
In this implementation, the `BackfillConfig` is part of an enum `ProcessorMode` that is used to determine the mode the processor is running in.
48+
In backfill mode, the processor starts from a different version and the progress is saved in a separate table.
5549

5650
### Updates to `config.yaml`
57-
Add the `backfill_config` section to `server_config` in your yaml file to set `backfill_alias`.
58-
59-
```yaml
60-
server_config:
61-
processor_config:
62-
type: "events_processor"
63-
...
64-
backfill_config:
65-
backfill_alias: "events_processor_backfill"
66-
```
67-
51+
Add the `backfill_config` section to `server_config` in your yaml file to set `backfill_alias`. [Example](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/example-backfill-config.yaml)
6852

6953
### Backfill Processor Status Table
70-
Use a separate table for backfill processor status to avoid write conflicts. This table (`backfill_processor_status_table`) uses `backfill_alias` as the primary key instead of `processor_name` to prevent conflicts with the main `processor_status` table when running head and backfill processors concurrently. Create multiple backfill processors with differing `backfill_alias` and transaction version ranges for a faster backfill.
71-
Expand on this [implementation](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/db/common/models/backfill_processor_status.rs). This model introduces a new state, `BackfillStatus`, which is either `InProgress` or `Complete` which will determine the backfilling restart behavior.
54+
Use a separate table for backfill processor status to avoid write conflicts. This table (`backfill_processor_status_table`) uses `backfill_alias` as the primary key instead of `processor_name` to prevent conflicts with the main `processor_status` table when running head and backfill processors concurrently.
55+
Create multiple backfill processors with differing `backfill_alias` and transaction version ranges for a faster backfill.
56+
Expand on this [implementation](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/db/backfill_processor_status.rs).
57+
This model introduces a new state, `BackfillStatus`, which is either `InProgress` or `Complete` which will determine the backfilling restart behavior.
7258

7359
### Updates to ProcessorStatusSaver
74-
Expand your `ProcessorStatusSaverEnum` to include a `Backfill` variant that extracts the `backfill_alias` from the `BackfillConfig`, and the `backfill_start_version` `backfill_end_version` from `IndexerProcessorConfig.transaction_stream_config` [like this](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L27). Update the corresponding write query to write to the new `backfill_processor_status` table.
60+
Expand your `ProcessorStatusSaver` implementation to include a `Backfill` variant that extracts the `backfill_alias` from the `BackfillConfig`, and the `backfill_start_version` `backfill_end_version` from `IndexerProcessorConfig.transaction_stream_config` [like this](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/processor_status_saver.rs#L96).
61+
Update the corresponding write query to write to the new `backfill_processor_status` table.
7562

7663
### Updates to get_starting_version
77-
Add a [statement](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/utils/starting_version.rs#L143) in your `get_starting_version` method to query the `backfill_processor_status_table` when the `BackfillConfig` field is present in `IndexerProcessorConfig` .
64+
Add a [statement](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/processor_status_saver.rs#L190) in your `get_starting_version` method to query the `backfill_processor_status_table` when the `BackfillConfig` field is present in `IndexerProcessorConfig` .
7865

0 commit comments

Comments
 (0)