You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 30, 2025. It is now read-only.
### Description
### Checklist
<!-- Read the Nextra Contributor's Guide here: https://aptos.dev/en/developer-platforms/contribute -->
- If any existing pages were renamed or removed:
- [ ] Were redirects added to [next.config.mjs](../apps/nextra/next.config.mjs)?
- [ ] Did you update any relative links that pointed to the renamed / removed pages?
- Do all Lints pass?
- [ ] Have you ran `pnpm fmt`?
- [ ] Have you ran `pnpm lint`?
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/apis/data-providers.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ Indexer layer on top of this will provide a [GRPC transaction stream](../indexer
16
16
17
17
On top of this transaction stream, we've built out some product logic tables that can be queried through [GraphQL](../indexer/)
18
18
19
-
Since the logic to parse out transaction is [public](https://github.com/aptos-labs/aptos-indexer-processors), some vendors have implemented similar parsing logic to create a subset of tables and made them available to query.
19
+
Since the logic to parse out transaction is [public](https://github.com/aptos-labs/aptos-indexer-processors-v2), some vendors have implemented similar parsing logic to create a subset of tables and made them available to query.
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/indexer-api/self-hosted.mdx
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -44,7 +44,7 @@ From here you will likely want to change the values of some of these fields. Let
44
44
A single instance of the service only runs a single processor. If you want to run multiple processors, you must run multiple instances of the service. In this case, it is up to you whether to use the same database or not.
45
45
</Callout>
46
46
47
-
This is the processor you want to run. You can see what processors are available [here](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/processors/mod.rs#L23). Some examples:
47
+
This is the processor you want to run. You can see what processors are available [here](https://github.com/aptos-labs/aptos-indexer-processors-v2/tree/main/processor/src/processors). Some examples:
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/indexer-sdk.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,4 +28,4 @@ Learn how to use the Indexer SDK through guides and documentation.
28
28
29
29
30
30
## Example Processors
31
-
As a reference, you can see all Aptos-Hosted processors that comprise the Indexer API [here](https://github.com/aptos-labs/aptos-indexer-processors).
31
+
As a reference, you can see all Aptos-Hosted processors that comprise the Indexer API [here](https://github.com/aptos-labs/aptos-indexer-processors-v2).
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/create-processor.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,7 @@ For basic cases, you can copy the [`IndexerProcessorConfig` from the `aptos-inde
37
37
It's used by `IndexerProcessorConfig.run()` to map the processor name to the right `ProcessorConfig`.
38
38
39
39
You can see a basic example of a `ProcessorConfig`[here](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/config/processor_config.rs).
40
-
An example of a more complex setup that includes multiple processors and configurations is [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/config/processor_config.rs#L84).
40
+
An example of a more complex setup that includes multiple processors and configurations is [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/config/processor_config.rs#L84).
41
41
42
42
## How to create a processor
43
43
Now that you've got the configuration pieces set up, the next step is to create the processor.
@@ -102,7 +102,7 @@ loop {
102
102
```
103
103
104
104
You can see a full example of a processor that indexes raw Aptos events in [`aptos-indexer-processor-example`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/processors/events/events_processor.rs).
105
-
As a reference, you can also see all of the processors that make up the [Indexer API](../../indexer-api.mdx) in [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors/tree/main/rust/sdk-processor/src/processors).
105
+
As a reference, you can also see all of the processors that make up the [Indexer API](../../indexer-api.mdx) in [`aptos-indexer-processors`](https://github.com/aptos-labs/aptos-indexer-processors-v2/tree/main/processor/src/processors).
106
106
107
107
## How to define `main.rs`
108
108
You may copy the [`main.rs`](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/main.rs) file from the `aptos-indexer-processor-example`.
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/define-schema.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ By using the indexer, a single indexer query can often replace multiple queries
17
17
## Aptos Core Processors
18
18
All data exposed by the [Indexer API](../../indexer-api.mdx) is initially indexed using custom processors.
19
19
Each core processor indexes a specific type of data.
20
-
You can explore the [full list of processors](https://github.com/aptos-labs/aptos-indexer-processors/tree/main/rust/sdk-processor/src/processors) and the [complete indexer data schema](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/db/postgres/schema.rs).
20
+
You can explore the [full list of processors](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/db/schema.rs).
21
21
22
22
The Aptos core processors and the [Quickstart Guide](../quickstart.mdx) use [PostgreSQL](https://www.postgresql.org/) as the database and [Diesel](https://diesel.rs/) as the ORM.
23
23
If you'd also like to use PostgreSQL and Diesel, you can follow the instructions in [PostgreSQL Installation](../quickstart.mdx#postgresql-installation).
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/version-tracking.mdx
+23-36Lines changed: 23 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ title: "Version Tracking"
5
5
# Version Tracking, Restart Behavior, and Backfilling
6
6
7
7
## Version Tracking
8
-
The `VersionTrackerStep` is a [common step in the SDK](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L29C12-L29C30) that checkpoints indexer progress. The struct requires implementation of the [`ProcessorStatusSaver` trait](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L16) and can be used with the same [`.connect_to()`](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/processors/events_processor.rs#L117) method as other steps. Upon a successfully processed batch, the `VersionTrackerStep` will [call](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L57) the trait implementation of `save_processor_status()`.
8
+
The `VersionTrackerStep` is a [common step in the SDK](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/events/events_processor.rs#L125) method as other steps. Upon a successfully processed batch, the `VersionTrackerStep` will [call](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/common_steps/version_tracker_step.rs#L57) the trait implementation of `save_processor_status()`.
9
9
10
10
### ProcessorStatusSaver
11
11
The `ProcessorStatusSaver` trait requires the implementation of the method `save_processor_status` with the following signature:
This method is where checkpointing should be written. Examples can be found [here](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L23). It is possible to checkpoint progress in different ways by using enums. [This example](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L131) inserts into Postgres using a simple [`processor_status` model](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/processor/src/db/postgres/models/processor_status.rs).
19
+
This method is where checkpointing should be written.
20
+
If you're writing to Postgres, you can use the SDK's Postgres implementation [here](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/utils/checkpoint.rs#L66).
21
+
It is possible to checkpoint progress in different ways by using enums.
22
+
The SDK's Postgres implementation inserts using a simple [`processor_status` model](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/models/processor_status.rs).
20
23
21
24
22
25
## Restart Behavior
23
-
Now that the processor successfully writes to the chosen store for version tracking, upon restarting it needs to retrieve the latest successful version from that store. [Here is an example](https://github.com/aptos-labs/aptos-indexer-processors/blob/29f4e9c2266db11cbec2177c17fc210d4c687d32/rust/sdk-processor/src/utils/starting_version.rs#L26) of a `get_starting_version()` method that returns the latest processed version saved. This `starting_version: u64` can then be used as below. If there is no checkpoint, the processor will start from the beginning of the chain.
26
+
Now that the processor successfully writes to the chosen store for version tracking, upon restarting it needs to retrieve the latest successful version from that store.
27
+
[Here is an example](https://github.com/aptos-labs/aptos-indexer-processor-sdk/blob/main/aptos-indexer-processors-sdk/sdk/src/postgres/utils/checkpoint.rs#L118) of a `get_starting_version()` method that returns the latest processed version saved.
28
+
This `starting_version: u64` can then be used as below.
29
+
If there is no checkpoint, the processor will start from the beginning of the chain.
@@ -32,47 +38,28 @@ Now that the processor successfully writes to the chosen store for version track
32
38
33
39
34
40
## Backfilling
35
-
To enable backfilling, `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` need some updates. Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor.
41
+
The SDK does not provide an implementation of `ProcessorStatusSaver` that will save backfill progress.
42
+
To enable saving backfill progress, `IndexerProcessorConfig`, `ProcessorStatusSaver` and `get_starting_version()` need some updates.
43
+
Without these changes, it is difficult to run a live processor at the latest transaction version as well as a backfill processor.
36
44
37
45
### Updates to Config
38
-
[Add an additional field](https://github.com/aptos-labs/aptos-indexer-processor-example/blob/main/aptos-indexer-processor-example/src/config/indexer_processor_config.rs#L20) on your `IndexerProcessorConfig` for a `BackfillConfig` which contains a single field, `backfill_alias`. The introduction of this field will be used to determine whether the processor is a backfill processor or regular one.
[Add an additional field](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/config/processor_mode.rs#L29) on your `IndexerProcessorConfig` for a `BackfillConfig`.
47
+
In this implementation, the `BackfillConfig` is part of an enum `ProcessorMode` that is used to determine the mode the processor is running in.
48
+
In backfill mode, the processor starts from a different version and the progress is saved in a separate table.
55
49
56
50
### Updates to `config.yaml`
57
-
Add the `backfill_config` section to `server_config` in your yaml file to set `backfill_alias`.
58
-
59
-
```yaml
60
-
server_config:
61
-
processor_config:
62
-
type: "events_processor"
63
-
...
64
-
backfill_config:
65
-
backfill_alias: "events_processor_backfill"
66
-
```
67
-
51
+
Add the `backfill_config` section to `server_config` in your yaml file to set `backfill_alias`. [Example](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/example-backfill-config.yaml)
68
52
69
53
### Backfill Processor Status Table
70
-
Use a separate table for backfill processor status to avoid write conflicts. This table (`backfill_processor_status_table`) uses `backfill_alias` as the primary key instead of `processor_name` to prevent conflicts with the main `processor_status` table when running head and backfill processors concurrently. Create multiple backfill processors with differing `backfill_alias` and transaction version ranges for a faster backfill.
71
-
Expand on this [implementation](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/db/common/models/backfill_processor_status.rs). This model introduces a new state, `BackfillStatus`, which is either `InProgress` or `Complete` which will determine the backfilling restart behavior.
54
+
Use a separate table for backfill processor status to avoid write conflicts. This table (`backfill_processor_status_table`) uses `backfill_alias` as the primary key instead of `processor_name` to prevent conflicts with the main `processor_status` table when running head and backfill processors concurrently.
55
+
Create multiple backfill processors with differing `backfill_alias` and transaction version ranges for a faster backfill.
56
+
Expand on this [implementation](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/db/backfill_processor_status.rs).
57
+
This model introduces a new state, `BackfillStatus`, which is either `InProgress` or `Complete` which will determine the backfilling restart behavior.
72
58
73
59
### Updates to ProcessorStatusSaver
74
-
Expand your `ProcessorStatusSaverEnum` to include a `Backfill` variant that extracts the `backfill_alias` from the `BackfillConfig`, and the `backfill_start_version` `backfill_end_version` from `IndexerProcessorConfig.transaction_stream_config` [like this](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/steps/common/processor_status_saver.rs#L27). Update the corresponding write query to write to the new `backfill_processor_status` table.
60
+
Expand your `ProcessorStatusSaver` implementation to include a `Backfill` variant that extracts the `backfill_alias` from the `BackfillConfig`, and the `backfill_start_version``backfill_end_version` from `IndexerProcessorConfig.transaction_stream_config`[like this](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/processor_status_saver.rs#L96).
61
+
Update the corresponding write query to write to the new `backfill_processor_status` table.
75
62
76
63
### Updates to get_starting_version
77
-
Add a [statement](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/sdk-processor/src/utils/starting_version.rs#L143) in your `get_starting_version` method to query the `backfill_processor_status_table` when the `BackfillConfig` field is present in `IndexerProcessorConfig` .
64
+
Add a [statement](https://github.com/aptos-labs/aptos-indexer-processors-v2/blob/main/processor/src/processors/processor_status_saver.rs#L190) in your `get_starting_version` method to query the `backfill_processor_status_table` when the `BackfillConfig` field is present in `IndexerProcessorConfig` .
0 commit comments