DAG indexer follows chain-reported finality, leaving ~2.6 days of uploads unservable

[jim-counter]

DAG indexer follows chain-reported finality, leaving ~2.6 days of uploads unservable

Summary

The DAG indexer targets the chain's reported finalized head instead of the chain tip. On Autonomys mainnet that head currently trails best by ~37,600 blocks (~2.6 days), so any file uploaded in that window has no row in "dag-indexer".nodes and GET /files/:cid/metadata returns 404.

The cause is a single boolean in services/dag-indexer/project.ts: historical: false silently forces unfinalizedBlocks to false, discarding the --unfinalized-blocks=true flag we pass on the command line. This has been the case since the indexer was first added (2025-07-30). Nothing was changed or mis-deployed; the wall-clock cost of the misconfiguration grew until it became user-visible.

Note that simply flipping that boolean is not a safe hotfix: SubQuery refuses to run unfinalized-block indexing without historical state, and historical state cannot be turned on for an existing schema. See "Fix options" for the three real routes.

Three separate layers of monitoring report healthy throughout, which is the second half of this issue.

Impact

• Files uploaded within roughly the last 2.6 days cannot be retrieved through the gateway, despite being fully archived on the DSN.
• Auto Drive surfaces this as 500 {"error":"Failed to retrieve data","details":"Error fetching file header: 404 Not Found"}, and because 404 reads as permanent, downstream consumers treat the objects as dead rather than retrying.
• The window is not fixed. It is one finality "jump" wide, and jump duration depends on chain throughput, so it can grow without any change on our side.

Evidence

Measured on auto-drive-gateway, 2026-08-03 ~12:30 UTC.

curl -s http://127.0.0.1:30001/meta:

{
  "currentProcessingHeight": 8926444,
  "targetHeight": 8926444,
  "bestHeight": 8964097,
  "indexerNodeVersion": "6.0.4",
  "uptime": 17297798
}

targetHeight is 37,653 blocks behind bestHeight, about 2.6 days at 6s blocks.

That targetHeight is exactly the chain's reported finalized head. Queried independently against the public RPC at the same time:

best=8964049  finalized=8926444  gap=37605

Our own node (http://subspace-node:9944, ghcr.io/autonomys/node:mainnet-2026-jan-20) returns the same finalized number as the public endpoint, so this is a consensus-level property, not a local node or pruning quirk.

The indexer is not stalled or under-resourced

It idles at the finalized head, then catches up in a burst when finality jumps. Observed live:

12:16:02  <UnfinalizedBlocks> Effective finalized header updated to: 8963778
12:16:02  Enqueuing blocks 8917625...8917749
...
12:21:16  Target height: 8,926,444. Current height: 8,926,444
12:21:31  Fully synced, waiting for new blocks

• Resumed at 8,917,625, exactly one past the finalized head measured 20 minutes earlier (8,917,624).
• The jump was 8,820 blocks, roughly 14.7 hours of chain time arriving in one step.
• It cleared those 8,820 blocks in about 5 minutes at ~28 blocks/s, then went idle again.

So throughput is not the problem, and --workers=1 is not the problem. Closing the current 37,653-block gap would take about 22 minutes.

Root cause

@subql/node decides its fetch target here (packages/node-core/src/indexer/fetch.service.ts):

private latestHeight(): number {
  return this.nodeConfig.unfinalizedBlocks ? this.latestBestHeight : this.latestFinalizedHeight;
}

latestFinalizedHeight comes from raw chain_getFinalizedHead. Per the forum explanation of Subspace finality, Autonomys has probabilistic finality and the reported finalized block is artificially delayed rather than a consistent -100, because Substrate only allows a fixed offset and not custom pruning logic. So the raw value is an unsuitable fetch target, and the unfinalizedBlocks ? latestBestHeight branch is the only sane one.

We pass --unfinalized-blocks=true in docker/dag-indexer/docker-compose.yml, but it never takes effect:

1. services/dag-indexer/project.ts declares historical: false in runner.node.options.

2. rebaseArgsWithManifest (packages/node-core/src/utils/configure.ts) copies most manifest options only when the CLI left them undefined, but handles historical separately and copies it unconditionally, so it overrides our --disable-historical=false:

   if (key === 'historical') {
     if (value !== undefined) {
       // THIS IS OPPOSITE
       argvs.historical = value === true ? 'height' : value;
     }
     return;
   }

3. NodeConfig.unfinalizedBlocks then gates on it:

   get unfinalizedBlocks(): boolean {
     if (this._isTest) return false;
     if (this._config.unfinalizedBlocks === false) {
       return false;
     }
     return this.historical !== false;   // historical is false, so this returns false
   }

   Our --unfinalized-blocks=true clears the first gate and is then discarded by the second, with no warning.

4. latestHeight() therefore takes the raw-finality branch.

Note that our subql-node-substrate fork already solves this problem correctly for a different consumer. UnfinalizedBlocksService.updateEffectiveFinalizedHeader() computes bestHeight - finalizedDepth (the Effective finalized header updated to: 8963778 line above, matching --finalized-depth=100), which is exactly what the forum post says finality means on this network. But that value is only used for fork detection and pruning; it never reaches the fetch target. The fork was built to be run with unfinalized-block indexing enabled, and our manifest disables it.

The manifest in use is not the one in git

@subql/node resolves DEFAULT_MANIFEST (project.yaml) for a directory target and never reads project.ts at runtime. project.yaml is gitignored (services/dag-indexer/.gitignore:30) and generated by subql codegen && subql build. The copy on the host is dated 2025-12-02 and its contents match project.ts:

# // Auto-generated , DO NOT EDIT
runner:
  node:
    options:
      unsafe: true
      historical: false
      unfinalizedBlocks: false

This is worth fixing on its own (see follow-ups): the configuration actually governing a production service cannot currently be reviewed, diffed, or reproduced from this repository, and reading project.ts or the compose command line gives a misleading answer.

Why nothing caught it

Three layers were green the whole time:

Signal	Reports
Docker healthcheck (curl -f /ready)	Up 6 months (healthy)
SubQuery's own logs	Fully synced, waiting for new blocks
Proposed /health/dag-indexer in PR #173	lagBlocks = targetHeight - lastProcessedHeight = 0, so status: ok, HTTP 200

All three are correct on their own terms. The indexer really has processed everything up to its target; the target is what is wrong. targetHeight is also only written to _metadata when fillNextBlockBuffer is about to fetch, so it converges on lastProcessedHeight precisely when fetching stops, making that subtraction structurally incapable of detecting this.

/meta has had targetHeight and bestHeight side by side the entire time. Nothing compared them.

Fix options

The obvious fix (set historical: true in the manifest so unfinalizedBlocks takes effect) cannot be applied to the existing schema. Verified against the fork at 1ae076f:

1. store.service.ts:192 sets _historical from getHistoricalStateEnabled(schema), which for an existing schema returns the stored value: store.service.ts:358 does return value ? 'height' : false, so our stored boolean false yields false. The config value is ignored, and only wins via the fresh-schema fallback at store.service.ts:366.

2. project.service.ts:88: isHistorical is storeService.historical, so also false.

3. project.service.ts:384 then hard-exits:

   if (this.nodeConfig.unfinalizedBlocks && !this.isHistorical) {
     exitWithError(
       'Unfinalized blocks cannot be enabled without historical. You will need to reindex your project to enable historical',
       logger
     );
   }

With restart: unless-stopped that is a crash loop. So there are three real options.

Option 1: patch the fork's fetch target (recommended)

Leave historical and unfinalizedBlocks off. Change what feeds the fetch target so it uses the fork's own effectiveFinalizedHeader (bestHeight - finalizedDepth, i.e. best - 100) instead of the raw chain_getFinalizedHead.

This is what the fork was already reaching for: updateEffectiveFinalizedHeader() (unfinalizedBlocks.service.ts:124) computes exactly the right number, and it is currently only consumed for fork detection and pruning, never for the fetch target.

Two changes are needed, not one:

• Source _latestFinalizedHeight (or the BlockTarget emission and nextEndBlockHeight clamp) from the effective header rather than the raw finalized header.
• Drive updateEffectiveFinalizedHeader() from the getBestBlockHead interval. Today it is only called when the raw finalized head advances (fetch.service.ts:123-128) or from processUnfinalizedBlockHeader, which is gated off while unfinalizedBlocks is false. So the effective header is itself ~15 hours stale, and repointing the target at it alone would change nothing.

Safe on the same premise the fork already relies on: best - 100 is below the chain's reorg floor. No re-index, no schema change, no historical storage, no rollback machinery, and no file-retriever changes. Cost is a fork build and image push.

Option 2: fresh schema with historical enabled, re-index

Set historical: true and unfinalizedBlocks: true in the manifest (boolean true, not 'height': the model declares @IsBoolean() historical?: boolean and the rebase maps true to 'height'), point at a fresh --db-schema, re-index, then cut over. Also remove the deprecated and contradictory --disable-historical=false from docker/dag-indexer/docker-compose.yml.

Prerequisites and consequences, all verified:

• btree_gist extension required, or store.service.ts:210 exits with 'Btree_gist extension is required to enable historical data, contact DB admin for support'. Needs the DB admin; the DB is remote.

• id stops being unique. Historical tables gain _id (uuid PK) and _block_range (int8range), and multiple rows may share an id (see sync-helper.test.ts:150-159). Sequelize masks this with a beforeFind hook injecting a _block_range filter, but the file-retriever queries Postgres directly via pg and gets no filter:

  • getDagNode's SELECT * FROM "dag-indexer".nodes WHERE id = $1 would take an arbitrary version.
  • The recursive CTE in getSortedChunksByCid joins n.cid = link_with_idx.cid; any node with two versions multiplies rows and silently corrupts the chunk list.

  DAG nodes are written once, so one version each in practice, but a rollback re-processing a block can create a second, and rollback becomes active under this option. Both queries need upper_inf(_block_range) filters before cutover.

• Per-block fork detection becomes active (indexer.manager.ts:130), adding RPC and write overhead, so the re-index will run slower than the 28 blocks/s measured.

• Re-index cost: ~8.9M blocks from block 1000, on the order of four days at best.

Option 3: accept the gap

Keep the current config and rely on the DSN fallback in PR #173 as the mitigation, accepting a permanent multi-day window served by reconstruction.

Deployment note (applies to any manifest change)

The deployed manifest is the generated project.yaml, not project.ts. Any manifest change must either be rebuilt (yarn dag-indexer build) as part of the deploy or applied to project.yaml on the host. Editing only project.ts has no effect until something rebuilds.

Image drift is not a risk for a restart: the locally cached :latest (sha256:41e2d3a8…) is identical to what the container is already running. It is about twelve hours older than the current tip of autonomys-subql-node, so a docker pull would change the binary.

Verification

Under option 1 or 2, expected recovery for the current 37,653-block gap is about 22 minutes at the observed rate.

curl -s http://127.0.0.1:30001/meta | python3 -c "import sys,json;m=json.load(sys.stdin);print('gap:', m['bestHeight']-m['targetHeight'])"

Should settle between 0 and ~100 rather than ~37,600.

Audit of what else the settings affect

For completeness, nodeConfig.historical has no consumers outside its own getter; every other historical decision reads storeService.historical. nodeConfig.unfinalizedBlocks has exactly these consumers:

Site	Effect when true
fetch.service.ts:165 latestHeight()	Target becomes best height (the intended change)
fetch.service.ts:377 nextEndBlockHeight	Batch end clamps to best, not finalized
fetch.service.ts:129 / :149	BlockTarget event sourced from best (affects /meta and logs)
indexer.manager.ts:130	Per-block fork detection and rollback becomes active
project.service.ts:384	Hard exit unless historical is enabled
unfinalizedBlocks.service.ts:71	Log line only

Follow-ups

• Alert on bestHeight - targetHeight from /meta, and on the frontier block's chain timestamp against wall clock. Do not alert on targetHeight - lastProcessedHeight, which is 0 under this failure mode.
• Stop the deployed manifest being an untracked build artifact: commit project.yaml, or generate it into the image, or have the deploy regenerate it. Log the resolved runner options at startup so the effective config appears in the logs.
• Upstream or fork: make --unfinalized-blocks=true either take effect or emit a warning when historical: false discards it. Silently ignoring an explicitly passed flag is what hid this for a year.
• Fork bug: fetch.service.ts:128 calls registerFinalizedBlock() without await, but the fork changed that method to async (unfinalizedBlocks.service.ts:212). Rejections inside updateEffectiveFinalizedHeader are unhandled and the caller proceeds before it completes. Not the cause of this incident.
• dag-indexer's handleCall catches every error with one generic log line and drops the node, which is the only remaining mechanism that can produce a genuine indexer gap once this is fixed. Surface it as a metric.

Relationship to PR #173

PR #173 was written to contain the symptoms of this bug, on the understanding that the indexer was wedged. It is not wedged, and the "root cause, now measured" section of that PR should be read in light of this issue.

Once the config is fixed, the DAG indexer sits within ~100 blocks (~10 minutes) of head. Object mappings are only published once the segment holding an object is archived, so a correctly configured DAG indexer is fresher than the object-mapping indexer can be. The DSN fallback therefore cannot serve the newest window at all, and reverts to covering genuine indexer gaps (dropped nodes) rather than being the primary read path for recent content.

Three fixes in that PR are independent of all of this and worth landing on their own:

• The link_order chunk-ordering bug in getSortedChunksByCid, which silently corrupts any file with a multi-level DAG (above ~106 MB) on the indexed path today.
• errorMiddleware being exported but never registered.
• Returning a retryable 503 with a reason code instead of a 404 on an indexer miss, since a derived index having no row is not evidence that content is absent from a permanent store.