feat(backend): add on-chain portfolio reconciliation and indexer repair jobs

docs/RECONCILIATION_RUNBOOK.md

@@ -0,0 +1,142 @@
+# BACKit On-Chain Portfolio Reconciliation & Indexer Repair Runbook
+
+This runbook outlines the operational workflow for quantifying off-chain database drift, performing dry-run evaluations, executing automated idempotent database repairs, and handling unrecoverable discrepancies.
+
+---
+
+## Overview
+
+BACKit uses PostgreSQL as its frontend read model and Soroban RPC contract events as the source of truth for on-chain prediction market state. Due to RPC interruptions, network re-orgs, indexer lag, or parsing errors, state drift can occur between Soroban events and PostgreSQL (`calls`, `stakes`, `payout_claims`).
+
+The **Reconciliation Service** compares indexed event activity against PostgreSQL records for a given ledger range (`fromLedger` to `toLedger`) without submitting any Soroban transactions.
+
+---
+
+## 1. Safety & Guarantees
+
+1. **Zero On-Chain Mutations**: Reconciliation **NEVER** submits transactions to the Stellar network or alters smart contract state.
+2. **Distributed Locking**: Only one active reconciliation job is permitted per network (`testnet`, `mainnet`) at any time.
+3. **Idempotency**: Re-running reconciliation over the same ledger range creates no duplicate records and triggers no duplicate user notifications.
+4. **Dry-Run Default**: All reconciliation runs default to `isDryRun: true`.
+5. **Quarantine Protection**: Unsafe mismatches (`UNRECOVERABLE`, corrupted data, conflicting states) are quarantined in `reconciliation_discrepancies` for admin investigation.
+
+---
+
+## 2. Discrepancy Classification
+
+Discrepancies are categorized into 5 distinct types:
+
+| Discrepancy Type     | Description                                                                                                        | Repair Strategy                                                                    |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
+| `MISSING_OFFCHAIN`   | On-chain event exists in Soroban RPC / event log, but record is absent in PostgreSQL.                              | Transactionally upserts missing record into `calls`, `stakes`, or `payout_claims`. |
+| `DUPLICATE_OFFCHAIN` | Multiple PostgreSQL records found matching a single stable event identity (`contractId:ledger:txHash:eventIndex`). | Quarantined for manual operator review.                                            |
+| `VALUE_MISMATCH`     | Record exists off-chain, but amounts, status, or addresses differ from on-chain event.                             | Simple status updates are auto-repaired; complex value conflicts are quarantined.  |
+| `UNKNOWN_CONTRACT`   | Event emitted by a contract ID not included in the configured BACKit deployment set.                               | Quarantined.                                                                       |
+| `UNRECOVERABLE`      | Event payload is unparseable or data is corrupted.                                                                 | Quarantined for developer / admin triage.                                          |
+
+---
+
+## 3. Operator Execution Workflow
+
+### Step 1: Execute Dry-Run Evaluation
+
+Before running repair mode, perform a dry-run to generate a structured discrepancy report:
+
+```bash
+POST /admin/reconciliation/run
+Header: Authorization: Bearer <ADMIN_JWT>
+Content-Type: application/json
+
+{
+  "network": "testnet",
+  "fromLedger": 100000,
+  "toLedger": 105000,
+  "isDryRun": true
+}
+```
+
+Response:
+
+```json
+{
+  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "network": "testnet",
+  "contractIds": ["CC..."],
+  "fromLedger": 100000,
+  "toLedger": 105000,
+  "isDryRun": true,
+  "status": "PENDING"
+}
+```
+
+### Step 2: Inspect Run Progress & Discrepancies
+
+Check run status:
+
+```bash
+GET /admin/reconciliation/runs/a1b2c3d4-e5f6-7890-abcd-ef1234567890
+```
+
+List detected discrepancies:
+
+```bash
+GET /admin/reconciliation/discrepancies?runId=a1b2c3d4-e5f6-7890-abcd-ef1234567890&limit=50
+```
+
+Review `scannedEventsCount`, `discrepancyCount`, and `discrepancyBreakdown`.
+
+### Step 3: Execute Repair Mode
+
+Once the dry-run report is reviewed and verified by an operator, execute repair mode over the target range:
+
+```bash
+POST /admin/reconciliation/run
+Header: Authorization: Bearer <ADMIN_JWT>
+Content-Type: application/json
+
+{
+  "network": "testnet",
+  "fromLedger": 100000,
+  "toLedger": 105000,
+  "isDryRun": false
+}
+```
+
+Repair mode will:
+
+- Upsert missing `Call`, `Stake`, and `PayoutClaim` records in PostgreSQL.
+- Mark auto-repaired items as `REPAIRED`.
+- Quarantine unsafe items as `QUARANTINED`.
+
+### Step 4: Re-Verify via Dry-Run
+
+Re-run the same range in dry-run mode:
+
+```bash
+POST /admin/reconciliation/run
+{
+  "network": "testnet",
+  "fromLedger": 100000,
+  "toLedger": 105000,
+  "isDryRun": true
+}
+```
+
+Verify that `discrepancyCount` for `MISSING_OFFCHAIN` is now `0`.
+
+---
+
+## 4. Incident Response & Rollback Procedures
+
+1. **Overlapping Lock Conflict**: If a run fails unexpectedly and leaves a stale lock key, clear the Redis lock key:
+   ```bash
+   redis-cli DEL reconciliation_lock:testnet
+   ```
+2. **Quarantined Discrepancies**: Query all `QUARANTINED` records:
+   ```bash
+   GET /admin/reconciliation/discrepancies?status=QUARANTINED
+   ```
+   Investigate cause (e.g. database constraint error or contract redeployment).
+3. **Database Transaction Safety**: All repairs execute within TypeORM database transactions (`entityManager.transaction(...)`). If a database error occurs during repair, the entire transaction rolls back cleanly.
+
+---

packages/backend/src/app.module.ts

@@ -32,6 +32,7 @@ import { StakesModule } from './stakes/stakes.module';
 import { EventStoreModule } from './event-store/event-store.module';
 import { GraphqlModule } from './graphql/graphql.module';
 import { WebhooksModule } from './webhooks/webhook.module';
+import { ReconciliationModule } from './reconciliation/reconciliation.module';
 
 @Module({
   imports: [
@@ -79,6 +80,7 @@ import { WebhooksModule } from './webhooks/webhook.module';
     EventStoreModule,
     GraphqlModule,
     WebhooksModule,
+    ReconciliationModule,
   ],
   controllers: [],
   providers: [{ provide: APP_INTERCEPTOR, useClass: LoggingInterceptor }],

packages/backend/src/common/queues/queues.constants.ts

@@ -2,9 +2,11 @@ export const QUEUE_IPFS_PINNING = 'ipfs-pinning';
 export const QUEUE_NOTIFICATIONS = 'notifications';
 export const QUEUE_ORACLE_SIGNING = 'oracle-signing';
 export const QUEUE_DEAD_LETTER = 'dead-letter';
+export const QUEUE_RECONCILIATION = 'reconciliation';
 
 export type QueueName =
   | typeof QUEUE_IPFS_PINNING
   | typeof QUEUE_NOTIFICATIONS
   | typeof QUEUE_ORACLE_SIGNING
-  | typeof QUEUE_DEAD_LETTER;
+  | typeof QUEUE_DEAD_LETTER
+  | typeof QUEUE_RECONCILIATION;

packages/backend/src/common/queues/queues.module.ts

@@ -6,6 +6,7 @@
   QUEUE_IPFS_PINNING,
   QUEUE_NOTIFICATIONS,
   QUEUE_ORACLE_SIGNING,
+  QUEUE_RECONCILIATION,
 } from './queues.constants';
 import { DeadLetterService } from './dead-letter.service';
 import { QueuesStatusService } from './queues.status.service';
@@ -17,7 +18,7 @@
     BullModule.forRootAsync({
       imports: [ConfigModule],
       inject: [ConfigService],
       useFactory: async (config: ConfigService) => {
         const redisUrl =
           config.get<string>('REDIS_URL') ?? 'redis://localhost:6379';
         return {
@@ -57,6 +58,15 @@
         removeOnFail: false,
       },
     }),
+    BullModule.registerQueue({
+      name: QUEUE_RECONCILIATION,
+      defaultJobOptions: {
+        attempts: 3,
+        backoff: { type: 'exponential', delay: 1000 },
+        removeOnComplete: { age: 60 * 60 * 24 },
+        removeOnFail: false,
+      },
+    }),
   ],
   controllers: [AdminQueuesController],
   providers: [DeadLetterService, QueuesStatusService],

packages/backend/src/reconciliation/dto/query-discrepancies.dto.ts

@@ -0,0 +1,41 @@
+import { IsOptional, IsInt, Min, Max, IsUUID, IsEnum } from 'class-validator';
+import { Type } from 'class-transformer';
+import { ApiPropertyOptional } from '@nestjs/swagger';
+import {
+  DiscrepancyType,
+  DiscrepancyStatus,
+} from '../entities/reconciliation-discrepancy.entity';
+
+export class QueryDiscrepanciesDto {
+  @ApiPropertyOptional({ default: 1 })
+  @IsOptional()
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  page?: number = 1;
+
+  @ApiPropertyOptional({ default: 20 })
+  @IsOptional()
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  @Max(100)
+  limit?: number = 20;
+
+  @ApiPropertyOptional({
+    description: 'Filter by specific reconciliation run ID',
+  })
+  @IsOptional()
+  @IsUUID()
+  runId?: string;
+
+  @ApiPropertyOptional({ enum: DiscrepancyType })
+  @IsOptional()
+  @IsEnum(DiscrepancyType)
+  type?: DiscrepancyType;
+
+  @ApiPropertyOptional({ enum: DiscrepancyStatus })
+  @IsOptional()
+  @IsEnum(DiscrepancyStatus)
+  status?: DiscrepancyStatus;
+}

packages/backend/src/reconciliation/dto/query-reconciliation-runs.dto.ts

@@ -0,0 +1,45 @@
+import {
+  IsOptional,
+  IsInt,
+  Min,
+  Max,
+  IsString,
+  IsEnum,
+  IsBoolean,
+} from 'class-validator';
+import { Type, Transform } from 'class-transformer';
+import { ApiPropertyOptional } from '@nestjs/swagger';
+import { ReconciliationRunStatus } from '../entities/reconciliation-run.entity';
+
+export class QueryReconciliationRunsDto {
+  @ApiPropertyOptional({ default: 1 })
+  @IsOptional()
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  page?: number = 1;
+
+  @ApiPropertyOptional({ default: 20 })
+  @IsOptional()
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  @Max(100)
+  limit?: number = 20;
+
+  @ApiPropertyOptional()
+  @IsOptional()
+  @IsString()
+  network?: string;
+
+  @ApiPropertyOptional({ enum: ReconciliationRunStatus })
+  @IsOptional()
+  @IsEnum(ReconciliationRunStatus)
+  status?: ReconciliationRunStatus;
+
+  @ApiPropertyOptional()
+  @IsOptional()
+  @Transform(({ value }) => value === 'true' || value === true)
+  @IsBoolean()
+  isDryRun?: boolean;
+}

packages/backend/src/reconciliation/dto/start-reconciliation.dto.ts

@@ -0,0 +1,57 @@
+import {
+  IsString,
+  IsArray,
+  IsOptional,
+  IsInt,
+  Min,
+  IsBoolean,
+} from 'class-validator';
+import { Type } from 'class-transformer';
+import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
+
+export class StartReconciliationDto {
+  @ApiPropertyOptional({
+    description: 'Target network (e.g. mainnet, testnet, futurenet)',
+    example: 'testnet',
+  })
+  @IsOptional()
+  @IsString()
+  network?: string = 'testnet';
+
+  @ApiPropertyOptional({
+    description:
+      'Target contract IDs to reconcile. If omitted, uses deployment defaults.',
+    type: [String],
+  })
+  @IsOptional()
+  @IsArray()
+  @IsString({ each: true })
+  contractIds?: string[];
+
+  @ApiProperty({
+    description: 'Start ledger sequence (inclusive)',
+    example: 1000,
+  })
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  fromLedger: number;
+
+  @ApiProperty({
+    description: 'End ledger sequence (inclusive)',
+    example: 2000,
+  })
+  @Type(() => Number)
+  @IsInt()
+  @Min(1)
+  toLedger: number;
+
+  @ApiPropertyOptional({
+    description:
+      'If true, produces discrepancy reports without DB mutations. Default is true.',
+    example: true,
+  })
+  @IsOptional()
+  @IsBoolean()
+  isDryRun?: boolean = true;
+}