Improve startup time of non-archiving ACS triggers

[meiersi-da]

Problem

An ACS triggers (actually an OnAssignedContractTrigger) reprocesses all active contracts on startup.

This works fine for triggers whose task completion always results in the archival of the contract that triggered. However there are triggers for which this is not the case:

• ReportSvStatusMetricsExportTrigger
• ReportValidatorLicenseMetricsExportTrigger
• ExecuteConfirmedActionTrigger
• GarbageCollectAmuletPriceVotesTrigger
• MergeMemberTrafficContractsTrigger
• MergeSvRewardStateTrigger
• MergeValidatorLicenseContractsTrigger
• ReconcileSequencerLimitWithMemberTrafficTrigger

For these triggers the reprocessing of the active contracts is usally an idempotent no-op that costs O(#active-contracts). We expect this to be a problem for the two traffic-related triggers when we introduce per-party traffic balances. There we expect to manage millions of active MemberTraffic contracts, which makes reprocessing them by default on startup prohibitive.

Proposal

Avoid reprocessing already processed active contracts as follows:

1. Introduce a trigger-key and trigger-id analogously to store-key and store-id.
   • use them to identify trigger-related data in the DB
   • use the same versioning strategy as for store-keys (both source code and config define a version number that is considered to be party of the key)
   • make config flag fine-grained enough to only trigger reprocessing of a single kind of trigger (identified by class-name)
2. Add a new table to store the last_processed_event_number per trigger_id after task completion in an OnAssignedContractTrigger.
   • make sure to only bump it when all tasks for contracts with a lower event_number have definitely been processed (i.e., successful completion or stopped retrying)
3. Start new instances of the trigger with a source that streams starting from the first contract after last_processed_event_number.

Thereby we get at-least-once processing for all assigned contracts. Reprocessing failed tasks works by bumping the trigger version in the config, and restarting the app.