High Performance Ethereum Indexer

Syncs events from Ethereum and indexes them for fast retrieval. This is useful and often essential for production Dapps.

• Support for 6 different persistence stores
• Includes "single-node friendly" stores such as flat-file JSON
• Comprehensive error retry logic when communicating with Ethereum node
• Maximum throughput via JSON RPC using batched parallel fetching

Todo

• Keep track of blockNumbers in persistence for fast-resume
• Benchmark documentation
• Support going forward syncing for Ethereum
• NPM package
• Improved failsafes for data consistency
• Split up stores into separate dependencies

Supported Indexing Stores and Benchmarks

The following indexing implementations have been benchmarked on an Ethereum blockchain section with a relatively large number of events (EtherDelta exchange blocks 4800000 - 5002718) on a single 8-core 2017 Macbook Pro:

Store Name	Store ID	Events Per Second	Blocks Per Second
Local Memory	memory	4072.7/s	464.5/s
Redis	redis	2049.55/s	235.25/s
Local Flat File	file	1798.75/s	201.75/s
MongoDB	mongodb	795.5/s	94.6/s
Elasticsearch	elasticsearch	452.25/s	57.1/s
LevelDB	level	207.35/s	26.95/s

Install Dependencies

yarn install

Testing & Linting

yarn test
yarn lint

Integration tests (requires synced ethereum node at localhost:8545):

yarn test:integration

Example: Sync trading and balance events from decentralized trading contract: EtherDelta

Contract address: https://etherscan.io/address/0x8d12a197cb00d4747a1fe03395095ce2a5cc6819

See examples/etherdelta/sync.js.

Define which events to index on which keys:

const indexing = {
  events: {
    Withdraw: {
      keys: ['user'],
    },
    Trade: {
      keys: ['tokenGive', 'tokenGet', 'get', 'give'],
    },
  },
};

Then create a store:

const store = new LevelStore(indexing, '/tmp/etherdelta.db');

Now boot up the indexer:

const indexer = new Indexer(store, EtherdeltaABI, '0x8d12a197cb00d4747a1fe03395095ce2a5cc6819');
await indexer.syncAll({
  fromBlock: 4906764,
});

Once indexing is complete events can be retrieved as follows:

const events = await store.get('Withdraw', 'user', '0x13d8d38421eb02973f3f923a71a27917bd483190');

Directory Structure

• package.json - Configure dependencies
• dist/* - Files generated by babel
• src - All source code
• src/*/__tests__ - Unit tests
• integration/__tests__ - Integration tests