Sample: Tuning Guidance for Consuming Events

[jsquire]

Summary

The samples for consuming and processing events do not provide guidance around the options for tuning performance, and the associated factors to consider.

Scope of Work

• The Event Hubs samples should be extended to include discussion of the options to tune performance when consuming, covering:

  • PrefetchCount (EventHubConsumerClient, PartitionReceiver, EventProcessor<T>)
  • CacheEventCount (EventHubConsumerClient)
  • MaximumEventCount (PartitionReceiver)
  • EventBatchMaximumCount (EventProcessor<T>)
    
    

• The Event Processor samples should be extended to include discussion of the options to tune performance when consuming, covering:

  • PrefetchCount
  • CacheEventCount
    
    

• The role and operation of the prefetch queue should be discussed, with its behavior highlighted. Specifically, it should be noted that:

  • The prefetch queue does not wait to build up the requested batch size
  • As soon as any event is present in the prefetch queue, whatever events are available will be returned, up to the requested batch size
  • Warming up the queue may help to drive more consistent throughput
  • Event size plays a big part in the queue behavior
    
    

• The Event Processor samples should touch on checkpointing with respect to throughput and performance. Trade-offs should be discussed, including:

  • More frequent checkpointing offers safety at the cost of performance
  • Checkpointing can help reduce duplication when scaling; if you can signal that you're about to scale and checkpoint, it may help
  • Checkpointing before you process events can lead to data loss if your processing fails

Success Criteria

• The topics detailed by the scope have been documented in samples in the relevant areas.

• All code in the samples are snippet-driven, with the code being built and smoke tested as part of nightly runs.

• The existing samples have been reviewed for accuracy and are in a good state for release of the Event Hubs client libraries.

• The tests necessary for sample validation have been created or adjusted and pass reliably.

• The existing test suite continues to produce deterministic results and pass reliably.