Right now the PersistenceAdapter interface declares these two methods for interacting with offsets:
void persistConsumerState(final String consumerId, final int partitionId, final long offset);
Long retrieveConsumerState(final String consumerId, final int partitionId);We made the assumption that an offset, or rather the state of the consumer, is always an integer, but as we're finding in other projects this doesn't always hold true.
To support more complex state tracking we need to make a shift here.
State is still going to need an offset for offset tracking, but I'm not starting to wonder if we need to create our own offset for the purposes of things like the PartitionOffsetManager. Basically, we could increment things coming off of the spout instance and track that as an ordered offset, but persist more complex data. Or maybe we leave this up to the specific consumer to implement if they're using a non-numeric piece of state.
I'm spit-balling here, but fundamentally we need to at least support a string for consumer state - probably should just consider a bag of bytes or something.
Let's come up with a design and figure out how to refactor the necessary parts of the framework for 0.10 milestone.
Right now the PersistenceAdapter interface declares these two methods for interacting with offsets:
We made the assumption that an offset, or rather the state of the consumer, is always an integer, but as we're finding in other projects this doesn't always hold true.
To support more complex state tracking we need to make a shift here.
State is still going to need an offset for offset tracking, but I'm not starting to wonder if we need to create our own offset for the purposes of things like the
PartitionOffsetManager. Basically, we could increment things coming off of the spout instance and track that as an ordered offset, but persist more complex data. Or maybe we leave this up to the specific consumer to implement if they're using a non-numeric piece of state.I'm spit-balling here, but fundamentally we need to at least support a string for consumer state - probably should just consider a bag of bytes or something.
Let's come up with a design and figure out how to refactor the necessary parts of the framework for 0.10 milestone.