Priority field for entityEvents resource

[grispin]

Large historical data patches (data remapping, DD version compliance changes, etc ) can create a large backlog (millions of records) of entity events the data consumers need to parse in order to catch up their replication state.

The proposal would be to add a bulkYN boolean field to all for consumers to have parallel replication streams; one for bulk data changes ( off-market, historical data) and one for the active operations happening on the data provider. This allows the data consumers to keep getting timely updates for on market changes while batch process historical changes to minimize impact to their operations.

This field could also be used by the webhook subscription filter to all data consumers to have differ endpoints for bulk vs non bulk endpoint changes and handle their processing velocity has their business needs demand.

[grispin]

It's has been suggestion at the RESO conference, we may want to make it bulkPriority with a numeric type so that more than 2 priority levels could be communicated. Do we need the increased complexity?

What would the preference be for bulk classification? Boolean or numeric

[bryanburgers]

As a consumer, I like this. Being able to get realtime updates as fast as possible but be able to control the rate of bulk updates would help us deliver to our customers.

The definition of BulkYN assumes there are exactly two categories – bulk and not bulk.

I haven’t thought this through deeply, but I wonder if there will be other categories in the future.

(To be clear, I’m not advocating that we add filtering to entity events.)

So I wonder if this should be some sort of “Priority” field that can be either “Realtime” or “Bulk” and leave ourselves open to a future third, fourth type. (Am I optimizing for a use case we don’t have yet? Or am I trying to make sure we don’t paint ourselves into a corner?)

[darnjo]

Could it just be an additional event type of Bulk Update?

…

On Wed, Oct 23, 2024, 09:07 Bryan Burgers ***@***.***> wrote: As a consumer, I like this. Being able to get realtime updates as fast as possible but be able to control the rate of bulk updates would help us deliver to our customers. The definition of BulkYN assumes there are exactly two categories – bulk and not bulk. I haven’t thought this through deeply, but I wonder if there will be other categories in the future. (To be clear, I’m *not* advocating that we add filtering to entity events.) So I wonder if this should be some sort of “Priority” field that can be either “Realtime” or “Bulk” and leave ourselves open to a future third, fourth type. (Am I optimizing for a use case we don’t have yet? Or am I trying to make sure we don’t paint ourselves into a corner?) — Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAECWPQ7CHC6VVGN2D4POA3Z47CU5AVCNFSM6AAAAABQPEQAQWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBTGE2DQMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** .com>

[grispin]

We could but I would prefer to have separate fields as they have a different purpose. The type is the action that occurred and the bulk field is the replication priority of that event. We would end up with the an enumeration have every combination of type, with and without bulk attached. I could see using bulk flag with every type. Bulk creations for the initialization of a data share in the data provider, bulk delete for data retention policies or the end of a data licensing agreement with the data provider

If people want the implementation to have more that two "lanes" provided by a boolean then we would have to have a set of types for each lane.

[darnjo]

The main reason for asking is to try and avoid creating extra fields, if possible, since the number of events can be extremely large. Would be nice to avoid bloating the payload with a field that isn't going to be used in the common case.

Having the Bulk Update as a possible event type allows only those who need it to use it and only in the cases when needed. It's fairly easy to query on a Bulk Update event (or not), and using event types is more extensible in the future, to speak to @bryanburgers's point, in case something else similar comes up. Adding a new YN field each time isn't.

[grispin]

We could just have the bulk/priority field not displayed by default unless selected as part of the standard which would make the payload no bigger than the current standard unless the bulk/priority field is requested. I agree the priority/bulk field is most useful for filters on queries and webhook subscriptions, not as data in the response payload. This behavioral difference between type and bulk/priority makes me want it as a separate field even more as the current type field is needed with every entity event record but the bulk/priority field is not. If we are concerned about payload size, the field visibility approach is also smaller than adding enumerations with bulk prefixes as the additional 4 characters for "bulk" prefix would be transmitted with each record where no additional characters are needed in the visibility approach.
Doing prioritization with a separate bulk field is also easier for clients as the only have to consider the prioritization field if they cannot ingest faster that the rate they need the data. For those that do not need near-time replication or have a high rate of consumption, they can setup their system and completely ignore the prioritization field. If we use type enumerations to provide bulk behaviour, the client not using prioritization still have to support it even if they are not using it.

The more I think about it, the more I like @bryanburgers suggestion to use a numeric so that tiered prioritization could be trivially communicated by data consumers and the clients can decide how granular to be through filter clauses. This change is a minimal impact to the logic proposed and provides both data publishers and clients more potential and flexibity during high impact or volume changes while only being more complex when the higher flexibility is used. All that changes between the boolean and a number is the field naming and the filter comparisons to true/false vs (>1 or =1).

Example:
Data provider could publish entity events with recent history events is a separate priority from much older and higher volume tier :

• Priority 1 active, real time entity events
• Priority 2 is last 12/24/36 month non active record updates
• Priority 3 is greater than all older listings
  Now consumers can define how they want to ingest
• Client 1: Consumer one cares mostly about real time display of data. they have two replication filters running, one priority=1 for real time events and one for priority>1 for all other events
• Client 2: doing weekly reports or analytics that are not time sensitive. They have no special filter
• Client 3:needs all the data as fast as possible with more current data being of higher value than others. The usage could have filters for priority=x down to 10 and then a final filter for priority>10.

Data providers can prioritize to as many priority tiers as desired as desired; patch size, impact, etc.. Clients can choose how much or not to use the priority tiering based on their need for real time data, their record ingestion rates, etc.

The key to this is that when the priority filter is used the lowest client priority filter should be a greater than last equals comparison to ensure the data consumer has fetched all priority levels that might have been used.

[darnjo]

This seems like a leaky abstraction. Implementation details about the internal system behavior are making their way into special flags the interface with the goal of trying to model batch processes with event streams. Seems like there are probably better ways to handle it. And not all event streams are meant to be monolithic and re-entrant replication logs. Different use cases can still use the interface of a log but with the events they're interested in, which is what filtering and event types are supposed to accomplish without Boolean variables. A batch update seems like more of an event type than its own independent variable. It's really a Bulk Update. Can discuss further in the workgroups. The goal is to use a few fields as possible, which is consistent with best practices for large event streams. I don't think any additional fields should be added unless absolutely necessary

…

On Wed, Oct 23, 2024, 22:49 Geoff Rispin ***@***.***> wrote: We could just have the bulk/priority field not displayed by default unless selected as part of the standard which would make the payload no bigger than the current standard unless the bulk/priority field is requested. I agree the priority/bulk field is most useful for filters on queries and webhook subscriptions, not as data in the response payload. This behavioral difference makes me want it as a separate field even more as the current type field is needed with every entity event record but the bulk/priority field is not. If we are concerned about payload size, the field visibility approach is also smaller than adding enumerations with bulk prefixes as the additional 4 characters for "bulk" prefix would be transmitted with each record where no additional characters are needed in the visibility approach. Doing it with a separate bulk field is also easier for clients as the only have to consider the bulk field if they need their entity event processing to be faster than their rate of consumption of the entity event resource. For those that do not need near-time replication or have a high rate of consumption, can setup their system and completely ignore the bulk/priority field. If we use bulk enumerations for each operation in type, the client now need to map the bulk enumeration values to base enumeration. The more I think about it, the more I like @bryanburgers <https://github.com/bryanburgers> suggestion to use a numeric so that tiered prioritization could be trivially communicated by data consumers and the clients has decide how granular to be through filter clauses. It's a minimal impact to the logic and provides both data publishers and clients potentially more flexibility if it is used especially if it is used and is only more complex for those that use the functionality. All that changes between the boolean and a number is the field naming and the filter comparisons to true/false vs (>1 or =1). Example: Data provider could publish entity events with recent history events is a separate priority from much older and higher volume tier : - Priority 1 active, real time entity events - Priority 2 is last 12/24/36 month non active record updates - Priority 3 is greater than all older listings Now consumers can define how they want to ingest - Client 1: Consumer one cares mostly about real time display of data. they have two replication filters running, one priority=1 for real time events and one for priority>1 for all other events - Client 2: doing weekly reports or analytics that are not time sensitive. They have no special filter - Client 3:needs all the data as fast as possible with more current data being of higher value than others. The usage could have filters for priority=x down to 10 and then a final filter for priority>10. Data providers can prioritize in to as many segments as desired based on patch size and impact. Clients can choose to use the tiering based on need for real time data, record ingestion rates to be as or not granular they want to be within the provided tiers of prioritized events. The key to this is that when the priority filter is used the lowest client priority filter should be a greater than last equals comparison to ensure the data consumer has fetched all priority levels that might be used. — Reply to this email directly, view it on GitHub <#154 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAECWPW6WESPSF2C5VBVZSLZ5CC5HAVCNFSM6AAAAABQPEQAQWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBTGY3DQMY> . You are receiving this because you commented.Message ID: ***@***.*** com>

[darnjo]

As a side note, @grispin, I don't have any doubts about the validity of the request and the use cases you've outlined.

Mostly just looking for ways to achieve the goals in a clean way with what we currently have available, if possible, before adding extra stuff.

Some potential options:

• Use EventTypes with an EventType that's both compact and meaningful that communicates that the client needs to refresh - something they can filter out and demux in another process.
• Make a recommendation for clients that if they see large jumps in records between the last time they synced and this time, like 100 went to 1000, they should keep up with the current stream and fill in the gaps with a batch process, ignoring anything they already have retrieved since then since they would have pulled the latest copy in that case. Defining "significant" is interesting, and perhaps we should be more prescriptive.
• Add a new resource called EntityEventRefresh where the ResourceRecordKey can be the changeset of keys to pick up and write only one entry to the log when they change rather than lots of events directly to the log. Not sure how much I'm really suggesting this - just thinking out loud a bit. That key set could be a LOT of records.

[grispin]

The main reason I brought this up was that I over heard a conversation at the RESO conference that gave me chills. "I'm going to treat entity events like I treat modification timestamp. If I don't update modification timestamp, I'm not going to add an entity event. This way I will not delay consumers' updates" . EntityEvents needs to be a proper, compacted, transaction log with no omissions or it's value is lost as it cannot be trust and we will have the re-sync, re-verfication processes everyone does to ensure they really have the current data.

After more thought and conversation with others, I've actually gone back to preferring a simple two-tier, bulk or not-bulk approach so that the client clearly knows that there are only two lanes that the clients know their purpose. While I can see it being nice for vendors to provide more lanes to the client so they can get historical data more timely without a way to communicate the purpose of the lane, it becomes problematic. @bryanburgers has suggested an enumeration which addresses the labeling of lanes but the needs would be data provider specific based on their role within the industry. What an aggregator would need is different from an AMA or an MLS, etc.

I can see the EventTypes approach working but it muddies the field purpose being about the action being preformed on the target resource with the priority of the entity event. I still think the separate default non-visible field provides a cleaner approach and does not increase the payload unless it is explicitly request and the bulk field's real purpose is for client to use a query filter and not payload data.

The clients should not be required to infer (guess) behaviors about what the server is doing based on the rate of change as this value varies by data source, time of day, etc. With an inferred bulk updated there may be one or more real-time transactions that were updated while the patch was running at the data provider that needs to be classified as real-time and not bulk. The client has no way make the bulk status determination on each entity event as the only knowledge they have is the data in the entity event.

We should avoid designs were the prioritization decision is only made on the client side. Having the result available via a OData filter allows the webhook subscriptions RCP to make use of what ever solution we devise and minimizes the time to next real-time record the client gets. While the entity event record is small having tens of millions of records touched should not be unexpected. The time just to pull and parse the records could be too long for some clients desired real time lag requirements.

The EntityEventRefresh resource is interesting but I don't think it is viable due to the complexity it introduces and creates a backwards breaking changes with the current EntityEvent specification for clients. For the data providers it creates a number of management issues.

• the compaction behaviours becomes much more complex when only a subset of a EntityEventRefresh records key list are affected. It's generally an anti-pattern to change the data in a historical transaction log, you write a new records and delete the old one but that would not be ideal either as it would re-replicate the non changing records as well when an EntityEventRefresh record needed it's key list subdivide. Other apporach are available but they require much more logic on both the client and the server to handle changes.
• I can also see a case where some systems will end up have a monolithic EntityEventRefresh record for each resource type that have a key list of everything in history as most very old, off-market listings will never change except for the large data patches. We lose the incremental nature that the transaction log gives us as well as those EntityEventRefresh records would be massive (many gigabytes) for a single refresh record in some cases.
• If the patch is written by someone that is not aware of entity event refresh behaviour, the patch could be organized across resources rather than by resources (iterating by listing for example (property, then it's media, then its rooms, etc.)), This would create a high number of entity events still and a high number of entity event refresh records.

[grispin]

So how do we want to proceed? @bryanburgers @darnjo

• do we want/need more then two priorities?

[psftc]

Did we have the discussion about separating streams rather than having a stream with a field on all events for bulk changes?

Having separate distinct endpoints would make a lot of sense - the data consumer (DC) can handle the two stream completely separately where the bulk update is blue/green type of flow (the DC does whatever work it needs to do to handle the bulk change while continuing to support events in the 'normal' event stream then switches over to the new data state).

At the risk of getting something thrown at me, it does bring us back to the System Resource where the data provider advertises what it can do. On the DC side, it is useful to know what the system supports. It definitely feels like we have enough cases of system specific behaviour that telling the DC in a metadata way would be helpful. In addition to different resources that may or may not be present on this system, rate limits and constraints, media handling, the event stream endpoints available and other system metadata might be useful.

That also allows for more endpoints/event streams where neat things like a DOM event could happen for the case where the provider doesn't update the DOM/modtms when no other modtms event happened to update the record. Essentially saying "For active records that did not have their modtms change, now is when we would increment the DOM".

[grispin]

@psftc We want all entity events in one resource so that DC only has to look in one location for records. Entity event no longer becomes an ordered transaction log if it is partitioned into multiple resources. Having the priority as a field in the same resource allows the DC to decide if they need to defer large data patches on historical listing or not based on the rate their solution can consume the data. The DC can choose their consumption by if they filter their queries on priority or not.

Having the field also allows the upcoming subscription webhook model to use the field optionally based on the DC's needs. If they need priority separation, they can have more than one subscription with filters on the new field for the delivery of each priority to a different webhook endpoints. The control of what is required is at the DC's discretion.

Anytime data providers change a data record and do not touch its timestamp is an anti-pattern. I know there are business reasons for doing this and I'm guilty of it as well but we need to look at those cases and try to finder a better mechanism. DOM is an excellent example of a better solution being available if we change the representation of the data. It's been discussed on the transport calls and was presented on stage at the conference. Switching from a server side incremented field to a timestamp based calculation removes the daily update and reduces the number of timestamp entries and entity event records.

And to scare you @psftc, I heard conversations at the conference about implementations that already exclude entity event generation for their DOM updates from their entity events resource. The justification given is the number of additional record fetches that the events would create for all replicators when only some today re-pull record for DOM nightly.