Resulting document after update is larger than 16777216

[lindaq1]

Description

Using default configuration of eiffel-backend-allevents component (from eiffel-easy2use) with AllEventsRules-Eiffel-Agen-Version.json rule set (need to aggregate each event separately).

Started to receive exceptions such as stated bellow on each event, which seems to cause failure in event subscription handling, although they are still persisted in mongo. Value of 16777216 seems to be maximum BSON document size (16 megabytes).

com.mongodb.MongoCommandException: Command failed with error 17419: 'Resulting document after update is larger than 16777216' on server 10.88.0.2:27017. The full response is { "ok" : 0.0, "errmsg" : "Resulting document after update is larger than 16777216", "code" : 17419, "codeName" : "Location17419" }
at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper$3.call(CommandOperationHelper.java:436) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.OperationHelper.withReleasableConnection(OperationHelper.java:433) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeRetryableCommand(CommandOperationHelper.java:429) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.BaseFindAndModifyOperation.execute(BaseFindAndModifyOperation.java:33) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:58) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.Mongo$3.execute(Mongo.java:837) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.executeFindOneAndUpdate(MongoCollectionImpl.java:765) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:748) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:743) ~[mongodb-driver-3.6.4.jar:na]
at com.ericsson.ei.mongo.MongoDBHandler.doFindAndModify(MongoDBHandler.java:320) ~[classes/:3.1.0]
at com.ericsson.ei.mongo.MongoDBHandler.findAndModify(MongoDBHandler.java:188) ~[classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.updateExistingMatchedSubscriptionWithAggrObjId(SubscriptionRepeatDbHandler.java:153) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.addMatchedAggrObjToSubscriptionId(SubscriptionRepeatDbHandler.java:82) [classes/:3.1.0]
at com.ericsson.ei.subscription.RunSubscription.runSubscriptionOnObject(RunSubscription.java:129) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.extractConditions(SubscriptionHandler.java:105) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.lambda$checkSubscriptionForObject$0(SubscriptionHandler.java:80) [classes/:3.1.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_181]
at com.ericsson.ei.subscription.SubscriptionHandler.checkSubscriptionForObject(SubscriptionHandler.java:79) [classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.postInsertActions(ObjectHandler.java:303) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:108) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:113) ~[classes/:3.1.0]
at com.ericsson.ei.jsonmerge.MergeHandler.addNewObject(MergeHandler.java:276) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ExtractionHandler.runExtraction(ExtractionHandler.java:89) ~[classes/:3.1.0]
at com.ericsson.ei.rules.IdRulesHandler.runIdRules(IdRulesHandler.java:63) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.eventReceived(EventHandler.java:57) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.onMessage(EventHandler.java:69) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler$$FastClassBySpringCGLIB$$e31fe186.invoke() ~[classes/:3.1.0]

Motivation

Avoid failed event subscription handling.

Exemplification

Benefits

Persistence of un-processed events.

Possible Drawbacks

[lindaq1]

It looks like the problem is due to fulfilled subscription persistence, which flooded subscription_repeat_handler collection in mongo.

Documentation:
It is possible to enable repeat, e.g. whether same subscription should be re-triggered for new additions to the aggregated object. If disabled, the notification will be triggered only the first time when requirements are fulfilled. It doesn't matter if you have multiple requirements, it will only be triggered once.

How these repeat subscriptions are being resolved from mongo event id entries to the actual re-trigger of the subscription? From what can I see, it looks like EI backend only inserts them.
Currently fix seems to be setting "Repeat" flag to "true", which prevents the insertion of fulfilled subscriptions in the code:
RunSubscription.java:

if (subscriptionRepeatFlag.equals("false") && id != null) {
    // the keyword 'synchronized' ensures that this part of the code run
    // synchronously. Thus avoids race condition.
    synchronized (this) {
        if (!subscriptionRepeatDbHandler.checkIfAggrObjIdExistInSubscriptionAggrIdsMatchedList(
                subscriptionName, requirementIndex, id)) {
            LOGGER.debug("Adding matched aggregated object to database:" + dataBaseName);
            subscriptionRepeatDbHandler.addMatchedAggrObjToSubscriptionId(subscriptionName,
                    requirementIndex, id);
        } else {
            conditionFulfilled = false;
        }
    }
}

Perhaps someone with broader background in Eiffel than me could elaborate what was the intention of storing ids of fulfilled event subscriptions? Thanks!

[tobiasake]

EI stores the AggregatedObject Id to keep track of which aggregated objects has triggered a subscription and it will only trigger one time. Aggregated Object Id is only stored when Repeat flag is set to false.
When Repeat flag is set to true, then the subscription should trigger for every aggregations updates with new event data. In this case there is no need to keep track of which aggregated object has matched and triggered subscription notification, since all aggregation objects updates should trigger the subscription.

I am not sure about BSON MongoDB Document size error,, I have seen that long time ago, but I don't remember if it was in this part of EI code.
Its hard to say why you get this error without any more information about the Subscriptions you use, etc.
It sounds like you have a subscription that has the repeat flag set to false and it matches a lot of different Aggregation objects and after some time SubscripotionRepeatHandler Document for subscriptions becomes to big for MongoDb to handle.