Fix[BMQ]: unsafe schemalearner use #618

dorjesinpo · 2025-02-19T16:32:31Z

Cannot use bmqp::SchemaLearner in both FSM and event threads.
Instead, call the leaner in FSM and cache the result (schema) in the generated event.

bmq-oss-ci

Build 2484 of commit 91d01b5 has completed with FAILURE

678098 · 2025-02-24T14:57:59Z

src/groups/bmq/bmqa/bmqa_message.h

@@ -60,6 +60,7 @@
 // BMQ

 #include <bmqa_queueid.h>
+#include <bmqp_messageproperties.h>


The only change in this header is this new field:
bmqp::MessageProperties::SchemaPtr d_schema_sp

It is a pointer type, so we don't need to know the class sizeof in advance. We can replace include to a heavy header with a forward declaration to this type. At some point in the future we might want to optimize includes to achieve a better compilation time.

If you think it's helpful, you can make this change now.

We can. No nested MessageProperties::Schema though, now it has to be MessageProperties_Schema

src/groups/bmq/bmqimp/bmqimp_event.h

678098 · 2025-02-24T17:00:07Z

src/groups/bmq/bmqp/bmqp_schemalearner.cpp

+
+    const SchemaPtr& result = contextHandle->d_schema_sp;
+
+    if (!result) {


Can we assume that it's an unlikely scenario? Or is it better to evaluate the expression without such assumptions?

678098 · 2025-02-24T17:01:12Z

src/groups/bmq/bmqp/bmqp_schemalearner.cpp

+        MessageProperties mps(d_allocator_p);
+        int               rc = mps.streamIn(blob, input, result);
+
+        if (rc == 0) {


If rc != 0 we will silently return an empty SchemaPtr. Should we log something or skip the message?

src/groups/bmq/bmqp/bmqp_messageproperties.h

src/groups/bmq/bmqp/bmqp_schemalearner.cpp

678098 · 2025-02-24T17:18:49Z

src/groups/bmq/bmqimp/bmqimp_queuemanager.cpp

+    bmqp::MessageProperties::SchemaPtr schema = schemaLearner().learn(
+        schemaLearner().createContext(queueId),
+        bmqp::MessagePropertiesInfo(info.d_header),
+        bdlbb::Blob());


Looking at learn implementation:

SchemaLearner::SchemaPtr SchemaLearner::learn(Context& context, const MessagePropertiesInfo& input, const bdlbb::Blob& blob) { const SchemaIdType inputId = input.schemaId(); if (!isPresentAndValid(inputId)) { // Nothing to do return SchemaPtr(); // RETURN }

We have an early return if schemaId is not present or valid. Does it happen for all messages that don't have message properties? If it can happen that frequently, we do a lot of unnecessary work here:

Pass args and call schemaLearner().createContext(queueId)

Within schemaLearner().createContext(queueId) make lookup in a map (we don't use its result)

Construct bdlbb::Blob() that we don't use

Pass args and call SchemaLearner::learn just to return early

Do you think it's worth to avoid these extra operations if we don't have schemaId?

bmq-oss-ci

Build 2518 of commit f0622b7 has completed with FAILURE

src/groups/bmq/bmqp/bmqp_schemalearner.h

src/groups/bmq/bmqp/bmqp_schemalearner.cpp

src/groups/bmq/bmqp/bmqp_eventutil.h

678098 · 2025-05-15T13:04:02Z

src/groups/bmq/bmqimp/bmqimp_queuemanager.cpp

+
+                    if (mps.streamIn(appData, input.isExtended()) == 0) {
+                        // Learn new schema.
+                        *schemaHolder = schema = mps.makeSchema(d_allocator_p);


Same concern about the number of SchemaPtr copies.

src/groups/bmq/bmqimp/bmqimp_queuemanager.h

src/groups/bmq/bmqimp/bmqimp_queuemanager.cpp

src/groups/bmq/bmqimp/bmqimp_event.h

src/groups/bmq/bmqa/bmqa_message.h

bmq-oss-ci

Build 2657 of commit 5b46679 has completed with SUCCESS

bmq-oss-ci

Build 2676 of commit 2ea8627 has completed with FAILURE

hallfox · 2025-06-05T16:29:37Z

src/groups/bmq/bmqa/bmqa_message.h

@@ -117,6 +121,8 @@ struct MessageImpl {
    /// SubscriptionHandle this message is associated with
    bmqt::SubscriptionHandle d_subscriptionHandle;

+    bsl::shared_ptr<const bmqp::MessageProperties_Schema> d_schema_sp;


This looks like a private component of MessageProperties, we should probably be referring to it through the typedef

Suggested change

bsl::shared_ptr<const bmqp::MessageProperties_Schema> d_schema_sp;

bmqp::MessageProperties::SchemaPtr d_schema_sp;

See #618 (comment)

We can return to that version, sure

bmq-oss-ci

Build 2723 of commit efcc512 has completed with SUCCESS

678098 · 2025-06-11T14:50:16Z

Hi @dorjesinpo, do you want to make any more changes to this PR?

dorjesinpo · 2025-06-11T16:22:09Z

Hi @dorjesinpo, do you want to make any more changes to this PR?

Hi. Other than the typedef change, no. Was waiting for the next round of your review

678098

Last comment

678098 · 2025-06-11T16:59:42Z

src/groups/bmq/bmqimp/bmqimp_queuemanager.cpp

+                    bmqp::EventUtilQueueInfo(msgIterator.header(),
+                                             subQueueInfos[0].id(),
+                                             msgIterator.applicationDataSize(),
+                                             schema));


There are still extra schema copies here, can do the same change you did in another file:

bmqp::MessageProperties::SchemaPtr* schema_p = 0; schema_p = d_schemaLearner.observe(d_schemaLearner.createContext(queueId), input); if (schema_p) { if (schema_p->get() == 0) { // Learn new Schema by reading all MessageProperties. bmqp::MessageProperties mps(d_allocator_p); if (mps.streamIn(d_appData, input.isExtended()) == 0) { // Learn new schema. *schema_p = mps.makeSchema(d_allocator_p); } } }

Signed-off-by: dorjesinpo <[email protected]>

bmq-oss-ci

Build 2742 of commit ef0bd0d has completed with FAILURE

678098 · 2025-06-12T09:56:10Z

src/groups/bmq/bmqimp/bmqimp_queuemanager.cpp

+        bmqp::MessageProperties::SchemaPtr schema;
+
+        if (schema_p) {
+            if (!schema_p->get() == 0) {


Is this ! a bug?

Suggested change

if (!schema_p->get() == 0) {

if (schema_p->get() == 0) {

Is this ! a bug?

it is! Amended

Signed-off-by: dorjesinpo <[email protected]>

bmq-oss-ci

Build 2748 of commit b7531cc has completed with FAILURE

678098

Looks good

678098 · 2025-06-13T14:01:36Z

src/groups/bmq/bmqimp/bmqimp_event.h

-    BSLS_ASSERT_SAFE(d_rawEvent.isPushEvent());
-    BSLS_ASSERT_SAFE(0 <= position);
-    BSLS_ASSERT_SAFE(static_cast<int>(d_correlationIds.size()) > position);
+    BSLS_ASSERT_SAFE(static_cast<int>(d_contexts.size()) > position);


Would prefer to reorder so the smaller value is on the left

Suggested change

BSLS_ASSERT_SAFE(static_cast<int>(d_contexts.size()) > position);

BSLS_ASSERT_SAFE(position < static_cast<int>(d_contexts.size()));

dorjesinpo added the bug Something isn't working label Feb 19, 2025

dorjesinpo requested a review from a team as a code owner February 19, 2025 16:32

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from a454b0d to 91d01b5 Compare February 19, 2025 17:33

bmq-oss-ci bot reviewed Feb 19, 2025

View reviewed changes

dorjesinpo assigned 678098 Feb 19, 2025

dorjesinpo requested a review from 678098 February 19, 2025 18:58

678098 changed the title ~~Fix unsafe scheamalearner use~~ Fix unsafe schemalearner use Feb 24, 2025

678098 changed the title ~~Fix unsafe schemalearner use~~ Fix[BMQ]: unsafe schemalearner use Feb 24, 2025

678098 reviewed Feb 24, 2025

View reviewed changes

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from 91d01b5 to f0622b7 Compare March 11, 2025 00:13

bmq-oss-ci bot reviewed Mar 11, 2025

View reviewed changes

dorjesinpo closed this Apr 8, 2025

dorjesinpo deleted the fix/unsafe-scheamalearner-use branch April 8, 2025 22:06

dorjesinpo restored the fix/unsafe-scheamalearner-use branch April 8, 2025 22:18

dorjesinpo reopened this Apr 8, 2025

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from f0622b7 to bbb706a Compare May 12, 2025 15:28

678098 requested changes May 15, 2025

View reviewed changes

bmq-oss-ci bot reviewed May 16, 2025

View reviewed changes

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch 2 times, most recently from 7fc32f5 to 2ea8627 Compare May 19, 2025 19:41

bmq-oss-ci bot reviewed May 20, 2025

View reviewed changes

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from 2ea8627 to efcc512 Compare June 5, 2025 15:10

hallfox reviewed Jun 5, 2025

View reviewed changes

bmq-oss-ci bot reviewed Jun 5, 2025

View reviewed changes

678098 assigned dorjesinpo and unassigned 678098 Jun 11, 2025

678098 reviewed Jun 11, 2025

View reviewed changes

Fixing thread-unsafe QueueMgr and SchemaLeaneruse

d2f9897

Signed-off-by: dorjesinpo <[email protected]>

dorjesinpo added 4 commits June 11, 2025 16:49

Fixing thread-unsafe SchemaLeaner use

c17c1dc

Signed-off-by: dorjesinpo <[email protected]>

Addressing review

c5abcd8

Signed-off-by: dorjesinpo <[email protected]>

learn schema while iterating event

ae5a20f

Signed-off-by: dorjesinpo <[email protected]>

engage schema learner in the flattener

66b7bab

Signed-off-by: dorjesinpo <[email protected]>

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from efcc512 to ef0bd0d Compare June 11, 2025 21:31

dorjesinpo assigned 678098 and unassigned dorjesinpo Jun 11, 2025

bmq-oss-ci bot reviewed Jun 12, 2025

View reviewed changes

678098 reviewed Jun 12, 2025

View reviewed changes

678098 assigned dorjesinpo and unassigned 678098 Jun 12, 2025

Addressing review

b7531cc

Signed-off-by: dorjesinpo <[email protected]>

dorjesinpo force-pushed the fix/unsafe-scheamalearner-use branch from ef0bd0d to b7531cc Compare June 12, 2025 21:31

dorjesinpo assigned 678098 and unassigned dorjesinpo Jun 12, 2025

bmq-oss-ci bot reviewed Jun 12, 2025

View reviewed changes

678098 approved these changes Jun 13, 2025

View reviewed changes

678098 assigned dorjesinpo and unassigned 678098 Jun 13, 2025

dorjesinpo merged commit 495dc34 into main Jun 13, 2025
39 of 40 checks passed

678098 deleted the fix/unsafe-scheamalearner-use branch June 13, 2025 14:34


		const SchemaPtr& result = contextHandle->d_schema_sp;

		if (!result) {

	bsl::shared_ptr<const bmqp::MessageProperties_Schema> d_schema_sp;
	bmqp::MessageProperties::SchemaPtr d_schema_sp;

	BSLS_ASSERT_SAFE(static_cast<int>(d_contexts.size()) > position);
	BSLS_ASSERT_SAFE(position < static_cast<int>(d_contexts.size()));

Fix[BMQ]: unsafe schemalearner use #618

Fix[BMQ]: unsafe schemalearner use #618

Uh oh!

Conversation

dorjesinpo commented Feb 19, 2025

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

678098 commented Jun 11, 2025

Uh oh!

dorjesinpo commented Jun 11, 2025

Uh oh!

678098 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmq-oss-ci bot left a comment

Choose a reason for hiding this comment

Uh oh!

678098 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!