Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle moved messages for unreads #1311

Merged
merged 10 commits into from
Mar 15, 2025
Merged

Handle moved messages for unreads #1311

merged 10 commits into from
Mar 15, 2025

Conversation

PIG208
Copy link
Member

@PIG208 PIG208 commented Jan 27, 2025

Work toward #901

I prioritized updating unreads data for moved messages because it is more visible after supporting the resolve/unresolve topic actions. Updating recent senders data will be a follow-up to this.

@PIG208 PIG208 force-pushed the pr-unreads branch 5 times, most recently from 683c713 to dad5aa0 Compare January 30, 2025 23:23
@PIG208 PIG208 marked this pull request as ready for review January 30, 2025 23:24
@PIG208 PIG208 added the maintainer review PR ready for review by Zulip maintainers label Jan 30, 2025
@PIG208 PIG208 requested a review from chrisbobbe January 30, 2025 23:24
@chrisbobbe
Copy link
Collaborator

chrisbobbe commented Feb 8, 2025

Actually @gnprice, I'd appreciate any thoughts you have from a skim of the first three commits:

model [nfc]: Add PropagateMode.fromRawString
api [nfc]: Extract and parse UpdateMessageMoveData from UpdateMessageEvent
api: Parse UpdateMessageMoveData with inter-related constraints

before I start my review, if that's OK; I think you designed/implemented similar logic in zulip-mobile.

@gnprice
Copy link
Member

gnprice commented Feb 19, 2025

Didn't catch the GitHub notification for this earlier, sorry — thanks for the ping in chat today.

The general strategy in those commits looks good. The high-level comments I have are:

  • Here in the API parsing code, when the response is malformed, instead of returning null, let's just throw — for example if something shouldn't be null, then just the ! operator is enough.

    That's what already happens for all kinds of other ways it can be malformed, like if json['stream_id'] is a string value (instead of a number or absent/null). Such exceptions get turned into MalformedServerResponseException by the generic code in ApiConnection.send, and in the case of events those get handled appropriately by the long-poll loop. (They cause a reload of the store, because if there's an event we weren't able to handle then we may no longer have accurate state.)

  • Let's have an invariant that we don't wind up with an UpdateMessageMoveData that doesn't represent a move: i.e. where the old and new channel ID are equal and the old and new topic are equal. In our code that updates data structures we might rely on that assumption (for example so that we're not mutating the same data structure twice concurrently as both the source and target of a move).

    That can be enforced by both an assert on the constructor, and the try-parse method either throwing or returning null before getting to the constructor in that case. (Compare the zulip-mobile logic quoted below.)

  • This logic about in what circumstances the various "orig" and "new" fields might be null, or are required to be present, feels pretty unsatisfying. Even though it's meant to accurately reflect guarantees that are in the API docs (https://zulip.com/api/get-events#update_message), and probably it does, it doesn't feel solid.

    In particular, facts like this one feel like quirks:

    orig_subject (aka origTopic) is documented to be present when either the stream or topic changed.

    and are asymmetric between the topic and the channel fields. Indeed this sort of fact has changed in the past (origStreamId aka stream_id became more widely present in Zulip Server 5), and just last year we had to edit the logic because the docs were wrong (020bfcb) about a fact of this kind which we had relied on.

    This logic in the zulip-mobile version feels more robust, because it's based on natural facts about the semantics the event is about:

    if (new_topic === orig_topic && new_stream_id === orig_stream_id) {
      // Stream and topic didn't change.
      return null;

    Perhaps better yet, though: we could just look to see if propagate_mode is present. (The zulip-mobile version didn't consider propagate_mode as part of this logic.) That's something that is necessary if there was a move, and clearly meaningless if there wasn't — so it seems like a very crisp and reliable indicator of whether there was a move. (And the API docs agree that it, too, is present just if there was a move.)

@gnprice
Copy link
Member

gnprice commented Feb 19, 2025

(So probably the right next steps are: @PIG208 go ahead and act on the feedback above, and then after that revision @chrisbobbe will do maintainer review.)

PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Feb 20, 2025
This drops the separate if-statements in favor of null-checks and
tries to rely on the natural semantics of the API.

See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
@PIG208
Copy link
Member Author

PIG208 commented Feb 20, 2025

Thanks for the review! The PR has been updated.

PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Feb 20, 2025
This drops the separate if-statements in favor of null-checks and
tries to rely on the natural semantics of the API.

This drops an expcetion that would have been thrown when propagate mode
is not present but the topic name/stream id changes.

See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
Copy link
Collaborator

@chrisbobbe chrisbobbe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'm excited for more correctness in the Unreads model! Small comments below.

Comment on lines 785 to 836
await store.handleEvent(eg.updateMessageEventMoveFrom(
origMessages: otherChannelMovedMessages,
newStreamId: otherStream.streamId,
newStreamId: thirdStream.streamId,
));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about adding an assert to eg.updateMessageEventMoveFrom, to catch invalid input?:

  final origStreamId = origMessage.streamId;
  final origTopic = origMessage.topic;
  assert(() {
    final streamChanged = newStreamId != null && newStreamId != origStreamId;
    final topicChanged = newTopic != null && newTopic != origTopic;
    return streamChanged || topicChanged;
  }());

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful before we pull out UpdateMessageMoveData, which actually helped me discover this issue in the first place. Having the assertions and check in one place (UpdateMessageMoveData) appears to be easier to maintain.

};

test('stream_id -> origStreamId', () {
test('stream_id -> origStreamId, subject = orig_subject', () {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

events test: Make test data more realistic

Could please you explain this commit a bit more? I don't understand the changes to the test descriptions. Their current descriptions are "A -> B" pairs, where A is input and B is expected output. I don't understand what the added pair "subject = orig_subject" means, and it doesn't follow that pattern. If I look at 'orig_subject' and 'subject' in the input, I see they're not equal; I guess that's not what the "=" is for, then.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's meant to represent that the topic does not change in this move. Changing it to a short phrase should help make it clear.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new check seems like it'd be clearest as a new separate test case. I'm not sure how it relates to the existing test, or really what it's meant to check.

Comment on lines 796 to 754
static Object? _readMoveData(Map<Object?, Object?> json, String key) {
// Parsing [UpdateMessageMoveData] requires `json`, not the default `json[key]`.
return json as Object?;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point we know json is a Map<String, dynamic>, because it was passed as the argument to UpdateMessageEvent.fromJson, which has that type. I think it would be neater to have _readMoveData encapsulate this, having Map<String, dynamic> for its return type instead of Object?.

For readValue, we're asked to implement a function that takes any Map, which is why we can't say Map<String, dynamic> json for _readMoveData. But that's fine; we can still have an assert:

--- lib/api/model/events.dart
+++ lib/api/model/events.dart
@@ -723,8 +723,7 @@ class UpdateMessageMoveData {
   /// [UpdateMessageEvent].
   ///
   /// Throws an error if the data is malformed.
-  factory UpdateMessageMoveData.fromJson(Object? json) {
-    json as Map<String, Object?>;
+  factory UpdateMessageMoveData.fromJson(Map<String, dynamic> json) {
     final origStreamId = (json['stream_id'] as num?)?.toInt();
     final newStreamId = (json['new_stream_id'] as num?)?.toInt();
     final propagateModeString = json['propagate_mode'] as String?;
@@ -793,9 +792,10 @@ class UpdateMessageEvent extends Event {
     required this.isMeMessage,
   });
 
-  static Object? _readMoveData(Map<Object?, Object?> json, String key) {
+  static Map<String, dynamic> _readMoveData(Map<dynamic, dynamic> json, String key) {
     // Parsing [UpdateMessageMoveData] requires `json`, not the default `json[key]`.
-    return json as Object?;
+    assert(json is Map<String, dynamic>); // value came through `fromJson` with this type
+    return json as Map<String, dynamic>;
   }
 
   factory UpdateMessageEvent.fromJson(Map<String, dynamic> json) =>

I know that the assert is redundant with the json as Map<String, dynamic>. My thought is to be clear that this is an invariant we own in our code; it's not about what we expect from the server (which is the case for most casts here, like json['propagate_mode'] as String?).

What do you think?

Comment on lines 704 to 705
/// Data structure representing a message move.
class UpdateMessageMoveData {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

api [nfc]: Extract and parse UpdateMessageMoveData from UpdateMessageEvent

At this commit the UpdateMessageMoveData dartdoc isn't accurate, because we still instantiate the class when there was no move, or when the fields have the correct types but are incoherent about whether there was a move (e.g. propagate_mode present but the stream/topic fields all absent).

How about:

/// Data structure holding the fields about a message move.

(or similar), until the "representing a message move" becomes accurate in a later commit.

Comment on lines 737 to 746
if (propagateMode == null) {
// There was no move.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

api: Simplify move data parsing

This drops the separate if-statements in favor of null-checks and
tries to rely on the natural semantics of the API.

This drops an expcetion that would have been thrown when propagate mode
is not present but the topic name/stream id changes.

See Greg's comment on this:
  https://github.com/zulip/zulip-flutter/pull/1311#issuecomment-2667327271

Signed-off-by: Zixuan James Li <[email protected]>

This new implementation follows the documented API, right? From that linked comment from Greg:

Perhaps better yet, though: we could just look to see if propagate_mode is present. (The zulip-mobile version didn't consider propagate_mode as part of this logic.) That's something that is necessary if there was a move, and clearly meaningless if there wasn't — so it seems like a very crisp and reliable indicator of whether there was a move. (And the API docs agree that it, too, is present just if there was a move.)

So let's explain it more confidently than "tries to rely on the natural semantics of the API". 🙂 Can just briefly say why we're not directly transcribing zulip-mobile code, but that this way is valid and more crisp, and link to Greg's comment.

Also how about squashing this commit into the previous:

api: Parse UpdateMessageMoveData with inter-related constraints

and perhaps even squash that into the one before:

api [nfc]: Extract and parse UpdateMessageMoveData from UpdateMessageEvent

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this part:

This drops an expcetion that would have been thrown when propagate mode
is not present but the topic name/stream id changes.

That would be easy to add back in, right, in the if (propagateMode == null)? Just that we'd throw if any of the four stream / topic fields are present.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be easy to add back in, right, in the if (propagateMode == null)? Just that we'd throw if any of the four stream / topic fields are present.

When there isn't a move, some of those four fields may still be present and it's complicated what exact subset is expected. (See "feel like quirks" and "changed in the past" in this comment: #1311 (comment)) So such a check would have to be more complicated than that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can have a check like:

// pseudo representation of the types of the fields
TopicName? origTopic = ...;
int? origStreamId = ...;
final newTopic ??= origTopic;
final newStreamId ??= origStreamId;

if (propagateMode == null) {
  if (origTopic != newTopic || origStreamId != newStreamId) {
    throw FormatException('move but no propagateMode'); // or "maybe move but no propagateMode"
  }
  // There was no move.
  return null;
}

This still throws if we have origTopic == null or origStreamId == null with a non-null newTopic or newStreamId, but that's malformed data anyway.

It is the negation of origStreamId == newStreamId && origTopic == newTopic, which is used later in this method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that could be fine if it doesn't involve making the other logic any more complicated to support it.

if (topics == null) return;
if (topics == null) return QueueList();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing we do with this is an .isEmpty check, right? How about returning null instead of creating a new QueueList (seems more efficient).

bool _handleMessageMove(UpdateMessageEvent event) {
final messageMove = event.moveData;
if (messageMove == null) {
// No moved messages or malformed event.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the "or malformed event" part is from an old revision—we throw if it's malformed, right?

Comment on lines 313 to 319
final topics = streams[messageMove.origStreamId];
if (topics == null || topics[messageMove.origTopic] == null) {
// No known unreads affected by move; nothing to do.
return false;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is redundant with the early returns at the top of _removeAllInStreamTopic, right?

// No moved messages or malformed event.
return false;
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could say

    final UpdateMessageMoveData(
      :origStreamId, :newStreamId, :origTopic, :newTopic) = messageMove;

to avoid repetition of messageMove. several times.

Comment on lines -299 to +308
// TODO(#901) handle moved messages
madeAnyUpdate |= _handleMessageMove(event);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unreads: Handle updates when there are moved messages

The commit message can get a "fixes" line. 🎉

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will get there soon! #901 also requires updates to recent senders data model.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh oops! Indeed!

PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Feb 24, 2025
This data structure encapsulates some checks so that we can make all
fields non-nullable, with reasonable fallback values.  As of writing,
we do not use origStreamId (a.k.a.: 'stream_id') when there was no
message move, even though it is present if there were content edits
This makes dropping 'stream_id' when parsing `moveData` into `null`
acceptable for now.

This also allows us to drop the `assert`'s and "TODO(log)"'s, because
the stacktrace we need can be retrieved after throwing these
`FormatException`'s.

This is similar to zulip-mobile code for parsing move data. The main
difference is that we check the value of `propagate_mode`, which is
documented to be present on message moves.  With this single indicator,
the logic is crisp for ruling out non-move message update events.
See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Feb 24, 2025
This data structure encapsulates some checks so that we can make all
fields non-nullable, with reasonable fallback values.  As of writing,
we do not use origStreamId (a.k.a.: 'stream_id') when there was no
message move, even though it is present if there were content edits
This makes dropping 'stream_id' when parsing `moveData` into `null`
acceptable for now.

This also allows us to drop the `assert`'s and "TODO(log)"'s, because
the stacktrace we need can be retrieved after throwing these
`FormatException`'s.

This is similar to zulip-mobile code for parsing move data. The main
difference is that we check the value of `propagate_mode`, which is
documented to be present on message moves.  With this single indicator,
the logic is crisp for ruling out non-move message update events.
See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Feb 25, 2025
This data structure encapsulates some checks so that we can make all
fields non-nullable, with reasonable fallback values.  As of writing,
we do not use origStreamId (a.k.a.: 'stream_id') when there was no
message move, even though it is present if there were content edits
This makes dropping 'stream_id' when parsing `moveData` into `null`
acceptable for now.

This also allows us to drop the `assert`'s and "TODO(log)"'s, because
the stacktrace we need can be retrieved after throwing these
`FormatException`'s.

This is similar to zulip-mobile code for parsing move data. The main
difference is that we check the value of `propagate_mode`, which is
documented to be present on message moves.  With this single indicator,
the logic is crisp for ruling out non-move message update events.
See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
@chrisbobbe
Copy link
Collaborator

It turns out there's a move-message edge case that we'll need to handle, either here or as a followup issue.

It's this TODO in Unreads.handleUpdateMessageEvent:

    // We assume this event can't signal a change in a message's 'read' flag.
    // TODO can it actually though, when it's about messages being moved into an
    //   unsubscribed stream?
    //   https://chat.zulip.org/#narrow/stream/378-api-design/topic/mark-as-read.20events.20with.20message.20moves.3F/near/1639957

The conclusion from an API-design discussion is that clients need to check if the messages were moved into a channel that is unsubscribed, and if so, drop them from unreads, since we won't be getting a separate event about that. (This is currently undocumented on update_message but should be soon.)

@PIG208
Copy link
Member Author

PIG208 commented Feb 25, 2025

Just read the discussion. Yeah, it seems straightforward enough to include in the main implementation commit. I removed the TODO and kept a link referring to that discussion at where this is handled.

From my understanding, the isRead flag gets stale and the claim that " We assume this event can't signal a change in a message's 'read' flag." holds, so there is nothing else to do at the TODO site; we handle this in _handleMessageMove.

@PIG208 PIG208 requested a review from chrisbobbe February 25, 2025 03:48
@PIG208
Copy link
Member Author

PIG208 commented Mar 6, 2025

Thanks for the review! This should be ready again. I made a new revision of tests in the last commit.

@PIG208
Copy link
Member Author

PIG208 commented Mar 6, 2025

I'm planning to update the test cases for the main implementation again, after having a call with Greg.

@PIG208
Copy link
Member Author

PIG208 commented Mar 7, 2025

This should be ready now.

Copy link
Member

@gnprice gnprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the revision! Comments below. In this round I've now read the whole thing.


final wasResolveOrUnresolve = (newStreamId == null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message [nfc]: Remove redundant parentheses

This change is trivial and self-explanatory, so it can just get squashed into the substantive change that's touchinig the same code.

final wasResolveOrUnresolve = newStreamId == null
&& MessageEditState.topicMoveWasResolveOrUnresolve(origTopic, newTopic!);
final wasResolveOrUnresolve = origStreamId == newStreamId
&& MessageEditState.topicMoveWasResolveOrUnresolve(origTopic!, newTopic!);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message: Enhance message move checks

Because of this simplification, it is easier to make sense of the
constraints that apply to the inter-related fields.

This commit doesn't seem to yet make sense on its own.

For example, the ! operators here don't seem justified. Why are these non-null?

I think the answer is "because they're documented to both be present if there was a move". That needs to be more explicit, though — otherwise it just looks like we potentially have a case where we crash. (When the logic lives inside our API parsing, it no longer needs to be so explicit because that's routine for that parsing code.)

In fact let's avoid throwing here in this handle-event method for malformed events; instead let's ignore such events (like the existing code does), and reserve the throwing for the parsing code. That's why at #1311 (comment) I requested:

In particular it'd be helpful if that second commit is NFC for debug builds — i.e. if its only functional change is that in some conditions where the old code would assert, the new code will throw while parsing.

That means that after the first commit, while this logic still lives in this handle-event method, it still asserts for malformed events rather than (in non-debug builds) throwing.

Comment on lines 320 to 322
final UpdateMessageMoveData(
:origStreamId, :newStreamId, :propagateMode, :origTopic, :newTopic,
) = event.moveData!;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: leave out unused variable

Suggested change
final UpdateMessageMoveData(
:origStreamId, :newStreamId, :propagateMode, :origTopic, :newTopic,
) = event.moveData!;
final UpdateMessageMoveData(
:origStreamId, :newStreamId, :origTopic, :newTopic) = event.moveData!;

Comment on lines 558 to 563
final retainedMessageIds = <int>[];
for (final id in messageIds) {
if (!incomingMessageIds.contains(id)) {
retainedMessageIds.add(id);
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just messageIds.where(…).toList(), right?

Comment on lines 567 to 563
// conversation are removed, which avoids making a copy of `messageId`
// unnecessarily.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// conversation are removed, which avoids making a copy of `messageId`
// unnecessarily.
// conversation are removed, which avoids making a copy of `messageIds`
// unnecessarily.

right?

Comment on lines 479 to 482
(_) => eg.streamMessage(
stream: origChannel, topic: origTopic, flags: [MessageFlag.read]));
final unreadMessages = List<StreamMessage>.generate(10,
(_) => eg.streamMessage(stream: origChannel, topic: origTopic));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: make the parallel parts visibly parallel:

Suggested change
(_) => eg.streamMessage(
stream: origChannel, topic: origTopic, flags: [MessageFlag.read]));
final unreadMessages = List<StreamMessage>.generate(10,
(_) => eg.streamMessage(stream: origChannel, topic: origTopic));
(_) => eg.streamMessage(stream: origChannel, topic: origTopic,
flags: [MessageFlag.read]));
final unreadMessages = List<StreamMessage>.generate(10,
(_) => eg.streamMessage(stream: origChannel, topic: origTopic));

Comment on lines 478 to 479
final readMessages = List<StreamMessage>.generate(10,
(_) => eg.streamMessage(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Messages can get mutated to handle events. So let's have these get generated fresh by each test that needs them.

(In particular if we refactored so that instead of calling Unreads.handleUpdateMessageEvent like other tests in this file, these called PerAccountStore.handleEvent like we do in some other areas of test/model/, I believe these would break as written.)

)).toList();
}

test('smoke', () async {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
test('smoke', () async {
test('moving a conversation exactly', () async {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can also check we have the optimization at #1311 (comment) , using identicalTo.

@@ -469,6 +469,145 @@ void main() {
}
}
});

group('moves', () {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other cases to test here, in order to properly exercise _popAllInStreamTopic:

  • the moved messages are an (inhabited) proper subset of the existing unreads in the conversation
  • the moved messages are a proper superset of the (inhabited) set of existing unreads in the conversation

PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Mar 11, 2025
Because of this simplification, it is easier to make sense of the
constraints that apply to the inter-related fields.

This also allows us to drop the `assert`'s and "TODO(log)"'s, because
the stacktrace we need can be retrieved after throwing these
`FormatException`'s.

This is similar to zulip-mobile code for parsing move data. The main
difference is that we now also check the value of `propagate_mode`,
which is documented to be present on message moves.
See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Mar 12, 2025
…hecks

This throws `AssertionError`s on debug builds when the event
data is malformed, and ignores it for release builds, ensuring that none
of the move-related fields are `null` by the time we look at their
values.

Because of this simplification, it is easier to make sense of the
constraints between the fields when there is a move.

This is similar to zulip-mobile code for parsing move data. The main
difference is that we now also check the value of `propagate_mode`,
which is documented to be present on message moves.
See Greg's comment on this:
  zulip#1311 (comment)

Signed-off-by: Zixuan James Li <[email protected]>
@PIG208
Copy link
Member Author

PIG208 commented Mar 12, 2025

Thanks for the review! Reorganized the move data change into 1 commit followed by 3 nfc ones, and addressed the comments on the main commit.

  • eeb89b4 message test [nfc]: Move tests to a new home
  • cf816ef api [nfc]: Simplify move data parsing with null-checks
  • 40ccd50 api [nfc]: Extract and parse UpdateMessageMoveData from UpdateMessageEvent
  • b9eafee message: Preface message move handling with more comprehensive null-checks

Copy link
Member

@gnprice gnprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Here's a partial review focused on that stretch of refactoring commits. I decided some of this would be most efficient for me to write instead of trying to describe — so I've just pushed a revision where those 4 commits you mentioned above are now these 5:
95fbd04 message: Identify moves by inequality new vs orig, not just non-nullity
4392beb message [nfc]: In moves, give newStreamId/newTopic their fallback values sooner
6b75bae wip api [nfc]: Extract and parse UpdateMessageMoveData from UpdateMessageEvent
5d2a8bb api [nfc]: Simplify move data parsing with null-checks
e79ea1e message test [nfc]: Move tests to a new home

The first two are new commits I just wrote (and one of them has you as a co-author), replacing your commit:
b9eafee message: Preface message move handling with more comprehensive null-checks

The third one I marked as "wip" because I'm not sure it's entirely coherent after I rebased it atop the other two.

The two after that I didn't touch.

I think this revision handles my comments below, except for the last one. So the open items for you are that last small comment, plus please take a look at the "wip" commit 6b75bae and see what adjustments it needs.

Comment on lines 208 to 204
assert(debugLog('Malformed UpdateMessageEvent: move but no propagateMode')); // TODO(log)
assert(false, 'Malformed UpdateMessageEvent: move but no propagateMode'); // TODO(log)
return;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm I see, I guess in #1311 (comment) when I wrote:

In particular it'd be helpful if that second commit is NFC for debug builds — i.e. if its only functional change is that in some conditions where the old code would assert, the new code will throw while parsing.

that was a bit misleading because I'd read the existing code too quickly. Sorry.

When I said "where the old code would assert", I meant what the existing code already does here — but in fact I see now that it's not really asserting, it's just logging a message in debug mode, and then ignoring the event.

This change from log-and-ignore to assert-or-ignore is fine, I guess, but isn't really necessary. The key thing I was going for here is just that the second change is NFC for the events that either its old or new code accepts; so old and new accept the same set of events, and have the same effect for those events, and the only functional change is in the specifics of what it looks like when the code rejects an event. (Plus that it's easy to tell by reading the change that it is indeed NFC in that respect, because the logic matches.)

The reason it's useful to separate that functional change is that it's something which should ideally happen in the same commit as where this logic gets moved from this file to the API parsing, because the old way (just ignore the event and maybe log) is appropriate here where we're already deep into processing the event, and the new way (throw) is appropriate in the API parsing code where it's not too late to reject the whole event up front.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I did in the later review at #1311 (comment) say:

That means that after the first commit, while this logic still lives in this handle-event method, it still asserts for malformed events

so there at least I indicated (with "still") that I expected this aspect wouldn't change in the first commit of this split.)

return;
}

if (newStreamId == null && newTopic == null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

message: Preface message move handling with more comprehensive null-checks

Does this really make the null-checks more comprehensive? The old null-checks look pretty comprehensive to me — there's the one on this line, and two more below it.

In particular, is there a case where this version rejects an event which the old code would have acted on?

It seems to me like the main point of this commit is the way it moves the two fallbacks like ?? origTopic up to the top of this logic, in order to make it closer to how it'll be after we move all the "find if and how the messages were moved" logic out from here into the API parsing code.

'stream_id': 1,
'new_stream_id': 2,
'orig_subject': null,
})).throws<AssertionError>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that these exception types keep changing in subsequent commits seems like a good sign that they're a detail we don't actually care about — so best to leave the specific types out.

@PIG208
Copy link
Member Author

PIG208 commented Mar 13, 2025

Thanks for helping with this! Pushed an update to the PR.

Copy link
Member

@gnprice gnprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This is quite close now — just small comments below.

// When parsing this, 'stream_id', which is also present when there was only
// a content edit, cannot be recovered if this ends up returning `null`.
// This may matter if we ever need 'stream_id' when no message move occurred.
static UpdateMessageMoveData? tryParseFromJson(Map<String, Object?> json) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

This is mostly NFC except that we were logging and ignoring malformed
event data before, and now start to throw errors with the parsing code.

s/with/within/ — i.e., we're now throwing errors from inside the parsing code


if (newStreamId == origStreamId && newTopic == origTopic) {
if (propagateMode != null) {
throw FormatException(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in commit message:

(It might also change what particular message gets printed to the
debug log; but that doesn't count as a behavior change. As a part of
that, we add an additional check for propagate mode, to succeed the
block of assertion.)

I see this second sentence as not so closely related to the first one; the first sentence is already true without any caveats related to the propagateMode check, since the only effect of that check is a log message.

Instead let's have a short paragraph at the end, after the "This change is useful" paragraph:

While here, we add a debug log message on unexpected propagateMode,
to further bring this closer to how it will look in the API parsing
code.

Comment on lines 765 to 768
final int origStreamId;
final int newStreamId;

final PropagateMode propagateMode;

final TopicName origTopic;
final TopicName newTopic;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let's follow the existing code in main by putting these in a more logical order, rather than this order (which I realize reflects the API docs). After all, by making newStreamId and newTopic non-null we're already deviating from the server API, and in a more substantive way 🙂

The existing code's order of stream, stream, topic, topic, propagateMode seems good.

(Then similarly for the parsing code below, and so on.)

newChannel: newChannel, newTopic: newTopic));
final newMessageIds =
model.streams[newChannel.streamId]![TopicName(newTopic)]!;
check(originalMessageIds).identicalTo(newMessageIds);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
check(originalMessageIds).identicalTo(newMessageIds);
// Check we successfully avoided making a copy of the list.
check(originalMessageIds).identicalTo(newMessageIds);

(otherwise it seems not quite self-explanatory why this check is relevant / why we'd care if it started failing)

check(originalMessageIds).identicalTo(newMessageIds);
});

test('moving all read and unread messages from a conversation', () async {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test seems redundant with the one above it, "moving some read and all unread messages from a conversation". So we can leave it out.

(The only difference is whether there are 2 or 10 read messages included in the list to move. In both cases there are 10 read messages included in fillWithMessages in setup; but read messages have no effect in fillWithMessages, so that doesn't matter.)

I guess in particular this means that for one of the two tests I asked for at #1311 (comment) , all that was needed was a rename to make clearer that an existing test covered that scenario 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha, I see. Yeah it occurred to me that the current test names aren't accurate enough about what they are doing, so it was no so clear whether they cover all the cases we care about.

Updated the names to express the relationships between the messages in terms of sets.

Comment on lines 578 to 580
checkMatchesMessages([
...copyMessagesWith(unreadMessages, newTopic: newTopic),
]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: no need for a list literal, can just use the copyMessagesWith result directly

Copy link
Member Author

@PIG208 PIG208 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! Updated the PR.

check(originalMessageIds).identicalTo(newMessageIds);
});

test('moving all read and unread messages from a conversation', () async {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha, I see. Yeah it occurred to me that the current test names aren't accurate enough about what they are doing, so it was no so clear whether they cover all the cases we care about.

Updated the names to express the relationships between the messages in terms of sets.

Copy link
Member

@gnprice gnprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good; merging.

]);
});

test('moved messages ∩ unread messages ≠ Ø, moved messages ∩ read messages ≠ Ø, moved messages ⊅ unread messages', () async {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
test('moved messages ∩ unread messages ≠ Ø, moved messages ∩ read messages ≠ Ø, moved messages ⊅ unread messages', () async {
test('moved messages ∩ unread messages ≠ Ø, moved messages ∩ read messages ≠ Ø, moved messages ⊅ unread messages', () async {

🙂 I'm not convinced this way is clearer than using words, but it does work.

PIG208 and others added 10 commits March 14, 2025 21:17
The streamId of the moved messages does not change as intended, and the
topic remains the same; this is not a valid message move.

We will later introduce a data structure that captures inconsistencies
of this kind.

Signed-off-by: Zixuan James Li <[email protected]>
The fields orig_subject and propagate_mode are always
present on message moves.

See also API documentation:
  https://zulip.com/api/get-events#update_message

Signed-off-by: Zixuan James Li <[email protected]>
This is NFC except in certain malformed cases: where either
newStreamId is present and equals origStreamId, or newTopic is
present and equals origTopic.  The API docs say neither of those
should happen.

This version is useful because it brings this code closer to how it
will look when we move this logic of "were the messages moved, and
how" into the API parsing code so that other parts of the data model
can reuse the results.
…ues sooner

To see that this is NFC, first observe that the only possible
behavior change this could cause is to change some event from being
ignored (hitting a `return` in this section of code) to being
processed by the code below, or vice versa.

(It might also change what particular message gets printed to the
debug log; but that doesn't count as a behavior change.)

Then both before and after this change, the events that this overall
set of conditions accepts are those that (a) have all of origTopic,
origStreamId, and propagateMode non-null, and (b) have at least one
of newTopic or newStreamId with a non-null value different from
origTopic or origStreamId respectively.

This change is useful because it brings this logic closer to how it
will look when we move it into the API parsing code.

While here, we add a debug log message on unexpected propagateMode,
to further bring this closer to how it will look in the API parsing
code.

Co-authored-by: Zixuan James Li <[email protected]>
This is mostly NFC except that we were logging and ignoring malformed
event data before, and now start to throw errors within the parsing
code.

This data structure encapsulates some checks so that we can make all
fields non-nullable, with reasonable fallback values.  As of writing,
we do not use origStreamId (a.k.a.: 'stream_id') when there was no
message move, even though it is present if there were content edits
This makes dropping 'stream_id' when parsing `moveData` into `null`
acceptable for now.

Signed-off-by: Zixuan James Li <[email protected]>
This does not correctly catch the case when
`origTopic='topic'`, and
`newTopic=origStreamId=newStreamId=null`.

Instead, remove it and rely on the assertions from
UpdateMessageMoveData's constructor.

Signed-off-by: Zixuan James Li <[email protected]>
We could have extend `_removeAllInStreamTopic` to fit the needs of
handling moved unreads, but it is used in a code path where messages
are marked as read (which will become hotter after zulip#81).

The new helper is designed to avoid making copies of message IDs
when possible.  As for the name, we use "pop" (inspired by Python)
to indicate that the removed items are returned, since "removeAll"
in Dart doesn't carry that meaning.

Signed-off-by: Zixuan James Li <[email protected]>
@gnprice gnprice merged commit 34e6201 into zulip:main Mar 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration review Added by maintainers when PR may be ready for integration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants