RealmSwift.List order not in sync on different clients

[divan84]

How frequently does the bug occur?

All the time

Description

We found the following problem with conflict resolution in RealmSwift.List: When changing the list order on several devices of which some are online and some are offline, the list order will stay different on the devices even after they were re-synced and are all back online.

We would expect the list order to be the same on all devices after sync. Right now, only if the app is deleted and reinstalled the order is the same again.

Stacktrace & log output

No response

Can you reproduce the bug?

Yes, always

Reproduction Steps

1. Create a simple app with device sync on
2. The app should display a simple data model like this:

class Project: Object, ObjectKeyIdentifiable {
    @Persisted(primaryKey: true) public var _id = ObjectId.generate()
    @Persisted public var comments: RealmSwift.List<Comment> = .init()
    @Persisted public var title: String
}
class Comment: Object, ObjectKeyIdentifiable {
    @Persisted(primaryKey: true) public var _id = ObjectId.generate()
    @Persisted public var text: String
}

3. The list items can be re-ordered
4. Log in with with at least two users on different devices
5. One user is offline and randomly changes list order several times
6. Another user does the same while online
7. The first user goes back online

Result
The list order on both devices is different and will stay so until the app is deleted and reinstalled.

Version

Realm 10.33.0, RealmDatabase 12.13.0

What SDK flavour are you using?

MongoDB Realm (i.e. Sync, auth, functions)

Are you using encryption?

No, not using encryption

Platform OS and version(s)

iOS 16.1, macOS 13.0

Build environment

Xcode version: 14.2
Dependency manager and version: SPM

[sync-by-unito]

➤ Jonathan Reams commented:

Sorry for the late response, between the holidays and a bunch of work coming back from the holidays I only got to this recently.

I tried to reproduce this in a C++ integration test in realm core and came up with the below test case, which I think does all the reproduction steps. This test case actually reproduced the issue until I added orig_realm->refresh() towards the end to force reading from the latest version of the database - which would suggest you're seeing stale data. You say in the description that you have to delete/re-install the app? Does that mean if you just close/re-open the app it doesn't update? You shouldn't have to do this for the realm to eventually update, but have you tried explicitly refreshing the realm to make sure you're seeing the latest data? Do you have any trace-level logs I could look at to get a better sense of exactly what's going on between your devices?

TEST_CASE("RCOCOA-1929 repro", "") {
    using namespace std::literals::string_literals;
    std::string base_url = get_base_url();
    REQUIRE(!base_url.empty());
    const auto partition = random_string(100);
    Schema schema = {{"Project",
                      {
                          {"_id", PropertyType::ObjectId, true},
                          {"title", PropertyType::String},
                          {"comments", PropertyType::Array | PropertyType::Object, "Comments"},
                      }},
                     {"Comments",
                      {
                          {"_id", PropertyType::Int, true},
                          {"text", PropertyType::String | PropertyType::Nullable},
                      }}};
    auto server_app_config = minimal_app_config(base_url, "rcocoa-1929", schema);
    TestAppSession test_session(create_app(server_app_config));
    std::string parition("default");
    auto project_id = ObjectId::gen();
    std::default_random_engine engine(std::random_device{}());
    const size_t expected_comments = 10;

    auto shuffle_list = [&](const SharedRealm& realm, List& list) {
        std::uniform_int_distribution<size_t> dist;
        using dist_param = decltype(dist)::param_type;
        size_t n = list.size() - 1;
        for (size_t idx = n - 1; idx > 0; --idx) {
            auto second_idx = dist(engine, dist_param(0, idx));
            if (idx == second_idx) {
                continue;
            }
            realm->begin_transaction();
            std::cerr << "=== Swapping idx " << idx << " with idx " << second_idx << " ===\n";
            list.move(idx, second_idx);
            realm->commit_transaction();
        }
    };

    auto get_comments_list = [&](const SharedRealm& realm) {
        CppContext c(realm);
        auto project_obj = Object::get_for_primary_key(c, realm, "Project", std::any(project_id));
        return List(project_obj, project_obj.get_object_schema().property_for_name("comments"));
    };

    auto comments_to_pks = [](List& list) {
        std::vector<int64_t> ret;
        for (size_t i = 0; i < list.size(); ++i) {
            ret.push_back(list.get_object(i).get_primary_key().get_int());
        }
        return ret;
    };

    create_user_and_log_in(test_session.app());
    auto user_1 = test_session.app()->current_user();
    SyncTestFile user_1_realm(user_1, partition, schema);
    auto orig_realm = Realm::get_shared_realm(user_1_realm);
    {

        std::cerr << "=== Creating initial object ===\n";
        CppContext c(orig_realm);
        orig_realm->begin_transaction();
        auto project_obj = Object::create(c, orig_realm, "Project",
                                          std::any(AnyDict{{"_id"s, project_id}, {"title"s, "test title"s}}));
        auto comments_list = get_comments_list(orig_realm);
        orig_realm->commit_transaction();

        std::cerr << "=== Creating comments ===\n";
        for (size_t i = 0; i < expected_comments; ++i) {
            orig_realm->begin_transaction();
            auto obj =
                Object::create(c, orig_realm, "Comments",
                               std::any(AnyDict{{"_id"s, int64_t(i)}, {"text"s, util::format("comment %1", i)}}));
            comments_list.add(obj.obj());
            orig_realm->commit_transaction();
        }
        wait_for_upload(*orig_realm);

        std::cerr << "=== Comments order " << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
        std::cerr << "=== Logging out original session ===\n";
        orig_realm->sync_session()->log_out();
    }

    create_user_and_log_in(test_session.app());
    auto user_2 = test_session.app()->current_user();
    SyncTestFile user_2_realm(user_2, partition, schema);
    {
        std::cerr << "=== Opening second realm ===\n";
        auto realm = Realm::get_shared_realm(user_2_realm);
        wait_for_download(*realm);

        std::cerr << "=== Shuffling second list while connected ===\n";
        auto comments_list = get_comments_list(realm);
        REQUIRE(comments_list.size() == expected_comments);
        shuffle_list(realm, comments_list);
        wait_for_upload(*realm);

        std::cerr << "=== Second realm is shuffled ===\n";
        std::cerr << "=== Second realm comments order after shuffle "
                  << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
    }

    {
        std::cerr << "=== Shuffling original list while disconnected ===\n";
        auto comments_list = get_comments_list(orig_realm);
        shuffle_list(orig_realm, comments_list);
        std::cerr << "=== Original realm comments order after shuffling "
                  << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
    }

    {
        std::cerr << "=== Re-opening first realm/uploading/downloading ===\n";
        orig_realm->sync_session()->revive_if_needed();
        wait_for_upload(*orig_realm);
        wait_for_download(*orig_realm);

        orig_realm->refresh();
        auto orig_comments_list = get_comments_list(orig_realm);
        std::cerr << "=== Original realm comments order after upload/download "
                  << ::Catch::Detail::stringify(comments_to_pks(orig_comments_list)) << "===\n";

        std::cerr << "=== Re-opening second realm/uploading/downloading ===\n";
        auto second_realm = Realm::get_shared_realm(user_2_realm);
        wait_for_upload(*second_realm);
        wait_for_download(*second_realm);

        auto second_comments_list = get_comments_list(second_realm);
        std::cerr << "=== Second realm comments order after upload/download "
                  << ::Catch::Detail::stringify(comments_to_pks(second_comments_list)) << "===\n";
        std::cerr << "=== Checking order ===\n";
        REQUIRE(orig_comments_list.size() == second_comments_list.size());
        REQUIRE(comments_to_pks(orig_comments_list) == comments_to_pks(second_comments_list));
    }
}

[divan84]

Refreshing the realm unfortunately doesn't help. No restart of the app either - only when the app is deleted and reinstalled, is the data correct again.

I am sending you the Xcode logs containing the realm log messages from a session where a user just goes offline and moves the list items (while another user does the same online), and goes online again - the order of the lists is different for both users and will remain so.

If one of the users then changes the list order, on the others users device it changes the element in the same place (but it is and remains the wrong element)

Xcode-log.txt

[sync-by-unito]

➤ Jonathan Reams commented:

I think I may have reproduced this, but I'm still trying to figure out exactly what's going on in this repro case. I'll update here when I know more.

[ianpward]

I don't think Sync or Core has ever guaranteed the unsorted list order to be consistent across different devices. In order for list order to be consistent you must apply the same sort method - are you sorting the same way across both devices?

[tgoyne]

Lists do guarantee that they are in the order you set them in.

[divan84]

Is there any update regarding this problem?