• Mongoid::Scroll
  • Compatibility
  • Demo
  • The Problem
  • Installation
  • Usage
    • Mongoid
    • Mongo-Ruby-Driver (Mongoid 5)
  • Indexes and Performance
  • Cursors
    • Standard Cursor
    • Base64 Encoded Cursor
  • Contributing
  • Copyright and License

Mongoid::Scroll

Mongoid extension that enables infinite scrolling for Mongoid::Criteria and Mongo::Collection::View.

Compatibility

This gem supports Mongoid 5, 6, 7 and 8.

Demo

Check out shows on artsy.net. Keep scrolling down.

There're also two code samples for Mongoid in examples. Run bundle exec ruby examples/mongoid_scroll_feed.rb.

The Problem

Traditional pagination does not work when data changes between paginated requests, which makes it unsuitable for infinite scroll behaviors.

• If a record is inserted before the current page limit, items will shift right, and the next page will include a duplicate.
• If a record is removed before the current page limit, items will shift left, and the next page will be missing a record.

The solution implemented by the scroll extension paginates data using a cursor, giving you the ability to restart pagination where you left it off. This is a non-trivial problem when combined with sorting over non-unique record fields, such as timestamps.

Installation

Add the gem to your Gemfile and run bundle install.

gem 'mongoid-scroll'

Usage

Mongoid

A sample model.

module Feed
  class Item
    include Mongoid::Document
    field :title, type: String
    field :position, type: Integer
    index({ position: 1, _id: 1 })
  end
end

Scroll by :position and save a cursor to the last item.

saved_cursor = nil
Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.

Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
  # each record, one-by-one
end

Mongo-Ruby-Driver (Mongoid 5)

Scroll a Mongo::Collection::View and save a cursor to the last item. You must also supply a field_type of the sort criteria.

saved_cursor = nil
client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Resume iterating using the previously saved cursor.

session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
  # each record, one-by-one
  saved_cursor = next_cursor
end

Indexes and Performance

A query without a cursor is identical to a query without a scroll.

# db.feed_items.find().sort({ position: 1 }).limit(7)
Feed::Item.desc(:position).limit(7).scroll

Subsequent queries use an $or to avoid skipping items with the same value as the one at the current cursor position.

# db.feed_items.find({ "$or" : [
#   { "position" : { "$gt" : 13 }},
#   { "position" : 13, "_id": { "$gt" : ObjectId("511d7c7c3b5552c92400000e") }}
# ]}).sort({ position: 1 }).limit(7)
Feed:Item.desc(:position).limit(7).scroll(cursor)

This means you need to hit an index on position and _id.

# db.feed_items.ensureIndex({ position: 1, _id: 1 })

module Feed
  class Item
    ...
    index({ position: 1, _id: 1 })
  end
end

Cursors

You can use Mongoid::Scroll::Cursor.from_record to generate a cursor. A cursor points at the last record of the previous iteration and unlike MongoDB cursors will not expire.

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"] })
# cursor or cursor.to_s can be returned to a client and passed into .scroll(cursor)

You can also a field_name and field_type instead of a Mongoid field.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })

When the include_current option is set to true, the cursor will include the record it points to:

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"], include_current: true })
Feed::Item.asc(:position).limit(1).scroll(cursor).first # record

If the field_name, field_type or direction options you specify when creating the cursor are different from the original criteria, a Mongoid::Scroll::Errors::MismatchedSortFieldsError will be raised.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })
Feed::Item.desc(:created_at).scroll(cursor) # Raises a Mongoid::Scroll::Errors::MismatchedSortFieldsError

Standard Cursor

The Mongoid::Scroll::Cursor encodes a value and a tiebreak ID separated by :, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options.

Base64 Encoded Cursor

The Mongoid::Scroll::Base64EncodedCursor can be used instead of Mongoid::Scroll::Cursor to generate a base64-encoded string (using RFC 4648) containing all the information needed to rebuild a cursor.

Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, next_cursor|
   # next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
end

Contributing

Fork the project. Make your feature addition or bug fix with tests. Send a pull request. Bonus points for topic branches.

Copyright and License

MIT License, see LICENSE for details.

(c) 2013-2023 Daniel Doubrovkine, based on code by Frank Macreery, Artsy Inc.