Gives the previous cursor in the scroll block (#38)

* Gives the previous cursor in the scroll block

* apply changes for mongo

* encrypt with the previous option

* take values before the first page

* keep the ordering when fetching previous records

* keep the ordering when fetching previous records

* add entry to the change log

* Fix with Ruby 2.6

* Update the README and CHANGELOG

* minor changes

* change to use type

* change to use an iterator object

* refactor

* Fix typos

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

* throw an error when type is unsupported

---------

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

Author: GCorbel

CHANGELOG.md

@@ -1,5 +1,6 @@
-### 1.0.2 (Next)
+### 2.0.0 (Next)
 
+* [#38](https://github.com/mongoid/mongoid-scroll/pull/38): Allow to reverse the scroll - [@GCorbel](https://github.com/GCorbel).
 * Your contribution here.
 
 ### 1.0.1 (2023/03/15)

README.md

@@ -69,27 +69,38 @@ end
 Scroll by `:position` and save a cursor to the last item.
 
 ```ruby
-saved_cursor = nil
-Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
+saved_iterator = nil
+
+Feed::Item.desc(:position).limit(5).scroll do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
-Resume iterating using the previously saved cursor.
+Resume iterating using saved cursor and save the cursor to go backward.
+
+```ruby
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
+  # each record, one-by-one
+  saved_iterator = iterator
+end
+```
+
+Loop over the first records again.
 
 ```ruby
-Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.previous_cursor) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
 The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.
 
 ```ruby
-Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
   # each record, one-by-one
+  saved_iterator = iterator
 end
 ```
 
@@ -98,19 +109,19 @@ end
 Scroll a `Mongo::Collection::View` and save a cursor to the last item. You must also supply a `field_type` of the sort criteria.
 
 ```ruby
-saved_cursor = nil
-client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
+saved_iterator = nil
+client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
 Resume iterating using the previously saved cursor.
 
 ```ruby
-session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
+session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_iterator.next_cursor, { field_type: DateTime }) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
@@ -179,15 +190,15 @@ Feed::Item.desc(:created_at).scroll(cursor) # Raises a Mongoid::Scroll::Errors::
 
 ### Standard Cursor
 
-The `Mongoid::Scroll::Cursor` encodes a value and a tiebreak ID separated by `:`, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options. 
+The `Mongoid::Scroll::Cursor` encodes a value and a tiebreak ID separated by `:`, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options.
 
 ### Base64 Encoded Cursor
 
 The `Mongoid::Scroll::Base64EncodedCursor` can be used instead of `Mongoid::Scroll::Cursor` to generate a base64-encoded string (using RFC 4648) containing all the information needed to rebuild a cursor.
 
 ```ruby
-Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, next_cursor|
-   # next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
+Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, iterator|
+   # iterator.next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
 end
 ```
 

UPGRADING.md

@@ -1,5 +1,25 @@
 # Upgrading
 
+## Upgrading to >= 2.0.0
+
+The second argument yielded in the block in `Mongoid::Criteria::Scrollable#scroll` and `Mongo::Scrollable#scroll` has changed from a cursor to an instance of `Mongoid::Criteria::Scrollable` which provides `next_cursor` and `previous_cursor`. The `next_cursor` method returns the same cursor as in versions prior to 2.0.0.
+
+For example, this code:
+
+```ruby
+Feed::Item.asc(field_name).limit(2).scroll(cursor) do |_, next_cursor|
+  cursor = next_cursor
+end
+```
+
+Should be updated to:
+
+```
+Feed::Item.asc(field_name).limit(2).scroll(cursor) do |_, iterator|
+  cursor = iterator.next_cursor
+end
+```
+
 ## Upgrading to >= 1.0.0
 
 ### Mismatched Sort Fields
@@ -9,6 +29,6 @@ Both `Mongoid::Criteria::Scrollable#scroll` and `Mongo::Scrollable` now raise a
 For example, the following code will now raise a `MismatchedSortFieldsError` because we set a different field name (`position`) from the `created_at` field used to sort in `scroll`.
 
 ```ruby
-cursor.field_name = "position" 
+cursor.field_name = "position"
 Feed::Item.desc(:created_at).scroll(cursor)
 ```

lib/config/locales/en.yml

@@ -27,4 +27,7 @@ en:
             message: "Unsupported field type."
             summary: "The type of the field '%{field}' is not supported: %{type}."
             resolution: "Please open a feature request in https://github.com/mongoid/mongoid-scroll."
-
+          unsupported_type:
+            message: "Unsupported type."
+            summary: "The type supplied in the cursor is not supported: %{type}."
+            resolution: "The cursor type can be either ':previous' or ':next'."

lib/mongo/scrollable.rb

@@ -16,21 +16,41 @@ def scroll(cursor_or_type = nil, options = nil, &_block)
       cursor_options = { field_name: scroll_field, direction: scroll_direction }.merge(options)
       cursor = cursor && cursor.is_a?(cursor_type) ? cursor : cursor_type.new(cursor, cursor_options)
       raise_mismatched_sort_fields_error!(cursor, cursor_options) if different_sort_fields?(cursor, cursor_options)
-      # make a view
-      view = Mongo::Collection::View.new(
-        view.collection,
-        view.selector.merge(cursor.criteria),
-        sort: (view.sort || {}).merge(_id: scroll_direction),
-        skip: skip,
-        limit: limit
-      )
+
+      records = nil
+      if cursor.type == :previous
+        # scroll backwards by reversing the sort order, limit and then reverse again
+        pipeline = [
+          { '$match' => view.selector.merge(cursor.criteria) },
+          { '$sort' => { scroll_field => -scroll_direction } },
+          { '$limit' => limit },
+          { '$sort' => { scroll_field => scroll_direction } }
+        ]
+        aggregation_options = view.options.except(:sort)
+        records = view.aggregate(pipeline, aggregation_options)
+      else
+        # make a view
+        records = Mongo::Collection::View.new(
+          view.collection,
+          view.selector.merge(cursor.criteria),
+          sort: (view.sort || {}).merge(_id: scroll_direction),
+          skip: skip,
+          limit: limit
+        )
+      end
       # scroll
       if block_given?
-        view.each do |record|
-          yield record, cursor_type.from_record(record, cursor_options)
+        previous_cursor = nil
+        records.each do |record|
+          previous_cursor ||= cursor_type.from_record(record, cursor_options.merge(type: :previous))
+          iterator = Mongoid::Criteria::Scrollable::Iterator.new(
+            previous_cursor: previous_cursor,
+            next_cursor: cursor_type.from_record(record, cursor_options)
+          )
+          yield record, iterator
         end
       else
-        view
+        records
       end
     end
   end

lib/mongoid-scroll.rb

@@ -11,5 +11,6 @@
 require 'mongoid/scroll/base64_encoded_cursor'
 require 'mongoid/criteria/scrollable/fields'
 require 'mongoid/criteria/scrollable/cursors'
+require 'mongoid/criteria/scrollable/iterator'
 require 'mongo/scrollable' if Object.const_defined?(:Mongo)
 require 'mongoid/criteria/scrollable'

lib/mongoid/criteria/scrollable.rb

@@ -12,13 +12,19 @@ def scroll(cursor_or_type = nil, &_block)
         cursor_options = build_cursor_options(criteria)
         cursor = cursor.is_a?(cursor_type) ? cursor : new_cursor(cursor_type, cursor, cursor_options)
         raise_mismatched_sort_fields_error!(cursor, cursor_options) if different_sort_fields?(cursor, cursor_options)
-        cursor_criteria = build_cursor_criteria(criteria, cursor)
+        records = find_records(criteria, cursor)
         if block_given?
-          cursor_criteria.order_by(_id: scroll_direction(criteria)).each do |record|
-            yield record, cursor_from_record(cursor_type, record, cursor_options)
+          previous_cursor = nil
+          records.each do |record|
+            previous_cursor ||= cursor_from_record(cursor_type, record, cursor_options.merge(type: :previous))
+            iterator = Mongoid::Criteria::Scrollable::Iterator.new(
+              previous_cursor: previous_cursor,
+              next_cursor: cursor_from_record(cursor_type, record, cursor_options)
+            )
+            yield record, iterator
           end
         else
-          cursor_criteria
+          records
         end
       end
 
@@ -60,10 +66,21 @@ def new_cursor(cursor_type, cursor, cursor_options)
         cursor_type.new(cursor, cursor_options)
       end
 
-      def build_cursor_criteria(criteria, cursor)
+      def find_records(criteria, cursor)
         cursor_criteria = criteria.dup
         cursor_criteria.selector = { '$and' => [criteria.selector, cursor.criteria] }
-        cursor_criteria
+        if cursor.type == :previous
+          pipeline = [
+            { '$match' => cursor_criteria.selector },
+            { '$sort' => { cursor.field_name => -cursor.direction } },
+            { '$limit' => criteria.options[:limit] },
+            { '$sort' => { cursor.field_name => cursor.direction } }
+          ]
+          aggregation = cursor_criteria.view.aggregate(pipeline)
+          aggregation.map { |record| Mongoid::Factory.from_db(cursor_criteria.klass, record) }
+        else
+          cursor_criteria.order_by(_id: scroll_direction(criteria))
+        end
       end
 
       def cursor_from_record(cursor_type, record, cursor_options)

lib/mongoid/criteria/scrollable/iterator.rb

@@ -0,0 +1,14 @@
+module Mongoid
+  class Criteria
+    module Scrollable
+      class Iterator
+        attr_accessor :previous_cursor, :next_cursor
+
+        def initialize(previous_cursor:, next_cursor:)
+          @previous_cursor = previous_cursor
+          @next_cursor = next_cursor
+        end
+      end
+    end
+  end
+end

lib/mongoid/scroll/base64_encoded_cursor.rb

@@ -18,7 +18,8 @@ def initialize(value, options = {})
             field_name: parsed['field_name'],
             direction: parsed['direction'],
             include_current: parsed['include_current'],
-            tiebreak_id: parsed['tiebreak_id'] && !parsed['tiebreak_id'].empty? ? BSON::ObjectId.from_string(parsed['tiebreak_id']) : nil
+            tiebreak_id: parsed['tiebreak_id'] && !parsed['tiebreak_id'].empty? ? BSON::ObjectId.from_string(parsed['tiebreak_id']) : nil,
+            type: parsed['type'].try(:to_sym)
           }
         else
           super nil, options
@@ -32,7 +33,8 @@ def to_s
           field_name: field_name,
           direction: direction,
           include_current: include_current,
-          tiebreak_id: tiebreak_id && tiebreak_id.to_s
+          tiebreak_id: tiebreak_id && tiebreak_id.to_s,
+          type: type
         }.to_json)
       end
     end

lib/mongoid/scroll/base_cursor.rb

@@ -1,7 +1,7 @@
 module Mongoid
   module Scroll
     class BaseCursor
-      attr_accessor :value, :tiebreak_id, :field_type, :field_name, :direction, :include_current
+      attr_accessor :value, :tiebreak_id, :field_type, :field_name, :direction, :include_current, :type
 
       def initialize(value, options = {})
         @value = value
@@ -10,6 +10,9 @@ def initialize(value, options = {})
         @field_name = options[:field_name]
         @direction = options[:direction] || 1
         @include_current = options[:include_current] || false
+        @type = options[:type] || :next
+
+        raise Mongoid::Scroll::Errors::UnsupportedTypeError.new(type: @type) if ![:previous, :next].include?(@type)
       end
 
       def criteria
@@ -86,20 +89,23 @@ def extract_field_options(options)
             field_type: field_type.to_s,
             field_name: field_name.to_s,
             direction: options[:direction] || 1,
-            include_current: options[:include_current] || false
+            include_current: options[:include_current] || false,
+            type: options[:type].try(:to_sym) || :next
           }
         elsif options && (field = options[:field])
           {
             field_type: field.type.to_s,
             field_name: field.name.to_s,
             direction: options[:direction] || 1,
-            include_current: options[:include_current] || false
+            include_current: options[:include_current] || false,
+            type: options[:type].try(:to_sym) || :next
           }
         end
       end
 
       def compare_direction
-        direction == 1 ? '$gt' : '$lt'
+        dir = type == :previous ? -direction : direction
+        dir == 1 ? '$gt' : '$lt'
       end
 
       def tiebreak_compare_direction