Add Map ItemBatcher/ItemSelector #294

agrare · 2024-11-05T20:06:04Z

The ItemBatcher provides a way to group large array inputs into larger batches for better performance

You can provide a MaxItemsPerBatch or MaxItemsPerBatchPath which limits the number of items in a batch, as well as a MaxInputBytesPerBatch and MaxInputBytesPerBatchPath which limits the size of the input in bytes.

The output will be a set of {"Items": []} payloads, one for each batch.

Batching Items

A Map State MAY have an "ItemBatcher" field, whose value MUST be a JSON object and is called the ItemBatcher Configuration. The ItemBatcher Configuration causes the interpreter to batch selected items into sub-arrays before passing them to each invocation. The interpreter will limit each sub-array to the maximum number of items specified by the "MaxItemsPerBatch" field, and to the maximum size in bytes specified by the "MaxInputBytesPerBatch" field.

The ItemBatcher Configuration MAY have a "BatchInput" field, whose value MUST be a Payload Template. An ItemBatcher Configuration MAY have a "MaxItemsPerBatch" field, whose value MUST be a positive integer. An ItemBatcher Configuration MAY have a "MaxInputBytesPerBatch" field, whose value MUST be a positive integer.

The default of "ItemBatcher" is the selected item. Put another way, the interpreter will not batch items if no "ItemBatcher" field is provided.

Both the "MaxItemsPerBatch" and "MaxInputBytesPerBatch" can be provided indirectly. A Map State may have "MaxItemsPerBatchPath" and "MaxInputBytesPerBatchPath" fields which MUST be Reference Paths which, when resolved, MUST select fields whose values are positive integers. A Map State MUST NOT include both "MaxItemsPerBatch" and "MaxItemsPerBatchPath" or both "MaxInputBytesPerBatch" and "MaxInputBytesPerBatchPath".

An ItemBatcher Configuration MUST contain at least one of "MaxItemsPerBatch", "MaxItemsPerBatchPath", "MaxInputBytesPerBatch", or "MaxInputBytesPerBatchPath".

https://docs.aws.amazon.com/step-functions/latest/dg/input-output-itembatcher.html
#241

lib/floe/workflow/item_batcher.rb

kbrock · 2025-01-17T13:54:39Z

lib/floe/workflow/item_batcher.rb

+        input.each_slice(max_items(context, state_input)).map do |batch|
+          output.merge("Items" => batch)
+        end


future: would be nice to not resolve all input at the start.

kbrock · 2025-01-17T13:56:27Z

lib/floe/workflow/item_batcher.rb

+      end
+
+      def validate!
+        if [max_items_per_batch, max_items_per_batch_path].all?(&:nil?)


optimization (read: feel free to ignore)
Count these and compare to 1?

You mean like [max_items_per_batch, max_items_per_batch_path].compact.count.zero?

yea

unless [max_items_per_batch, max_items_per_batch_path].compact.count == 1 parser_error!("must have one of \"MaxItemsPerBatch\", \"MaxItemsPerBatchPath\"") end

kbrock · 2025-01-17T13:57:44Z

lib/floe/workflow/item_batcher.rb

+        invalid_field_error!("MaxItemsPerBatch", max_items_per_batch, "must be a positive integer")            if max_items_per_batch && max_items_per_batch <= 0
+        invalid_field_error!("MaxInputBytesPerBatch", max_input_bytes_per_batch, "must be a positive integer") if max_input_bytes_per_batch && max_input_bytes_per_batch <= 0


not sure if we care, but these will throw a ruby runtime error if these are not an integer. I really want to avoid that. Maybe do a to_i?

I can add a kind_of?(Integer) check, I don't want to "blindly" to_i it because that would hide someone passing in a float which is kind of unexpected.

kbrock

This looks good. I did not run.

I had put in a bunch of thoughts, but I do want some form of protection around invalid references to avoid ruby runtime errors. to_i is just one suggestion for a fix.

kbrock · 2025-01-29T15:32:50Z

lib/floe/workflow/item_batcher.rb

+        return if max_items_per_batch_path.nil?
+
+        result = max_items_per_batch_path.value(context, state_input)
+        raise runtime_field_error!("MaxItemsPerBatchPath", result, "must be a positive integer") if result <= 0


please use a to_i

kbrock · 2025-01-29T15:33:24Z

lib/floe/workflow/item_batcher.rb

+        invalid_field_error!("MaxItemsPerBatch", max_items_per_batch, "must be a positive integer")            if max_items_per_batch && max_items_per_batch <= 0
+        invalid_field_error!("MaxInputBytesPerBatch", max_input_bytes_per_batch, "must be a positive integer") if max_input_bytes_per_batch && max_input_bytes_per_batch <= 0


to_i for both of these (to avoid the runtime error)

kbrock · 2025-01-29T15:36:06Z

spec/workflow/item_batcher_spec.rb

+        it "raises an exception" do
+          expect { subject.value(context, input, state_input) }
+            .to raise_error(Floe::ExecutionError, "Map.ItemBatcher field \"MaxItemsPerBatchPath\" value \"0\" must be a positive integer")
+        end


Not sure if we want a test with invalid values. (are we doing tests for these?)

e.g.:

let(:state_input) { {"batchSize" => "a", "items" => input} }

agrare · 2025-01-29T16:48:58Z

Added Integer type checks and tests for floats and strings

agrare requested a review from Fryguy as a code owner November 5, 2024 20:06

agrare force-pushed the map_state_item_batcher branch 4 times, most recently from 43adee0 to a521ed3 Compare November 6, 2024 15:51

kbrock reviewed Nov 6, 2024

View reviewed changes

lib/floe/workflow/item_batcher.rb Outdated Show resolved Hide resolved

agrare changed the title ~~[WIP] Add Map ItemBatcher~~ Add Map ItemBatcher Nov 6, 2024

agrare force-pushed the map_state_item_batcher branch from 4ab7b13 to 494553e Compare November 6, 2024 18:19

agrare changed the title ~~Add Map ItemBatcher~~ Add Map ItemBatcher/ItemSelector Nov 8, 2024

agrare mentioned this pull request Nov 8, 2024

Support Map #241

Open

7 tasks

agrare added the enhancement New feature or request label Nov 8, 2024

agrare force-pushed the map_state_item_batcher branch from 209a1da to 2cd4ab2 Compare November 12, 2024 14:35

agrare assigned kbrock Nov 21, 2024

kbrock reviewed Jan 17, 2025

View reviewed changes

kbrock approved these changes Jan 29, 2025

View reviewed changes

agrare added 11 commits January 29, 2025 11:29

Add Map ItemBatcher

3fe9069

ItemBatcher should return {"Items": []}

fb6e75a

Support ItemBatcher Paths

f903a25

Add BatchInput support

a8c6620

Check that MaxItemsPerBatch/MaxInputBytesPerBatch are positive

9ac64ee

Raise execution error if path batch is invalid

3cbf010

Extract max_items logic to a method

2168a7a

Move ItemBatcher validations to method

289ac33

Only implement MaxItemsPerBatch

4fea6cc

Add ItemSelector support

596436d

Add Integer type checking to ItemBatcher

eb89144

agrare force-pushed the map_state_item_batcher branch from 2cd4ab2 to eb89144 Compare January 29, 2025 17:20

kbrock approved these changes Jan 30, 2025

View reviewed changes

kbrock merged commit 814c859 into ManageIQ:master Jan 30, 2025
4 of 5 checks passed

agrare deleted the map_state_item_batcher branch January 30, 2025 00:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Map ItemBatcher/ItemSelector #294

Add Map ItemBatcher/ItemSelector #294

agrare commented Nov 5, 2024 •

edited

Loading

kbrock Jan 17, 2025

kbrock Jan 17, 2025

agrare Jan 29, 2025

kbrock Jan 30, 2025

kbrock Jan 17, 2025

agrare Jan 29, 2025

kbrock left a comment

kbrock Jan 29, 2025

kbrock Jan 29, 2025

kbrock Jan 29, 2025

agrare commented Jan 29, 2025

		invalid_field_error!("MaxItemsPerBatch", max_items_per_batch, "must be a positive integer") if max_items_per_batch && max_items_per_batch <= 0
		invalid_field_error!("MaxInputBytesPerBatch", max_input_bytes_per_batch, "must be a positive integer") if max_input_bytes_per_batch && max_input_bytes_per_batch <= 0

Add Map ItemBatcher/ItemSelector #294

Add Map ItemBatcher/ItemSelector #294

Conversation

agrare commented Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kbrock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agrare commented Jan 29, 2025

agrare commented Nov 5, 2024 •

edited

Loading