When can I use a Reducer? What does it emit?

[spagdoon0411]

Please let me know if there's a resource I missed that might answer this!

To start, here's the Reducer documentation.

From the documentation, I wasn't able to get a complete understanding of when a Reducer can or can't be used prior to another node.

For instance, say I'm trying to split a slideshow into images, then give them all to Claude and ask for a summary of the presentation. The Reducer documentation states this node allows us to "group relevant documents into a single semantical envelope and perform combined operations on them." Broadly, that seems right for this use case.

But I'm also doubtful of whether that's possible: it doesn't look like there's a spot to pass a funclet to an AnthropicTextProcessor Builder that defines how to perform the reduction correctly for Claude's input. (In contrast, the podcast generator example features a withIntent call on an FfmpegProcessor Builder that takes a concat funclet; see line 231 here).

// ...

import { concat } from './funclets/ffmpeg';

// ...

const ffmpeg = new FfmpegProcessor.Builder()
      .withScope(this)
      .withIdentifier('FfmpegProcessor')
      .withCacheStorage(cache)
      .withVpc(vpc)
      .withSource(voiceReducer)
      .withIntent(concat)
      .build();

Information that might help is what CloudEvent(s) is/are emitted into SNS. (E.g., the following diagram notes "aggregated events"--but in what sense are they aggregated and how are the CloudEvents logically organized?)

[HQarroum]

Hey @spagdoon0411!

That's a good question. The TLDR; is that the reducer aggregates the CloudEvents passed by upstream middlewares and combines them into a JSON document (a JSON array, in fact) that contains the aggregated cloudevents. It does that aggregation based on the given reducer strategy (TimeWindowStrategy, StaticCounterStrategy, ConditionalStrategy). The output of a reducer is always application/cloudevents+json. So the output of a reducer would like like this :

[{
  "specversion": "1.0",
  "id": "1780d5de-fd6f-4530-98d7-82ebee85ea39",
  "type": "document-created",
  ...
}, {
  // Another cloudevent
}]

Now regarding your question of when a reducer can be used prior to another middleware is it depends on whether that middleware supports application/cloudevents+json as an input type. Today, only the following middlewares that support composite inputs are the following :

• Anthropic Text Processor
• Structured Entity Extractor
• FFMPEG Processor
• Zip Processor
• Tar Inflate Processor

Hope I answered your question.

[spagdoon0411]

Hi Halim! The doc page does list application/cloudevents+json as an output MIME type; I probably could have paid closer attention to that.

I also saw you were working on a guide for custom middleware modules--I think defining complex reducer behavior would be a great use case.

Thanks for the answer!