pulsar_function: window functions cannot be created — windowConfig is not exposed

[david-streamlio]

Community Note

• Please vote on this issue by adding a
  👍 reaction to the original
  issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra
  noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

pulsar_function cannot create a window function. WindowConfig is the one substantial Pulsar Functions capability with no representation in the resource at all — not a tuning knob, but a different processing model: a window function receives a collection of messages over a tumbling or sliding window rather than one message at a time.

utils.FunctionConfig has a WindowConfig *WindowConfig field and the admin client models the type fully:

type WindowConfig struct {
	WindowLengthCount             *int
	WindowLengthDurationMs        *int64
	SlidingIntervalCount          *int
	SlidingIntervalDurationMs     *int64
	LateDataTopic                 *string
	MaxLagMs                      *int64
	WatermarkEmitIntervalMs       *int64
	TimestampExtractorClassName   *string
	ActualWindowFunctionClassName *string
	ProcessingGuarantees          *string
}

marshalFunctionConfig never populates it (grep -c WindowConfig pulsar/resource_pulsar_function.go returns 0), so it is always sent nil. A user who needs windowing has to create the function with pulsar-admin and leave it outside Terraform — which for anyone managing Pulsar as code means that function is unmanaged, and a subsequent terraform apply cannot see or preserve its window settings.

I found this while auditing every utils.FunctionConfig field against what the resource sets: 45 fields, 35 populated. Of the 10 that are not, two are addressed by open PRs (#219 for InputSpecs, #223 for ProducerConfig), three look derived rather than user-facing (FQFN, Runtime, FunctionType), and four are minor tuning knobs (MaxPendingAsyncRequests, ExposePulsarAdminClientEnabled, RuntimeFlags, the top-level BatchBuilder). WindowConfig is the only one that gates a capability rather than a setting, which is why it is filed on its own.

No upstream dependency. The type is already in the pinned admin client, so this is purely provider work.

Scope: window functions are Java-only

Worth stating before anyone implements this, because it narrows the value and suggests a validation the provider should carry.

Windowing is not a per-runtime capability that Python and Go merely lack — it is a Java-runtime implementation. In FunctionConfigUtils.convert(), a non-null windowConfig causes the broker to stash the config into userConfig under WINDOW_CONFIG_KEY, move the user's class into actualWindowFunctionClassName, and then replace className with org.apache.pulsar.functions.windowing.WindowFunctionExecutor — a Java class no other runtime can load.

Accordingly the broker refuses it outright for the other two:

// FunctionConfigUtils.doPythonChecks()
if (functionConfig.getWindowConfig() != null) {
    throw new IllegalArgumentException("There is currently no support windowing in python");
}

// FunctionConfigUtils.doGolangChecks()
if (functionConfig.getWindowConfig() != null) {
    throw new IllegalArgumentException("Windowing is not supported in Go function yet");
}

Two consequences:

1. This issue only benefits Java functions. Still worth doing — a jar-based function is a first-class case for the resource — but it is not the general capability the original description implied.
2. The provider should reject the combination at plan time. A configuration setting window_config alongside py or go is guaranteed to fail at apply with the broker's exception. Catching it in CustomizeDiff turns that into a plan-time error naming the actual constraint, which is the same treatment input_specs gets in feat(function): expose input_specs for per-topic consumer config #219 for the schema/serde exclusivity the broker enforces.

Note also that the broker forces processingGuarantees to MANUAL on the function and moves the configured value onto the window config, and rejects EFFECTIVELY_ONCE and MANUAL outright when windowing is set. That interacts with the existing top-level processing_guarantees attribute and is the substance of point 4 below.

New or Affected Resource(s)

• pulsar_function

Potential Terraform Configuration

A window_config block, following the sink_config / source_config precedent already in the resource:

resource "pulsar_function" "windowed" {
  provider = pulsar

  tenant    = "public"
  namespace = "default"
  name      = "windowed"

  jar       = "function://public/default/api-examples@v1"
  classname = "org.apache.pulsar.functions.api.examples.WindowDurationFunction"

  inputs = ["persistent://public/default/input"]
  output = "persistent://public/default/output"

  window_config {
    window_length_duration_ms    = 60000
    sliding_interval_duration_ms = 30000
    late_data_topic              = "persistent://public/default/late"
    max_lag_ms                   = 5000
    watermark_emit_interval_ms   = 1000
    timestamp_extractor_classname = "com.acme.MyTimestampExtractor"
  }
}

Points worth settling in review:

1. Count versus duration are mutually exclusive per axis. windowLengthCount and windowLengthDurationMs cannot both be set, and neither can the two sliding-interval fields. Pulsar validates this server-side; catching it at plan time would give a better error, in the same way input_specs validates schema/serde exclusivity in feat(function): expose input_specs for per-topic consumer config #219.
2. Every field is a pointer in the admin model, so unset and zero are distinguishable. The schema should preserve that rather than collapsing unset to zero — the same trap that made receiver_queue_size = 0 unrepresentable until fix(pulsaradmin): preserve explicit zero receiver queue size apache/pulsar-client-go#1529.
3. actualWindowFunctionClassName is set by the worker, not the user — it records the user's class while className becomes the framework's window wrapper. It probably wants to be Computed rather than configurable, but that is worth confirming against the broker's behaviour rather than assumed.
4. ProcessingGuarantees appears both here and at the top level. Which wins, and whether the block's copy should be exposed at all, needs deciding.

References

• pulsar_function: inputSpecs / receiverQueueSize not exposed, making per-topic consumer config unreachable from IaC #217 / feat(function): expose input_specs for per-topic consumer config #219 — input_specs, the same class of gap on the consumer side
• pulsar_function: producerConfig not exposed (compression_type, batch_builder, max_pending_messages), unlike pulsar_source #220 / feat(function): expose producer configuration on pulsar_function #223 — producerConfig, the same class of gap on the producer side