Skip to content

KAFKA-16505: Add source raw bytes in processorContex #17960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

Dabz
Copy link

@Dabz Dabz commented Nov 27, 2024

Pull request to implement KIP-1034, it adds the records source byte in the processor context

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch from 7cd809b to 33a294b Compare January 28, 2025 18:44
@loicgreffier
Copy link
Contributor

@cadonna @Dabz Just linking the following PR #16684 that has been reverted.

We faced some issues with store caches. I suppose these issues are still relevant in this PR. Going to check and keep this feed updated.

@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch 2 times, most recently from 132aaab to 82d61c7 Compare January 29, 2025 10:37
@loicgreffier
Copy link
Contributor

loicgreffier commented Jan 29, 2025

From my tests, the following topology:

builder
  .stream()
  .groupByKey()
  .windowedBy(...) // Does not really matter. NPE thrown windowing or not
  .aggregate(...)
  .mapValues(value -> throw new RuntimeException(...))

Is not throwing a NPE anymore (compared to the previous PR #16684), since we're using sourceRawKey and sourceRawValue in ProcessorNode instead of rawRecord().key and rawRecord().value.

However, sourceRawKey and sourceRawValue are lost and not forwarded to the ProcessingExceptionHandler. Opening a new PR to address this case: #18739.

@loicgreffier loicgreffier force-pushed the KAFKA-16505-SourceByte-In-ProcessorContext branch from 9769124 to 207c365 Compare February 2, 2025 15:09
Copy link

github-actions bot commented May 4, 2025

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions bot added the stale Stale PRs label May 4, 2025
@cadonna cadonna closed this May 28, 2025
@cadonna
Copy link
Member

cadonna commented May 28, 2025

Superseded by #18739

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants