[Pull-based Ingestion] Fix poller bug and remove persisted pointers

[varunbharadwaj]

Description

This PR includes the following bug fixes.

1. Removes persisted pointer concept to fix correctness corner cases related to user initiated consumer rewind (as described in the issue) when versioning is not used. Pull-based ingestion without versioning will provide atleast once processing guarantees. Versioning must be used to ensure consistent view of documents on rewind/replay, if eventually consistent view is not acceptable.
2. Setauto.offset.reset Kafka setting to none by default to throw errors on out-of-bounds offsets. This could be possible if the ingestion was paused and exceeds Kafka retention period.
3. Update the Kafka polling logic to use batch size setting based on message count, instead of offset windows. This prevents ingestion getting stuck, if user configures auto.offset.reset to earliest/latest and offset is out-of-bounds.

Related Issues

Resolves #19591

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 4eb43e8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for cb49bb0: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.12%. Comparing base (2695c93) to head (cb49bb0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/plugin/kafka/KafkaPartitionConsumer.java	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19607      +/-   ##
============================================
+ Coverage     73.06%   73.12%   +0.05%     
- Complexity    70630    70672      +42     
============================================
  Files          5723     5723              
  Lines        323513   323484      -29     
  Branches      46852    46848       -4     
============================================
+ Hits         236384   236537     +153     
+ Misses        68013    67840     -173     
+ Partials      19116    19107       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.