Add dfs transformation function in XContentMapValues

[jmazanec15]

Description

In KNN, we are developing feature to remove vectors from source from documents before serialization. Originally, we were using the XContentMapValues.filter function to remove the vector fields from the source objects (ref). However, the problem with filter is that if a filter produces an empty object, then it will be removed from the array if its in an array. This alters the structure of the source map. Thus, putting the vectors back becomes very complex for nested cases (opensearch-project/k-NN#2583).

After some discussion, we are pivoting to an approach that instead of filtering the fields, we will mask the field with a smaller representation on write, and vice versa on read. I added PoC here: https://github.com/jmazanec15/k-NN-1/tree/mask-derived-poc. This change adds a transform method to XContentMapValues that performs depth first traversal on a map, potentially applying transformations to different values along the way. I could add in k-NN plugin, but thought it might be useful in future in core. The DFS nature works well with the architecture of nested documents.

Related Issues

Resolves opensearch-project/k-NN#2377

Check List

• Functionality includes testing.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 4617895: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 91.93548% with 5 lines in your changes missing coverage. Please review.

Project coverage is 72.37%. Comparing base (dcad6b8) to head (4e06dc5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...rch/common/xcontent/support/XContentMapValues.java	91.93%	0 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17612      +/-   ##
============================================
- Coverage     72.46%   72.37%   -0.10%     
+ Complexity    65757    65713      -44     
============================================
  Files          5311     5311              
  Lines        305001   305073      +72     
  Branches      44230    44243      +13     
============================================
- Hits         221022   220796     -226     
- Misses        65932    66116     +184     
- Partials      18047    18161     +114     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 6e6b7ae: SUCCESS

[github-actions]

❕ Gradle check result for 4e06dc5: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testZeroLagOnCreateIndex

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[jed326]

Thanks @jmazanec15 , had one suggestion for an additional test case but otherwise LGTM

[jmazanec15]

Thanks @jed326 - added the test

[jed326]

LGTM, thanks! @msfroh do you want to take another pass at this?

[github-actions]

❌ Gradle check result for f9b40ec: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f9b40ec: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f9b40ec: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jed326]

❌ Gradle check result for f9b40ec: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

I'm not sure what is actually failing here, @jmazanec15 maybe try rebasing your branch if the next retry fails again

[jmazanec15]

Sure @jed326 , just rebased

[github-actions]

❌ Gradle check result for f43e1d6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f43e1d6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?