[RFS] Add additional logging when bulk requests fail

[lewijacn]

Description

Our current pattern for handling failures with bulk requests doesn't print information regarding what caused the failure (for the default log level) until all retries are completed, which can cause users and our tests around a 10 minute delay to identify the cause. This change provides the bulk request response error message on failures to give quicker detection here

Previously:

2025-04-28 19:19:59,817 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [78172]: 2025-04-28 19:19:59,817 INFO o.o.m.b.c.DocumentReindexer [DocumentBatchReindexer-1] Batch Id:71d98779-cce8-47c8-8282-5e420a489000, 1 documents in current bulk request.
2025-04-28 19:19:59,827 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [78172]: 2025-04-28 19:19:59,826 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request on index 'geonames', 0 more documents have succeeded, 1 remain
2025-04-28 19:20:02,724 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [78172]: 2025-04-28 19:20:02,724 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request on index 'geonames', 0 more documents have succeeded, 1 remain
2025-04-28 19:20:05,144 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [78172]: 2025-04-28 19:20:05,143 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request on index 'geonames', 0 more documents have succeeded, 1 remain
2025-04-28 19:20:12,117 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [78172]: 2025-04-28 19:20:12,117 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request on index 'geonames', 0 more documents have succeeded, 1 remain

With change:

2025-04-28 19:10:55,140 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [76748]: 2025-04-28 19:10:55,140 INFO o.o.m.b.c.DocumentReindexer [DocumentBatchReindexer-1] Batch Id:d5aea48a-3eb7-40a4-a074-1fea5104eccf, 1 documents in current bulk request.
2025-04-28 19:10:55,150 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [76748]: 2025-04-28 19:10:55,150 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request attempt 1 on index 'geonames', 0 more documents have succeeded, 1 remain. The error response message was: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
2025-04-28 19:10:57,633 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [76748]: 2025-04-28 19:10:57,633 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request attempt 2 on index 'geonames', 0 more documents have succeeded, 1 remain. The error response message was: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
2025-04-28 19:11:00,714 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [76748]: 2025-04-28 19:11:00,714 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request attempt 3 on index 'geonames', 0 more documents have succeeded, 1 remain. The error response message was: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
2025-04-28 19:11:05,775 INFO o.o.m.b.SourceTestBase [Thread-24] from sub-process [76748]: 2025-04-28 19:11:05,774 WARN o.o.m.b.c.OpenSearchClient [reactor-http-kqueue-2] After bulk request attempt 4 on index 'geonames', 0 more documents have succeeded, 1 remain. The error response message was: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Issues Resolved

N/A

Testing

Local testing

Check List

• New functionality includes testing
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peternied]

Lets add test cases for this

[peternied]

Can we reuse CloseableLogSetup for this test case?

[lewijacn]

Oof this is exactly what I was looking for and didn't see, have adjusted to use now and removed my created class

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (86186dc) to head (2013659).
Report is 4 commits behind head on main.

Additional details and impacted files

@@     Coverage Diff      @@
##   main   #1484   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.