[BUG] Stop using detail APIs in Java code

[vyasr]

Describe the bug
There are a large number of libcudf tests that rely on detail APIs. There are parts of the JNI that use detail APIs. detail APIs are internal functionality that should not be called externally.

Expected behavior
We should rewrite the offending code to not use detail APIs. If this proves too difficult, that probably indicates that we have functionality hidden in detail namespaces that should be exposed publicly. I suspect that in at least a few cases we will need to add new public APIs to support users, but hopefully not too many of them.

UPDATE 6/28/2024
Upon further discussion, we have decided to continue allowing testing of detail APIs because in some contexts that can be useful for unit testing small, internal components of larger APIs. Therefore this issue is now primarily about avoiding usage of detail APIs in the JNI layer.

[vyasr]

I should clarify. It may be acceptable to write tests of detail APIs. Some detail APIs are important enough and used by multiple public APIs, so they may benefit from dedicated testing. However, I suspect that in many cases we are using detail APIs to construct tests for public APIs. One specific instance that I know occurs is the use of vector factories in the detail namespace. Those uses are suspect and should be analyzed. The JNI should certainly not need to rely on detail APIs.

[ttnghia]

Note that some/most of the detail APIs used in Java are due to not having their public version to call. In order to fulfill the guideline, we also need to implement the public interface for them.

[vyasr]

Yup I agree, I think there will be cases where we need new public APIs. That indicates a hole in libcudf's API though, not a reason to use the detail APIs externally.

[jlowe]

Related to the comment about needing public APIs, note there is a growing amount of Spark-specific JNI code leveraging libcudf at https://github.com/NVIDIA/spark-rapids-jni/tree/branch-22.12/src/main/cpp. There's a number of detail references in that code, including:

cudf::detail::bitmask_and
cudf::detail::gather
cudf::detail::make_counting_transform_iterator
cudf::detail::negative_index_policy
cudf::detail::sequence
cudf::detail::single_lane_block_sum_reduce
cudf::detail::size_in_bits
cudf::detail::valid_if
cudf::detail::warp_size
cudf::strings::detail::make_offsets_child_column
cudf::test::detail::column_wrapper
cudf::test::detail::make_null_mask
numeric::detail::abs

I haven't looked at these in detail (no pun intended). Some of these may be using the detail interface solely to specify a specific stream to use, and that will be addressed when libcudf exposes streams in the public APIs. Just wanted to point out another source base that can indicate what might need to be promoted to a public API.

[vyasr]

@jlowe thanks! That definitely looks like a mixed bag: bitmask_and is something that we should expose publicly (I know that Python could use it too), detail::gather is most likely just being used to pass a stream, make_counting_transform_iterator and warp_size are conveniences that the JNI probably shouldn't be relying on, etc. Once stream APIs exist we can probably trim out some of these and reassess the rest.

[GregoryKimball]

Update June 2024: We had a discussion about symbol visibility today and agreed that libcudf unit testing of detail APIs is useful. I propose that we update this issue to focus only on JNI uses of detail APIs, and leave libcudf testing uses of detail APIs as a separate topic.