[improve][pip] PIP-390 Improve the reusability of Pulsar test code and best practice for unit and integration tests for Pulsar

pip/pip-390.md

@@ -0,0 +1,79 @@
+# PIP-390: Improve the reusability of Pulsar test code and best practices for unit and integration tests in Pulsar
+
+## Background
+
+- Pulsar integration tests are located in the `tests` module, which run on [testcontainers](https://github.com/testcontainers).
+- Pulsar unit tests are located in each module under the test scope, with many tests running on mocked (in-memory) Pulsar clusters. 
+- Currently, Pulsar tests run on the `testng` framework.
+
+## Motivation
+
+Apache Pulsar has several reusable test code, such as client test cases, container setup, and other helper functions in the `tests` module. However, since this module is only accessible within the "test" scope, it is difficult to reuse these utilities in other projects. 
+
+The Pulsar community has discussed the ideal practice for unit and integration tests (see discussion [here](https://lists.apache.org/thread/fn3rk1x7v9291qh3g6vf4jxhvq6zc4mm)). This PIP aims to document the results of the discussion for future reference.
+
+
+## Goals
+
+### In Scope
+
+1. Improve the reusability of Pulsar test code, especially for the `tests` module.
+2. Document best practices for unit and integration tests in Pulsar. 
+
+### Out of Scope
+
+- Propose the next generation of the test framework and improvement plan(This can be a separate pip). 
+
+## High-Level Design
+
+### 1. Improve the Reusability of Pulsar Test Code in the `tests` Module
+- Decouple client test logic from cluster setup and test runner code, allowing the test logic to be maintained independently of test environment and test runner framework changes. 
+
+## Detailed Design
+
+### 1. Improve the Reusability of Pulsar Test Code in the `tests` Module
+
+Refactor the `tests` module as follows:
+
+- `tests`: Contains the main integration test runners built under the "test" scope (e.g., test runner classes using TestNG, dependent on `pulsar-inttest-lib`, `pulsar-inttest-client`, and other modules).
+- `pulsar-inttest-lib`: Contains common libraries for integration tests (e.g., test utilities, test containers, and other reusable test setups).
+- `pulsar-inttest-client`: Contains common client-side integration test cases (should include only client-side test logic and assertions, depending on `PulsarClient` and `PulsarAdmin`). These test cases can be reused in `tests` and other modules.
+
+### 2. Document Best Practices for Unit and Integration Tests in Pulsar
+
+Based on the [community discussion](https://lists.apache.org/thread/fn3rk1x7v9291qh3g6vf4jxhvq6zc4mm), Pulsar contributors are expected to consider the following guidelines when adding unit and integration tests:
+
+- **Reuse Test Classes**: When possible, add new test cases as `@Test` annotations within existing test classes instead of creating new classes, as additional classes can increase test overhead.
+
+- **Unit Tests Per Class**: Contributors should add unit tests for each new class they introduce, focusing on the behavior of the class’s public methods. Deep state checks using reflection tools should be avoided, as they may break with future implementation changes. Reference: [Testing on the Toilet - Test Behavior, Not Implementation](https://testing.googleblog.com/2013/08/testing-on-toilet-test-behavior-not.html?m=1).  
+  If testing public methods in specific class states is necessary, a constructor with the `@VisibleForTesting` annotation can be added to allow injection of (mocked) member variables.
+
+- **Use In-Memory Test Cluster**: For tests requiring a Pulsar cluster, consider using an in-memory test cluster (e.g., by extending `MockedPulsarServiceBaseTest` or `PulsarTestContext`). This approach is also useful for "mocking" Pulsar behavior.
+
+- **Use Container Test Cluster for Integration Tests**: When the test can run solely with Pulsar client logic and no Pulsar behavior mocking is needed, consider adding it as an integration test using a container-based test cluster. However, container-based tests are heavier than in-memory tests, as they require a containerized cluster setup. Integration tests are typically reserved for core features and Pulsar sink/source/function IO tests.
+
+- **Add Shared Client Logic to `pulsar-inttest-client`**: Include/move common test client logic here if it can be reused. [AssertionJ](https://github.com/assertj/assertj) can be used for assertion statements, which is compatible with both Junit and TestNg frameworks.
+- **Add Shared Libraries to `pulsar-inttest-lib`**: Place reusable test setups or test utilities in this module.
+
+## Public-Facing Changes
+
+### Configuration
+
+- N/A
+
+### API Changes
+
+- N/A
+
+## Backward & Forward Compatibility
+
+These changes apply only to test classes, so they do not impact backward or forward compatibility.
+
+## Security Considerations
+
+- N/A
+
+## Links
+
+- [Initial mailing list discussion thread](https://lists.apache.org/thread/fn3rk1x7v9291qh3g6vf4jxhvq6zc4mm)
+- Mailing list PIP voting thread (to be added)