Replies: 1 comment 1 reply
-
|
Hi @Yuukadesu , I had one thought regarding the tests. If I understand it correctly, the proposal currently tries to test both the Java client itself and the load-balancing behavior in the same E2E setup. Would it maybe make sense to split this into two separate tests? For example:
That way the client test stays relatively simple and failures might be easier to interpret. Also one small question about the broker: since NATS is the default broker since the last release, would it make sense to use NATS instead of Kafka in the test setup? These are just some thoughts I am happy to hear your opinion on this. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Proposal: Implementing Integration E2E Tests for Java Client
Motivation
To ensure the stability and reliability of the StreamPipes Java client library, this implementation adopts an end-to-end testing strategy aligned with the Go Client E2E. First, a test environment closely resembling production is set up (a load-balanced topology based on Docker Compose), including the Backend, multiple Extension service instances, Kafka, CouchDB, and other required dependencies, so that test results are authentic and reproducible. Second, test cases are designed to cover two main objectives: data path correctness and load balancing effectiveness. They verify not only the end-to-end data flow (Kafka topic-in → adapter/source → processor → sink → Kafka topic-out), but also assert under the load of 70 pipelines and 70 adapters that adapters and pipeline elements are each distributed across multiple extension instances, with each instance’s share roughly equal (within a configurable deviation, default 20% from the average). To improve test efficiency and consistency, automated tests are written with JUnit 5, and connection parameters are injected via custom startup scripts and system properties. Finally, E2E tests are integrated into GitHub Actions CI: on every PR, images are built, the load-balanced environment is started, and Java Client E2E is executed, enabling continuous monitoring and timely feedback on the Java client and load-balancing behavior, and delivering a reliable and efficient Java client library for StreamPipes users.
Goals
sensor_fault_flags == falseare passed through).Implementation
The implementation ensures that the Java E2E test flow is consistent with the Go E2E test flow. A Docker-based environment is used with custom startup scripts that accept host, port, and other configuration parameters. E2E tests are triggered automatically on each PR via a GitHub Actions workflow and a script chain: the workflow builds images, starts the load-balanced stack with
docker-compose-loadbalancer.yml, then runsstart-streampipes-client-e2e.sh -t java-client-e2e.sh, which performs login, API key retrieval, andmvn testwith the appropriate system properties; any failure fails the workflow so that code quality and load-balancing behavior are continuously monitored.Beta Was this translation helpful? Give feedback.
All reactions