We evaluate the accuracy of Roseau using multiple API evolution datasets from various sources.
Each dataset consists of a set of cases. Each case includes: a baseline API (v1), an updated version of that API (v2), and a main() method that uses symbols from the baseline API (client).
Each case is validated by systematically compiling the baseline API, the updated API, and the client against the baseline API.
The client is then recompiled and relinked against the updated API. If the compiler reports an error, the case is marked as source-incompatible. If the linker reports an error, the case is marked as binary-incompatible.
Each tool is given the two API versions and must identify source- or binary-breaking changes. Tool results are compared against the ground truth to compute accuracy metrics. The tools do not have access to the clients.
- Jezek (310 cases): presented in API Evolution and Compatibility: A Data Corpus and Tool Evaluation by Jezek and Dietrich. We manually fixed some buggy cases and significantly strengthened the clients to address false negatives.
- Roseau (422 cases): the cases are automatically extracted from Roseau's test suite
- The benchmark uses the Java 25 compiler and linker (
OpenJDK Runtime Environment (build 25.0.3)). - The benchmark strictly focuses on source- and binary-breaking changes; behavioral/semantic breaking changes are not considered.
- When client code breaks, it is guaranteed that the case is indeed breaking, as stated by the compiler and/or linker. When client code doesn't break, however, it is possible that this particular client doesn't break, but that a different one using the baseline API differently would. We do our best to write exhaustive clients that fully exercise the baseline API in all possible ways.
- The baseline API and client code are located in different Java packages. Therefore, the benchmark assumes that package-private symbols are not part of the API.
- The benchmark evaluates whether the tools identify some breaking changes with the right compatibility level (source or binary). However, it does not evaluate whether the breaking change kind reported by the tools (e.g.,
CLASS_NOW_FINAL) indeed corresponds to the case.
| Dataset | Category | Metric | Roseau | japicmp | Revapi |
|---|---|---|---|---|---|
| Jezek | Breaking | Precision | 0.98 | 0.89 | 0.82 |
| Recall | 1.00 | 0.83 | 0.97 | ||
| F1 | 0.99 | 0.86 | 0.89 | ||
| Source | Precision | 0.90 | 0.78 | 0.74 | |
| Recall | 1.00 | 0.82 | 0.96 | ||
| F1 | 0.95 | 0.80 | 0.84 | ||
| Binary | Precision | 0.95 | 0.91 | 0.92 | |
| Recall | 1.00 | 1.00 | 0.97 | ||
| F1 | 0.98 | 0.95 | 0.94 | ||
| Roseau | Breaking | Precision | 0.98 | 0.70 | 0.75 |
| Recall | 0.99 | 0.85 | 0.91 | ||
| F1 | 0.98 | 0.76 | 0.82 | ||
| Source | Precision | 0.98 | 0.65 | 0.71 | |
| Recall | 0.99 | 0.84 | 0.88 | ||
| F1 | 0.98 | 0.73 | 0.79 | ||
| Binary | Precision | 0.84 | 0.76 | 0.70 | |
| Recall | 1.00 | 0.98 | 0.95 | ||
| F1 | 0.91 | 0.85 | 0.81 |