Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix exploration tests #7719

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix exploration tests #7719

wants to merge 1 commit into from

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Oct 3, 2024

What Does This Do

update the registry for pushing the image
install jdk 21
use the jsoup library and patch it for running test with instrument-the-world mode
add include file to specify which packages/classes we want to include into the instrumentation instead of instrumenting everything (like build tools)
do not collect static inherited fields to avoid duplicate class definition for enums.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@jpbempel jpbempel added comp: testing Testing tag: no release notes Changes to exclude from release notes comp: debugger Dynamic Instrumentation labels Oct 3, 2024
@jpbempel jpbempel requested review from a team as code owners October 3, 2024 12:38
@jpbempel jpbempel requested review from ojung and manuel-alvarez-alvarez and removed request for a team October 3, 2024 12:38
@jpbempel jpbempel force-pushed the jpbempel/fix-exploration-tests2 branch 2 times, most recently from 7ae25da to 38a78ce Compare October 3, 2024 12:56
@pr-commenter
Copy link

pr-commenter bot commented Oct 3, 2024 •
edited
Loading

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1727971296 1727971671
end_time 2024-10-03T16:02:51 2024-10-03T16:09:06
git_branch master jpbempel/fix-exploration-tests2
git_commit_sha 85b316b 7482440
start_time 2024-10-03T16:01:37 2024-10-03T16:07:52
See matching parameters
Baseline Candidate
ci_job_id 660464330 660464330
ci_pipeline_id 45729951 45729951
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1727970725 1727970725

Summary

Found 0 performance improvements and 4 performance regressions! Performance is the same for 4 metrics, 7 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop worse
[+204.590µs; +567.082µs] or [+2.016%; +5.589%]		 worse
[+213.385µs; +587.280µs] or [+2.071%; +5.700%]		 worse
[+208.892µs; +605.091µs] or [+2.010%; +5.822%]		 worse
[+188.827µs; +717.680µs] or [+1.765%; +6.707%]		 unsure
[-4.521op/s; -0.820op/s] or [-4.724%; -0.857%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-29.295µs; +47.342µs] or [-11.208%; +18.113%]		 unstable
[-41.959µs; +61.871µs] or [-13.903%; +20.500%]		 unstable
[-51.295µs; +74.199µs] or [-16.273%; +23.539%]		 unstable
[-281.435µs; +665.575µs] or [-46.493%; +109.954%]		 unstable
[-142.549op/s; +142.549op/s] or [-5.346%; +5.346%]
scenario:basic same same same unstable
[-33.863µs; +67.111µs] or [-6.200%; +12.288%]		 unstable
[-167.107op/s; +167.107op/s] or [-6.517%; +6.517%]
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (301.807 µs) : 267, 336
.   : milestone, 302,
basic (303.178 µs) : 294, 313
.   : milestone, 303,
loop (10.303 ms) : 10286, 10319
.   : milestone, 10303,
section candidate
noprobe (311.763 µs) : 251, 373
.   : milestone, 312,
basic (307.192 µs) : 293, 321
.   : milestone, 307,
loop (10.703 ms) : 10458, 10948
.   : milestone, 10703,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 301.807 µs [267.147 µs, 336.467 µs]
basic 303.178 µs [293.703 µs, 312.653 µs]
loop 10.303 ms [10.286 ms, 10.319 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 311.763 µs [250.91 µs, 372.616 µs]
basic 307.192 µs [293.463 µs, 320.921 µs]
loop 10.703 ms [10.458 ms, 10.948 ms]

@pr-commenter
Copy link

pr-commenter bot commented Oct 3, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-exploration-tests2
git_commit_date 1727968048 1727970725
git_commit_sha 85b316b 7482440
release_version 1.41.0-SNAPSHOT~85b316b1c0 1.41.0-SNAPSHOT~7482440b04
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1727973130 1727973130
ci_job_id 660464324 660464324
ci_pipeline_id 45729951 45729951
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 11 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.073 s) : 0, 1073421
Total [baseline] (10.327 s) : 0, 10327396
Agent [candidate] (1.078 s) : 0, 1078024
Total [candidate] (10.495 s) : 0, 10494588
section appsec
Agent [baseline] (1.208 s) : 0, 1207611
Total [baseline] (10.597 s) : 0, 10597478
Agent [candidate] (1.208 s) : 0, 1207937
Total [candidate] (10.617 s) : 0, 10617131
section iast
Agent [baseline] (1.2 s) : 0, 1200133
Total [baseline] (10.851 s) : 0, 10850644
Agent [candidate] (1.198 s) : 0, 1198155
Total [candidate] (10.875 s) : 0, 10874732
section profiling
Agent [baseline] (1.266 s) : 0, 1265504
Total [baseline] (10.555 s) : 0, 10555343
Agent [candidate] (1.268 s) : 0, 1267893
Total [candidate] (10.652 s) : 0, 10652118
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.073 s -
Agent appsec 1.208 s 134.191 ms (12.5%)
Agent iast 1.2 s 126.713 ms (11.8%)
Agent profiling 1.266 s 192.083 ms (17.9%)
Total tracing 10.327 s -
Total appsec 10.597 s 270.082 ms (2.6%)
Total iast 10.851 s 523.248 ms (5.1%)
Total profiling 10.555 s 227.947 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent appsec 1.208 s 129.914 ms (12.1%)
Agent iast 1.198 s 120.132 ms (11.1%)
Agent profiling 1.268 s 189.87 ms (17.6%)
Total tracing 10.495 s -
Total appsec 10.617 s 122.543 ms (1.2%)
Total iast 10.875 s 380.144 ms (3.6%)
Total profiling 10.652 s 157.53 ms (1.5%) 
gantt
    title petclinic - break down per module: candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.068 ms) : 0, 685068
BytebuddyAgent [candidate] (687.981 ms) : 0, 687981
GlobalTracer [baseline] (311.969 ms) : 0, 311969
GlobalTracer [candidate] (313.11 ms) : 0, 313110
AppSec [baseline] (54.213 ms) : 0, 54213
AppSec [candidate] (54.613 ms) : 0, 54613
Remote Config [baseline] (665.112 µs) : 0, 665
Remote Config [candidate] (670.963 µs) : 0, 671
Telemetry [baseline] (7.751 ms) : 0, 7751
Telemetry [candidate] (7.837 ms) : 0, 7837
section appsec
BytebuddyAgent [baseline] (702.436 ms) : 0, 702436
BytebuddyAgent [candidate] (701.803 ms) : 0, 701803
GlobalTracer [baseline] (308.33 ms) : 0, 308330
GlobalTracer [candidate] (308.77 ms) : 0, 308770
AppSec [baseline] (161.806 ms) : 0, 161806
AppSec [candidate] (163.086 ms) : 0, 163086
Remote Config [baseline] (649.229 µs) : 0, 649
Remote Config [candidate] (643.158 µs) : 0, 643
Telemetry [baseline] (8.209 ms) : 0, 8209
Telemetry [candidate] (9.282 ms) : 0, 9282
IAST [baseline] (24.753 ms) : 0, 24753
IAST [candidate] (21.102 ms) : 0, 21102
section iast
BytebuddyAgent [baseline] (797.875 ms) : 0, 797875
BytebuddyAgent [candidate] (797.855 ms) : 0, 797855
GlobalTracer [baseline] (302.331 ms) : 0, 302331
GlobalTracer [candidate] (300.322 ms) : 0, 300322
AppSec [baseline] (54.75 ms) : 0, 54750
AppSec [candidate] (55.474 ms) : 0, 55474
Remote Config [baseline] (618.188 µs) : 0, 618
Remote Config [candidate] (613.793 µs) : 0, 614
Telemetry [baseline] (7.042 ms) : 0, 7042
Telemetry [candidate] (7.097 ms) : 0, 7097
IAST [baseline] (23.736 ms) : 0, 23736
IAST [candidate] (23.019 ms) : 0, 23019
section profiling
BytebuddyAgent [baseline] (674.525 ms) : 0, 674525
BytebuddyAgent [candidate] (675.643 ms) : 0, 675643
GlobalTracer [baseline] (392.682 ms) : 0, 392682
GlobalTracer [candidate] (394.095 ms) : 0, 394095
AppSec [baseline] (54.832 ms) : 0, 54832
AppSec [candidate] (54.637 ms) : 0, 54637
Remote Config [baseline] (657.768 µs) : 0, 658
Remote Config [candidate] (663.534 µs) : 0, 664
Telemetry [baseline] (7.451 ms) : 0, 7451
Telemetry [candidate] (7.486 ms) : 0, 7486
ProfilingAgent [baseline] (96.681 ms) : 0, 96681
ProfilingAgent [candidate] (96.578 ms) : 0, 96578
Profiling [baseline] (96.704 ms) : 0, 96704
Profiling [candidate] (96.602 ms) : 0, 96602
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.077 s) : 0, 1077015
Total [baseline] (8.606 s) : 0, 8606206
Agent [candidate] (1.07 s) : 0, 1070483
Total [candidate] (8.564 s) : 0, 8564249
section iast
Agent [baseline] (1.204 s) : 0, 1204382
Total [baseline] (9.103 s) : 0, 9102982
Agent [candidate] (1.214 s) : 0, 1213996
Total [candidate] (9.207 s) : 0, 9206716
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.203 s) : 0, 1202931
Total [baseline] (9.087 s) : 0, 9086570
Agent [candidate] (1.197 s) : 0, 1197234
Total [candidate] (9.072 s) : 0, 9071591
section iast_TELEMETRY_OFF
Agent [baseline] (1.193 s) : 0, 1193388
Total [baseline] (9.093 s) : 0, 9093030
Agent [candidate] (1.194 s) : 0, 1193658
Total [candidate] (9.064 s) : 0, 9063787
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent iast 1.204 s 127.367 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.203 s 125.916 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.193 s 116.372 ms (10.8%)
Total tracing 8.606 s -
Total iast 9.103 s 496.776 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.087 s 480.364 ms (5.6%)
Total iast_TELEMETRY_OFF 9.093 s 486.824 ms (5.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.07 s -
Agent iast 1.214 s 143.512 ms (13.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.197 s 126.75 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.194 s 123.174 ms (11.5%)
Total tracing 8.564 s -
Total iast 9.207 s 642.467 ms (7.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.072 s 507.343 ms (5.9%)
Total iast_TELEMETRY_OFF 9.064 s 499.539 ms (5.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.983 ms) : 0, 685983
BytebuddyAgent [candidate] (683.372 ms) : 0, 683372
GlobalTracer [baseline] (314.142 ms) : 0, 314142
GlobalTracer [candidate] (310.931 ms) : 0, 310931
AppSec [baseline] (54.652 ms) : 0, 54652
AppSec [candidate] (54.105 ms) : 0, 54105
Remote Config [baseline] (681.034 µs) : 0, 681
Remote Config [candidate] (662.62 µs) : 0, 663
Telemetry [baseline] (7.793 ms) : 0, 7793
Telemetry [candidate] (7.647 ms) : 0, 7647
section iast
BytebuddyAgent [baseline] (802.574 ms) : 0, 802574
BytebuddyAgent [candidate] (808.948 ms) : 0, 808948
GlobalTracer [baseline] (301.285 ms) : 0, 301285
GlobalTracer [candidate] (303.629 ms) : 0, 303629
AppSec [baseline] (52.809 ms) : 0, 52809
AppSec [candidate] (53.207 ms) : 0, 53207
Remote Config [baseline] (603.913 µs) : 0, 604
Remote Config [candidate] (619.666 µs) : 0, 620
Telemetry [baseline] (7.053 ms) : 0, 7053
Telemetry [candidate] (7.191 ms) : 0, 7191
IAST [baseline] (26.197 ms) : 0, 26197
IAST [candidate] (26.41 ms) : 0, 26410
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.151 ms) : 0, 802151
BytebuddyAgent [candidate] (796.732 ms) : 0, 796732
GlobalTracer [baseline] (300.007 ms) : 0, 300007
GlobalTracer [candidate] (300.119 ms) : 0, 300119
AppSec [baseline] (56.016 ms) : 0, 56016
AppSec [candidate] (54.296 ms) : 0, 54296
Remote Config [baseline] (616.02 µs) : 0, 616
Remote Config [candidate] (615.482 µs) : 0, 615
Telemetry [baseline] (7.123 ms) : 0, 7123
Telemetry [candidate] (7.069 ms) : 0, 7069
IAST [baseline] (23.095 ms) : 0, 23095
IAST [candidate] (24.537 ms) : 0, 24537
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (794.101 ms) : 0, 794101
BytebuddyAgent [candidate] (793.878 ms) : 0, 793878
GlobalTracer [baseline] (299.401 ms) : 0, 299401
GlobalTracer [candidate] (299.402 ms) : 0, 299402
AppSec [baseline] (54.405 ms) : 0, 54405
AppSec [candidate] (52.07 ms) : 0, 52070
Remote Config [baseline] (615.13 µs) : 0, 615
Remote Config [candidate] (637.129 µs) : 0, 637
Telemetry [baseline] (6.916 ms) : 0, 6916
Telemetry [candidate] (6.926 ms) : 0, 6926
IAST [baseline] (24.134 ms) : 0, 24134
IAST [candidate] (26.884 ms) : 0, 26884
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-10-03T16:02:38 2024-10-03T16:09:29
git_branch master jpbempel/fix-exploration-tests2
git_commit_date 1727968048 1727970725
git_commit_sha 85b316b 7482440
release_version 1.41.0-SNAPSHOT~85b316b1c0 1.41.0-SNAPSHOT~7482440b04
start_time 2024-10-03T16:02:25 2024-10-03T16:09:16
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1727972114 1727972114
ci_job_id 660464325 660464325
ci_pipeline_id 45729951 45729951
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.99 µs) : 348, 387
.   : milestone, 368,
iast (480.43 µs) : 459, 502
.   : milestone, 480,
iast_FULL (550.831 µs) : 530, 572
.   : milestone, 551,
iast_GLOBAL (511.586 µs) : 489, 534
.   : milestone, 512,
iast_HARDCODED_SECRET_DISABLED (479.643 µs) : 459, 501
.   : milestone, 480,
iast_INACTIVE (446.871 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (468.568 µs) : 448, 489
.   : milestone, 469,
tracing (444.317 µs) : 423, 465
.   : milestone, 444,
section candidate
no_agent (371.748 µs) : 352, 391
.   : milestone, 372,
iast (479.575 µs) : 458, 501
.   : milestone, 480,
iast_FULL (555.344 µs) : 534, 577
.   : milestone, 555,
iast_GLOBAL (501.045 µs) : 480, 522
.   : milestone, 501,
iast_HARDCODED_SECRET_DISABLED (489.888 µs) : 468, 511
.   : milestone, 490,
iast_INACTIVE (447.408 µs) : 426, 469
.   : milestone, 447,
iast_TELEMETRY_OFF (470.95 µs) : 450, 492
.   : milestone, 471,
tracing (447.37 µs) : 427, 468
.   : milestone, 447,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.99 µs [348.485 µs, 387.495 µs] -
iast 480.43 µs [459.25 µs, 501.61 µs] 112.44 µs (30.6%)
iast_FULL 550.831 µs [529.733 µs, 571.928 µs] 182.841 µs (49.7%)
iast_GLOBAL 511.586 µs [489.226 µs, 533.945 µs] 143.596 µs (39.0%)
iast_HARDCODED_SECRET_DISABLED 479.643 µs [458.614 µs, 500.671 µs] 111.653 µs (30.3%)
iast_INACTIVE 446.871 µs [425.958 µs, 467.783 µs] 78.881 µs (21.4%)
iast_TELEMETRY_OFF 468.568 µs [447.766 µs, 489.371 µs] 100.578 µs (27.3%)
tracing 444.317 µs [423.201 µs, 465.432 µs] 76.327 µs (20.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 371.748 µs [352.106 µs, 391.39 µs] -
iast 479.575 µs [458.438 µs, 500.711 µs] 107.827 µs (29.0%)
iast_FULL 555.344 µs [534.098 µs, 576.59 µs] 183.596 µs (49.4%)
iast_GLOBAL 501.045 µs [480.125 µs, 521.965 µs] 129.297 µs (34.8%)
iast_HARDCODED_SECRET_DISABLED 489.888 µs [468.354 µs, 511.423 µs] 118.14 µs (31.8%)
iast_INACTIVE 447.408 µs [426.221 µs, 468.595 µs] 75.66 µs (20.4%)
iast_TELEMETRY_OFF 470.95 µs [449.847 µs, 492.053 µs] 99.202 µs (26.7%)
tracing 447.37 µs [426.583 µs, 468.156 µs] 75.622 µs (20.3%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.695 ms) : 1671, 1720
.   : milestone, 1695,
appsec_no_iast (1.714 ms) : 1691, 1738
.   : milestone, 1714,
iast (1.483 ms) : 1460, 1506
.   : milestone, 1483,
profiling (1.495 ms) : 1469, 1520
.   : milestone, 1495,
tracing (1.467 ms) : 1443, 1491
.   : milestone, 1467,
section candidate
no_agent (1.331 ms) : 1312, 1351
.   : milestone, 1331,
appsec (1.698 ms) : 1674, 1723
.   : milestone, 1698,
appsec_no_iast (1.724 ms) : 1700, 1748
.   : milestone, 1724,
iast (1.472 ms) : 1449, 1495
.   : milestone, 1472,
profiling (1.483 ms) : 1459, 1508
.   : milestone, 1483,
tracing (1.445 ms) : 1420, 1470
.   : milestone, 1445,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.695 ms [1.671 ms, 1.72 ms] 353.707 µs (26.4%)
appsec_no_iast 1.714 ms [1.691 ms, 1.738 ms] 372.635 µs (27.8%)
iast 1.483 ms [1.46 ms, 1.506 ms] 141.219 µs (10.5%)
profiling 1.495 ms [1.469 ms, 1.52 ms] 152.924 µs (11.4%)
tracing 1.467 ms [1.443 ms, 1.491 ms] 125.239 µs (9.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.312 ms, 1.351 ms] -
appsec 1.698 ms [1.674 ms, 1.723 ms] 366.952 µs (27.6%)
appsec_no_iast 1.724 ms [1.7 ms, 1.748 ms] 392.557 µs (29.5%)
iast 1.472 ms [1.449 ms, 1.495 ms] 140.518 µs (10.6%)
profiling 1.483 ms [1.459 ms, 1.508 ms] 151.702 µs (11.4%)
tracing 1.445 ms [1.42 ms, 1.47 ms] 113.446 µs (8.5%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-exploration-tests2
git_commit_date 1727968048 1727970725
git_commit_sha 85b316b 7482440
release_version 1.41.0-SNAPSHOT~85b316b1c0 1.41.0-SNAPSHOT~7482440b04
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1727972663 1727972663
ci_job_id 660464326 660464326
ci_pipeline_id 45729951 45729951
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.092 s) : 15092000, 15092000
.   : milestone, 15092000,
appsec (15.194 s) : 15194000, 15194000
.   : milestone, 15194000,
iast (18.848 s) : 18848000, 18848000
.   : milestone, 18848000,
iast_GLOBAL (18.118 s) : 18118000, 18118000
.   : milestone, 18118000,
profiling (15.299 s) : 15299000, 15299000
.   : milestone, 15299000,
tracing (15.459 s) : 15459000, 15459000
.   : milestone, 15459000,
section candidate
no_agent (15.847 s) : 15847000, 15847000
.   : milestone, 15847000,
appsec (15.317 s) : 15317000, 15317000
.   : milestone, 15317000,
iast (19.292 s) : 19292000, 19292000
.   : milestone, 19292000,
iast_GLOBAL (18.173 s) : 18173000, 18173000
.   : milestone, 18173000,
profiling (15.206 s) : 15206000, 15206000
.   : milestone, 15206000,
tracing (15.225 s) : 15225000, 15225000
.   : milestone, 15225000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.092 s [15.092 s, 15.092 s] -
appsec 15.194 s [15.194 s, 15.194 s] 102.0 ms (0.7%)
iast 18.848 s [18.848 s, 18.848 s] 3.756 s (24.9%)
iast_GLOBAL 18.118 s [18.118 s, 18.118 s] 3.026 s (20.1%)
profiling 15.299 s [15.299 s, 15.299 s] 207.0 ms (1.4%)
tracing 15.459 s [15.459 s, 15.459 s] 367.0 ms (2.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.847 s [15.847 s, 15.847 s] -
appsec 15.317 s [15.317 s, 15.317 s] -530.0 ms (-3.3%)
iast 19.292 s [19.292 s, 19.292 s] 3.445 s (21.7%)
iast_GLOBAL 18.173 s [18.173 s, 18.173 s] 2.326 s (14.7%)
profiling 15.206 s [15.206 s, 15.206 s] -641.0 ms (-4.0%)
tracing 15.225 s [15.225 s, 15.225 s] -622.0 ms (-3.9%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.41.0-SNAPSHOT~7482440b04, baseline=1.41.0-SNAPSHOT~85b316b1c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.342 ms) : 2301, 2382
.   : milestone, 2342,
iast (2.08 ms) : 2028, 2132
.   : milestone, 2080,
iast_GLOBAL (2.124 ms) : 2071, 2176
.   : milestone, 2124,
profiling (1.943 ms) : 1901, 1985
.   : milestone, 1943,
tracing (1.917 ms) : 1878, 1957
.   : milestone, 1917,
section candidate
no_agent (1.463 ms) : 1452, 1475
.   : milestone, 1463,
appsec (2.312 ms) : 2272, 2353
.   : milestone, 2312,
iast (2.073 ms) : 2021, 2124
.   : milestone, 2073,
iast_GLOBAL (2.119 ms) : 2067, 2171
.   : milestone, 2119,
profiling (1.947 ms) : 1906, 1988
.   : milestone, 1947,
tracing (1.918 ms) : 1879, 1957
.   : milestone, 1918,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.461 ms, 1.484 ms] -
appsec 2.342 ms [2.301 ms, 2.382 ms] 869.184 µs (59.0%)
iast 2.08 ms [2.028 ms, 2.132 ms] 607.226 µs (41.2%)
iast_GLOBAL 2.124 ms [2.071 ms, 2.176 ms] 651.425 µs (44.2%)
profiling 1.943 ms [1.901 ms, 1.985 ms] 470.612 µs (32.0%)
tracing 1.917 ms [1.878 ms, 1.957 ms] 444.979 µs (30.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.463 ms [1.452 ms, 1.475 ms] -
appsec 2.312 ms [2.272 ms, 2.353 ms] 849.185 µs (58.0%)
iast 2.073 ms [2.021 ms, 2.124 ms] 609.453 µs (41.7%)
iast_GLOBAL 2.119 ms [2.067 ms, 2.171 ms] 655.717 µs (44.8%)
profiling 1.947 ms [1.906 ms, 1.988 ms] 483.91 µs (33.1%)
tracing 1.918 ms [1.879 ms, 1.957 ms] 454.819 µs (31.1%)

evanchooly
evanchooly reviewed Oct 3, 2024
View reviewed changes
dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests
@@ -16,25 +17,36 @@ RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && \
yes | sdk install java $JAVA_8_VERSION && \
yes | sdk install java $JAVA_11_VERSION && \
yes | sdk install java $JAVA_17_VERSION && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you commit a .sdkmanrc file this all can be one line.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok thanks will improve that later

@jpbempel
Fix exploration tests
7482440 
update the registry for pushing the image
install jdk 21
use the jsoup library and patch it for running test with
instrument-the-world mode
add include file to specify which packages/classes we want to include
into the instrumentation instead of instrumenting everything
(like build tools)
do not collect static inherited fields to avoid duplicate class
definition for enums.
@jpbempel jpbempel force-pushed the jpbempel/fix-exploration-tests2 branch from 38a78ce to 7482440 Compare October 3, 2024 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@evanchooly evanchooly evanchooly left review comments

@ojung ojung Awaiting requested review from ojung ojung is a code owner automatically assigned from DataDog/debugger-java

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/apm-java

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jpbempel @evanchooly