| Setting | Run #11 (base) | Run #14 (target) |
|---|---|---|
| Description | — | — |
| JVM heap | -Xms128m -Xmx256m | -Xms128m -Xmx256m |
| JVM args | -Xgcpolicy:optthruput -Xgcthreads1 -Xminf0.1 -Xmaxf0.2 | -Xgcpolicy:optthruput -Xgcthreads1 -Xminf0.1 -Xmaxf0.2 |
| Java | semeru-openj9-25.0.2+10_openj9-0.57.0 | semeru-openj9-25.0.2+10_openj9-0.57.0 |
| Iterations | 7 | 7 |
| Started | 2026-05-12 15:40:01.0 | 2026-05-14 10:51:06.0 |
| Deploy commit | fc104f4 | fc104f4 |
Of 24 (runtime × metric) comparisons: target wins 11, base wins 13. Biggest improvement on target: quarkus-semeru · Load RSS (-5.6%). Biggest regression: quarkus-semeru · Throughput (-3.2%).
Cells show mean ± stddev with CV% (run-to-run consistency: <5% stable · 5-15% moderate · >15% noisy). Hover a cell for the 95% CI. Δ% is target relative to base. d is Cohen's effect size; Welch p is the two-tailed p-value (p<0.001, p<0.05). Green marks the better mean per row; grey marks rows where one side has no data.
| Metric | Run #11 | Run #14 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 9,871 ±275 (CV 2.8%) | 10,127 ±261 (CV 2.6%) | +256 | +2.6% | 0.95 large | 0.0998 |
| TTFR (ms) | 4,002 ±134 (CV 3.4%) | 3,959 ±106 (CV 2.7%) | -43 | -1.1% | -0.35 small | 0.5211 |
| rps / MiB (rps/MiB) | 28.14 ±0.85 (CV 3.0%) | 29.08 ±0.90 (CV 3.1%) | +0.95 | +3.4% | 1.08 large | 0.0656 |
| Startup RSS (MiB) | 264.8 ±0.2 (CV 0.08%) | 265.8 ±0.1 (CV 0.05%) | +1.0 | +0.4% | 5.50 large | <0.0001 |
| 1st-req RSS (MiB) | 279.5 ±1.0 (CV 0.34%) | 279.7 ±0.3 (CV 0.10%) | +0.2 | +0.1% | 0.25 small | 0.6600 |
| Load RSS (MiB) | 350.9 ±7.5 (CV 2.1%) | 348.3 ±6.7 (CV 1.9%) | -2.6 | -0.7% | -0.36 small | 0.5081 |
| Latency avg (ms) | — | 9.83 ±0.26 (CV 2.6%) | — | — | — | — |
| Latency max (ms) | — | 136.9 ±15.9 (CV 12%) | — | — | — | — |
| Metric | Run #11 | Run #14 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 9,713 ±120 (CV 1.2%) | 9,398 ±165 (CV 1.8%) | -315 | -3.2% | -2.18 large | 0.0018 |
| TTFR (ms) | 3,248 ±77 (CV 2.4%) | 3,251 ±94 (CV 2.9%) | +3 | +0.1% | 0.03 negligible | 0.9511 |
| rps / MiB (rps/MiB) | 29.10 ±0.76 (CV 2.6%) | 29.83 ±0.92 (CV 3.1%) | +0.73 | +2.5% | 0.87 large | 0.1301 |
| Startup RSS (MiB) | 254.4 ±0.3 (CV 0.10%) | 255.0 ±0.4 (CV 0.16%) | +0.6 | +0.2% | 1.63 large | 0.0119 |
| 1st-req RSS (MiB) | 263.9 ±1.1 (CV 0.42%) | 265.3 ±0.7 (CV 0.26%) | +1.3 | +0.5% | 1.44 large | 0.0219 |
| Load RSS (MiB) | 334.0 ±6.8 (CV 2.0%) | 315.2 ±8.1 (CV 2.6%) | -18.7 | -5.6% | -2.50 large | 0.0006 |
| Latency avg (ms) | — | 10.59 ±0.18 (CV 1.7%) | — | — | — | — |
| Latency max (ms) | — | 138.0 ±13.8 (CV 10.0%) | — | — | — | — |
| Metric | Run #11 | Run #14 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 10,134 ±274 (CV 2.7%) | 10,279 ±180 (CV 1.7%) | +145 | +1.4% | 0.63 medium | 0.2672 |
| TTFR (ms) | 3,986 ±81 (CV 2.0%) | 3,949 ±107 (CV 2.7%) | -37 | -0.9% | -0.39 small | 0.4769 |
| rps / MiB (rps/MiB) | 33.13 ±0.96 (CV 2.9%) | 33.18 ±0.86 (CV 2.6%) | +0.05 | +0.2% | 0.06 negligible | 0.9137 |
| Startup RSS (MiB) | 264.9 ±0.3 (CV 0.10%) | 265.7 ±0.1 (CV 0.03%) | +0.8 | +0.3% | 4.06 large | 0.0001 |
| 1st-req RSS (MiB) | 280.0 ±1.0 (CV 0.36%) | 280.6 ±0.4 (CV 0.15%) | +0.7 | +0.2% | 0.88 large | 0.1393 |
| Load RSS (MiB) | 305.9 ±1.4 (CV 0.46%) | 309.9 ±5.7 (CV 1.8%) | +4.0 | +1.3% | 0.96 large | 0.1175 |
| Latency avg (ms) | — | 9.68 ±0.17 (CV 1.8%) | — | — | — | — |
| Latency max (ms) | — | 67.2 ±4.5 (CV 6.7%) | — | — | — | — |
| Metric | Run #11 | Run #14 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 9,958 ±204 (CV 2.0%) | 9,693 ±113 (CV 1.2%) | -264 | -2.7% | -1.60 large | 0.0143 |
| TTFR (ms) | 3,337 ±116 (CV 3.5%) | 3,289 ±95 (CV 2.9%) | -48 | -1.4% | -0.45 small | 0.4122 |
| rps / MiB (rps/MiB) | 33.52 ±0.72 (CV 2.2%) | 33.10 ±0.38 (CV 1.1%) | -0.42 | -1.2% | -0.72 medium | 0.2085 |
| Startup RSS (MiB) | 254.2 ±0.2 (CV 0.08%) | 255.0 ±0.3 (CV 0.12%) | +0.8 | +0.3% | 3.10 large | 0.0001 |
| 1st-req RSS (MiB) | 264.1 ±1.1 (CV 0.43%) | 265.5 ±1.1 (CV 0.41%) | +1.4 | +0.5% | 1.28 large | 0.0335 |
| Load RSS (MiB) | 297.1 ±2.1 (CV 0.69%) | 292.8 ±0.8 (CV 0.28%) | -4.2 | -1.4% | -2.70 large | 0.0010 |
| Latency avg (ms) | — | 10.27 ±0.12 (CV 1.2%) | — | — | — | — |
| Latency max (ms) | — | 123.5 ±46.3 (CV 38%) | — | — | — | — |