| Setting | Run #13 (base) | Run #12 (target) |
|---|---|---|
| Description | — | — |
| JVM heap | -Xms128m -Xmx256m | -Xms128m -Xmx256m |
| JVM args | -XX:+UseSerialGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 | -XX:+UseSerialGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 |
| Java | temurin-25.0.2 | temurin-25.0.2 |
| Iterations | 7 | 7 |
| Started | 2026-05-13 16:15:15.0 | 2026-05-13 09:29:38.0 |
| Deploy commit | fc104f4 | fc104f4 |
Of 32 (runtime × metric) comparisons: target wins 18, base wins 14. Biggest improvement on target: quarkus-jvm · Latency max (-6.2%). Biggest regression: quarkus-virtual · Latency max (+16.0%).
Cells show mean ± stddev with CV% (run-to-run consistency: <5% stable · 5-15% moderate · >15% noisy). Hover a cell for the 95% CI. Δ% is target relative to base. d is Cohen's effect size; Welch p is the two-tailed p-value (p<0.001, p<0.05). Green marks the better mean per row; grey marks rows where one side has no data.
| Metric | Run #13 | Run #12 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 12,023 ±442 (CV 3.7%) | 12,369 ±565 (CV 4.6%) | +346 | +2.9% | 0.68 medium | 0.2273 |
| TTFR (ms) | 2,636 ±41 (CV 1.5%) | 2,538 ±78 (CV 3.1%) | -98 | -3.7% | -1.57 large | 0.0163 |
| rps / MiB (rps/MiB) | 26.48 ±1.00 (CV 3.8%) | 27.57 ±1.61 (CV 5.9%) | +1.09 | +4.1% | 0.81 large | 0.1591 |
| Startup RSS (MiB) | 234.1 ±3.6 (CV 1.5%) | 227.9 ±6.0 (CV 2.6%) | -6.2 | -2.7% | -1.26 large | 0.0400 |
| 1st-req RSS (MiB) | 250.2 ±3.6 (CV 1.4%) | 243.5 ±5.9 (CV 2.4%) | -6.6 | -2.6% | -1.35 large | 0.0308 |
| Load RSS (MiB) | 454.1 ±8.9 (CV 1.9%) | 448.9 ±10.5 (CV 2.3%) | -5.2 | -1.1% | -0.53 medium | 0.3386 |
| Latency avg (ms) | 8.28 ±0.30 (CV 3.6%) | 8.04 ±0.37 (CV 4.6%) | -0.23 | -2.8% | -0.70 medium | 0.2161 |
| Latency max (ms) | 64.3 ±15.3 (CV 24%) | 60.4 ±7.5 (CV 12%) | -4.0 | -6.2% | -0.33 small | 0.5534 |
| Metric | Run #13 | Run #12 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 11,391 ±584 (CV 5.1%) | 11,383 ±537 (CV 4.7%) | -8 | -0.1% | -0.01 negligible | 0.9791 |
| TTFR (ms) | 853 ±37 (CV 4.3%) | 853 ±27 (CV 3.2%) | -0 | -0.0% | -0.01 negligible | 0.9812 |
| rps / MiB (rps/MiB) | 27.39 ±1.25 (CV 4.6%) | 27.31 ±1.63 (CV 6.0%) | -0.08 | -0.3% | -0.05 negligible | 0.9232 |
| Startup RSS (MiB) | 217.9 ±2.4 (CV 1.1%) | 218.9 ±2.3 (CV 1.1%) | +1.0 | +0.5% | 0.42 small | 0.4435 |
| 1st-req RSS (MiB) | 223.6 ±1.9 (CV 0.85%) | 227.0 ±3.2 (CV 1.4%) | +3.3 | +1.5% | 1.26 large | 0.0410 |
| Load RSS (MiB) | 416.0 ±11.3 (CV 2.7%) | 417.3 ±15.9 (CV 3.8%) | +1.3 | +0.3% | 0.10 negligible | 0.8603 |
| Latency avg (ms) | 8.74 ±0.43 (CV 5.0%) | 8.75 ±0.40 (CV 4.6%) | +0.01 | +0.1% | 0.02 negligible | 0.9701 |
| Latency max (ms) | 64.2 ±11.8 (CV 18%) | 60.8 ±5.6 (CV 9.2%) | -3.3 | -5.2% | -0.36 small | 0.5170 |
| Metric | Run #13 | Run #12 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 14,832 ±290 (CV 2.0%) | 15,051 ±397 (CV 2.6%) | +219 | +1.5% | 0.63 medium | 0.2641 |
| TTFR (ms) | 2,683 ±85 (CV 3.2%) | 2,664 ±51 (CV 1.9%) | -19 | -0.7% | -0.27 small | 0.6198 |
| rps / MiB (rps/MiB) | 35.10 ±0.65 (CV 1.9%) | 36.40 ±1.42 (CV 3.9%) | +1.30 | +3.7% | 1.18 large | 0.0569 |
| Startup RSS (MiB) | 233.7 ±4.1 (CV 1.8%) | 231.4 ±3.9 (CV 1.7%) | -2.2 | -1.0% | -0.56 medium | 0.3139 |
| 1st-req RSS (MiB) | 256.5 ±6.3 (CV 2.4%) | 256.3 ±2.7 (CV 1.0%) | -0.2 | -0.1% | -0.03 negligible | 0.9519 |
| Load RSS (MiB) | 422.6 ±7.7 (CV 1.8%) | 413.7 ±9.6 (CV 2.3%) | -8.9 | -2.1% | -1.02 large | 0.0824 |
| Latency avg (ms) | 6.71 ±0.13 (CV 2.0%) | 6.61 ±0.18 (CV 2.7%) | -0.10 | -1.4% | -0.61 medium | 0.2750 |
| Latency max (ms) | 36.1 ±3.4 (CV 9.3%) | 41.9 ±7.0 (CV 17%) | +5.8 | +16.0% | 1.05 large | 0.0829 |
| Metric | Run #13 | Run #12 | Δ | Δ% | d | Welch p |
|---|---|---|---|---|---|---|
| Throughput (rps) | 14,553 ±306 (CV 2.1%) | 14,160 ±184 (CV 1.3%) | -393 | -2.7% | -1.55 large | 0.0159 |
| TTFR (ms) | 866 ±28 (CV 3.3%) | 868 ±62 (CV 7.2%) | +2 | +0.2% | 0.04 negligible | 0.9403 |
| rps / MiB (rps/MiB) | 38.38 ±1.24 (CV 3.2%) | 37.69 ±1.95 (CV 5.2%) | -0.70 | -1.8% | -0.43 small | 0.4444 |
| Startup RSS (MiB) | 222.0 ±4.0 (CV 1.8%) | 226.6 ±4.2 (CV 1.8%) | +4.5 | +2.0% | 1.11 large | 0.0598 |
| 1st-req RSS (MiB) | 231.5 ±4.7 (CV 2.0%) | 235.6 ±5.0 (CV 2.1%) | +4.0 | +1.7% | 0.83 large | 0.1462 |
| Load RSS (MiB) | 379.4 ±11.4 (CV 3.0%) | 376.5 ±16.3 (CV 4.3%) | -3.0 | -0.8% | -0.21 small | 0.7014 |
| Latency avg (ms) | 6.84 ±0.15 (CV 2.1%) | 7.03 ±0.09 (CV 1.3%) | +0.19 | +2.8% | 1.55 large | 0.0159 |
| Latency max (ms) | 37.3 ±3.1 (CV 8.4%) | 42.9 ±10.7 (CV 25%) | +5.6 | +15.0% | 0.71 medium | 0.2265 |