Benchmark run #9

Started2026-05-02 22:09:36.0
Stopped2026-05-03 04:40:27.0
NoteRun 9: Serial GC + aggressive heap return, -Xms128m -Xmx256m, JDK 25, 7 iter (post dotnet validation-filter parity fix)
Tests runmeasure-time-to-first-request,measure-rss,run-load-test
Iterations7
HostHetzner vServer · AMD EPYC-Genoa Processor (16 cpus) · 30Gi
OSUbuntu 24.04.3 LTS · 6.8.0-110-generic
Javatemurin-25.0.2
GraalVMgraalvm-community-25.0.2
Mandrelmandrel-25.0.2.0-Final+java25
dotnet10.0.100
JVM heap-Xms128m -Xmx256m
JVM args-XX:+UseSerialGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20
CPU pinning (app/db/load/otel)0-3 · 4-6 · 10-12 · 7-9
Deploy commit090f8e2 · main
Deploy run_id9b6f2b9d-16de-4445-83eb-9a2f6cc6100d
Deployed at2026-05-02 22:08:11.0

dotnet vs Quarkus comparison

Per metric, the ratio in parentheses is quarkus / dotnet: values above 1× mean Quarkus produced a higher number, below 1× means dotnet did. Green marks the per-row winner within each category. Red marks RSS values that exceed the runtime's configured memory budget. Two importance-weighted scores are reported per category: the Overall winner uses the table-row order (Throughput → TTFR → rps/MiB → memory rows → build time → errors), while the Efficiency-weighted winner reorders to rps/MiB → TTFR → throughput → RSS-under-load. In both, the leading metric outweighs every later metric combined, so ranking is effectively lexicographic in the chosen order.

dotnet-aspnet-ef vs JVM-mode Quarkus variants

Overall winner: Quarkus (JVM-mode) (metric wins: dotnet-aspnet-ef=5, Quarkus (JVM-mode)=2, ties=2; tiebreaker: Throughput)

Efficiency-weighted winner (rps/MiB → TTFR → throughput → RSS load): dotnet-aspnet-ef (metric wins: dotnet-aspnet-ef=2, Quarkus (JVM-mode)=2; tiebreaker: rps / MiB)

Metricdotnet-aspnet-efquarkus-jvmquarkus-leydenquarkus-virtualquarkus-virtual-leyden
valueratiovalueratiovalueratiovalueratio
Throughput (rps)6,53012,0231.84×10,5941.62×15,7992.42×14,6792.25×
TTFR (ms)1,395.02,637.71.89×999.00.716×2,613.41.87×877.20.629×
rps / MiB (rps/MiB)29.2626.730.914×25.500.872×28.330.968×28.220.965×
RSS @ startup (MiB)74.8229.73.07×220.62.95×230.63.08×228.73.06×
RSS @ 1st req (MiB)133.2247.31.86×240.31.80×250.91.88×240.21.80×
RSS under load (MiB)223.2449.92.02×415.61.86×557.82.50×520.22.33×
Build time (s)2.208.253.75×52.6723.9×8.303.77×52.6323.9×
Conn err (count)00—0—0—0—
Timeouts (count)00—0—0—0—

dotnet-aspnet-ef vs Native Quarkus variants

Overall winner: Quarkus (Native) (metric wins: dotnet-aspnet-ef=2, Quarkus (Native)=5, ties=2; tiebreaker: Throughput)

Efficiency-weighted winner (rps/MiB → TTFR → throughput → RSS load): Quarkus (Native) (metric wins: dotnet-aspnet-ef=0, Quarkus (Native)=4; tiebreaker: rps / MiB)

Metricdotnet-aspnet-efquarkus-nativequarkus-native-mandrel
valueratiovalueratio
Throughput (rps)6,5307,5491.16×7,5001.15×
TTFR (ms)1,395.0108.20.078×104.40.075×
rps / MiB (rps/MiB)29.2637.261.27×36.811.26×
RSS @ startup (MiB)74.889.31.19×89.61.20×
RSS @ 1st req (MiB)133.295.70.718×95.50.717×
RSS under load (MiB)223.2202.60.908×203.80.913×
Build time (s)2.20156.1070.9×153.2969.6×
Conn err (count)00—0—
Timeouts (count)00—0—

Statistical comparisons (Java/Quarkus vs dotnet)

Per-metric pairwise comparisons. Cohen's d sign reflects (Quarkus − dotnet): positive = Quarkus higher, negative = dotnet higher. Welch's two-tailed t-test does not assume equal variances. p < 0.001 · p < 0.05 · not significant.

Throughput (rps) — higher is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm6,53012,02322.75 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden6,53010,59422.45 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native6,5307,5499.98 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel6,5307,5006.13 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual6,53015,79931.66 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden6,53014,67929.12 large<0.0001 ✓

TTFR (ms) — lower is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm1,395.02,637.717.14 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden1,395.0999.0-9.61 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native1,395.0108.2-36.78 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel1,395.0104.4-36.82 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual1,395.02,613.420.10 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden1,395.0877.2-13.75 large<0.0001 ✓

rps / MiB (rps/MiB) — higher is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm29.2626.73-3.58 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden29.2625.50-4.81 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native29.2637.2612.05 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel29.2636.817.91 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual29.2628.33-1.25 large0.0387 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden29.2628.22-1.54 large0.0137 ✓

RSS @ startup (MiB) — lower is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm74.8229.772.55 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden74.8220.631.90 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native74.889.3153.32 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel74.889.6154.38 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual74.8230.654.72 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden74.8228.760.52 large<0.0001 ✓

RSS @ 1st req (MiB) — lower is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm133.2247.338.73 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden133.2240.322.26 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native133.295.7-269.09 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel133.295.5-268.60 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual133.2250.932.91 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden133.2240.238.49 large<0.0001 ✓

RSS under load (MiB) — lower is better

Comparisondotnet meanQuarkus meanCohen's dWelch p
dotnet-aspnet-ef vs quarkus-jvm223.2449.924.08 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-leyden223.2415.627.43 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native223.2202.6-7.81 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-native-mandrel223.2203.8-5.77 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual223.2557.869.47 large<0.0001 ✓
dotnet-aspnet-ef vs quarkus-virtual-leyden223.2520.249.05 large<0.0001 ✓

Workload simulation

Closed-form models that translate per-runtime metrics into deployment-archetype outcomes. Inputs come from the measured columns in the per-runtime summary; the math is shown alongside the score so readers can sanity-check it.

Serverless function

Simulated workload: 1,000,000 invocations, cold-start probability 20%. Per-invocation cost = avg latency × RSS-after-first-request (warm) + pcold × TTFR × RSS (cold tax). Result is total billable GB·seconds; lower is better.

RankRuntimeWarm cost
MiB·ms / req		Cold tax
MiB·ms / req		Total
GB·seconds		Ratio vs winner
1quarkus-native-mandrel1,2691,9943,1871.00×
2quarkus-native1,2632,0693,2541.02×
3dotnet-aspnet-ef2,02437,15638,26212.01×
4quarkus-virtual-leyden1,62942,14042,74313.41×
5quarkus-leyden2,25648,00949,08715.40×
6quarkus-jvm2,046130,479129,41940.61×
7quarkus-virtual1,580131,126129,59540.66×

Long-running service

Simulated workload: 10,000 rps for 24 hours, instances pinned to 80% utilization, Service Level Objective (SLO) target latency 100 ms — i.e. the maximum request latency the service commits to. The simulator uses the measured max latency as a proxy: full credit at the SLO, zero credit at 2× the SLO, linear in between. Score = SLO-compliant requests served per GB·hour of cluster memory; higher is better.

RankRuntimeSLO complianceEffective
rps		Instances
required		Cluster RSS
MiB		Score
SLO-rps / GB·hour		Ratio vs winner
1quarkus-virtual100%15,7991558104,405,9831.00×
2quarkus-virtual-leyden98%14,4111520102,126,6830.98×
3quarkus-native-mandrel93%6,973240863,056,6270.60×
4quarkus-native82%6,198240556,383,8990.54×
5dotnet-aspnet-ef100%6,530244653,918,6510.52×
6quarkus-jvm100%12,023290049,258,1050.47×
7quarkus-leyden100%10,594283146,981,8630.45×

Native closed-world analysis

Output of GraalVM/Mandrel native-image's build-time reachability analysis. Under the closed-world assumption only types/fields/methods reached at build time end up in the binary; anything dynamic (reflection, JNI, resources) must be explicitly registered. The reflection columns show registered count and as a percentage of total reachable.

RuntimeBinary size
MiB		Build RSS
GiB		Classes reachableFields reachableMethods reachableClasses registered
for reflection		Fields registered
for reflection		Methods registered
for reflection
quarkus-native121.84.9627,58737,248135,4819,051 (32.8%)1,816 (4.9%)17,176 (12.7%)
quarkus-native-mandrel122.45.1027,58737,248135,4829,051 (32.8%)1,816 (4.9%)17,176 (12.7%)

Per-runtime summary

Cells show mean ± sample standard deviation across iterations.

RuntimeThroughput
rps		TTFR
ms		rps / MiB
rps/MiB		RSS @ startup
MiB		RSS @ 1st req
MiB		RSS under load
MiB		Build time
s		Conn err
count		Timeouts
count
dotnet-aspnet-ef6,530 ± 111 (CV 1.7%) (n=7)1,395.0 ± 47.5 (CV 3.4%) (n=7)29.26 ± 0.68 (CV 2.3%) (n=7)74.8 ± 0.1 (CV 0.18%) (n=7)133.2 ± 0.2 (CV 0.15%) (n=7)223.2 ± 3.3 (CV 1.5%) (n=7)2.20 ± 1.50 (CV 68%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-jvm12,023 ± 323 (CV 2.7%) (n=7)2,637.7 ± 90.9 (CV 3.4%) (n=7)26.73 ± 0.73 (CV 2.7%) (n=7)229.7 ± 3.0 (CV 1.3%) (n=7)247.3 ± 4.2 (CV 1.7%) (n=7)449.9 ± 12.9 (CV 2.9%) (n=7)8.25 ± 0.16 (CV 1.9%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-leyden10,594 ± 231 (CV 2.2%) (n=7)999.0 ± 33.8 (CV 3.4%) (n=7)25.50 ± 0.87 (CV 3.4%) (n=7)220.6 ± 6.5 (CV 2.9%) (n=7)240.3 ± 6.8 (CV 2.8%) (n=7)415.6 ± 9.4 (CV 2.3%) (n=7)52.67 ± 0.18 (CV 0.34%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-native7,549 ± 92 (CV 1.2%) (n=7)108.2 ± 13.9 (CV 13%) (n=7)37.26 ± 0.65 (CV 1.7%) (n=7)89.3 ± 0.0 (CV 0.02%) (n=7)95.7 ± 0.0 (CV 0.02%) (n=7)202.6 ± 1.8 (CV 0.90%) (n=7)156.10 ± 1.87 (CV 1.2%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-native-mandrel7,500 ± 194 (CV 2.6%) (n=7)104.4 ± 14.2 (CV 14%) (n=7)36.81 ± 1.16 (CV 3.2%) (n=7)89.6 ± 0.0 (CV 0.03%) (n=7)95.5 ± 0.0 (CV 0.03%) (n=7)203.8 ± 3.5 (CV 1.7%) (n=7)153.29 ± 1.87 (CV 1.2%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-virtual15,799 ± 399 (CV 2.5%) (n=7)2,613.4 ± 71.4 (CV 2.7%) (n=7)28.33 ± 0.81 (CV 2.9%) (n=7)230.6 ± 4.0 (CV 1.7%) (n=7)250.9 ± 5.1 (CV 2.0%) (n=7)557.8 ± 6.0 (CV 1.1%) (n=7)8.30 ± 0.15 (CV 1.8%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)
quarkus-virtual-leyden14,679 ± 380 (CV 2.6%) (n=7)877.2 ± 24.1 (CV 2.7%) (n=7)28.22 ± 0.67 (CV 2.4%) (n=7)228.7 ± 3.6 (CV 1.6%) (n=7)240.2 ± 3.9 (CV 1.6%) (n=7)520.2 ± 7.9 (CV 1.5%) (n=7)52.63 ± 0.46 (CV 0.88%) (n=7)0 ± 0 (n=7)0 ± 0 (n=7)

Charts

Bar = mean across iterations. Error bar = sample standard deviation. Runtimes that didn't produce a metric are omitted from that chart.

Time to first request
Load test throughput
RSS at startup
RSS during load test
Build time
Per-iteration raw values
RuntimeMetriciter 0iter 1iter 2iter 3iter 4iter 5iter 6
dotnet-aspnet-efThroughput6,7436,5436,4466,5866,5106,4096,475
dotnet-aspnet-efTTFR1,434.51,349.01,362.81,325.31,432.41,426.71,434.4
dotnet-aspnet-efrps / MiB30.4829.4229.5929.1728.6228.4129.10
dotnet-aspnet-efRSS @ startup74.875.074.974.974.674.774.9
dotnet-aspnet-efRSS @ 1st req133.1133.5133.2133.3133.0132.9133.2
dotnet-aspnet-efRSS under load221.2222.4217.8225.8227.4225.6222.5
dotnet-aspnet-efBuild time5.611.721.621.591.611.651.61
dotnet-aspnet-efConn err0000000
dotnet-aspnet-efTimeouts0000000
quarkus-jvmThroughput12,35611,72212,03711,68511,95012,55111,862
quarkus-jvmTTFR2,635.42,650.22,689.42,502.92,789.92,631.42,564.6
quarkus-jvmrps / MiB26.7726.0325.6026.8727.3527.7326.78
quarkus-jvmRSS @ startup228.6227.4230.1229.1233.5233.5225.4
quarkus-jvmRSS @ 1st req247.6243.9249.5243.5250.9253.4242.6
quarkus-jvmRSS under load461.5450.3470.2434.9436.9452.6442.9
quarkus-jvmBuild time8.488.218.298.388.258.048.07
quarkus-jvmConn err0000000
quarkus-jvmTimeouts0000000
quarkus-leydenThroughput10,40410,47010,70410,57910,72310,29810,983
quarkus-leydenTTFR1,036.11,045.3950.1980.4993.31,010.0977.9
quarkus-leydenrps / MiB25.3424.1525.8325.8325.3625.0026.99
quarkus-leydenRSS @ startup212.8212.6219.4221.2230.7224.6222.8
quarkus-leydenRSS @ 1st req230.1238.1236.9236.2244.8249.3246.6
quarkus-leydenRSS under load410.5433.5414.3409.6422.8411.8406.9
quarkus-leydenBuild time52.8952.4552.9452.5452.6352.6052.64
quarkus-leydenConn err0000000
quarkus-leydenTimeouts0000000
quarkus-nativeThroughput7,5987,5287,6047,5927,6367,3617,521
quarkus-nativeTTFR100.5130.8114.1106.688.898.5117.8
quarkus-nativerps / MiB37.8737.5637.8037.3637.4036.0436.78
quarkus-nativeRSS @ startup89.289.389.389.289.389.389.3
quarkus-nativeRSS @ 1st req95.695.795.795.695.795.795.7
quarkus-nativeRSS under load200.6200.4201.2203.2204.2204.3204.5
quarkus-nativeBuild time154.89155.01154.49158.94154.30157.85157.23
quarkus-nativeConn err0000000
quarkus-nativeTimeouts0000000
quarkus-native-mandrelThroughput7,5527,4037,5387,8607,2167,4817,452
quarkus-native-mandrelTTFR89.0116.085.5120.797.1105.0117.5
quarkus-native-mandrelrps / MiB37.1635.7935.9738.7135.3537.4037.26
quarkus-native-mandrelRSS @ startup89.689.589.689.689.689.689.6
quarkus-native-mandrelRSS @ 1st req95.595.495.595.595.595.595.5
quarkus-native-mandrelRSS under load203.2206.8209.5203.1204.1200.0200.0
quarkus-native-mandrelBuild time156.16150.10153.30154.41153.38153.47152.19
quarkus-native-mandrelConn err0000000
quarkus-native-mandrelTimeouts0000000
quarkus-virtualThroughput15,75415,80316,29415,64315,18616,31615,598
quarkus-virtualTTFR2,621.92,602.42,555.42,686.42,698.42,495.12,634.2
quarkus-virtualrps / MiB28.4728.6528.7427.9026.9729.5628.00
quarkus-virtualRSS @ startup225.4228.8227.8234.2235.2227.8234.8
quarkus-virtualRSS @ 1st req243.1254.3253.6252.8254.1244.0254.3
quarkus-virtualRSS under load553.4551.6567.0560.8563.1552.0557.1
quarkus-virtualBuild time8.348.328.048.478.388.378.17
quarkus-virtualConn err0000000
quarkus-virtualTimeouts0000000
quarkus-virtual-leydenThroughput14,31814,77714,65614,13115,09114,60215,179
quarkus-virtual-leydenTTFR867.4913.4864.5852.0853.3888.2901.6
quarkus-virtual-leydenrps / MiB28.0528.7428.2226.8228.7128.4328.56
quarkus-virtual-leydenRSS @ startup229.8233.4225.3227.4225.3233.4226.1
quarkus-virtual-leydenRSS @ 1st req238.8241.7245.0239.5232.6241.1242.6
quarkus-virtual-leydenRSS under load510.4514.1519.3526.9525.6513.6531.4
quarkus-virtual-leydenBuild time52.7152.5652.2153.0253.1152.9651.85
quarkus-virtual-leydenConn err0000000
quarkus-virtual-leydenTimeouts0000000