D1886R0.0
Error speed benchmarking

1. Introduction

2. Measuring methodology

Gathering representative timings for very small pieces of code is difficult on modern hardware. One type of micro-architectural quirk that makes these measurements very painful is code alignment [Bakhvalov]. Each call and each branch can introduce stalls if jumping to a poorly aligned location. These stalls can dominate the timings. The author has seen code alignment differences swing a benchmark running at 451 cycles per iteration to 310 cycles per iteration, for a 45% increase in execution time. This involved no system calls, no atomic operations, no divisions, or any other highly expensive operations being performed.

This research used a novel technique to get representative timings across a wide range of jump alignments. At the beginning of each function, a random number of nop instructions (0 to 31 inclusive) were inserted. A matching set of nop instructions were inserted at a later point in the execution path so that a total of 31 user inserted nop instructions were always executed. 1024 instances of this program, all with different layouts of nops, were then built and run.

Tests were built to optimize for speed. The test case optimizes away in the face of link-time optimizations (LTO) / whole program optimizations, so they were not used. Some compilers only support profile-guided optimizations (PGO) after turning on link-time optimizations, so PGO wasn’t used either. Only the exception cases were built with exceptions turned on.

For all but the exception failure cases, 16K warm-up iterations were run before starting timing. Within the timer bounds, 512K iterations of the test case were run. The exception failure cases only had 256 warm-up iterations and 16K timed iterations, due to just how slow throwing an exception can be.

All tests were run on the same machine. The processor was an Intel Core i7-3770 running at 3.3915 GHz. The Core i7-3770 has an Ivy Bridge microarchitecture, and was released in April 2012. C-states, Hyper-Threading, and Turbo Boost were all disabled in order to improve benchmark reproducibility. MSVC benchmarks were run on the 64-bit version of Windows 10. Clang and GCC benchmarks were run on the Slax Linux distribution. (TODO: verify)

3. Starter test case for on-line analysis

struct Dtor[N] {
  extern ~Dtor[N]() {
    /* Random number of NOPs K[N] := [0,31] */
  }
};
struct inline_nops {
  inline inline_nops() {
    /* Random number of NOPs Z := [0,31] */
  }
  inline ~inline_nops() {
    /* 31 - Z NOPs */
  }
};
int global_int = 0;
int callee(bool do_err) {
  inline_nops n;
  if(do_err)
    return 1;
  return 0;
}
int caller[N](bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(do_err);
  if(e) return e;
  global_int = 0;
  return 0;
}
int main() {
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](false);
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  /* Test sad path */
  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](true);
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

• callee() will raise an error based off of runtime state

• caller[N]() needs to clean up the d object in error and non-error conditions.

• caller[N]() should only set global_int in success cases.

• The inserted NOPs allow us to sample various alignments while keeping the same program structure.

In the actual tests all the function bodies are in separate .cpp files, and link-time / whole-program optimizations aren’t used. If they had been used, the entire program would get optimized away, removing our ability to measure error handling differences.

The initial part of this paper is only looking at three types of error handling.

• abort: When an error is encountered, kill the program with std::abort. No "sad" path benchmarks were taken for this case.

• return_val: Return an integer error code, where zero represents success.

• throw_exception: Throw an exception deriving from std::exception, that contains only an int. This should represent typical exception usage cases. The exception is only caught with a catch(...).

The actual code used for the benchmark can be found on my github.

4. Happy path measurements

Abort and x64 exceptions are consistently faster than return codes. Windows x86 exceptions are slower than both return codes and abort semantics. While return codes are slower than abort and x64 exceptions, they aren’t much slower. The cost ranges from less than a cycle per function to four cycles per function in my experiments.

4.1. Write to a global

Write to global: Median cycles per iteration. A histogram can be found here

4.2. Write through an argument

int callee(int *val, bool do_err) {
  (void)val;
  inline_nops n;
  if(do_err)
    return 1;
  return 0;
}
int caller[N](int *val, bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(val, do_err);
  if(e) return e;
  *val = 0;
  return 0;
}

Write through argument: Median cycles per iteration. A histogram can be found here

4.3. "Return" a value

int callee(int *val, bool do_err) {
  inline_nops n;
  if(do_err)
    return 1;
  *val = 0;
  return 0;
}
int caller[N](int *val, bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  int out_val = 0;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(&out_val, do_err);
  if(e) return e;
  *val = out_val + 1;
  return 0;
}

Return a value: Median cycles per iteration. A histogram can be found here

5. Sad path measurements

Sad path timings for abort aren’t shown, as the recovery path is very system specific. Throwing an exception is more than 100x slower than returning an error code.

5.1. Write to a global

Write to a global, sad path: Median cycles per iteration. A histogram can be found here

5.2. Write through an argument

Write through an argument, sad path: Median cycles per iteration. A histogram can be found here

5.3. "Return" a value

Return a value, sad path: Median cycles per iteration. A histogram can be found here

6. Off-line analysis

6.1. LLVM-MCA happy path off-line analysis

LLVM-MCA [MCA] is a tool that will analyze assembly and provide statistics on a hypothetical execution on a specific simulated CPU. This can be used to provide a hardware independent analysis of generated code. There is at least one substantial limitation though, in that calls and returns are typically not modeled. Regardless, the analysis is still useful.

I focused on analyzing the difference between exception builds and abort / no exception builds, all in an error neutral function. In the off-line analysis, I’m only focused on 64-bit x86-64 (Clang, GCC, and MSVC) and 64-bit ARM (Clang and GCC). Exceptions on 32-bit x86 on Windows is so much slower in the happy path that it isn’t interesting to dive in at the assembly level.

In general, there are some minor missed optimization opportunities in some of the smallest error handling examples.

I will be looking at this code snippet:

struct Dtor {~Dtor();};
void callee();
int global;
void caller() {
  Dtor d;
  callee();
  global = 0;
}

sub rsp, 40
call void callee(void)
; Note the ordering difference
mov DWORD PTR int global, 0
lea rcx, QWORD PTR d$[rsp]
call Dtor::~Dtor(void)
add rsp, 40
ret 0

sub rsp, 40
call void callee(void)
; Note the ordering difference
lea rcx, QWORD PTR d$[rsp]
mov DWORD PTR int global, 0
call Dtor::~Dtor(void)
add rsp, 40
ret 0

pushq %rbx
subq $16, %rsp
callq callee()
movl $0, global(%rip)
leaq 8(%rsp), %rdi
callq Dtor::~Dtor()
addq $16, %rsp
popq %rbx
retq

pushq %rax
; no subq here
callq callee()
movl $0, global(%rip)
movq %rsp, %rdi ; movq instead of leaq
callq Dtor::~Dtor()
; no addq here
popq %rax
retq

pushq %rbx
subq $16, %rsp
call callee()
; ordering difference
movl $0, global(%rip)
leaq 15(%rsp), %rdi
call Dtor::~Dtor()
addq $16, %rsp
popq %rbx
ret

; no pushq
subq $24, %rsp
call callee()
; ordering difference
leaq 15(%rsp), %rdi
movl $0, global(%rip)
call Dtor::~Dtor()
addq $24, %rsp
; no popq
ret

In my timings, GCC’s abort code was 2 cycles faster than GCC’s exception code. Clang’s abort code was 5 cycles faster than Clang’s exception code.

str x19, [sp, #-32]!
stp x29, x30, [sp, #16]
add x29, sp, #16
bl callee()
adrp x8, global
add x0, sp, #8
str wzr, [x8, :lo12:global]
bl Dtor::~Dtor()
ldp x29, x30, [sp, #16]
ldr x19, [sp], #32
ret

sub sp, sp, #32 // sub instead of str
stp x29, x30, [sp, #16]
add x29, sp, #16
bl callee()
adrp x8, global
add x0, sp, #8
str wzr, [x8, :lo12:global]
bl Dtor::~Dtor()
ldp x29, x30, [sp, #16]
add sp, sp, #32 // add instead of ldr
ret

stp x29, x30, [sp, -48]!
mov x29, sp
bl callee()
adrp x1, .LANCHOR0
add x0, sp, 40
str wzr, [x1, #:lo12:.LANCHOR0]
bl Dtor::~Dtor()
ldp x29, x30, [sp], 48
ret

stp x29, x30, [sp, -32]! // different constant
mov x29, sp
bl callee()
adrp x1, .LANCHOR0
add x0, sp, 24 // different constant
str wzr, [x1, #:lo12:.LANCHOR0]
bl Dtor::~Dtor()
ldp x29, x30, [sp], 32 // different constant
ret

6.2. Sad path affecting the happy path and macro-benchmark theorizing

struct Dtor {~Dtor();};
void callee();
int global;
void caller() {
  Dtor d;
  callee();
  global = 0;
}

push    rbx
sub     rsp, 16
call    callee()
mov     dword ptr [rip + global], 0
lea     rdi, [rsp + 8]
call    Dtor::~Dtor()
add     rsp, 16
pop     rbx
ret
; Begin landing pad
mov     rbx, rax
lea     rdi, [rsp + 8]
call    Dtor::~Dtor()
mov     rdi, rbx
call    _Unwind_Resume

push    rbx
sub     rsp, 16
call    callee()
lea     rdi, [rsp+15]
mov     DWORD PTR global[rip], 0
call    Dtor::~Dtor()
add     rsp, 16
pop     rbx
ret
; Begin landing pad
lea     rdi, [rsp+15]
mov     rbx, rax
call    Dtor::~Dtor()
mov     rdi, rbx
call    _Unwind_Resume

push    rbp
sub     rsp, 16
call    callee()
mov     DWORD PTR global[rip], 0
lea     rdi, [rsp+15]
call    Dtor::~Dtor()
add     rsp, 16
pop     rbp
ret
; Begin landing pad
mov     rbp, rax
jmp     .L2
caller() [clone .cold]:
.L2:
lea     rdi, [rsp+15]
call    Dtor::~Dtor()
mov     rdi, rbp
call    _Unwind_Resume

In all three cases, some cold landing pad code is directly adjacent to hot happy path code. This doesn’t cause any problems in microbenchmarks, but in theory, it could reduce instruction cache utilization in larger benchmarks. Starting in GCC 8.1, GCC will only put a tiny trampoline in the landing pad. The bulk of the landing pad code is moved into a different, cold location. This should, in theory, improve instruction cache utilization in the happy path compared to Clang and earlier GCCs, though it will still be at a minor disadvantage compared to terminate based implementations. I have no measurements to confirm or reject this theory though. [Spencer18] measured a small, non-zero performance degradation ( <1% ) in the happy path in a AAA game when turning on exceptions.

The MSVC x64 /d2FH4 implementation does not place any exception handling code near the happy path. The MSVC x86 sad path exception handling code is also far away from the happy path, but the happy path does exception bookkeeping.

7. Conclusion

8. Acknowledgments

Charts generated with [ECharts].

Appendix A: Additional error handling types

In each of these cases, the struct involved is of the following form:

struct error_struct {
  void *error = nullptr;
  void *domain = nullptr;
};

• ref_struct: Pass an error_struct by reference, and check error after each call.

• return_struct: Like return_val, but with an error_struct instead.

• return_nt_struct: Like return_struct, but error_struct has a destructor of the form ~error_struct() {}. "nt" means "non-trivial".

• expected_struct: This uses Sy Brand’s [Expected] library with error_struct as the error type. This should closely resemble the proposed std::expected.

• outcome_std_error: This uses a branch of Niall Douglas’s [Outcome] library. This branch is specially optimized for GCC.

In the sad path graphs, exceptions are omitted so that the remaining values can be easily compared.

Write to a global (happy path)

Write to global: Median cycles per iteration. A histogram can be found here

Write through an argument (happy path)

Write to global: Median cycles per iteration. A histogram can be found here

"Return" a value (happy path)

Write to global: Median cycles per iteration. A histogram can be found here

Write to a global (sad path)

Write to global: Median cycles per iteration. A histogram can be found here

Write through an argument (sad path)

Write to global: Median cycles per iteration. A histogram can be found here

"Return" a value (sad path)

Write to global: Median cycles per iteration. A histogram can be found here

Appendix B: The build flags

MSVC

/d2FH4 is a critical flag for these benchmarks. Without it, the results are drastically different. See [MoFH4] for a description of this flag.

Compiler marketing version: Visual Studio 2019

Compiler toolkit version: 14.22.27905

cl.exe version: 19.22.27905

Compiler codegen flags (no exceptions): /GR /Gy /Gw /O2 /MT /d2FH4 /std:c++latest /permissive- /DNDEBUG

Compiler codegen flags (with exceptions): /EHs /GR /Gy /Gw /O2 /MT /d2FH4 /std:c++latest /permissive- /DNDEBUG

Compiler codegen flags (outcome): /GR- /Gy /Gw /O2 /MT /d2FH4 /std:c++latest /permissive- /DNDEBUG

Linker flags: /OPT:REF /release /subsystem:CONSOLE /incremental:no /OPT:ICF /NXCOMPAT /DYNAMICBASE *.obj

Clang x64

• Clang 8.0.0 and libc++

• System linker from Ubuntu 14.04.3’s GCC 4.8.4 installation

Compiler codegen flags (no exceptions): -fno-exceptions -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -stdlib=libc++ -static -DNDEBUG

Compiler codegen flags (exceptions): -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -stdlib=libc++ -static -DNDEBUG

Compiler codegen flags (outcome): -fno-exceptions -fno-rtti -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -stdlib=libc++ -static -DNDEBUG

Linking flags: -Wl,--gc-sections -pthread -static -static-libgcc -stdlib=libc++ *.o libc++abi.a

GCC x64

Compiler codegen flags (no exceptions): -fno-exceptions -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -static -DNDEBUG

Compiler codegen flags (exceptions): -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -static -DNDEBUG

Compiler codegen flags (outcome): -fno-exceptions -fno-rtti -O2 -mtune=ivybridge -ffunction-sections -fdata-sections -std=c++17 -static -DNDEBUG

Linking flags: -Wl,--gc-sections -pthread -static -static-libgcc -static-libstdc++ *.o

Appendix C: The code

Common support code

struct Dtor[N] {
  extern ~Dtor[N]() {
    /* Random number of NOPs K[N] := [0,31] */
  }
};
struct inline_nops {
  inline inline_nops() {
    /* Random number of NOPs Z := [0,31] */
  }
  inline ~inline_nops() {
    /* 31 - Z NOPs */
  }
};

int main() {
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](false);
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](true);
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

class err_exception : public std::exception {
public:
  int val;
  explicit err_exception(int e) : val(e) {}
  const char *what() const noexcept override { return ""; }
};
int main() {
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    try { (void)caller[0](false); } catch(...){}
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  /* Test sad path */
  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    try { (void)caller[0](true); } catch(...){}
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

Write to a global

int global_int = 0;

int callee(bool do_err) {
  inline_nops n;
  if(do_err)
    return 1;
  return 0;
}
int caller[N](bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(do_err);
  if(e) return e;
  global_int = 0;
  return 0;
}

void caller[N](bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  NEXT_FUNC(do_err);
  global_int = 0;
}

void callee(bool do_err) {
  inline_nops n;
  if(do_err)
    abort();
}

void callee(bool do_err) {
  inline_nops n;
  if(do_err)
    throw err_exception(1);
}

Write through an argument

int main() {
  int val=0;
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](&val, false);
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  /* Test sad path */
  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    (void)caller[0](&val, true);
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

int callee(int *, bool do_err) {
  inline_nops n;
  if(do_err)
    return 1;
  return 0;
}
int caller[N](int *val, bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(val, do_err);
  if(e) return e;
  *val = 0;
  return 0;
}

void caller[N](int *val, bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  NEXT_FUNC(val, do_err);
  *val = 0;
}

void callee(int *, bool do_err) {
  inline_nops n;
  if(do_err)
    abort();
}

void callee(int *, bool do_err) {
  inline_nops n;
  if(do_err)
    throw err_exception(1);
}
int main() {
  int val=0;
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    try { (void)caller[0](&val, false); } catch(...) {}
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  /* Test sad path */
  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    try { (void)caller[0](&val, true); } catch(...) {}
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

"Return" a value

int callee(int *val, bool do_err) {
  inline_nops n;
  if(do_err)
    return 1;
  *val = 0;
  return 0;
}
int caller[N](int *val, bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  int out_val = 0;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  int e = NEXT_FUNC(&out_val, do_err);
  if(e) return e;
  *val = out_val + 1;
  return 0;
}
int main() {
  /* Test happy path */
  startTimer();
  /* X NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    int val=0;
    (void)caller[0](&val, false);
  }
  /* 31 - X NOPs */
  endTimer();
  print(duration() / ITERATIONS);

  /* Test sad path */
  startTimer();
  /* Y NOPs */
  for (int i = 0; i < ITERATIONS; ++i) {
    int val=0;
    (void)caller[0](&val, true);
  }
  /* 31 - Y NOPs */
  endTimer();
  print(duration() / ITERATIONS);
}

int caller[N](bool do_err) {
  /* 31 - K[N] NOPs */
  Dtor[N] d;
  /* NEXT_FUNC is callee when N == 7, or caller[N+1] otherwise */
  return 1 + NEXT_FUNC(do_err);
}

int callee(bool do_err) {
  inline_nops n;
  if(do_err)
    abort();
  return 0;
}

int callee(bool do_err) {
  inline_nops n;
  if(do_err)
    throw err_exception(1);
  return 0;
}

Appendix D: Histograms

Happy path: Write to a global

Write to global: Cycles per iteration histogram

Happy path: Write through an argument

Write through an argument: Cycles per iteration histogram

Happy path: "Return" a value

Return a value: Cycles per iteration histogram

Exception sad path: Write to a global

Write to a global: Cycles per iteration histogram. throw_exception and return_val only.

Exception sad path: Write through an argument

Write through an argument: Cycles per iteration histogram. throw_exception and return_val only.

Exception sad path: "Return" a value

Return a value: Cycles per iteration histogram. throw_exception and return_val only.

Other sad path: Write to a global

Write to a global: Cycles per iteration histogram. throw_exception omitted.

Other sad path: Write through an argument

Write through an argument: Cycles per iteration histogram. throw_exception omitted.

Other sad path: "Return" a value

Return a value: Cycles per iteration histogram. throw_exception omitted.