retry option for flaky benchmark tests
Use the retry option in test() to automatically retry failing benchmark tests. This helps mitigate benchmark noise.
benchmarking/stability
10 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Example of retry option for benchmarks
test('performance comparison', { retry: 3 }, async ({ bench }) => { const result = await bench.compare( bench('lib1', () => { lib1() }), bench('lib2', () => { lib2() }), ) expect(result.get('lib1')).toBeFasterThan(result.get('lib2')) })
Benchmark stability considerations
Benchmarks are inherently flaky: CPU load, thermal throttling, GC pressure, and background processes all affect results. Vitest takes several steps to minimize this: benchmark files run in a separate project, tests within a benchmark file run sequentially, and benchmark files run one at a time.
Dead code elimination pitfall in benchmarks
JavaScript engines can optimize away code that has no observable side effects. If your benchmark function doesn't use its result, the engine may skip the computation entirely, producing misleadingly fast numbers. Consume the result in the benchmark function.
Example of dead code elimination pitfall
test('parsing', async ({ bench }) => { // BAD: the engine may eliminate the work await bench('parse', () => { JSON.parse(input) }).run() // GOOD: the result is consumed await bench('parse', () => { const result = JSON.parse(input) doSomething(result) }).run() })
Module runner overhead in benchmarks
By default, Vitest runs tests in Node.js using Vite's module runner which transforms all module exports into getters. This overhead is negligible in regular tests but can dominate measurements in benchmarks where functions are called millions of times.
Bypass getter overhead by storing reference locally
Store imported function references locally to bypass the getter wrapper when benchmarking. This avoids the overhead of every call to an imported function going through a getter.
Example of storing reference to bypass getter overhead
import { parse } from './parser.js' const _parse = parse test('parsing', async ({ bench }) => { // BAD: every call to `parse` goes through a getter await bench('parse', () => { parse(input) }).run() // GOOD: store the reference locally to bypass the getter await bench('parse', () => { _parse(input) }).run() })
Benchmark pre-built artifact instead of source
Import the library through its package name (which resolves to its built output) instead of reaching into its source. The built file has already collapsed internal imports into direct references, so Vite's module runner sees a single module with no internal getters.
Disable module runner for benchmarks with experimental.viteModuleRunner
Disable experimental.viteModuleRunner for the benchmark project if the benchmark does not need Vite transforms, mocks, or Vitest module interception. This allows Node to run native ESM directly without the getter overhead. This only affects Node.js mode.