benchmark/custom-provider

8 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Custom benchmark provider setup in config

Set the benchmark.provider configuration option to the path of your provider module. Relative paths are resolved from the project root. The module must have a default export with an object that implements BenchmarkProvider.

BenchmarkProvider.run() method signature and parameters

The provider.run() method is called when a registration's .run() method is called or once for all runnable registrations passed to bench.compare(). The group parameter contains: test (the test that registered the benchmarks, with test.context.signal aborted when run is cancelled), config (the resolved benchmark configuration for the current project), registrations (runnable benchmarks in registration order, each containing name, fn, and optional fnOpts for lifecycle hooks), and options (benchmark run options passed to .run() or bench.compare() if any).

Provider responsibilities for running benchmarks

The provider is responsible for honoring the run and registration options and for running every benchmark function and its beforeAll, beforeEach, afterEach, and afterAll hooks according to the benchmarking engine's lifecycle. If execution fails, throw the error to fail the test.

BenchResult return value requirements

The run method must resolve to one BenchResult for every runnable registration. Results are matched to registrations by name and are the source for .run() return values, comparison tables, reporters, and saved benchmark results. A custom engine must convert its measurements into the Tinybench-compatible BenchResult shape exported by vitest.

Registrations from bench.from() are not passed to provider

Registrations created by bench.from() are loaded by Vitest and are not passed to the provider.

Provider module lifetime and caching

Vitest imports the provider module on first use and caches its default export for the lifetime of the worker. The API does not have separate setup or teardown hooks; keep worker-scoped state on the provider object when needed.

Custom benchmark provider wrapping Tinybench example

Example showing how to wrap Tinybench in a custom provider: ```ts import type { BenchmarkProvider } from 'vitest' import { Bench } from 'tinybench' const provider = { async run({ test, config, registrations, options }) { const bench = new Bench({ signal: test.context.signal, retainSamples: config.retainSamples, ...options, }) for (const { name, fn, fnOpts } of registrations) { bench.add(name, fn, fnOpts) } await bench.run() return bench.tasks.map((task) => { const result = task.result if (result.state === 'errored') { throw result.error } if (result.state !== 'completed') { throw new Error(`Benchmark "${task.name}" ended in the "${result.state}" state`) } return { ...result, name: task.name, } }) }, } satisfies BenchmarkProvider export default provider ``` This demonstrates how registrations and results flow through a provider.

Vitest config example for custom benchmark provider

Example vitest.config.ts configuration to set a custom benchmark provider: ```ts import { defineConfig } from 'vitest/config' export default defineConfig({ test: { benchmark: { provider: './benchmark-provider.ts', }, }, }) ```

Give your agent this brain