benchmarking/persistence

8 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

writeResult option to save benchmark results

Pass writeResult as a per-bench option to write the result to a JSON file every time the benchmark runs. The path is resolved against the project root. The benchmark always runs and the file is overwritten on every successful run. If the function throws, the file is not written.

Example of writeResult option

test('parse', async ({ bench }) => { await bench( 'parse', { writeResult: './benchmarks/parse.json' }, () => parse(largeInput), ).run() })

bench.from() to read stored benchmark results

bench.from(name, source) is a registration that doesn't execute a function. It reads a previously stored result and feeds it into bench.compare() or returns it directly when you call .run(). The source can be a path (relative to the project root) or a function that returns the result data, including a Promise.

Example of bench.from() with multiple sources

test('compare against the stored baseline', async ({ bench }) => { const result = await bench.compare( bench( 'current', { writeResult: './benchmarks/parse.json' }, () => parse(largeInput), ), bench.from('previous', './benchmarks/parse.json'), bench.from('remote', () => fetch('https://path/to/external/file.json').then(r => r.json())), ) expect(result.get('current')).toBeFasterThan(result.get('previous')) })

Versioning benchmark artifacts

You can keep historical artifacts for older versions and compare them against the current implementation. Because bench.from() never invokes the function that produced the file, the original benchmark code can be deleted once the artifact is committed.

Example of comparing parser versions

test('compare parser versions', async ({ bench }) => { const input = '{"key":"value"}' await bench.compare( bench.from('v1', './benchmarks/parse.v1.json'), bench.from('v2', './benchmarks/parse.v2.json'), bench( 'current', { writeResult: './benchmarks/parse.current.json' }, () => customParser(input), ), ) })

Conditionally refresh or compare benchmark baseline

Gate the write behind an environment variable so the same test either refreshes the artifact or compares against it. Run with the environment variable set to refresh, and without it to compare.

Example of conditional benchmark baseline refresh

test('compare parser versions', async ({ bench }) => { if (import.meta.env.VITE_WRITE_BENCH) { const baseline = bench('baseline', { writeResult: './my-bench.json' }, () => fn()) await baseline.run() } else { const baseline = bench.from('baseline', './my-bench.json') await bench.compare(bench('current', () => fn()), baseline) } }) // Run VITE_WRITE_BENCH=1 vitest bench to refresh the stored result // Run vitest bench to compare the current implementation against it

Give your agent this brain