Writing Benchmarks

When to Use This Skill

Comparing implementations: Measuring old vs new approach after an optimization
Regression testing: Verifying a refactor doesn't degrade performance
Comparing with published version: Benchmarking workspace code against the latest published npm package

Do NOT Use This Skill For

General app-level performance optimization (use
```
jazz-performance
```
)
Profiling or debugging slow user-facing behavior

Directory Structure

All benchmarks live in the

bench/

directory at the repository root:

bench/
├── package.json              # Dependencies: cronometro, cojson, jazz-tools, vitest
├── jazz-tools/               # jazz-tools benchmarks
│   └── *.bench.ts

File Naming

Benchmark files follow the pattern:

<subject>.<operation>.bench.ts

Each file should focus on a single benchmark comparing multiple implementations (e.g.,

@latest

@workspace

Examples:

```
comap.create.jazz-tools.bench.ts
```
— benchmarks CoMap creation
```
filestream.getChunks.bench.ts
```
— benchmarks FileStream.getChunks()
```
filestream.asBase64.bench.ts
```
— benchmarks FileStream.asBase64()
```
binaryCoStream.write.bench.ts
```
— benchmarks binary stream writes

Benchmark Library: cronometro

Benchmarks use cronometro, which runs each test in an isolated worker thread for accurate measurement.

Basic Template

import cronometro from "cronometro";

const TOTAL_BYTES = 5 * 1024 * 1024;
let data: SomeType;

await cronometro(
  {
    "operation - @latest": {
      async before() {
        // Setup — runs once before the test iterations
        data = prepareTestData(TOTAL_BYTES);
      },
      test() {
        // The code being benchmarked — runs many times
        latestImplementation(data);
      },
      async after() {
        // Cleanup — runs once after all iterations
        cleanup();
      },
    },
    "operation - @workspace": {
      async before() {
        data = prepareTestData(TOTAL_BYTES);
      },
      test() {
        workspaceImplementation(data);
      },
      async after() {
        cleanup();
      },
    },
  },
  {
    iterations: 50,
    warmup: true,
    print: {
      colors: true,
      compare: true,
    },
    onTestError: (testName: string, error: unknown) => {
      console.error(`\nError in test "${testName}":`);
      console.error(error);
    },
  },
);

Single Cronometro Instance Per Benchmark

Each benchmark file should have a single
cronometro()
call that compares multiple implementations of the same operation. This makes results easier to read and compare:

import cronometro from "cronometro";

const TOTAL_BYTES = 5 * 1024 * 1024;
let data: InputType;

await cronometro(
  {
    "operationName - @latest": {
      async before() {
        data = generateInput(TOTAL_BYTES);
      },
      test() {
        latestImplementation(data);
      },
      async after() {
        cleanup();
      },
    },
    "operationName - @workspace": {
      async before() {
        data = generateInput(TOTAL_BYTES);
      },
      test() {
        workspaceImplementation(data);
      },
      async after() {
        cleanup();
      },
    },
  },
  {
    iterations: 50,
    warmup: true,
    print: { colors: true, compare: true },
    onTestError: (testName: string, error: unknown) => {
      console.error(`\nError in test "${testName}":`);
      console.error(error);
    },
  },
);

Key principles:

One file = one benchmark (e.g.,
```
getChunks
```
,
```
asBase64
```
,
```
write
```
)
One cronometro call comparing
```
@latest
```
vs
```
@workspace
```
(or old vs new)
Fixed data size at the top of the file (e.g.,
```
const TOTAL_BYTES = 5 * 1024 * 1024
```
)
Descriptive test names with format
```
"operation - @implementation"
```

Comparing workspace vs published package

To compare current workspace code against the latest published version:

1. Add npm aliases to
bench/package.json
:

json

{
  "dependencies": {
    "cojson": "workspace:*",
    "cojson-latest": "npm:cojson@0.20.7",
    "jazz-tools": "workspace:*",
    "jazz-tools-latest": "npm:jazz-tools@0.20.7"
  }
}

Then run

pnpm install

bench/

2. Import both versions:

import * as localTools from "jazz-tools";
import * as latestPublishedTools from "jazz-tools-latest";
import { WasmCrypto as LocalWasmCrypto } from "cojson/crypto/WasmCrypto";
import { WasmCrypto as LatestPublishedWasmCrypto } from "cojson-latest/crypto/WasmCrypto";

3. Use
@ts-expect-error
when passing the published package since the types won't match the workspace version:

ctx = await createContext(
  // @ts-expect-error version mismatch
  latestPublishedTools,
  LatestPublishedWasmCrypto,
);

Benchmarking with a Jazz context

When benchmarking CoValues (not standalone functions), create a full Jazz context. Use this helper pattern:

async function createContext(tools: typeof localTools, wasmCrypto: typeof LocalWasmCrypto) {
  const ctx = await tools.createJazzContextForNewAccount({
    creationProps: { name: "Bench Account" },
    peers: [],
    crypto: await wasmCrypto.create(),
    sessionProvider: new tools.MockSessionProvider(),
  });
  return { account: ctx.account, node: ctx.node };
}

Key points:

Pass
```
peers: []
```
— benchmarks don't need network sync
Use
```
MockSessionProvider
```
— avoids real session persistence

Call

(ctx.node as any).gracefulShutdown()

after()

to clean up

Test data strategy

Define a fixed data size constant at the top of the file, then generate test data inside the

before

hook:

const TOTAL_BYTES = 5 * 1024 * 1024; // 5MB

let chunks: Uint8Array[];

await cronometro({
  "operationName - @workspace": {
    async before() {
      chunks = makeChunks(TOTAL_BYTES, CHUNK_SIZE);
    },
    test() {
      doWork(chunks);
    },
  },
}, options);

Choose a size large enough to measure meaningfully. Small data (e.g., 100KB) may complete so fast that measurement noise dominates. 5MB is typically a good default for file/stream operations.

All fixture generation must be done inside the
before
hook, not at module level. This ensures data is created in the same worker thread that runs the test.

Running Benchmarks

Add a script entry to

bench/package.json

json

{
  "scripts": {
    "bench:mytest": "node --experimental-strip-types --no-warnings ./jazz-tools/mytest.jazz-tools.bench.ts"
  }
}

Then run from the

bench/

directory:

cd bench
pnpm run bench:mytest

Critical Gotchas

1. Use

node --experimental-strip-types

, NOT

tsx

Cronometro spawns worker threads that re-import the benchmark file. Workers don't inherit tsx's custom ESM loader, so the TypeScript import fails silently and the benchmark hangs forever.

Use

node --experimental-strip-types --no-warnings

instead:

json

"bench:foo": "node --experimental-strip-types --no-warnings ./jazz-tools/foo.bench.ts"

before

after

hooks MUST be

async

or accept a callback

Cronometro's lifecycle hooks expect either:

An async function (returns a Promise)
A function that accepts and calls a callback parameter

A plain synchronous function that does neither will silently prevent the test from ever starting, causing the benchmark to hang indefinitely:

// BAD — test never starts, benchmark hangs
{
  before() {
    data = generateInput();  // sync, no callback, no promise
  },
  test() { ... },
}

// GOOD — async function returns a Promise
{
  async before() {
    data = generateInput();
  },
  test() { ... },
}

// ALSO GOOD — callback style
{
  before(cb: () => void) {
    data = generateInput();
    cb();
  },
  test() { ... },
}

test()

can be sync or async

Unlike

before

after

, the

test

function works correctly as a plain synchronous function. Make it

async

only if the code under test is genuinely asynchronous.

4. TypeScript constraints under

--experimental-strip-types

Node's type stripping handles annotations,

as

casts, and

assertions. But it does not support:

```
enum
```
declarations (use
```
const
```
objects instead)
```
namespace
```
declarations
Parameter properties in constructors (
```
constructor(private x: number)
```
)
Legacy
```
import =
```
/
```
export =
```
syntax

Keep benchmark files to simple TypeScript that only uses type annotations, interfaces, type aliases, and casts.

Example: Full Benchmark

This example shows a benchmark comparing

getChunks()

between the published package and workspace code:

import cronometro from "cronometro";
import * as localTools from "jazz-tools";
import * as latestPublishedTools from "jazz-tools-latest";
import { WasmCrypto as LocalWasmCrypto } from "cojson/crypto/WasmCrypto";
import { cojsonInternals } from "cojson";
import { WasmCrypto as LatestPublishedWasmCrypto } from "cojson-latest/crypto/WasmCrypto";

const CHUNK_SIZE = cojsonInternals.TRANSACTION_CONFIG.MAX_RECOMMENDED_TX_SIZE;
const TOTAL_BYTES = 5 * 1024 * 1024;

function makeChunks(totalBytes: number, chunkSize: number): Uint8Array[] {
  const chunks: Uint8Array[] = [];
  let remaining = totalBytes;
  while (remaining > 0) {
    const size = Math.min(chunkSize, remaining);
    const chunk = new Uint8Array(size);
    for (let i = 0; i < size; i++) {
      chunk[i] = Math.floor(Math.random() * 256);
    }
    chunks.push(chunk);
    remaining -= size;
  }
  return chunks;
}

type Tools = typeof localTools;

async function createContext(tools: Tools, wasmCrypto: typeof LocalWasmCrypto) {
  const ctx = await tools.createJazzContextForNewAccount({
    creationProps: { name: "Bench Account" },
    peers: [],
    crypto: await wasmCrypto.create(),
    sessionProvider: new tools.MockSessionProvider(),
  });
  return { account: ctx.account, node: ctx.node, FileStream: tools.FileStream };
}

function populateStream(ctx: Awaited<ReturnType<typeof createContext>>, chunks: Uint8Array[]) {
  let totalBytes = 0;
  for (const c of chunks) totalBytes += c.length;
  const stream = ctx.FileStream.create({ owner: ctx.account });
  stream.start({ mimeType: "application/octet-stream", totalSizeBytes: totalBytes });
  for (const chunk of chunks) stream.push(chunk);
  stream.end();
  return stream;
}

const benchOptions = {
  iterations: 50,
  warmup: true,
  print: { colors: true, compare: true },
  onTestError: (testName: string, error: unknown) => {
    console.error(`\nError in test "${testName}":`);
    console.error(error);
  },
};

let readCtx: Awaited<ReturnType<typeof createContext>>;
let readStream: ReturnType<typeof populateStream>;

await cronometro(
  {
    "getChunks - @latest": {
      async before() {
        readCtx = await createContext(
          // @ts-expect-error version mismatch
          latestPublishedTools,
          LatestPublishedWasmCrypto,
        );
        readStream = populateStream(readCtx, makeChunks(TOTAL_BYTES, CHUNK_SIZE));
      },
      test() {
        readStream.getChunks();
      },
      async after() {
        (readCtx.node as any).gracefulShutdown();
      },
    },
    "getChunks - @workspace": {
      async before() {
        readCtx = await createContext(localTools, LocalWasmCrypto);
        readStream = populateStream(readCtx, makeChunks(TOTAL_BYTES, CHUNK_SIZE));
      },
      test() {
        readStream.getChunks();
      },
      async after() {
        (readCtx.node as any).gracefulShutdown();
      },
    },
  },
  benchOptions,
);

Checklist

One benchmark file per operation (e.g.,
```
filestream.getChunks.bench.ts
```
)
Single
```
cronometro()
```
call comparing
```
@latest
```
vs
```
@workspace
```
Fixed data size constant at top of file (e.g.,
```
const TOTAL_BYTES = 5 * 1024 * 1024
```
)
Benchmark file placed in
```
bench/jazz-tools/
```
with
```
*.bench.ts
```
naming

Script added to

bench/package.json

using

node --experimental-strip-types --no-warnings

```
before
```
/
```
after
```
hooks are
```
async
```
(not plain sync)
```
iterations
```
set to at least 50 for stable results
```
warmup: true
```
enabled
```
onTestError
```
handler included to surface worker failures

Test names follow format

"operation - @implementation"

(e.g.,

"getChunks - @workspace"

)

When comparing vs published: npm aliases added to
```
bench/package.json
```
and
```
pnpm install
```
run
When using Jazz context:
```
gracefulShutdown()
```
called in
```
after()
```
hook
Test data generated inside
```
before()
```
hooks (not at module level or inside
```
test()
```
)

benchmarking

NPX Install

Tags

SKILL.md Content

Writing Benchmarks

When to Use This Skill

Do NOT Use This Skill For

Directory Structure

File Naming

Benchmark Library: cronometro

Basic Template

Single Cronometro Instance Per Benchmark

Comparing workspace vs published package

Benchmarking with a Jazz context

Test data strategy

Running Benchmarks

Critical Gotchas

1. Use
`node --experimental-strip-types`
, NOT
`tsx`

2.
`before`
/
`after`
hooks MUST be
`async`
or accept a callback

3.
`test()`
can be sync or async

4. TypeScript constraints under
`--experimental-strip-types`

Example: Full Benchmark

Checklist

benchmarking

NPX Install

Tags

SKILL.md Content

Writing Benchmarks

When to Use This Skill

Do NOT Use This Skill For

Directory Structure

File Naming

Benchmark Library: cronometro

Basic Template

Single Cronometro Instance Per Benchmark

Comparing workspace vs published package

Benchmarking with a Jazz context

Test data strategy

Running Benchmarks

Critical Gotchas

1. Use node --experimental-strip-types, NOT tsx

2. before/after hooks MUST be async or accept a callback

3. test() can be sync or async

4. TypeScript constraints under --experimental-strip-types

Example: Full Benchmark

Checklist

1. Use
`node --experimental-strip-types`
, NOT
`tsx`

2.
`before`
/
`after`
hooks MUST be
`async`
or accept a callback

3.
`test()`
can be sync or async

4. TypeScript constraints under
`--experimental-strip-types`