Testing Web Performance with Web Test Runner

When testing and measuring the performance of a web application or library, it can be tricky to ensure the right metrics are tested and tracked. Web Performance contains many different factors but the most concerns all under these four categories:

1. Time to first interaction - how fast it takes to start using the app
2. Render - how fast an element is updated it the UI
3. Bundle Size - How much code is required to load or update the UI
4. Jank - how smooth is the application or little to no lag/stuttering

Each facet of performance plays an important part in a high-performing web app. However, each requires a different approach to test and measure. In this post, we will learn how to use the @web/test-runner test runner and the web-test-runner-performance plugin to measure bundle and render performance in a basic JavaScript library.

Web Test Runner

The @web/test-runner package is a fantastic test runner to run and unit test modern JavaScript. The @web/test-runner package is part of the modern-web.dev project, which provides several high-quality tools for modern web development.

While @web/test-runner is a generic test runner for unit tests, it has a great plugin API for adding customized plugins for your test suite. To create some tests to measure performance, we will be using the web-test-runner-performance plugin.

Setup

To set up our tests, we will create a web-test-runner.config.mjs file, which will have the configuration setup for our tests. Next, we will need to install the following packages to our package.json:

• @web/test-runner
• @web/dev-server-esbuild
• @web/test-runner-playwright
• web-test-runner-performance

// web-test-runner.performance.mjs
import { defaultReporter } from '@web/test-runner';
import { esbuildPlugin } from '@web/dev-server-esbuild';
import { playwrightLauncher } from '@web/test-runner-playwright';

export default ({
  concurrency: 1,
  concurrentBrowsers: 1,
  nodeResolve: true,
  testsFinishTimeout: 20000,
  files: ['./src/*.performance.ts'],
  browsers: [playwrightLauncher({ product: 'chromium', launchOptions: { headless: false } })],
  reporters: [
    defaultReporter({ reportTestResults: true, reportTestProgress: true })
  ],
  plugins: [
    esbuildPlugin({ ts: true, json: true, target: 'auto', sourceMap: true })
  ],
});

Our performance tests should be run independently of any of our other tests, as well as having the concurrency and concurrentBrowsers options set to 1. This helps provide a more consistent test environment. The more parallel work running while the performance tests are running, the less accurate the measurements will be due to resource allocation by the operating system.

To ensure our configuration is appropriately set up, we can make a basic test.

//demo.performance.ts
import { expect } from '@esm-bundle/chai';

describe('basic test', () => {
  it(`should add`, async () => {
    expect(2 + 3).to.be(5);
  });
});

To run our tests, in the root directory of our web-test-runner.performance.mjs, we can run the command:

$ web-test-runner

Measuring Bundle Performance

Now that we have a basic test runner setup, we can create tests to measure bundle performance. First, to begin testing bundle sizes of our library, we need to add the bundle plugin:

// web-test-runner.performance.mjs
...
import { bundlePerformancePlugin } from 'web-test-runner-performance';

export default ({
  ...
  plugins: [
    bundlePerformancePlugin(),
  ]
});

Once the plugin is added, we can create a test that accepts an import module path of a library or local module.

// component.performance.js
import { expect } from '@esm-bundle/chai';
import { testBundleSize } from 'web-test-runner-performance/browser.js';

describe('performance', () => {
  it('should meet maximum css bundle size limits (0.2kb brotli)', async () => {
    expect((await testBundleSize('./demo-module/index.css')).kb).to.below(0.2);
  });

  it('should meet maximum js bundle size limits (0.78kb brotli)', async () => {
    expect((await testBundleSize('./demo-module/index.js')).kb).to.below(0.8);
  });
});

The web-test-runner-performance plugin will take the module's path, which can be JavaScript or CSS, and bundle/minify through rollup. Once bundled, it will measure the file size using Brotli compression and return a final file size in kilobytes. Bundle tests like these can help prevent performance regressions with third-party code and ensure tree shaking compatibility of modules.

Bundles can be multiple entry points by listing multiple imports at once.

// component.performance.js
import { expect } from '@esm-bundle/chai';
import { testBundleSize } from 'web-test-runner-performance/browser.js';

describe('performance', () => {
  it('should meet maximum js bundle size limits (2kb brotli)', async () => {
    const bundle = `
      import './demo-module/index.js';
      import './demo-module-two/index.js';
    `;
    expect((await testBundleSize(bundle)).kb).to.below(2);
  });
});

It gives the flexibility to test independent module bundle sizes or combinations of modules to determine what code is reused/shared within the bundle.

Measuring Render Performance

The renderPerformancePlugin will measure the render time of a given custom element in milliseconds. This measurement is from the time the element is added to the DOM to the first paint of the element. The measurement of time is determined using the Performance Timing API and the ResizeObserver API, combined giving us a relatively accurate timeframe of the first render.

// web-test-runner.performance.mjs
...
import { renderPerformancePlugin } from 'web-test-runner-performance';

export default ({
  ...
  plugins: [
    renderPerformancePlugin(),
    ...
  ],
});

Once the plugin is added, a test can check how quickly an element can be rendered.

// component.performance.js
import { expect } from '@esm-bundle/chai';
import { testBundleSize, testRenderTime, html } from 'web-test-runner-performance/browser.js';
import 'demo-module/my-element.js';

describe('performance', () => {
  it(`should meet maximum render time 1000 <p> below 50ms`, async () => {
    const result = await testRenderTime(html`<my-element>hello world</my-element>`, { iterations: 1000, average: 10 });
    expect(result.averages.length).to.eql(10);
    expect(result.duration).to.below(50);
  });
});

The testRenderTime takes a component template to render and will return how long the element took to render its first paint in milliseconds. The config accepts an iterations property to render the template n+ iterations. By default, the test runner will render three times and average the results. This can be customized using the average property.

By measuring render performance at a component level with a small and fast test, we can quickly narrow and isolate performance early on in development.

Reporting

While it's great to have tests to measure and prevent performance regressions, it is even better to track them over time. The web-test-runner-performance plugin provides a custom reporter to write a JSON file report of the test results. This file then can be easily logged or stored to track the performance of your library or application over time.

// web-test-runner.performance.mjs
import { playwrightLauncher } from '@web/test-runner-playwright';
import { esbuildPlugin } from '@web/dev-server-esbuild';
import { defaultReporter } from '@web/test-runner';
import { bundlePerformancePlugin, renderPerformancePlugin, performanceReporter } from 'web-test-runner-performance';

export default ({
  concurrency: 1,
  concurrentBrowsers: 1,
  nodeResolve: true,
  testsFinishTimeout: 20000,
  files: ['./src/*.performance.ts'],
  browsers: [playwrightLauncher({ product: 'chromium', launchOptions: { headless: false } })],
    reporters: [
    defaultReporter({ reportTestResults: true, reportTestProgress: true }),
    performanceReporter({ writePath: `${process.cwd()}/dist/performance` })
  ],
  plugins: [
    esbuildPlugin({ ts: true, json: true, target: 'auto', sourceMap: true }),
    renderPerformancePlugin(),
    bundlePerformancePlugin(),
  ]
});

// dist/performance/report.json
{
  "renderTimes": [
    {
      "testFile": "/src/index.performance.ts",
      "duration": 27.559999999999995
    }
  ],
  "bundleSizes": [
    {
      "testFile": "/src/index.performance.ts",
      "compression": "brotli",
      "kb": 0.193
    },
    {
      "testFile": "/src/index.performance.ts",
      "compression": "brotli",
      "kb": 0.78
    }
  ]
}

With the @web/test-runner and web-test-runner-performance packages, we can test for performance in very isolated small cases making it easier to track regressions during our development cycles. Check out the example in the repo below!