Test Shard Runtime Calculator

Target {{ targetBadge }} Waves {{ waveBadge }} Speedup {{ speedupBadge }} Tail {{ tailBadge }}

Suite name:

Identify one CI test job or suite.

Total tests:

Use the same unit your sharding tool distributes.

Average test time:

Use test execution time when setup is modeled separately.

sec/test

Per-shard setup time:

Repeated setup can erase gains from adding shards.

sec/shard

Shards:

Model the exact shard total you plan to run.

shards

Max concurrent jobs:

Shard counts above this limit run in queued waves.

jobs

Split balance:

Choose the closest match to the production splitter.

Timing coverage:

Use a lower value after rewrites, new tests, or stale timing files.

Runner efficiency:

Use 100% for ideal isolated runners and reduce it for contention.

Target wall time:

The target drives status, audit guidance, and the runtime chart line.

min

Extra imbalance buffer:

Neutral default: 0%.

Retry or rerun uplift:

Neutral default: 0%; leave it there when averages already include retries.

Serial setup time:

Neutral default: 0 seconds.

sec

Report merge time:

Neutral default: 0 seconds.

sec

Metric	Value	Detail	Copy
{{ row.cells[0] }}	{{ row.cells[1] }}	{{ row.cells[2] }}

Check	Status	Action	Copy
{{ row.cells[0] }}	{{ row.cells[1] }}	{{ row.cells[2] }}

Shards	Active jobs	Waves	Wall time	Runner minutes	Speedup	Note	Copy
{{ cell }}

The chart renderer is unavailable. The same values remain in Shard scenarios.

The chart renderer is unavailable. Runner-minute values remain in Shard scenarios.

Splitting a test suite across several jobs can shorten continuous integration (CI) feedback, but shard count is not the same as useful parallelism. Each shard repeats setup work, the slowest shard determines when its wave ends, and jobs above the available concurrency wait for a later wave.

Wall time and runner use answer different planning questions. Wall time measures how long the test stage delays a developer or deployment. Runner minutes add the occupied time across all shard jobs. More shards may reduce the first measure while increasing the second through repeated dependency setup, image pulls, service startup, and test-runner initialization.

Shard: One partition of the suite assigned to a separate test job.
Wave: A group of shard jobs that can run concurrently; excess shards wait for the next wave.
Slowest-shard tail: Extra time carried by the least balanced shard beyond the modeled average shard.
Setup tax: The share of the modeled slowest shard job spent on repeated per-shard setup.

Balance quality often matters more than adding another worker. Historical test timings can distribute long and short tests more evenly, while count- or name-based splits ignore runtime differences. Timing data loses value after major rewrites, new tests, changed fixtures, or missing timing records, so the slowest shard needs a larger allowance when coverage is incomplete.

Runner efficiency represents contention and overhead inside each job. CPU throttling, shared caches, database bottlenecks, service limits, and noisy neighboring workloads can keep twelve nominal runners from delivering twelve units of test throughput. Retry uplift adds expected rerun work, while serial setup and report merging remain outside the parallel waves.

A runtime estimate is a capacity model, not a queue forecast. It is most useful when calibrated against recent CI runs from the same suite, runner type, split method, and cache policy. Recheck the assumptions whenever the test topology or execution environment changes.

How to Use This Tool:

Model one CI suite or job at a time with recent timing data from the runner class you plan to use.

Enter Total tests, Average test time, and Per-shard setup time. Keep setup separate from test execution time.
Set Shards and Max concurrent jobs. When shards exceed concurrency, the estimate adds queued waves.
Choose the closest Split balance, then set Timing coverage, Runner efficiency, and any extra imbalance. Use lower timing coverage after substantial suite changes or when many tests lack history.
Enter a Target wall time. Use Advanced only when retries, serial setup, or report merging contribute material time.
Compare modeled wall time, waves, setup tax, and runner-minute cost across shard scenarios. Validate the preferred count with actual CI timings before changing production concurrency.

Interpreting Results:

Estimated wall time includes serial setup, every queued wave's modeled slowest shard, and report merge time. Runner-minute cost sums modeled slow-shard job time across all shards, so it can rise even while wall time falls.

Within target means modeled wall time is less than or equal to the entered target. It does not include external queue delay.
Queued waves means shard count is above maximum concurrency. Adding shards within the same wave may help; crossing into another wave can cause a step increase.
Fastest shards minimizes modeled wall time among the displayed scenarios. Cheapest target shards instead minimizes runner minutes among scenarios that meet the target.
A high Setup tax means repeated setup is consuming much of the slowest job. Reduce setup, reuse artifacts, or stop adding shards before assuming more concurrency will help.

Technical Details:

The runtime model starts with total retry-adjusted test work, divides it evenly across shards, reduces throughput for runner inefficiency, and inflates the average shard by a split-balance allowance. Every wave is assumed to last as long as that modeled slow shard.

Formula Core

Let N be test count, a average seconds per test, r retry uplift as a decimal, S shard count, and η runner efficiency as a decimal. Total test work W and average shard test time A are:

W = N a (1 + r) A = \frac{W}{S η}

If I is effective imbalance as a decimal and P is per-shard setup seconds, slow-shard job time J becomes:

J = P + A (1 + I)

With maximum concurrent jobs M, serial setup U, and report merge time G, wave count Q, wall time T, and runner minutes R are:

Q = ⌈ \frac{S}{M} ⌉ T = U + Q J + G R = \frac{S J + U + G}{60}

Speedup divides the serial baseline by T. Setup tax is 100 × P ÷ J. Target gap is target seconds minus T; positive values are under target and negative values are over.

Split Strategy Rule Core

Effective imbalance is a repo-authored planning allowance, not a CI-provider guarantee. Let b be the strategy base, c timing coverage in percent, p the coverage penalty, and e the extra imbalance entered by the user.

I_{%} = \min (400, b + (100 - c) p + e)

Repo-authored imbalance assumptions for each split strategy
Split strategy	Base imbalance	Penalty per missing coverage point
Timing-weighted split	5%	0.12%
File timing split	12%	0.18%
Count or name split	24%	0.08%
Manual or high-variance split	38%	0.05%

The model marks setup-heavy plans at setup tax of 35% or more. Scenario notes use a stricter 45% threshold after checking current shard count, target attainment, and queued waves. Calculations retain full precision; seconds, minutes, ratios, and percentages are rounded only for display.

Accuracy Notes:

The equations are deterministic, but the split imbalance and efficiency terms are planning assumptions. They do not model every source of CI variation.

External queue wait, runner provisioning jitter, cache hit rates, network variance, and provider throttling are excluded.
One unsplittable test file, fixture, database lock, or shared service can dominate a shard beyond the selected allowance.
Retries are spread uniformly as total uplift, even though flaky tests may cluster on one shard.
All waves use the same slow-shard duration, and dynamic work stealing is not represented.
Calibrate average test time, setup time, efficiency, and imbalance against recent measured runs before making a capacity commitment.

Worked Examples:

Twelve-way timing split

A 2,400-test suite averaging 1.8 seconds has 4,320 seconds of test work. With 12 shards, 12 concurrent jobs, 82% runner efficiency, 75 seconds of setup per shard, and 90% timing coverage, the timing-weighted allowance is 6.2%. The modeled slow shard job lasts about 541.2 seconds, so wall time is about 9.02 minutes and runner use is about 108.25 minutes. That is roughly 58.8 seconds inside a 10-minute target, before any external queue delay.

References:

Use dynamic test splitting, CircleCI Documentation.
Running variations of jobs in a workflow, GitHub Docs.