{{ resultsReady ? 'Estimated model compute' : 'Inputs need attention' }}
{{ summaryValue }}

{{ summaryLine }}

GPU-hours {{ resultsReady ? fmt(values.gpu_hours) : '—' }} Days {{ resultsReady ? fmt(values.wall_days) : '—' }} Rental {{ resultsReady ? usd(values.cost_usd) : '—' }}
Billions of parameters. For MoE, include shared and attention weights in the active count.
B
Dense scaling comparisons are suppressed for mixture-of-experts models.
Billions of tokens processed, including repeats; 1,000B = 1T. 20N is a dense pretraining heuristic, not a quality target.
B
Dense Tensor Core peaks, without sparsity. H100, H200 and A100 specifications. FP8 needs compatible mixed-precision training.
Use the non-sparse peak for the precision your MFU refers to.
TFLOP/s
Whole GPUs. Timing assumes the same MFU at this cluster size; memory fit is not checked.
GPUs
{{ mfu_percent }}%
40% is illustrative. PaLM's MFU convention; this 6ND estimate omits explicit attention costs.
%
USD quote for one GPU. The default is an example, not a live market price.
USD/GPU-h
0 uses this run’s GPU-hours. A positive budget drives the Budget capacity scenarios.
GPU-h
0 omits energy. Use average measured draw, not MFU × TDP.
1 adds no overhead. GPU energy × PUE excludes the host, network and storage IT load.

{{ firstIssueMessage }}

Budget itemEstimateUnitBasisCopy
{{ row[0] }}{{ row[1] }}{{ row[2] }}{{ row[3] }}
Loading chart…

Chart unavailable. Your run budget is still available.

Budget (GPU-h)Tokens at your N (B)20N model (B)20N tokens (B)Rental (USD)Copy
{{ fmt(row.gpu_hours) }}{{ fmt(row.fixed_tokens_b) }}{{ fmt(row.balanced_parameters_b) }}{{ fmt(row.balanced_tokens_b) }}{{ usd(row.cost_usd) }}
Training arithmetic
C ≈ 6ND · GPU-hours = C / (peak × MFU × 3,600)
  1. {{ row.label }}{{ row.value }}

Similar tools

Training a dense language model with twice as many parameters on the same number of tokens roughly doubles its main arithmetic workload. Doubling both the model size and the token count raises that workload about fourfold. A useful training budget therefore starts with how much learning work is planned, before assigning a GPU cluster or promising a finish date.

Parameters are the learned weights of the model. Tokens are the units of text processed during training, and the training total counts repeated passes through the same data again. For transformer pretraining, a common approximation allows about six floating-point operations per parameter per token, covering a forward pass and backpropagation. For continued pretraining, the workload covers the additional tokens used to train an existing model.

Training FLOPs
The estimated number of floating-point operations needed for the planned run. This measures work, while FLOP/s measures the rate at which work is performed.
GPU-hours
The sum of time used across all GPUs. Eight GPUs running for ten hours consume 80 GPU-hours.
Elapsed training time
The time from starting to finishing the run. Sharing work across more GPUs can shorten this duration without reducing the total GPU-hours.

Peak accelerator throughput is a hardware rating. Model FLOPs utilization, or MFU, describes the fraction of that peak delivered as useful model-training work. The training setup, numerical precision, and communication between GPUs affect sustained performance. Adding GPUs shortens the schedule proportionally only while the same MFU can be maintained. Rental spending follows the total GPU-hours and the quoted price for each hour.

With a dense transformer, the parameter count represents the model whose weights participate in the computation. A mixture-of-experts model, or MoE, activates selected experts for each token. Its approximate arithmetic budget therefore uses the active parameters per token, including shared and attention projection weights. The much larger collection of stored expert weights still matters when planning memory capacity.

Model size and training length also compete for the same compute budget. Compute-optimal dense pretraining aims for the lowest language-model loss within a fixed training compute budget. Roughly twenty training tokens per parameter is a starting guide from research into that allocation. It is not a minimum dataset size or a guaranteed quality target. Training a smaller model for longer can be deliberate when its later inference cost matters. MoE architectures require their own allocation analysis.

These estimates help compare a proposed pretraining run, an extended token budget, or a shorter delivery schedule. They cannot establish the quality of the training data, predict the finished model's capabilities, or confirm that the model and its training state fit in GPU memory.

How to Use This Tool:

Start with the model and token plan, then replace the example hardware assumptions with those for the cluster you intend to use. Results recalculate as you edit.

  1. Choose Architecture and enter the parameter count in billions. Dense transformer uses Model parameters; Mixture of experts (MoE) uses Active parameters per token, including shared and attention weights.
  2. Enter Training tokens in billions, including repeats. For a dense comparison, Use 20 tokens per parameter replaces the token count with the reference amount. That action is unavailable in MoE mode. A trillion tokens is entered as 1,000B.
  3. Select GPU and precision. If the accelerator is not listed, choose Custom dense peak and enter its non-sparse throughput in Dense peak per GPU.
  4. Enter GPU count, Model FLOPs utilization, and Price per GPU-hour. The starting 40% MFU and $2 per GPU-hour are illustrative assumptions. Replace them with a relevant measured MFU or planning estimate and your USD quote for one GPU.
  5. Open Advanced if you want to compare a different GPU-hour allocation or add energy estimates. Alternative GPU-hour budget controls the capacity scenarios. Average power per GPU accepts watts or kilowatts, and Power usage effectiveness adds a facility-overhead multiplier to GPU energy.
  6. Review Estimated model compute and Run budget, then compare the schedule in MFU sensitivity before using a deadline row for an allocation request.
    If the summary says Inputs need attention, correct the field identified in its message. Invalid optional settings leave the main estimate available: an invalid alternative budget uses the current run's GPU-hours, invalid power omits energy, and invalid PUE uses 1. Check the warning under Advanced before relying on those secondary results.

Interpreting Results:

Changing only GPU count changes estimated days and cluster token throughput. Model compute, total GPU-hours, and rental remain unchanged because MFU is held fixed. Before accepting a 7-, 30-, or 90-day GPU request, check memory requirements and the proposed distribution of work, then measure throughput on a representative cluster.

MFU sensitivity shows the schedule at 10% through 80% in ten-percentage-point steps, plus your exact MFU if it differs. These are alternative assumptions, not a measured confidence interval. Cluster throughput is likewise an estimate of training tokens per second, not an inference benchmark.

In Budget capacity, compare the token count for your existing model with the dense-only alternative that resizes both model and training length to twenty tokens per parameter. These are separate plans for the same GPU-hour allocation. Changing the alternative budget leaves the current run's token count and main estimate unchanged.

Technical Details:

The dominant parameter-dependent work in transformer training comes from matrix multiplication. Counting a multiplication and an addition as two operations gives roughly two operations per parameter per token for the forward pass. Backpropagation computes gradients for both inputs and weights, contributing roughly four more.

Formula Core

The operation estimate becomes a time estimate when divided by sustained model throughput. In these equations, C is training FLOPs, Peff is useful FLOP/s per GPU, H is total GPU-hours, t is elapsed days, and K is GPU rental in USD.

C≈6×N×D Peff=P×u H=CP×u×3600 t=H24×G K=H×r

The factors 3,600 and 24 convert seconds to hours and hours to days. Parameter and token entries use decimal billions, and accelerator throughput uses decimal trillions of operations per second.

Symbol Quantity Value used in the equations
N Model parameters, or active parameters per token for MoE The entered billions multiplied by 109
D Training tokens processed The entered billions multiplied by 109
P Dense peak throughput per GPU TFLOP/s multiplied by 1012
u Model FLOPs utilization MFU percentage divided by 100
G GPU count A positive whole number of GPUs
r Price per GPU-hour USD for one GPU running for one hour

Precision-specific throughput

The named GPU choices use the approximate non-sparse Tensor Core peaks below. The H100 and H200 values approximate half the vendor’s sparse ratings, without assuming a sparsity benefit. BF16 is a 16-bit floating-point format; FP8 uses 8-bit formats for supported arithmetic.

GPU BF16 TFLOP/s FP8 TFLOP/s
A100 SXM 80GB 312 Not offered
H100 SXM 80GB 989 1,979
H200 SXM 141GB 989 1,979

H100 and H200 give the same estimates at the same precision and other settings because their assumed arithmetic peaks are equal. Their memory differences are not modeled. FP8 requires compatible mixed-precision training and an MFU measured against the FP8 peak; selecting it does not establish twice the actual training speed.

For the starting dense run of 7B parameters and 140B tokens, an H100 SXM BF16 peak of 989 TFLOP/s at 40% MFU gives 395.6 effective TFLOP/s per GPU. Substitution gives the estimated work and aggregate GPU time below.

C≈6×(7×109)×(140×109)=5.88×1021 FLOPs H=5.88×1021989×1012×0.40×3600≈4128.75 GPU-hours

Dividing the unrounded GPU-hours across 64 GPUs gives about 2.69 days. At the illustrative rate of $2 per GPU-hour, the GPU rental estimate is $8,257.50. These are consequences of the stated assumptions, not measurements of a completed training run.

MFU, recomputation, and omitted arithmetic

MFU counts required model work relative to peak throughput. Activation recomputation repeats calculations to save memory, but those repeated operations do not count as additional useful model work. Their effect on elapsed time is already reflected in a measured MFU. Consequently, adding a separate recomputation multiplier while retaining that MFU would count the same penalty twice.

The parameter count includes attention projection weights. The six-operation approximation omits the additional attention arithmetic that depends on sequence length, and the MoE approximation adds no explicit routing term. An MFU taken from another report must use a compatible FLOP-counting convention and the same precision denominator. A GPU activity percentage alone does not establish this model-throughput ratio.

Token throughput and deadline requests

Dividing aggregate useful throughput by the approximate work per token gives training tokens per second, q. A deadline of d days requires the estimated GPU-hours to fit within that many days of continuous operation.

q=G×P×u6×N Grequest=max(1,⌈H24×d⌉)

The ceiling brackets mean round upward to a whole GPU, with a minimum of one. The available deadline comparisons use 7, 30, and 90 days. The 7B example needs 25 GPUs for a seven-day deadline under unchanged MFU; this is an arithmetic allocation, not a verified memory or parallelism requirement.

Dense token reference

The twenty-token reference scales the training token count with the dense parameter count. The tokens-per-parameter ratio describes training length, while the reference multiple compares it with twenty tokens per parameter.

Dref=20×N Tokens per parameter=DN Reference multiple=D20×N

A reference multiple below one means fewer tokens than this guide, exactly one means the same count, and above one means more. These comparisons have no pass/fail or model-quality threshold. They do not account for the value of repeated data, domain adaptation, or the intended inference workload.

Allocating a GPU-hour budget

Let Hb be a GPU-hour allocation. A positive alternative budget supplies it; zero uses the current run's estimated GPU-hours. Half, full, and double allocations are compared at the same peak throughput and MFU. Their compute allowance, Cb, determines how many tokens, Db, fit at a fixed parameter count.

Cb=3600×Hb×P×u Db=Cb6×N

If both the dense model size and token count may change, imposing twenty tokens per parameter gives a different allocation. The factor 120 comes from multiplying the six-operation estimate by the twenty-token condition.

N20=Cb120 D20=20×N20

The resulting parameter and token counts are divided by a billion for display. Along this conditional allocation, quadrupling compute doubles both quantities. It is not a search for the largest model that fits in memory or a prediction of the best architecture for that budget. No corresponding dense allocation is calculated for MoE.

GPU energy and facility overhead

For average power w in watts per GPU, the summed GPU-hours determine GPU energy in kilowatt-hours. Power usage effectiveness, or PUE, is the ratio of total data-center energy to IT-equipment energy. Applying it to GPU energy provides only the GPU contribution with a proportional overhead allowance.

EGPU=H×w1000 Eadjusted=EGPU×PUE

Both energy results are in kWh. GPU-hours already include every GPU, so multiplying by GPU count again would overcount energy. Use average draw for the planned workload, rather than multiplying a thermal-design power rating by MFU. Switching the power unit preserves the quantity: 700 W is 0.7 kW.

Zero power omits the energy results; it does not represent a run that consumes no electricity. PUE of 1 adds no overhead. Host, network, and storage IT energy remain excluded, so PUE-adjusted GPU energy is not a complete facility-energy estimate. Energy is not converted into an electricity bill or added to the GPU rental estimate.

Accepted ranges and displayed precision

The ranges below are inclusive entry limits. They describe accepted numbers, not recommendations or proof that a cluster is feasible. Required numeric entries must be finite and nonblank; GPU count must also be a whole number.

Quantity Minimum Maximum
Parameters, billions0.001100,000
Training tokens, billions0.00110,000,000
Custom dense peak, TFLOP/s per GPU0.001100,000
GPU count, whole GPUs11,000,000
MFU, percent1100
Price, USD per GPU-hour01,000
Alternative budget, GPU-hours01,000,000,000,000
Average GPU power, W010,000
PUE13

The exact MFU entry accepts decimals across 1% to 100%; the quick slider covers 10% to 80% in whole percentage points. The power ceiling is equivalently 10 kW. Custom peak throughput affects the calculation only when the custom hardware choice is selected.

Calculations retain full floating-point intermediate precision and round for presentation. Most values display up to two decimal places; nonzero values below 0.01 use three significant figures. Total FLOPs use scientific notation with four significant figures, and USD amounts show cents. Deadline GPU counts round upward before display. Small costs can therefore display as $0.00 even when positive. Rounded output does not imply equivalent accuracy in the underlying assumptions.

Limitations and Privacy:

The estimate covers the proposed model-training workload. A complete project plan needs additional allowances for work and charges that are not represented by the entered assumptions.

  • Data preparation, evaluation, separate experimental runs, failed runs, recovery, and checkpoint writing receive no separate time allowance. Include any time outside the interval represented by your measured MFU when planning the schedule.
  • Communication delays and other execution overhead affect the answer only through the chosen MFU. There is no prediction of how those costs change with cluster size, sequence length, or expert routing.
  • Rental is the entered GPU-hour price multiplied by estimated GPU-hours. Separate storage, host, network, data-transfer, tax, and provider billing-rounding charges are not added. A zero rate makes only this rental line zero.

Arithmetic runs in your browser using the entered numbers; no model weights or dataset upload is needed. Edited settings are carried in the page URL and can be restored when it is reopened, so sharing that URL shares the assumptions it contains. Displaying the page and its chart can still require network access, including an external chart resource.

Worked Examples:

A 1,000 GPU-hour allocation

Keep a 7B dense model, H100 SXM BF16 at 989 TFLOP/s, 40% MFU, and a $2 per GPU-hour quote, then set Alternative GPU-hour budget to 1,000. The full-budget row in Budget capacity funds approximately 33.91B tokens at the existing 7B model size for $2,000. If model size may change, the alternative twenty-token allocation is approximately 3.44B parameters trained on 68.90B tokens. Choose between those plans before revising the proposed run; entering the alternative budget alone leaves its main estimate unchanged.

A lower measured utilization

For the 7B model trained on 140B tokens with 64 H100 SXM BF16 GPUs, reducing MFU from 40% to 20% increases estimated duration from 2.69 to 5.38 days. At the same $2 rate, rental rises from $8,257.50 to $16,515.00. Increasing the cluster to 128 GPUs would restore approximately 2.69 days if 20% MFU holds, while rental stays at $16,515.00. Meeting the original date therefore does not recover the original budget; the throughput assumption needs to be settled before requesting the allocation.

References: