Log Sampling Rate Calculator

Introduction

Logging systems often receive far more routine events than a storage budget can retain. Request successes, health checks, debug messages, retries, and repeated application notices may dominate daily volume even though investigations depend on a much smaller set of errors, security records, audit events, or unusual traces.

Sampling reduces a chosen portion of that stream by keeping each eligible event with a probability. A 10% sampling rate means roughly one eligible event in ten is expected to remain over a large population. It does not promise one event from every block of ten, and it can miss a rare event entirely. Records that must be complete therefore need an always-keep rule before sampling is applied.

Daily volume depends on more than events per second. Average event size, traffic bursts, enrichment fields, indexing overhead, and the time basis of the rate all change the byte total. Measurements should come from a representative pre-sampling counter and a realistic encoded event size, not from a quiet interval or an uncompressed message sample.

A storage-fitting rate is a capacity plan rather than a complete observability policy. It can show how much of the sampleable stream fits after protected logs are reserved. It cannot decide which event classes are safe to sample, preserve related requests together, or guarantee enough evidence for incident response and compliance.

How to Use This Tool:

Measure the unsampled stream first, then reserve logs that must remain complete before solving the eligible sampling rate.

1. Enter Incoming log rate with the correct per-second, per-minute, per-hour, or per-day basis.
2. Enter the average encoded event size and the target retained volume per day. Keep the binary units consistent with the storage quote.
3. Set Full-fidelity reserve to the share that bypasses sampling, then add a burst multiplier and storage overhead when the measured average understates peak or indexed volume.
4. Read Sampleable keep rate, Overall keep rate, and the one-in-N guide together. A budget conflict means the protected share alone exceeds the target.
5. Translate the percentage into the actual logging system and validate retained bytes over a representative day before treating the target as stable.

Interpreting Results:

Sampleable keep rate applies only to logs outside the protected reserve. Overall keep rate includes both protected and sampled events, so it will be higher whenever some records bypass sampling. The retained and dropped volume estimates describe the modeled day after burst and overhead adjustments.

A low percentage is not automatically wrong, but it raises the chance that uncommon eligible events will be absent. Protect critical classes explicitly, and check whether related records require consistent sampling by request, trace, user, or another stable key.

Technical Details:

All rates are converted to events per second, sizes to bytes per event, and target capacity to bytes per day. The burst multiplier increases the modeled event rate, while overhead increases bytes after event count and average size are combined. Binary storage factors are used: 1 KiB is 1,024 bytes, 1 MiB is 1,024² bytes, and 1 GiB is 1,024³ bytes.

Formula Core

The protected share is reserved first. Only the remaining bytes are reduced by the solved sampling probability.

When raw volume already fits, r is capped at 1. When protected bytes exceed the target, the rate is capped at 0 and the result reports a budget conflict because sampling cannot remove protected records. One-in-N is the reciprocal of r whenever the rate is above zero. Full precision is kept through the calculation; the display setting rounds only visible values.

Rule Core

Accuracy Notes:

Expected volume is only as sound as the measured rate and size distribution. Large stack traces, payload fragments, enrichment fields, multiline events, and index replicas can make an average message sample too small.

• Measure before sampling and across a period that includes normal peaks.
• Keep compliance, security, billing, and incident-critical classes outside probabilistic sampling when completeness matters.
• Recalculate after parser, enrichment, indexing, routing, or retention policy changes.

Worked Examples:

References:

• Processors, OpenTelemetry documentation.
• Probabilistic Sampling Processor, OpenTelemetry Collector Contrib documentation.
• Log exclusion resources, Google Cloud Logging documentation.