{{ summaryValue }}

{{ summaryLine }}

Uses the first non-empty line; LOG/TXT stays local.
Loads an editable sample.
Auto detect reports the resolved token in the field ledger.
Inference rules:
The defaults preserve the primary generated result.
The neutral default is off and leaves the Grok pattern unchanged.
{{ include_pipeline ? 'Pipeline artifact included' : 'Pattern only' }}
Use the event field containing the original line, normally message or log.original.
Leave blank to preserve ECS names; a value is normalized to one trailing dot.
Optional metadata for the downloaded pipeline configuration.
Optional event.module value added as a second ingest processor.
{{ computation.values.grok_pattern }}

{{ warningMessages.join(' ') }}

{{ computation.values.pipeline_json }}
FieldGrok tokenSampleSignalCopy
{{ row.name }}{{ row.token }}{{ row.sample || '—' }}{{ row.known ? 'Recognized' : 'Review name' }}

The chart renderer is unavailable. The exact token counts remain available in the chart CSV export after the renderer loads.

Application Log Pattern Analyzer Group application logs into stable fingerprints, compare severity and time clusters, and triage recurring patterns without uploading the source.
Regex Sample Matches Checker Test a JavaScript regex against lines that should match or fail with whole-line checks plus match evidence, input caps and backtracking guards.
Regex Tester Test JavaScript regular expressions against text or files with match and capture-group evidence, replacement previews, and timeout-protected runs.
YAML Validator Validate YAML locally with selectable scalar schemas, strict duplicate-key checks, and bounded path evidence for single or multi-document input.

Plain-text logs are readable to people but difficult to search as data. Grok describes the recurring shape of a line and assigns names to captured parts such as a timestamp, client address, request method, status code, or message. Once those fields are extracted, dashboards and alerts can query values instead of searching one long string.

A Grok expression combines reusable tokens with literal separators. In %{NUMBER:http.response.status_code:int}, NUMBER describes the accepted text, http.response.status_code names the captured field, and int requests integer conversion. Literal spaces, brackets, quotes, and punctuation around tokens must still match the source line.

  • A specific token rejects values with the wrong shape, but a legitimate format variation may stop matching.
  • DATA handles flexible text without consuming more than necessary; its surrounding delimiters must remain stable.
  • GREEDYDATA captures an open-ended tail and can swallow later fields when placed too early.
  • An ECS field name aligns a common concept across events only when the captured value truly has that meaning.

One representative line can draft a pattern, but it cannot prove that the pattern covers a whole log family. Applications often emit startup lines, stack traces, proxy errors, health checks, and older formats alongside normal requests. Test several real examples, including at least one line for every known variation.

Field names deserve the same care as matching. Elastic Common Schema (ECS) gives common events a shared vocabulary, but source-specific values may need custom fields. A familiar-looking key such as duration still needs unit and meaning checks before it is mapped to an ECS field.

Logs can contain account names, URLs, tokens, addresses, and message text. Use a representative line that preserves structure without exposing secrets, then validate the final pattern in the same Elastic version and ingest path used in production.

How to Use This Tool:

Choose a known log family when it truly matches; otherwise let a representative custom line reveal the separators and fields.

  1. Paste one sanitized log line or load a LOG or TXT file. Only the first non-empty line is used, so remove unrelated lines before judging the pattern.
  2. Select the closest Log family. Presets cover common Apache, Nginx, HAProxy, IIS, PostgreSQL CSV, and syslog shapes; use Custom application log for other formats.
  3. Leave Timestamp hint on Auto unless it chooses the wrong date shape. Keep ECS names and key/value detection enabled when the line contains recognizable fields.
  4. Enable integer and float suffixes only when numeric captures should become numbers in Elasticsearch rather than remain strings.
  5. Optionally include an ingest pipeline, then confirm the Match field contains the original log text. A field prefix changes capture names but leaves @timestamp unprefixed.
  6. Check the generated pattern, sample-match result, warnings, and field ledger. If a selected preset falls back to custom inference, either correct the sample or switch to Custom deliberately.
  7. Simulate the pattern in Elasticsearch with several matching and non-matching lines before connecting it to live Filebeat traffic.

Interpreting Results:

Sample matched means a local regular-expression approximation accepted the chosen line. It is a useful drafting check, not proof that Elasticsearch Grok will compile the pattern or that other lines will match.

  • Known fields counts captures mapped to recognized names. Coverage is known fields divided by all captured fields.
  • Confidence is a repo-authored heuristic based on the selected preset and inferred tokens. It is not an Elastic score or a probability of production success.
  • A preset warning means the exact preset shape failed and custom inference produced the result instead.
  • An ignored-lines warning means extra non-empty lines were present but were not tested.
  • Before rollout, confirm field types, units, optional sections, failure handling, and performance with the Elastic Grok processor or Grok Debugger.

Technical Details:

Grok is a regular-expression dialect with named reusable patterns. An ingest processor applies one or more expressions to a source field and writes named captures into the event. Anchoring an expression with ^ and $ requires the whole representative line to match rather than allowing an unexplained prefix or tail.

Transformation Core:

Stage Rule
Choose source Trim blank lines and keep the first non-empty line; report how many additional lines were ignored.
Apply preset Use the selected known expression only when its complete source shape matches. Otherwise continue with custom inference and add a warning.
Find structured spans Detect the selected timestamp form, then non-overlapping key=value pairs, followed by recognizable generic tokens.
Name captures Map known keys to ECS-style names, place unknown normalized keys under custom., and add numeric suffixes when a name repeats.
Preserve separators Escape literal text between captures and wrap the final expression with start and end anchors.
Draft pipeline When requested, place the same pattern in one Grok processor for the selected match field and optionally add event.module.

Custom inference gives timestamps first priority so later token scans do not split them. Key/value values may become NUMBER with int or float conversion. IP addresses, URLs, paths, log levels, HTTP methods, three-digit status values, and quoted user-agent-like strings receive specialized tokens when their shapes are recognized.

Diagnostic Formula Core:

Known-field coverage is the share of captured fields that received a recognized mapping.

C=100×KF

For custom inference, confidence is the arithmetic mean of the fixed weight assigned to each captured field.

H=∑i=1FhiF

Here K is the recognized-field count, F is the total capture count, C is coverage percent, h is a repo-authored token weight, and H is custom confidence percent. Presets use fixed confidence values instead of averaging inferred weights. These diagnostics describe the draft, not the probability that a production stream will match.

Rule Core:

Observed value Generated token Important limit
RFC 3339-like date and time TIMESTAMP_ISO8601 Matching does not parse or normalize the timestamp by itself.
IPv4-shaped text IP The local shape check does not reject octets above 255.
Known severity word LOGLEVEL Only the built-in listed severity words are recognized by custom inference.
Path beginning with / URIPATH or URIPATHPARAM A question mark selects the path-and-query form.
Unstructured tail GREEDYDATA Used as the whole-line fallback when no other marker is found.

The optional ingest pipeline uses the chosen event field, one generated pattern, ignore_missing: true, and disabled match tracing. A pipeline ID is added as metadata in the exported JSON; it does not store or install the pipeline.

Because built-in Grok patterns and ECS behavior can vary by Elastic version and compatibility mode, production validation must use the target cluster. Test failure cases as well as successful examples so unexpected lines have an intentional path.

Accuracy and Privacy Notes:

Pattern generation, file reading, and the sample-match approximation run in the current browser. The selected log text is not sent to a server by this generator.

  • Sanitize secrets, tokens, personal data, private hostnames, and customer content before pasting a line.
  • Do not treat heuristic field names or type conversions as authoritative without checking the source system.
  • Validate the final pattern in Elasticsearch and monitor real ingest failures after deployment.

Worked Examples:

Key/value application line

For 2024-10-20T14:52:11Z api-prod-1 level=INFO duration=35 path=/api/items status=200, custom inference can identify the timestamp, log level, numeric duration, URL path, and numeric status while preserving the hostname and separators. Inspect the ledger before accepting the duration mapping because the line does not declare whether 35 means milliseconds, seconds, or another unit.

Preset shape does not fit

Selecting Apache combined for a shortened access line without referrer and user-agent fields causes the preset match to fail. The result then uses custom inference and reports that fallback. Choose a matching preset or build a custom pattern, and test both complete and shortened lines in Elasticsearch.

References: