JUnit Test Report Analyzer

Continuous integration systems need structured test results because a console log is difficult to compare across jobs and commits. JUnit-style XML supplies that structure through test suites, test cases, durations, and outcome elements. Many languages and test runners emit the format, but their dialects are not perfectly uniform, so the producer and the CI importer can disagree about optional fields or retry markers.

A test run has several outcomes that should not be collapsed into one green percentage. A failure usually records an unmet assertion. An error often points to setup, fixtures, environment, teardown, or an uncaught exception. A skipped case did not provide pass evidence, and a retry-marked pass can hide instability even though its final outcome is successful.

• Pass rate shows the share of all reported tests that passed after failures, errors, and skips are removed.
• Hard failures combine failures and errors because either can block the local release gate.
• Slow passes identify successful cases whose reported duration reaches the chosen cutoff.
• Flaky signals come from recognized retry elements or configured words found in case names and messages.

A high pass rate can still hide a release blocker. One failed checkout test among thousands of passes may matter more than the percentage suggests, which is why a hard-failure-aware gate checks the count before the target. Conversely, a report below the target can contain no failures at all when enough cases were skipped.

The XML artifact is evidence of what was written, not proof that every planned test ran. A crashed shard, wrong artifact pattern, truncated report, or job that never produced XML can leave the imported report smaller than expected. Compare suite names and total tests with the CI job definition and recent runs before approving a release.

Duration is also contextual. Test time can include application work, fixtures, network calls, emulator startup, or runner contention depending on the producer. Keep the same runner, environment, sharding, and timing policy when comparing builds.

How to Use This Tool:

Use one XML artifact from the CI run you intend to judge, then verify its suite and test totals against that job.

1. Paste or load the JUnit XML report. The source must be well-formed XML and contain a suite or test case.
2. Set the Pass-rate target from 0% to 100%. This is the local review gate, not a replacement for the CI job's own exit status.
3. Choose the Slow test cutoff. A passing case is called slow when its duration is at or above this value; 0 disables slow-pass triage.
4. Edit Flaky signal terms only when the report producer uses known retry or instability language. Broad terms can create false positives through simple text matching.
5. Add a Report label if the results need a job or suite name, then read Suite health and inspect every row in Case triage.
6. If parsing fails or counts are inconsistent, re-export the XML. Do not infer a release result from a repaired fragment or incomplete aggregate.

Interpreting Results:

Gate fails: hard failures takes precedence whenever at least one failure or error is reported. With zero hard failures, Gate passes requires pass rate to be greater than or equal to the target; otherwise the gate is below target.

Case triage includes failures, errors, skips, retry-marked or term-matched passes, and slow passes. A flaky signal is a search clue rather than statistical proof of flakiness. Review the case history and rerun behavior before labeling it unstable.

A passing local gate does not prove the CI job passed, because CI products can treat imported reports separately from the command exit status. Confirm the job status, expected shards, artifact path, and total test count.

Technical Details:

Element names are compared without namespace prefixes. Leaf suites with direct test cases are calculated from those cases rather than trusting their aggregate attributes. A suite without direct cases uses its non-negative tests, failures, errors, skipped, and time attributes. If failures, errors, and skips exceed tests, the aggregate is rejected.

Transformation Core:

Formula Core:

Passes are the reported tests left after failures, errors, and skips. Retries remain a separate signal and do not reduce the pass count.

When the total test count is zero, pass rate is 0%. Durations remain in seconds internally; a case is slow only when it passes, the selected cutoff is greater than 0 ms, and reported seconds multiplied by 1,000 are greater than or equal to the cutoff.

Rule Core:

Source is limited to 500,000 characters, loaded files to 500,000 bytes, flaky terms to 500 characters, and report labels to 80 characters.

Privacy and Limitations:

JUnit XML is processed locally, and raw source is excluded from exports. Failure messages and case names can still reveal code paths, customer data, internal services, or secrets that a test accidentally printed.

• The parser supports a practical JUnit-style subset rather than every producer-specific extension.
• Text terms are case-insensitive substring matches and can overstate flaky evidence.
• Missing reports, duplicate cases handled by a CI importer, and results from shards not included in the XML cannot be detected.
• Reported duration is accepted as supplied and cannot separate execution time from fixture or environment overhead.

References:

• Maven Surefire test report XML schema, Apache Software Foundation.
• Unit test reports, GitLab.
• JUnit plugin pipeline step, Jenkins.
• How to generate a JUnit report from Detox tests, Simplified Guide.