OpenAPI Operation Coverage Checker

OpenAPI operation coverage asks which documented HTTP operations have evidence in a test inventory and which documented operations still have no matching signal. In an OpenAPI description, operations sit under paths and HTTP methods, often with operationId values and tags that test suites, generated clients, and release reviews reuse.

A coverage gap can mean a new endpoint has no contract test yet, a test list uses an old path, a retired operation is still counted, or the API contract drifted away from the service. The percentage is useful, but the path-by-path ledger matters more because a single missing DELETE, PATCH, or webhook operation can carry more release risk than several covered read-only routes.

Operation coverage is not the same as API quality. A match shows that the inventory referenced the operation through an identifier or a method and path. It does not prove that the test has useful assertions, checks every response status, covers authorization, or exercises the deployed service.

operationId evidence and method-path evidence complement each other. Operation identifiers survive base URL changes and path-parameter examples, while method and path evidence can be extracted from logs, cURL commands, CSV rows, and plain manifest lines. Mismatches between the two often reveal stale tests, renamed routes, or contract entries that need owner review.

Technical Details:

OpenAPI represents endpoint shape with path items and operation objects. A path item describes the HTTP operations available on one path, and each operation can carry an operationId, tags, summary text, response definitions, and a deprecation flag. The operationId value is intended to be unique and case-sensitive, which makes it useful for matching contract test manifests when teams keep identifiers stable.

The coverage denominator is the set of operations in scope for the review. Deprecated operations, top-level webhooks, and ignored HTTP methods can change that denominator before any coverage percentage is calculated. Path templates also matter because GET /orders/{orderId} should match a concrete inventory line such as GET /orders/ord_123, but it should not silently match a stale base path such as GET /v1/orders/ord_123 when the contract path is /orders/{orderId}.

Rule Core

The checker builds a documented operation set, parses inventory evidence, then marks each operation covered when at least one inventory entry maps to it under the selected match mode. Warnings stay separate from the percent because a high number can still be misleading when duplicate identifiers, duplicate operation keys, unresolved references, or unparsed inventory lines are present.

Coverage Formula

Coverage is a simple ratio after the operation scope has been built and filters have been applied.

Everyday Use & Decision Guide:

Start with the API contract you intend to hold tests against. Paste or drop OpenAPI YAML or JSON into OpenAPI source, then paste the test evidence into Test inventory. A useful first inventory can be a contract-test manifest, a small access-log sample, cURL commands from smoke tests, Newman output, CSV rows, or one operationId per line.

Use API name to label the run and Coverage target as the release gate. The default 90% target is a reasonable audit signal for many APIs, but high-risk interfaces often need every state-changing operation covered before release. For a first pass, keep Coverage match mode on Operation ID or method + path so both identifier-heavy manifests and path-heavy logs can contribute evidence.

• Leave Deprecated operations included when retired endpoints still need regression protection or migration visibility.
• Turn Deprecated operations off when retired endpoints are intentionally outside the active test gate and the release note explains that choice.
• Turn Webhook path items on when incoming webhook templates should be checked with the same coverage gate as ordinary API paths.
• Use Ignored HTTP methods for methods such as OPTIONS or HEAD only when they are generated by infrastructure and intentionally excluded from active coverage.
• Use Normalize after dropping text with trailing spaces or mixed indentation; then confirm the source badge still reports the expected operation count.

Read the summary first, then move through Coverage Gate Metrics, Operation Coverage Ledger, and Coverage Gap Queue. The metrics table gives the percent, denominator, target gap, parser mode, deprecated count, inventory count, and warning count. The ledger ties each operation to status, evidence, and recommended action. The queue is where missing operations, unmatched inventory, duplicate keys, duplicate IDs, and parser warnings become work items.

Do not treat a strong percentage as approval by itself. A 100.0% run means every in-scope operation had at least one matching inventory entry under the current settings. It does not prove test depth, live endpoint health, authorization coverage, or schema assertions. Use the JSON tab or table rows as a handoff only after the queue rows match the story you want to tell in release review.

Step-by-Step Guide:

Review one API contract and one coherent test inventory at a time so the coverage target, ledger, queue, and method chart all describe the same release scope.

1. Enter API name, such as orders-api, so copied rows and JSON identify the service under review.
2. Set Coverage target. If the target is lower than the organization gate, the summary may say target met while the release still fails the real policy.
3. Choose Coverage match mode. Start with Operation ID or method + path, then switch to Method + path only if duplicate operation IDs are causing false comfort.
4. Paste, browse, or drop the API contract into OpenAPI source. The badge should show the expected operation count and a parser mode such as YAML document or JSON document.
5. Paste, browse, or drop evidence into Test inventory. The inventory badge should show parsed entries and unmatched entries; a high unmatched count usually means stale paths, wrong base paths, or tests for operations missing from the contract.
6. Open Advanced and set Deprecated operations, Webhook path items, and Ignored HTTP methods before comparing results. These controls change the denominator.
7. Check the warning area above the results. If OpenAPI operations are required or At least one test inventory entry is required appears, fix the missing source before relying on any percent.
8. Open Coverage Gap Queue and clear missing operations first. Then review unmatched inventory lines, duplicate operation keys, duplicate operationId values, and parser warnings.
9. Open Method Coverage Matrix when you need a quick method view. A missing PATCH, DELETE, or webhook-heavy group can be easier to spot there than in a long ledger.

A clean handoff has a nonzero operation count, coverage at or above the target, no unexplained parser warnings, and a gap queue that either has no rows or has owner-ready remediation notes.

Interpreting Results:

The most important output is the first unresolved row in Coverage Gap Queue. The percentage helps triage, but a queue item such as PATCH /orders/{orderId}, Duplicate operationId syncOrder, or Unmatched inventory line 17 tells you what must be fixed or explained.

A perfect coverage percentage can still hide weak tests. Use it as a map of documented operations to test evidence, then review assertion depth, negative cases, authorization rules, schema checks, and production routing separately.

Worked Examples:

Orders API below a 90 percent gate

An orders contract defines GET /orders, POST /orders, GET /orders/{orderId}, PATCH /orders/{orderId}, and deprecated DELETE /orders/{orderId}. The inventory contains operationId=listOrders, POST /orders, a log line for GET /orders/ord_123, a dry-run delete URL, and GET /orders/search. Operation Coverage Gate shows 80.0%, Operation Coverage Ledger marks PATCH /orders/{orderId} as Coverage gap, and Coverage Gap Queue includes both the missing patch operation and the unmatched search line.

Retired endpoint changes the denominator

The same contract can read differently when Deprecated operations changes. If the inventory covers list, create, and get operations but omits both patch and deprecated delete, keeping deprecated operations included counts 3 covered out of 5. Turning the deprecated switch off removes delete from the denominator, so the run becomes 3 covered out of 4. The next action is still to decide whether PATCH /orders/{orderId} needs a test; the higher percent did not add coverage.

Duplicate operation ID needs cleanup

A contract with POST /orders/{orderId}/sync and POST /orders/{orderId}/resync both using operationId=syncOrder can make an identifier-based inventory look better than it should. A single inventory line operationId=syncOrder may map to more than one operation, while Coverage Gap Queue adds Duplicate operationId syncOrder. Switch to Method + path only for the review or make the identifiers unique before accepting the evidence.

Base path mismatch creates an unmatched row

A test log may contain GET /v1/orders/ord_123 while the OpenAPI path is /orders/{orderId} because the API gateway adds /v1. The concrete URL does not match the template in that run, so Inventory entries reports an unmatched entry and Coverage Gap Queue points at the stale or prefixed line. Fix the contract path, strip the gateway prefix before review, or paste inventory rows that use the same path basis as OpenAPI.

FAQ:

Glossary:

Operation

An HTTP method on an OpenAPI path, such as GET /orders or PATCH /orders/{orderId}.

operationId

A case-sensitive identifier for an OpenAPI operation, often used by generated code, documentation systems, and test manifests.

Path template

An OpenAPI path with variable segments in braces, such as /orders/{orderId}, that can match concrete inventory paths.

Coverage denominator

The total number of operations counted after deprecated, webhook, and ignored-method settings are applied.

Inventory entry

One parsed line or row from the test evidence, such as an operation ID, method-path pair, log line, cURL command, or CSV row.

Parser warning

A review cue showing that part of the source or inventory needs attention before the coverage number should be trusted.

References:

• OpenAPI Specification v3.1.2, OpenAPI Initiative.
• API Endpoints, OpenAPI Learn.
• Providing Webhooks, OpenAPI Learn.