“The scan passed with zero findings” can describe at least five different states: the relevant code was analyzed and produced no findings; the code was excluded; its language was unsupported; the build needed for extraction failed; or the analysis never ran. Those outcomes are not interchangeable.
A useful scanner should therefore publish two products: findings and a coverage manifest. Findings state what the tool detected. The manifest states the boundary within which absence of findings can be interpreted.
Successful execution is not complete analysis
GitHub documents a concrete example. In one advanced CodeQL configuration, implicit language detection can analyze only the compiled language with the most source files unless the workflow declares a language matrix. A green workflow can therefore coexist with an unanalyzed supported language. GitHub CodeQL troubleshooting
Build mode creates another boundary. GitHub distinguishes none, autobuild, and manual modes and describes different completeness and accuracy properties. Generated code, dependency discovery, non-standard builds, and the files actually compiled can all change what reaches the analysis database. GitHub compiled-language guidance
These are not arguments against CodeQL. They demonstrate why the analysis configuration is part of the security evidence.
The minimum coverage manifest
For each analyzer invocation, record:
| Field | Why it matters |
|---|---|
| Tool and version | Rules and supported behavior change. |
| Rule or query pack | “Scanned” does not mean every rule ran. |
| Repository revision | Results must bind to immutable input. |
| Languages and ecosystems detected | Discovery is evidence, not an assumption. |
| Languages selected | Detected and selected can differ. |
| Build mode and commands | Extraction can depend on what was built. |
| Included and excluded paths | Policy exclusions must be visible. |
| Generated, vendored, and test-code policy | These categories often have different treatment. |
| Unsupported surfaces | A known unsupported area is different from clean coverage. |
| Failures and warnings | Partial failure must not collapse into success. |
| Files or artifacts analyzed | Enables coverage comparison and regression review. |
| Start/end time and exit status | Establishes when and whether the invocation completed. |
SARIF can carry findings together with invocation status, configuration, and tool notifications. Its executionSuccessful field answers whether the engineering system believes the tool completed successfully; it does not, by itself, prove that every intended artifact was analyzed. SARIF 2.1.0
The following synthetic YAML shows the shape of a coverage record. It is an illustrative contract, not an existing BaitaPhish, RepoNavi, GitHub, or SARIF schema:
schema: example.scan-coverage.v1
repositoryRevision: 8f41c2d
analyzer:
name: example-code-scanner
version: 4.2.0
ruleset: security-extended@2026-08
invocation:
startedAt: 2026-08-12T18:04:00Z
executionSuccessful: true
buildMode: manual
coverage:
typescript:
state: analyzed
files: 184
rust:
state: unsupported
reason: no Rust extractor configured
terraform:
state: unknown
reason: analyzer job absent from required matrix
exclusions:
- path: generated/**
rationale: generated client code; generator scanned separately
The important property is not this serialization. It is the refusal to collapse unsupported or unknown into analyzed merely because the invocation itself succeeded.
Report five states, not one
A practical UI or API should distinguish:
- Analyzed, no finding: the intended analyzer completed over the identified surface.
- Finding: the analyzer reported a result that still requires triage.
- Excluded by policy: a documented configuration intentionally omitted the surface.
- Unsupported or failed: the tool could not analyze the surface.
- Unknown: the evidence cannot establish which state applies.
“Unknown” is a valid result. Replacing it with “clean” creates false assurance and makes regression invisible.
A polyglot repository example
Consider a repository with TypeScript services, a Rust command-line tool, Terraform, container definitions, generated API clients, and deployment workflows. A JavaScript security scan may be valuable, but its result cannot speak for the Rust build, Terraform plan, container image, action dependencies, or deployed policies. Even a collection of scanners needs a combined manifest showing which tool owns each surface and where coverage overlaps or remains empty.
The coverage graph should also bind to revision and configuration. If a workflow changes its language matrix or path exclusions, a comparison against the previous run should show that the evidence boundary changed—even when the finding count did not.
CI decisions
CI should fail or require explicit review when:
- a required analyzer did not run;
- a required language or path changed from analyzed to unknown;
- the build/extraction phase partially failed;
- a new language or infrastructure surface has no assigned analyzer;
- an exclusion expands without an approved rationale;
- the result cannot be bound to the expected revision or configuration.
CI does not need to fail merely because an analyzer has a documented unsupported surface. It does need to prevent that surface from being represented as clean.
Practical takeaways
Count findings only after establishing the coverage boundary. Treat new unknowns and lost coverage as security regressions. Preserve analyzer, revision, configuration, warnings, and exclusions so “zero findings” means something reviewable.
Continue with Threat Modeling as Testable Control Claims for a deeper method that binds control claims to verification evidence and failure handling.
Limitations
Scanner platforms expose different evidence and may not provide file-level coverage. A manifest can document the analysis boundary but cannot prove detector completeness, rule correctness, runtime reachability, or the absence of an unknown vulnerability. This Article proposes a reporting contract; it does not claim RepoNavi or BaitaPhish currently implements it.