SAST Accuracy
SAST accuracy encompasses both precision (false-positive rate) and recall (false-negative rate), along with detection coverage, framework awareness, and analysis depth.
Primary question: What does SAST accuracy mean and how should it be measured?
Direct answer
SAST accuracy is a multi-dimensional concept that includes precision (the proportion of reported findings that are genuine vulnerabilities), recall (the proportion of genuine vulnerabilities that are detected), detection coverage, and analysis depth — no single metric fully captures accuracy
SAST accuracy cannot be reduced to a single metric. It encompasses precision (how many reported findings are genuine), recall (how many genuine vulnerabilities are detected within a defined scope), detection coverage (which vulnerability classes are covered), and analysis depth (how thoroughly data flows are traced). A tool with high precision but low recall reports few findings but misses many. A tool with high recall but low precision reports many findings but generates significant false-positive burden.
Accuracy is context-dependent. A tool's precision and recall vary by technology stack, code quality, rule configuration, and vulnerability class. Organizations should evaluate accuracy against their specific codebase, not against generic benchmarks. Scanner agreement between two tools is supporting evidence that both independently detected the same finding — it does not prove that either tool is fully accurate or that a unique finding is a false positive.
Definitions
SAST accuracy
The degree to which a SAST tool correctly identifies genuine vulnerabilities and avoids reporting non-vulnerabilities, measured across precision, recall, detection coverage, and analysis depth.
Precision (SAST)
The proportion of SAST-reported findings that are genuine vulnerabilities. Precision = true positives / (true positives + false positives). Precision is measured against a defined evaluation scope and ground truth.
Recall (SAST)
The proportion of genuine vulnerabilities within a defined evaluation scope that a SAST tool detects. Recall = true positives / (true positives + false negatives). Recall requires a sufficiently complete ground truth for the evaluated scope.
Analysis depth
The extent to which a SAST tool can trace data flows, control flows, and taint propagation across procedures, modules, and files.
The engineering problem
Organizations often evaluate SAST tools based only on false-positive rate, ignoring false-negative rate and detection coverage, which provides an incomplete picture of accuracy.
Organizations should evaluate SAST accuracy against their own codebase rather than relying on vendor marketing claims.
In practice, increasing recall often decreases precision, though the relationship depends on the specific tool, configuration, and evaluation scope.
Finding count is not accuracy. A tool that reports more findings may have better recall, worse precision, or both — finding count alone does not indicate accuracy.
Security controls
Each control inspects a different artifact and produces evidence for an engineering decision.
Multi-metric evaluation
Accuracy assessment- Artifact
- Evaluation of a SAST tool across precision, recall, detection coverage, and analysis depth on the organization's representative codebase.
- Risk
- Evaluating a SAST tool based on a single metric and making a procurement decision based on incomplete evidence.
- Output
- A detailed accuracy profile that supports informed evaluation and comparison.
Evidence:
Codebase-specific testing
Representative codebase analysis- Artifact
- Analysis of a representative sample of the organization's codebases, including diverse technology stacks, code patterns, and known vulnerabilities.
- Risk
- Drawing conclusions from unrepresentative codebases that do not reflect the organization's actual development practices.
- Output
- Evidence about how the SAST tool performs on the organization's specific code, not on generic or synthetic code.
Evidence:
Verification workflow
- Define the evaluation scope, acceptance criteria, and measurement methodology.
- Select a representative sample of the organization's codebases, including diverse technology stacks and vulnerability classes.
- Establish ground truth through manual review of a carefully sampled set of findings and code areas, or through seeded vulnerability testing with known limitations.
- Run the SAST tool and collect findings.
- Calculate precision (true positives / total reported findings) and recall (true positives / total known vulnerabilities within scope), document coverage gaps, and assess analysis depth.
- Compare results against acceptance criteria and identify areas for improvement.
- Treat scanner agreement between tools as supporting evidence and scanner disagreement as an opportunity for further investigation — neither is proof of accuracy.
DerScanner · Enterprise
Relationship to DerScanner
DerScanner provides SAST analysis that can be evaluated on the organization's representative codebase to assess accuracy across multiple dimensions.
DerScanner performs static analysis of supported source and binary formats, providing configurable analysis that can be evaluated against the organization's specific codebase. [derscanner-sast]
Limits of verification
- True accuracy requires knowing the ground truth — which vulnerabilities exist and which findings are genuine. In practice, ground truth is rarely complete, making accuracy assessment inherently approximate.
- Seeded vulnerability testing only measures detection of known, injected vulnerabilities and does not fully capture detection of real-world vulnerabilities with complex data flows or unusual patterns.
- Manual review of a sample does not establish complete ground truth for an entire codebase.
- Benchmarks and third-party evaluations may not reflect performance on the organization's specific codebase.
- Accuracy varies over time as the tool's rule set evolves and as the codebase changes.
- Two SAST tools may share blind spots, use similar rule patterns, or be based on common assumptions, meaning agreement between tools is not proof of complete accuracy.
Related knowledge
SAST False Positives
The precision dimension of SAST accuracy
Relationship: related-toSAST False Negatives
The recall dimension of SAST accuracy
Relationship: related-toSAST Detection Coverage
The coverage dimension of SAST accuracy
Relationship: related-toHow to Validate SAST Results
Methodology for validating SAST analysis quality
Relationship: related-toCanonical terms used: SAST accuracy; SAST precision; SAST recall; Analysis depth.
Evidence and references
- DerScanner SAST documentationDerScanner performs static analysis of supported source and binary formats, provides configuration-file analysis, and reports and compares analysis results. DerTriage and DerCodeFix are available within static analysis.
derscanner-sast
SAST accuracy