SA-APEX · evidence doctrine
A fluent explanation is not a finding.
Key doctrine
AI confidence is not evidence. Multiple agents repeating the same unsupported claim are not independent confirmation. A finding exists only when the candidate can show the affected asset and code path, the preconditions, the controlled action, the observed result, the security boundary crossed, the impact, and a repeatable method that reproduces it.
Finding lifecycle
Nothing reaches a report from the left-hand side of this table.
| State | Definition | Allowed in the final report? |
|---|---|---|
| Hypothesis | A potential issue proposed by a human, a model, an analyzer or a test. | Not in the report |
| Candidate | Code or runtime evidence suggests a plausible attack path. | Not in the report |
| Validated | Independent reproduction crosses the claimed security boundary. | In the report |
| Conditionally validated | Exploitability is proven under clearly stated conditions; constraints remain. | In the report, conditions stated up front |
| Rejected | Negative testing disproved the claim, or the evidence was insufficient. | Not in the report; retained internally for scoring |
| Duplicate or variant | Same root cause or attack primitive as another finding. | Merged; affected paths preserved |
The platform enforces this rather than trusting it. A candidate cannot promote a finding without the required evidence fields present, and the report generator reads only records that have already passed the gates. Rejected hypotheses stay available to reviewers — how honestly a candidate handles something that did not pan out is itself scored.
The validation packet
Twelve fields, every one of them required.
- Included:Finding identifier, title, affected component, environment and discovery source.
- Included:Exact source locations or runtime endpoints, including commit or build identity.
- Included:Attack preconditions and privilege assumptions.
- Included:Minimal reproducible steps and a safe proof-of-concept artifact.
- Included:Raw and normalised evidence, hashes, timestamps and capture provenance.
- Included:Expected secure behaviour set against observed behaviour.
- Included:Negative controls and a documented attempt to falsify the claim.
- Included:Impact demonstration without unnecessary harm to the target.
- Included:Root-cause explanation and the variants searched for.
- Included:Severity rationale kept separate from exploitability confidence.
- Included:Remediation and retest criteria.
- Included:Reviewer identity and disposition.
What independent means
Confirmation using a meaningfully separate method, context, actor or deterministic oracle. Asking a second model whether the first model was right is not independent, and it is not accepted.
Metrics
We publish the definitions. We do not publish promotional numbers.
| Metric | Definition | Why it matters |
|---|---|---|
| Validated precision | Validated unique findings ÷ all findings submitted for reporting. | The primary false-positive measure. |
| Validation yield | Validated hypotheses ÷ hypotheses sent to validation. | Measures discovery-stage selectivity. |
| Duplicate rate | Duplicate or variant submissions ÷ submitted findings. | Detects a swarm amplifying one root cause into many findings. |
| Critical recall proxy | Required critical challenge issues discovered ÷ discoverable critical challenge issues. | Measures consequential misses in a controlled exam. |
| Evidence sufficiency | Findings passing every evidence gate ÷ validated findings. | Stops weak proof reaching a client. |
| Replay success | Findings reproduced by an independent reviewer ÷ findings selected for replay. | Measures defensibility. |
| Calibration error | Difference between stated confidence bands and observed validation rate. | Punishes unjustified certainty. |
Why precision alone is gameable
Submit one finding you are certain of and your precision is perfect. That is why precision is always paired with the critical recall proxy, coverage obligations, and a rule that missing a designated critical issue is a fail. Reporting very little is not a strategy here.
Why duplicate rate is scored
A swarm will happily turn one root cause into eleven findings that all look like work. Root-cause grouping and duplicate rate exist to make that behaviour cost points instead of earning them.
Thresholds are versioned by exam form and reviewed for fairness. Until enough exam and practice data supports stable benchmarks, we publish what each metric means and nothing more.
Beyond CVEs
Original, derived from the system, or it does not count.
| Qualifying evidence | Non-qualifying substitute |
|---|---|
| A business-logic or authorization flaw derived from code and runtime behaviour. | A scanner saying IDOR or BOLA may exist. |
| An unsafe interaction between components that are individually reasonable. | A dependency with a known CVE. |
| A state-machine, race, cache, tenancy, trust-boundary, parser or workflow weakness. | A generic missing-header observation. |
| A candidate-built proof crossing a security boundary with controlled impact. | Model-generated exploit text that was never understood or executed. |
| Variant analysis showing related reachable paths and a shared root cause. | One copied request with no explanation. |
The exam does not require a literal undisclosed zero-day in third-party software. It uses original, non-public challenge defects that measure the same discovery skill safely. If a candidate does find a genuinely transferable unknown issue in a third-party component during controlled work, it goes through coordinated disclosure — it does not become exam content.
Critical-miss review
Completeness cannot be established by asking the model that wrote the report.
The candidate maintains an attack-surface inventory and a test-obligation graph, maps each obligation either to evidence or to an explicit limitation, uses independent analysis paths, performs variant analysis around every root cause, and reconciles code review against runtime observation.
In the exam, hidden critical issues give an objective recall measure. Missing a designated critical is a fail even when every submitted finding is accurate — because that is exactly the outcome a client cannot survive.
The report has to name what was not tested. Silence about coverage is scored as a gap, not as completeness.