APEX · evidence doctrine
A fluent explanation is not a finding.
Key doctrine
AI confidence is not evidence. Multiple agents repeating the same unsupported claim are not independent confirmation. A finding exists only when the candidate can show the affected asset and code path, the preconditions, the controlled action, the observed result, the security boundary crossed, the impact, and a repeatable method that reproduces it.
The hard line
Prove it, or it is not a finding.
Every issue submitted in the exam carries a proof of concept the candidate wrote, that a reviewer executes independently, and that demonstrably crosses the security boundary being claimed. Not a description of an attack. Not a screenshot. Not a severity rating. Not "could be done". Code that runs.
It runs
A reviewer takes the artifact into a clean replay environment and executes it. If it does not run, the finding does not exist — regardless of how plausible the write-up is.
It crosses the boundary
The proof has to demonstrate the security boundary actually being crossed, not merely the presence of a condition that might allow it. Reachability is part of the proof, not an assumption in the narrative.
The candidate wrote it and can explain every line
The oral defense samples the code unpredictably, with generative AI disabled. A candidate who cannot explain their own proof fails on the spot, whatever it does when it runs.
It reproduces
Same artifact, same environment, same result — for somebody who was not there when it was written. Preconditions, privilege assumptions and build identity are recorded with it.
It proves impact without causing it
Controlled demonstration, not collateral damage. A proof that would predictably harm the target beyond what the demonstration requires is an immediate fail.
It comes with the attempt to disprove it
Negative controls showing what does not work, and why the result is the vulnerability rather than a coincidence of the environment.
Not accepted as proof
- Excluded:A description of how an attacker could exploit it.
- Excluded:A scanner finding with a CVSS score attached.
- Excluded:A screenshot of a response, with no artifact behind it.
- Excluded:A request copied from a write-up, with no explanation of why it works here.
- Excluded:Model-generated exploit code the candidate did not run or cannot explain.
- Excluded:A theoretical chain that was never assembled and executed end to end.
- Excluded:"Exploitable in principle." Either it is proven or it is a hypothesis.
And when they could not do it
It also means something when they could NOT exploit it. An engineer who has proven they can drive AI across an entire attack surface and still could not cross the boundary has told you something a scanner never can: it was genuinely covered, and it held.
Why this is set where it is
Most penetration test reports contain a “proof of concept” section that is a screenshot and a paragraph. That was defensible when writing a convincing paragraph took expertise. It is not defensible now, because a model will write you a very convincing paragraph about a vulnerability that is not there.
Executable code that a second person runs is the part that did not get easier. So that is where the bar is.
Finding lifecycle
Twelve claims go in. Four come out.
Every gate below kills claims, and that is the point of having them. What survives is not what sounded most convincing — it is what somebody other than the author could run.
Hypothesis
12
Raised by a human, a model, an analyzer or a test.
Dropped here: No affected asset or code path (+3 more)
Candidate
0
Code or runtime evidence suggests a real path.
Validated
0
Independently reproduced across the boundary.
In the report
0
Everything else stayed internal.
8 of 12 claims never reached the report. The 4 that did each carry code a reviewer ran and watched cross the boundary. A fluent explanation does not survive this, which is the entire reason the false-positive rate stays where it does.
| State | Definition | Allowed in the final report? |
|---|---|---|
| Hypothesis | A potential issue proposed by a human, a model, an analyzer or a test. | Not in the report |
| Candidate | Code or runtime evidence suggests a plausible attack path. | Not in the report |
| Validated | Independent reproduction crosses the claimed security boundary. | In the report |
| Conditionally validated | Exploitability is proven under clearly stated conditions; constraints remain. | In the report, conditions stated up front |
| Rejected | Negative testing disproved the claim, or the evidence was insufficient. | Not in the report; retained internally for scoring |
| Duplicate or variant | Same root cause or attack primitive as another finding. | Merged; affected paths preserved |
The platform enforces this rather than trusting it. A candidate cannot promote a finding without the required evidence fields present, and the report generator reads only records that have already passed the gates. Rejected hypotheses stay available to reviewers — how honestly a candidate handles something that did not pan out is itself scored.
The validation packet
Twelve fields, every one of them required.
- Included:Finding identifier, title, affected component, environment and discovery source.
- Included:Exact source locations or runtime endpoints, including commit or build identity.
- Included:Attack preconditions and privilege assumptions.
- Included:Minimal reproducible steps and a safe proof-of-concept artifact.
- Included:Raw and normalized evidence, hashes, timestamps and capture provenance.
- Included:Expected secure behavior set against observed behavior.
- Included:Negative controls and a documented attempt to falsify the claim.
- Included:Impact demonstration without unnecessary harm to the target.
- Included:Root-cause explanation and the variants searched for.
- Included:Severity rationale kept separate from exploitability confidence.
- Included:Remediation and retest criteria.
- Included:Reviewer identity and disposition.
What independent means
Confirmation using a meaningfully separate method, context, actor or deterministic oracle. Asking a second model whether the first model was right is not independent, and it is not accepted.
Metrics
We publish the definitions. We do not publish promotional numbers.
| Metric | Definition | Why it matters |
|---|---|---|
| Validated precision | Validated unique findings ÷ all findings submitted for reporting. | The primary false-positive measure. |
| Validation yield | Validated hypotheses ÷ hypotheses sent to validation. | Measures discovery-stage selectivity. |
| Duplicate rate | Duplicate or variant submissions ÷ submitted findings. | Detects a swarm amplifying one root cause into many findings. |
| Critical recall proxy | Required critical challenge issues discovered ÷ discoverable critical challenge issues. | Measures consequential misses in a controlled exam. |
| Evidence sufficiency | Findings passing every evidence gate ÷ validated findings. | Stops weak proof reaching a client. |
| Replay success | Findings reproduced by an independent reviewer ÷ findings selected for replay. | Measures defensibility. |
| Calibration error | Difference between stated confidence bands and observed validation rate. | Punishes unjustified certainty. |
Why precision alone is gameable
Submit one finding you are certain of and your precision is perfect. That is why precision is always paired with the critical recall proxy, coverage obligations, and a rule that missing a designated critical issue is a fail. Reporting very little is not a strategy here.
Why duplicate rate is scored
A swarm will happily turn one root cause into eleven findings that all look like work. Root-cause grouping and duplicate rate exist to make that behavior cost points instead of earning them.
Thresholds are versioned by exam form and reviewed for fairness. Until enough exam and practice data supports stable benchmarks, we publish what each metric means and nothing more.
Beyond CVEs
Original, derived from the system, or it does not count.
| Qualifying evidence | Non-qualifying substitute |
|---|---|
| A business-logic or authorization flaw derived from code and runtime behavior. | A scanner saying IDOR or BOLA may exist. |
| An unsafe interaction between components that are individually reasonable. | A dependency with a known CVE. |
| A state-machine, race, cache, tenancy, trust-boundary, parser or workflow weakness. | A generic missing-header observation. |
| A candidate-built proof crossing a security boundary with controlled impact. | Model-generated exploit text that was never understood or executed. |
| Variant analysis showing related reachable paths and a shared root cause. | One copied request with no explanation. |
The exam does not require a literal undisclosed zero-day in third-party software. It uses original, non-public challenge defects that measure the same discovery skill safely. If a candidate does find a genuinely transferable unknown issue in a third-party component during controlled work, it goes through coordinated disclosure — it does not become exam content.
Critical-miss review
Completeness cannot be established by asking the model that wrote the report.
The candidate maintains an attack-surface inventory and a test-obligation graph, maps each obligation either to evidence or to an explicit limitation, uses independent analysis paths, performs variant analysis around every root cause, and reconciles code review against runtime observation.
In the exam, hidden critical issues give an objective recall measure. Missing a designated critical is a fail even when every submitted finding is accurate — because that is exactly the outcome a client cannot survive.
The report has to name what was not tested. Silence about coverage is scored as a gap, not as completeness.