LLMs in Security Writing: A Defender's Governance Guide for AI-Assisted Content Workflows

A recent piece by Thomas Ptacek — amplified by Simon Willison's widely-read weblog — articulates a discipline that security organizations should treat as more than a writing tip: use LLMs as copyeditors, never as ghostwriters. Ptacek's Rule Number One is blunt: you may not use a single word an LLM suggests to you. Any specific turn of phrase the model offers is off-limits. Willison takes a similar position: LLMs are permitted for fact-checking, spelling, grammar, and thesaurus duty — but never for generating the content itself.

Why does a writing-philosophy debate matter to a SOC, an IR team, or a CISO's office? Because the same LLM workflows now being adopted for blog posts are being quietly adopted for incident reports, threat intelligence summaries, client deliverables, board communications, and compliance evidence. In a security context, the failure modes of unconstrained LLM writing — fabricated facts, invented citations, homogenized voice, and data leakage — are not stylistic problems. They are integrity and confidentiality failures that can poison an investigation record, mislead a regulator, or expose case details to a third-party model provider.

This post translates the Ptacek/Willison position into a defensible organizational policy for AI-assisted security content in 2026.

Analysis: Why This Matters for Security Operations

The Integrity Risk: Hallucinated Technical Facts

In our IR practice, the single most dangerous LLM behavior is confident fabrication. When an LLM is asked to draft or even "polish" technical content, it will happily:

• Invent plausible-looking CVE identifiers that do not exist, or attach the wrong CVE to a vulnerability
• Fabricate command-line flags, registry paths, file hashes, and MITRE ATT&CK technique mappings
• Misstate patch version numbers — for example, asserting a fix exists in a release that predates the vulnerability
• Conflate two distinct campaigns or threat actors into a single, confidently wrong narrative

In a blog post, that's embarrassing. In an incident report submitted to a client, a cyber-insurance carrier, or a regulator under HIPAA or PCI-DSS obligations, it's a liability event. Ptacek's rule — never adopt the model's phrasing — functions as what he calls "intellectual personal protective equipment": it forces a human to remain the authoritative source of every factual and stylistic claim.

The Confidentiality Risk: Where Does the Text Go?

Willison's own usage pattern — proofreading prompts, fact-checking passes — involves sending draft text to a hosted model. For a personal blog, that's fine. For a draft incident timeline containing customer identifiers, IOCs tied to an ongoing investigation, attorney-client privileged analysis, or pre-disclosure vulnerability details, it is a third-party data disclosure. Unless your organization has an enterprise agreement with explicit non-training/no-retention guarantees and your team can verify which endpoint the text actually reaches, pasting draft IR content into a consumer chat interface is functionally equivalent to emailing it to an outside contractor without an NDA.

The Adversarial Risk: Prompt Injection via "Fact-Checking"

Even the conservative use case — "LLM as fact-checker" — carries a 2026-relevant threat. If analysts paste external source material (advisories, forum posts, PDF reports) into a model for summarization or verification, that untrusted text can carry indirect prompt injection payloads: instructions embedded in the content that steer the model's output. A fact-check pass that quietly drops a paragraph, softens a severity assessment, or inserts an attacker-friendly claim is a supply-chain attack on your documentation pipeline. Human review of the model's diff, not just its output, is mandatory.

The Provenance Risk: Undisclosed Machine Authorship

Compliance frameworks care about evidence integrity. If LLM-generated prose enters audit workpapers, incident timelines, or attestation documents without disclosure, you have a provenance problem: the organization can no longer certify who — or what — authored a statement of fact. Ptacek's strict rule has a compliance corollary: if no LLM phrasing survives into the final document, provenance questions largely evaporate.

Executive Takeaways

This is a governance and workflow topic rather than a technical vulnerability, so the defensive value here is policy, not detection rules. We recommend the following:

1. Codify the "copyeditor, not ghostwriter" rule in writing. Publish an explicit AI-assisted content policy: LLMs may be used for grammar, spelling, readability critique, structural feedback, and fact-verification prompts — but no sentence, phrase, or factual claim may enter a deliverable without a named human author writing and validating it. Ptacek's Rule Number One is a strong default.

2. Mandate human verification of every technical claim. Any CVE identifier, version number, hash, registry key, log path, or ATT&CK mapping that an LLM touches during editing must be re-verified against the primary source (vendor advisory, NVD, CISA KEV) by the author before publication or delivery. Treat model output as an unvetted junior analyst's notes.

3. Classify content before it touches a model. Apply your existing data classification scheme to LLM inputs. Incident reports, client data, pre-disclosure vulnerability information, and privileged communications must only go to organizationally-approved, contractually-protected model endpoints — or be redacted before any external LLM use. When in doubt, use a locally-hosted model for sensitive drafts.

4. Guard the fact-checking pipeline against indirect prompt injection. Never let a model ingest untrusted third-party content and then edit your authoritative document in the same session without a human reviewing the specific changes. Require the model to output a change list rather than rewriting text inline, and diff every suggested modification.

5. Require disclosure and preserve provenance. Maintain a lightweight log of where LLM assistance was used in formal deliverables (IR reports, compliance evidence, client communications). If a regulator, court, or client ever questions authorship, you need a defensible answer — and "a human wrote every word; the model only flagged typos" is the strongest one available.

6. Train analysts on hallucination recognition. Run tabletop exercises where a draft report contains two or three deliberately planted LLM-style fabrications (a wrong CVE, a nonexistent patch version, a misattributed TTP). Analysts who routinely catch these are your control; analysts who don't need coaching before they touch client deliverables.

Remediation: Standing Up the Policy This Quarter

• Week 1–2: Draft the AI-assisted content policy. Reuse your data classification and third-party disclosure frameworks rather than inventing new categories. Get Legal and Compliance sign-off early.
• Week 3–4: Inventory where LLMs are already being used informally — report drafting, ticket summaries, threat intel curation. You cannot govern usage you haven't mapped. Expect to find more than you assumed.
• Week 5–6: Stand up an approved tooling path: an enterprise model endpoint with contractual no-retention/no-training terms, or a self-hosted model for sensitive material, with DLP rules alerting on classified markers (case numbers, client names, IOC formats) flowing to unapproved endpoints.
• Week 7–8: Deliver analyst training covering hallucination patterns, prompt injection awareness, and the verification checklist for technical claims. Update your IR report QA checklist to include a "technical claims re-verified against primary sources" sign-off line.
• Ongoing: Audit quarterly. Sample published deliverables, verify the LLM-use log is being maintained, and review DLP alerts for policy drift.

The Ptacek/Willison position isn't anti-AI — it's a recognition that in 2026, the model is a powerful tool with non-negotiable failure modes. Security organizations that adopt the copyeditor-only discipline get the productivity benefit while keeping factual authority, confidentiality, and evidentiary provenance exactly where they belong: with accountable humans.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub