Unauthorized PHI Uploads to Generative AI Tools: Detection and Prevention Guide for Healthcare Security Teams

Two healthcare providers have recently disclosed data privacy incidents with a common and increasingly dangerous thread: workforce members uploaded protected health information (PHI) to generative artificial intelligence tools without authorization. These are not external attacks. There is no exploit chain, no malware, no adversary infrastructure to block. The threat actor in these incidents was an authorized user with legitimate access to patient data and a browser tab open to an AI chatbot.

This pattern — often called shadow AI — is now one of the fastest-growing sources of reportable HIPAA breaches. Staff under time pressure paste clinical notes into ChatGPT to draft discharge summaries, upload spreadsheets containing patient demographics to AI tools for analysis, or feed transcripts into summarization services. Each of those actions is a disclosure of PHI to an entity with no Business Associate Agreement (BAA), no HIPAA risk analysis, and in most consumer-grade services, terms of service that explicitly permit the provider to retain and potentially train on submitted data.

If you run security for a covered entity or business associate, assume this is already happening in your environment. Your job is to find it, stop it, and build the governance and technical controls that make the sanctioned path easier than the unsanctioned one.

Technical Analysis

What Happened

Per the incidents reported via The HIPAA Journal, two separate healthcare providers discovered that patient data had been transmitted to AI tools without organizational authorization. While the specific service names and record counts vary by incident, the mechanics are consistent with what we are seeing across IR engagements in the healthcare vertical:

1. Authorized data access — A workforce member with legitimate EHR, billing, or clinical systems access copies or exports patient data (names, MRNs, diagnoses, clinical narratives, insurance details).
2. Unsanctioned transmission — The data is pasted into a web-based AI chat interface, uploaded as a document, or sent via a browser extension or desktop AI application.
3. Third-party retention — The AI provider now holds PHI on infrastructure outside the covered entity's control, subject to that provider's data retention and model-training policies.
4. Delayed discovery — The disclosure is found weeks or months later via DLP alerting, audit log review, employee self-report, or — worst case — a patient complaint.

Why This Is a Breach, Not a Policy Violation

Under HIPAA, an impermissible acquisition, access, use, or disclosure of PHI is presumed to be a breach unless a risk assessment under 45 CFR § 164.402 demonstrates a low probability that the PHI has been compromised. Key considerations in that assessment:

• The nature and extent of PHI involved — Clinical narratives and identifiers together are high-sensitivity.
• The unauthorized person who received it — A commercial AI provider with no BAA and data-processing terms that permit retention.
• Whether the PHI was actually acquired or viewed — Submission logs from the AI provider (if obtainable) matter here.
• The extent to which risk has been mitigated — Deletion requests under the provider's data controls, contractual attestations, and account-level purges.

For incidents affecting 500 or more individuals, notification to HHS OCR, affected individuals, and prominent media outlets is required. Even sub-500 incidents must be logged and reported to OCR annually. State attorneys general and, in some states, patients themselves have private causes of action under state law.

The Attack Surface: How PHI Reaches AI Tools

From a defender's perspective, the exfiltration channels to enumerate and control are:

Note the last two rows: in several recent healthcare incidents we have worked, the PHI did not leave through a chatbot website at all — it left through a sanctioned SaaS product whose vendor quietly enabled an AI subprocessor, or through an AI meeting notetaker a department head signed up for with a work email. Your data flow map must account for AI subprocessors in your existing vendor stack, not just consumer chatbots.

Exploitation Status

There is no CVE and no adversary exploitation here — this is an active, in-the-wild data exposure pattern confirmed by multiple breach notifications. HHS OCR has published guidance on the use of online tracking technologies and AI, and regulators are treating unsanctioned PHI disclosures to AI platforms as reportable events. Expect enforcement activity and civil monetary penalties in this category to increase through 2026.

Detection & Response

You cannot detect a clipboard paste into a browser text box with an endpoint rule. What you can detect is the network session to the AI service, the presence of unsanctioned AI software, and bulk upload behavior at the proxy. Layer these detections and scope them to your highest-risk assets: clinical workstations, EHR-adjacent systems, and accounts with bulk PHI access.

Tuning note: If your organization has sanctioned AI tools, do not blanket-alert on all AI domains — you will drown your SOC. Scope rules to devices and user groups that are not covered by an approved AI use exception, and alert on the non-sanctioned service list specifically.

---
title: Outbound Connection to Unsanctioned Generative AI Service
description: Detects endpoint network connections to consumer generative AI platforms from Windows systems. Scope to clinical workstations or non-exempt users to reduce noise in environments with sanctioned AI usage.
references:
  - https://attack.mitre.org/techniques/T1567/
  - https://attack.mitre.org/techniques/T1048/
author: Security Arsenal
id: 8f3a2c71-4b6d-4e9a-b1c7-2d5e8f0a9b3c
status: experimental
date: 2026/02/09
tags:
  - attack.exfiltration
  - attack.t1567
logsource:
  category: network_connection
  product: windows
detection:
  selection:
    DestinationHostname|contains:
      - 'chatgpt.com'
      - 'chat.openai.com'
      - 'claude.ai'
      - 'gemini.google.com'
      - 'perplexity.ai'
      - 'deepseek.com'
      - 'copilot.microsoft.com'
      - 'grok.com'
      - 'poe.com'
      - 'character.ai'
  filter_sanctioned_browser:
    Image|endswith:
      - '\msedge.exe'
    Initiated: 'true'
  condition: selection and not filter_sanctioned_browser
falsepositives:
  - Approved enterprise Copilot usage (tune copilot.microsoft.com out if licensed)
  - Marketing or research staff with documented AI-use exceptions
level: medium
---
title: DNS Query for Consumer Generative AI Platform
description: Detects DNS resolution of consumer generative AI domains, useful for environments without endpoint network connection logging. Deploy against DNS server or forwarded DNS telemetry; scope by subnet for clinical VLANs.
references:
  - https://attack.mitre.org/techniques/T1567/
author: Security Arsenal
id: 3c7e1b92-6a4f-4d8c-a5e1-9f2b4d6c8e0a
status: experimental
date: 2026/02/09
tags:
  - attack.exfiltration
  - attack.t1567
logsource:
  category: dns
detection:
  selection:
    query|contains:
      - 'chatgpt.com'
      - 'openai.com'
      - 'claude.ai'
      - 'anthropic.com'
      - 'perplexity.ai'
      - 'deepseek.com'
      - 'poe.com'
      - 'huggingface.co'
  condition: selection
falsepositives:
  - Environments with sanctioned API integrations against OpenAI or Anthropic (exclude service accounts/subnets)
  - Security research activity
level: low
---
title: Local LLM or AI Desktop Application Execution on Clinical Endpoint
description: Detects execution of local large-language-model runtimes and consumer AI desktop applications. Clinical endpoints have no legitimate use case for local model hosting; these tools bypass web gateway controls entirely.
references:
  - https://attack.mitre.org/techniques/T1059/
  - https://attack.mitre.org/techniques/T1105/
author: Security Arsenal
id: 5a9d4e83-7c1b-4f2a-8d6e-1b3c5a7f9e2d
status: experimental
date: 2026/02/09
tags:
  - attack.execution
  - attack.t1059
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith:
      - '\ollama.exe'
      - '\ollama app.exe'
      - '\lm-studio.exe'
      - '\lmstudio.exe'
      - '\jan.exe'
      - '\gpt4all.exe'
      - '\chatbox.exe'
      - '\ChatGPT.exe'
      - '\Claude.exe'
  condition: selection
falsepositives:
  - Data science or IT innovation teams with sanctioned local AI development (scope by OU)
level: high

The following Sentinel query hunts for outbound sessions to consumer AI services and, critically, joins in proxy telemetry to surface large uploads — a single HTTPS GET to chatgpt.com is browsing; a multi-megabyte POST is a document upload and a very different conversation with the user's manager.

let AIDomains = dynamic(["chatgpt.com","openai.com","claude.ai","gemini.google.com","perplexity.ai","deepseek.com","copilot.microsoft.com","poe.com","grok.com","huggingface.co"]);
// Endpoint visibility: which devices and users are reaching AI services
let EndpointHits =
    DeviceNetworkEvents
    | where TimeGenerated > ago(7d)
    | where RemoteUrl has_any (AIDomains)
    | summarize FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), Connections=count(),
                URLs=make_set(RemoteUrl, 20)
        by DeviceName, InitiatingProcessAccountName, InitiatingProcessFileName;
// Proxy visibility: identify large uploads (potential document/PHI submission)
let ProxyUploads =
    CommonSecurityLog
    | where TimeGenerated > ago(7d)
    | where DestinationHostName has_any (AIDomains) or RequestURL has_any (AIDomains)
    | where SentBytes > 100000  // >100KB outbound to an AI service
    | summarize TotalSentBytes=sum(SentBytes), RequestCount=count(),
                SampleURLs=make_set(RequestURL, 10)
        by SourceIP, SourceUserID, DestinationHostName;
EndpointHits
| join kind=fullouter ProxyUploads on $left.DeviceName == $right.SourceIP
| project TimeRange = strcat(FirstSeen, " -> ", LastSeen), DeviceName, SourceIP,
          User = coalesce(InitiatingProcessAccountName, SourceUserID),
          Connections, TotalSentBytes, URLs, SampleURLs
| order by TotalSentBytes desc

For IR scoping after a suspected disclosure, Velociraptor gives you ground truth on what the endpoint was actually doing: which AI desktop tools are installed, and whether browser history corroborates the reported timeline.

-- Scope unauthorized AI tool usage on a suspect endpoint
-- 1) Look for installed/running local LLM and AI desktop applications
SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE Exe =~ '(?i)(ollama|lm.?studio|gpt4all|jan\\.exe|chatbox|ChatGPT|Claude)'

-- 2) Enumerate AI application install artifacts in user profiles
SELECT FullPath, Size, Mtime
FROM glob(globs='C:/Users/*/AppData/**/programs/{ollama,lm-studio,LM Studio,gpt4all,jan,ChatGPT,Claude}/*.exe')

-- 3) Identify active connections from browser processes (correlate with proxy logs for AI domains)
SELECT Pid, Name, Status, LocalAddr, LocalPort, RemoteAddr, RemotePort
FROM netstat()
WHERE Status =~ 'ESTABLISHED'
  AND Name =~ '(?i)(chrome|msedge|firefox|brave)'

The script below performs a rapid endpoint audit for installed AI applications and browser extensions, then optionally applies hosts-file sinkholing as an emergency containment measure while your DNS/web-gateway blocks are rolled out properly. Hosts-file blocking is a stopgap — it is trivially bypassed and does not cover mobile or off-network devices — but in the first hours of a confirmed disclosure it buys you time.

#Requires -RunAsAdministrator
# Audit and emergency-contain unsanctioned AI tool usage on a Windows endpoint
# Security Arsenal - Healthcare Shadow AI Response

$AIAppPatterns = 'ollama','lm-studio','lmstudio','gpt4all','jan','chatbox','ChatGPT','Claude','Perplexity'
$AIDomains = @(
    'chatgpt.com','www.chatgpt.com','chat.openai.com','api.openai.com',
    'claude.ai','api.anthropic.com','gemini.google.com','perplexity.ai',
    'www.perplexity.ai','deepseek.com','chat.deepseek.com','poe.com','grok.com'
)
$ReportPath = "$env:TEMP\ShadowAI_Audit_$(Get-Date -Format yyyyMMdd_HHmmss).txt"

# --- Step 1: Audit installed AI desktop applications ---
"=== Installed AI Application Audit ===" | Out-File $ReportPath
$installed = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*,
             HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*,
             HKCU:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* -ErrorAction SilentlyContinue |
    Where-Object { $p = $_.DisplayName; $AIAppPatterns | Where-Object { $p -match $_ } } |
    Select-Object DisplayName, DisplayVersion, InstallDate, PSPath
$installed | Format-Table -AutoSize | Out-File $ReportPath -Append
if ($installed) { Write-Warning "AI desktop applications found - review $ReportPath" }

# --- Step 2: Audit browser extensions (Chrome/Edge) for AI assistants ---
"=== Browser Extension Audit ===" | Out-File $ReportPath -Append
Get-ChildItem 'C:\Users\*\AppData\Local\Google\Chrome\User Data\*\Extensions',
              'C:\Users\*\AppData\Local\Microsoft\Edge\User Data\*\Extensions' -ErrorAction SilentlyContinue |
    ForEach-Object {
        $manifest = Get-ChildItem $_.FullName -Recurse -Filter manifest.json -ErrorAction SilentlyContinue | Select-Object -First 1
        if ($manifest) {
            $m = Get-Content $manifest.FullName -Raw | ConvertFrom-Json
            if ($m.name -match '(?i)(ai|gpt|claude|copilot|assistant|summar)') {
                "$($m.name) :: $($manifest.FullName)" | Out-File $ReportPath -Append
            }
        }
    }

# --- Step 3 (optional): Emergency hosts-file sinkhole - run only with -Block switch ---
param([switch]$Block)
if ($Block) {
    $hosts = "$env:SystemRoot\System32\drivers\etc\hosts"
    Copy-Item $hosts "$hosts.bak_$(Get-Date -Format yyyyMMdd)" -Force
    foreach ($d in $AIDomains) {
        if (-not (Select-String -Path $hosts -Pattern "\b$([regex]::Escape($d))\b" -Quiet)) {
            Add-Content $hosts "0.0.0.0`t$d`t# ShadowAI containment $(Get-Date -Format yyyy-MM-dd)"
        }
    }
    ipconfig /flushdns | Out-Null
    Write-Host "Sinkholed $($AIDomains.Count) AI domains via hosts file. Replace with DNS/SWG policy ASAP." -ForegroundColor Yellow
}

Write-Host "Audit complete: $ReportPath" -ForegroundColor Green

Remediation

If You Have Confirmed an Unauthorized Disclosure

1. Preserve evidence before you remediate. Capture proxy/firewall logs for the user and destination service, endpoint browser history, and the AI service's account data if the user authenticates with corporate credentials. You will need this for the breach risk assessment and potential OCR inquiry.
2. Engage privacy/legal immediately. The four-factor risk assessment under § 164.402 drives whether this is a reportable breach. Do not let IT make that call alone.
3. Request deletion from the AI provider. Major providers have data-deletion and enterprise privacy channels. Document the request and any attestation received — this directly supports the mitigation factor of your risk assessment.
4. Check the account type. If the employee used a corporate email, determine whether the account is a consumer-tier or enterprise-tier account; enterprise tiers (e.g., ChatGPT Enterprise/Team, Claude for Work, Gemini for Workspace) typically contractually prohibit training on inputs, which materially changes the risk assessment.
5. Meet notification deadlines. 60 days to individuals for reportable breaches; HHS OCR within 60 days for 500+ records; media notice for 500+ in a single state; annual log submission for smaller incidents. Several states impose shorter clocks.

Preventive Controls (Priority Order)

1. Governance first, technology second. Publish an AI acceptable-use policy that names sanctioned tools, prohibited data types (all PHI absent a BAA and risk analysis), and consequences. A policy that only says "no" without offering an approved alternative guarantees continued shadow usage.
2. Provide a sanctioned path. Stand up an enterprise AI service under a BAA (Azure OpenAI with a Microsoft BAA, AWS Bedrock under your AWS BAA, Google Cloud Vertex AI under your Google BAA) with PHI-aware usage policies. Clinicians adopted shadow AI because it solves real workflow problems — give them the workflow without the breach.
3. Block at the network layer. Enforce DNS filtering and secure web gateway categories for unsanctioned AI services, with a documented exception process. The hosts-file stopgap in the script above is for emergency containment only.
4. DLP with AI-aware rules. Configure your DLP/CASB to inspect uploads to AI domains for PHI classifiers (MRN formats, ICD/CPT codes alongside identifiers, SSNs). Tune against pasting of small text fragments — that is the hardest channel and where browser-isolation or clipboard DLP products earn their cost.
5. Application and extension control. AppLocker/WDAC policies blocking local LLM runtimes and consumer AI desktop apps on clinical endpoints; browser extension allowlisting via Group Policy or MDM.
6. Vendor and subprocessor review. Inventory every SaaS platform that touches PHI and identify AI features and AI subprocessors. Update BAAs where vendors have added AI processing. Review AI meeting notetakers, transcription services, and ambient documentation tools — these are the current leading edge of unintentional disclosure.
7. Training that reflects reality. Annual HIPAA training must now include concrete AI scenarios: "drafting a discharge summary in ChatGPT," "uploading a claims spreadsheet to an AI analyzer," "inviting an AI notetaker to a telehealth visit." Abstract policy language does not change behavior; specific scenarios do.
8. Detect continuously. Deploy the detections above, scoped to clinical VLANs and PHI-access user groups, and review AI-domain network telemetry weekly during your initial rollout to baseline legitimate vs. shadow usage.

Related Resources

Security Arsenal Healthcare Cybersecurity AlertMonitor Platform Book a SOC Assessment healthcare Intel Hub