Securing Healthcare AI: How to Mitigate Risks in Rapid Innovation Environments

Introduction

Recent reports from Northern Ohio highlight a surge in collaboration to drive healthcare AI innovation, aiming to improve patient outcomes and operational efficiency. However, as healthcare organizations in the "Buckeye State" and beyond rush to adopt Artificial Intelligence and Machine Learning (ML), the attack surface expands significantly. For defenders, this means safeguarding not just the Electronic Health Record (EHR), but the vast datasets fueling these models and the APIs serving them. The integration of AI introduces risks such as data poisoning, model inversion attacks, and the inadvertent exposure of Protected Health Information (PHI) through third-party AI tools—often referred to as "Shadow AI."

Technical Analysis

While the Ohio initiative focuses on the beneficial application of AI, the underlying technology relies on massive ingestion of sensitive clinical data. The security risks generally fall into three categories:

1. Data Privacy and Leakage: AI models often require large datasets. If these datasets are not properly sanitized or anonymized before training, PHI can be memorized by the model and leaked through inference attacks. Additionally, staff may input patient data into public generative AI tools (e.g., ChatGPT) to assist with administrative tasks, bypassing corporate DLP controls.
2. Supply Chain Vulnerabilities: AI frameworks (like TensorFlow or PyTorch) and pre-trained models are often sourced from open-source repositories. These dependencies can be vulnerable to typical software supply chain attacks (e.g., dependency confusion or malicious package uploads).
3. Model Exploitation: Attackers may attempt "prompt injection" attacks against AI-powered chatbots integrated into hospital portals to bypass security controls or extract training data.

Executive Takeaways

For security leaders in healthcare navigating this wave of innovation:

• Governance First: Establish an AI Governance Committee that includes InfoSec before any AI tool is piloted. Collaboration should not bypass compliance.
• Data Egress Control: The "collaboration" mentioned in the news often implies data sharing. Ensure strict Data Loss Prevention (DLP) policies are applied to any interface moving data to external AI partners or cloud environments.
• Inventory of Models: You cannot protect what you cannot see. Maintain an inventory of all AI models in production, their data sources, and their access interfaces.

Defensive Monitoring

Detecting the use of unauthorized AI tools and monitoring the integrity of ML workloads is critical. Below are detection rules and hunts to secure your environment against "Shadow AI" and suspicious model activity.

SIGMA Rules

---
title: Potential Shadow AI Usage via Public Generative AI Domains
id: 4c8f1a2b-3d4e-4f5a-8b9c-0d1e2f3a4b5c
status: experimental
description: Detects network connections to known public Generative AI domains which may indicate employees uploading sensitive data to unauthorized AI tools.
references:
  - https://attack.mitre.org/techniques/T1567/002/
author: Security Arsenal
date: 2024/05/21
tags:
  - attack.exfiltration
  - attack.t1567.002
logsource:
  category: network_connection
  product: windows
detection:
  selection:
    DestinationHostname|contains:
      - 'chatgpt.com'
      - 'openai.com'
      - 'anthropic.com'
      - 'bard.google.com'
      - 'copilot.microsoft.com'
  condition: selection
falsepositives:
  - Authorized use of AI tools by marketing or research teams
level: medium
---
title: Suspicious Machine Learning Framework Execution in User Directory
id: 5e9f2c3d-4e5f-5a6b-9c0d-1e2f3a4b5c6d
status: experimental
description: Detects execution of Python scripts often associated with ML training (TensorFlow/PyTorch) originating from user profiles, suggesting unauthorized development or shadow AI work.
references:
  - https://attack.mitre.org/techniques/T1059/006/
author: Security Arsenal
date: 2024/05/21
tags:
  - attack.execution
  - attack.t1059.006
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith: '\python.exe'
    ParentImage|contains: '\Users\'
    CommandLine|contains:
      - 'import tensorflow'
      - 'import torch'
      - 'import sklearn'
      - 'pandas.read_csv'
  condition: selection
falsepositives:
  - Data scientists working in local environments
level: low

KQL (Microsoft Sentinel/Defender)

// Hunt for connections to known AI providers
DeviceNetworkEvents
| where RemoteUrl has_any ("chatgpt.com", "openai.com", "api.openai.com", "anthropic.com", "huggingface.co")
| project Timestamp, DeviceName, InitiatingProcessAccountName, RemoteUrl, RemotePort, BytesSent, BytesReceived
| summarize count() by DeviceName, RemoteUrl
| order by count_ desc

// Detect large file transfers that might indicate dataset exfiltration
DeviceFileEvents
| where ActionType == "FileUploaded"
| where FileName endswith ".csv" or FileName endswith "." or FileName endswith ".pkl" or FileName endswith ".h5"
| project Timestamp, DeviceName, InitiatingProcessAccountName, FileName, FolderPath, FileSize
| where FileSize > 10000000 // Greater than 10MB

Velociraptor VQL

-- Hunt for ML model artifacts in user directories
SELECT FullPath, Size, Mtime, Mode
FROM glob(globs='C:\Users\*\**\*.pkl')
  OR glob(globs='C:\Users\*\**\*.h5')
  OR glob(globs='C:\Users\*\**\*.onnx')
WHERE Mtime > now() - 30d

-- Hunt for Python processes importing common ML libraries
SELECT Pid, Name, CommandLine, Exe, Username
FROM pslist()
WHERE CommandLine =~ 'tensorflow'
   OR CommandLine =~ 'torch'
   OR CommandLine =~ 'sklearn'
   OR CommandLine =~ 'keras'

PowerShell

# Audit registry for AI-related software installations
Get-ItemProperty "HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*", 
                "HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*" |
Where-Object { $_.DisplayName -match "Python|Anaconda|PyTorch|TensorFlow" } |
Select-Object DisplayName, DisplayVersion, InstallDate, InstallSource

Remediation

To protect patient data while enabling AI innovation, healthcare organizations should implement the following controls:

1. Implement Strict Egress Filtering: Configure firewalls and proxies to block access to unauthorized public generative AI APIs. Allowlist only specific, vetted AI services required for business operations.
2. Data Sanitization Pipelines: Ensure that any data exported for AI development or collaboration undergoes rigorous de-identification and anonymization (per HIPAA Safe Harbor standards) before leaving the secure EHR environment.
3. Private LLM Deployment: Where possible, utilize self-hosted or cloud-private instances of LLMs (e.g., Azure OpenAI Service within your private tenant) to ensure data residency and prevent PHI from being used to train public models.
4. Browser Extensions Policy: Ban the installation of unauthorized browser extensions that integrate AI capabilities (like ChatGPT sidebar extensions) on endpoints accessing PHI.

Related Resources

Security Arsenal Healthcare Cybersecurity AlertMonitor Platform Book a SOC Assessment healthcare Intel Hub