Hugging Face Supply-Chain Attack & PHANTOM-B: Threat Modeling Lessons for Defending AI/ML Pipelines

When Adam Shostack — arguably the most authoritative voice in threat modeling alive — says he was "blown away" by an attack disclosure, defenders should pay attention. In a recent interview with Dark Reading, Shostack reacted to OpenAI's revelations about an attack targeting Hugging Face, the de facto public repository for machine learning models, and introduced his new threat model for large language model systems, dubbed PHANTOM-B, which he describes as "lightweight yet still usable."

This story matters on two levels. First, the Hugging Face incident is a concrete reminder that AI/ML supply chains — model hubs, tokenizer libraries, serialization formats, and the pipelines that pull artifacts into production — are now first-class attack surfaces. Nation-state operators and criminal groups understand that a poisoned model or compromised model-hub credential is a shortcut past your perimeter: the "payload" arrives as a trusted artifact, loaded by a trusted process, inside your own VPC. Second, Shostack's PHANTOM-B framework signals that the security community is still catching up on how to systematically model these threats rather than reacting to them ad hoc.

If your organization fine-tunes models, pulls base models or datasets from public hubs, or deploys LLM-powered applications, this is your supply chain problem now. This post breaks down the defensive implications and gives your SOC concrete detection content for the most dangerous technique in this space: malicious model artifacts achieving code execution at load time.

Technical Analysis

The Attack Surface: Why Model Hubs Are High-Value Targets

Hugging Face hosts millions of models, datasets, and demo "Spaces." The platform's role in modern MLOps mirrors what npm and PyPI are to software development — which means it inherits the same attack classes, plus some uniquely dangerous ones:

• Malicious serialized models. Python's pickle format — the default serialization for PyTorch .pt/.pth/.bin weights — executes arbitrary code during deserialization. A model file is not inert data; torch.load() on a crafted file runs attacker code with the privileges of the loading process. This has been publicly demonstrated for years, and scanners (including Hugging Face's own Pickle Scanner and third-party tools like fickling and modelscan) exist precisely because this is exploited in practice.
• Compromised hub credentials and tokens. Prior Hugging Face incidents (including the 2024 Spaces secrets exposure) demonstrated that stolen API tokens let attackers overwrite popular models or inject malware into trusted namespaces. Downstream consumers who pin only a model name — not a revision hash — inherit the compromise automatically.
• Typosquatted and namespace-jacked models. Attackers publish look-alike repos mimicking popular models, counting on from_pretrained("org/model") typos or automated pipeline pulls.
• Poisoned datasets and LoRA adapters. Adapters and fine-tune datasets are lower-privilege but can embed backdoored behavior — models that behave normally in testing but exfiltrate or misbehave on a trigger token.
• LLM application-layer threats. Prompt injection, indirect prompt injection via retrieved content, and agent tool-abuse — the class of issues that OpenAI's own disclosures around the incident highlighted and that PHANTOM-B aims to structure.

No CVE identifier is associated with this news item, and I won't fabricate one. The relevant framing is MITRE ATT&CK T1195 (Supply Chain Compromise) and T1195.002 (Compromise Software Supply Chain), alongside MITRE ATLAS techniques for ML supply chain and model evasion. Exploitation status for malicious-model delivery is confirmed in the wild as a technique class: security researchers have repeatedly found live malicious pickles on public hubs, and Hugging Face operates malware-scanning infrastructure for exactly this reason.

What PHANTOM-B Means for Your Program

Shostack's point — that a threat model for LLMs must be lightweight yet usable — is a direct rebuke to the tendency to produce 80-page LLM risk frameworks nobody operationalizes. The defensive takeaway: you don't need a perfect ontology of AI threats to act. You need to answer four questions for every model and dataset entering your environment:

1. Where did this artifact come from, and can I verify it? (Provenance, signed commits, pinned revision SHAs.)
2. What happens when it's loaded? (Serialization format, sandboxing, scanner results.)
3. What can it reach? (Network egress from training/inference hosts, credential access, tool permissions for agents.)
4. What does the model do with untrusted input? (Prompt injection exposure, RAG content trust, tool-call guardrails.)

Organizations that can answer those four questions with evidence — not aspiration — are ahead of most of the industry. The remainder of this post focuses on the detection and hardening controls that produce that evidence.

Detection & Response

The highest-fidelity, lowest-noise detections in this space focus on the moment a model artifact becomes executable: deserialization and the anomalous behavior of ML framework processes afterward. These rules are grounded in the malicious-pickle and hub-compromise techniques described above, not speculative indicators.

Sigma Rules

---
title: Python or ML Framework Process Spawning Unexpected Child Process
id: 3f7c2a91-8e44-4b6d-9a12-5d8e1f3a9c07
status: experimental
description: Detects python, torchserve, triton, or notebook kernels spawning shells, script interpreters, or download tools — consistent with arbitrary code execution during malicious model (pickle) deserialization or poisoned notebook artifacts.
references:
  - https://attack.mitre.org/techniques/T1195/002/
  - https://www.darkreading.com/vulnerabilities-threats/adam-shostack-talks-hugging-face-phantom-b
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.execution
  - attack.t1059
  - attack.t1195.002
logsource:
  category: process_creation
  product: windows
detection:
  selection_parent:
    ParentImage|endswith:
      - '\python.exe'
      - '\pythonw.exe'
      - '\torchserve.exe'
      - '\jupyter.exe'
      - '\ipython.exe'
  selection_child:
    Image|endswith:
      - '\cmd.exe'
      - '\powershell.exe'
      - '\pwsh.exe'
      - '\wscript.exe'
      - '\cscript.exe'
      - '\mshta.exe'
      - '\curl.exe'
      - '\certutil.exe'
      - '\bitsadmin.exe'
  condition: selection_parent and selection_child
falsepositives:
  - Legitimate ML pipelines invoking shell commands (e.g., data preprocessing wrappers); tune per pipeline host
level: high
---
title: Model Weight File Written Outside Designated Model Store
id: 9b1d4e62-2c58-4a71-bf93-7e2c5d8a1b46
status: experimental
description: Detects serialized model artifacts (.pt, .pth, .pkl, .bin, .ckpt, .safetensors) being written to temp, user profile, or other non-standard directories — a common staging pattern when pipelines pull unvetted models from public hubs at runtime.
references:
  - https://attack.mitre.org/techniques/T1195/002/
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.initial_access
  - attack.t1195.002
logsource:
  category: file_event
  product: windows
detection:
  selection_ext:
    TargetFilename|endswith:
      - '.pt'
      - '.pth'
      - '.pkl'
      - '.pickle'
      - '.ckpt'
  selection_path:
    TargetFilename|contains:
      - '\AppData\Local\Temp\'
      - '\AppData\Roaming\'
      - '\Users\Public\'
      - '\Downloads\'
  condition: selection_ext and selection_path
falsepositives:
  - Data scientists experimenting in notebooks; whitelist known research workstations or redirect them to a governed model store
level: medium
---
title: Linux ML Process Loading Suspicious Module or Executing Shell After Model Load
id: 5c8a3f17-9d2b-4e65-a841-3f6b9c2d7e15
status: experimental
description: Detects python processes on Linux ML hosts spawning interactive shells or network utilities — a hallmark of pickle-deserialization payloads executing at torch.load() time.
references:
  - https://attack.mitre.org/techniques/T1195/002/
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.execution
  - attack.t1059.006
  - attack.t1195.002
logsource:
  category: process_creation
  product: linux
detection:
  selection_parent:
    ParentImage|endswith:
      - '/python'
      - '/python3'
      - '/torchserve'
  selection_child:
    Image|endswith:
      - '/sh'
      - '/bash'
      - '/dash'
      - '/curl'
      - '/wget'
      - '/nc'
      - '/ncat'
      - '/socat'
      - '/base64'
  condition: selection_parent and selection_child
falsepositives:
  - Legitimate preprocessing scripts; baseline per host and alert on deviations
level: high

KQL — Microsoft Sentinel / Defender

This query hunts the same behavior across both Windows and Linux endpoints via Defender process telemetry, plus hub-download activity visible in network logs. Run it as a scheduled analytic rule on ML workstations and inference hosts, with a wider net during threat hunts.

// Hunt: ML framework processes spawning shells or download tools (pickle deserialization payloads)
let ml_parents = dynamic(["python.exe", "pythonw.exe", "python", "python3", "torchserve", "jupyter", "ipython"]);
let suspicious_children = dynamic(["cmd.exe", "powershell.exe", "pwsh.exe", "mshta.exe", "wscript.exe", "cscript.exe", "certutil.exe", "bitsadmin.exe", "curl.exe", "curl", "wget", "sh", "bash", "nc", "ncat", "socat", "base64"]);
DeviceProcessEvents
| where TimeGenerated > ago(7d)
| where InitiatingProcessFileName in~ (ml_parents)
| where FileName in~ (suspicious_children)
| project TimeGenerated, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, FileName, ProcessCommandLine, AccountName, InitiatingProcessFolderPath
| order by TimeGenerated desc;

// Hunt: outbound pulls from model hubs on hosts that are NOT designated ML build/research systems
DeviceNetworkEvents
| where TimeGenerated > ago(7d)
| where RemoteUrl has_any ("huggingface.co", "hf.co", "cdn-lfs.huggingface.co")
| join kind=leftanti (
    DeviceInfo
    | where DeviceName has_any ("ml-build", "ds-workstation", "training")
    | summarize by DeviceName
) on DeviceName
| summarize FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), Connections=count() by DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, RemoteUrl
| order by FirstSeen desc;

Velociraptor VQL

Use this artifact for fleet-wide triage when you suspect a poisoned model was introduced — it surfaces model artifacts staged in suspicious locations together with the processes that touched them.

-- Hunt for serialized model artifacts staged in user/temp locations and recently-modified
SELECT FullPath, Size, Mtime, Btime
FROM glob(globs=[
  'C:/Users/*/AppData/Local/Temp/**/*.p*',
  'C:/Users/*/Downloads/**/*.{pt,pth,pkl,ckpt,bin}',
  '/tmp/**/*.{pt,pth,pkl,ckpt}',
  '/home/*/.cache/huggingface/**'
])
WHERE Mtime > (now() - 604800)  -- last 7 days
ORDER BY Mtime DESC

-- Correlate: python processes with model-loading or hub-download command lines
SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE CommandLine =~ '(torch\.load|from_pretrained|huggingface_hub|hf_hub_download|snapshot_download)'
   OR Exe =~ '(python|torchserve)'

Remediation & Verification Script

The following Bash script audits a Linux ML host for the most common exposure patterns: unsafe torch.load usage, unpinned model pulls, and presence of a model-artifact scanner. Adapt paths to your environment.

#!/bin/bash
# audit_ml_supplychain.sh — verify ML host hardening against malicious model artifacts
set -u

echo "=== [1] Scanning codebases for unsafe deserialization patterns ==="
# torch.load without weights_only=True allows arbitrary pickle execution (PyTorch >= 2.0)
grep -rn --include='*.py' -E 'torch\.load\(' /opt /srv /home 2>/dev/null | \
  grep -v 'weights_only' | head -50
echo "-> Any hits above lack weights_only=True; enforce it or migrate to safetensors."

echo "=== [2] Checking for unpinned Hugging Face pulls ==="
# from_pretrained without a revision= pins to mutable 'main' — a supply chain risk
grep -rn --include='*.py' -E 'from_pretrained\(' /opt /srv /home 2>/dev/null | \
  grep -v 'revision=' | head -50

echo "=== [3] Verifying model scanner availability ==="
if command -v modelscan >/dev/null 2>&1; then
  echo "modelscan present: $(modelscan --version 2>/dev/null || echo 'version unknown')"
else
  echo "MISSING: install modelscan (pip install modelscan) and gate all inbound artifacts through it."
fi

echo "=== [4] Auditing HF token exposure on host ==="
find /home /root /opt -name 'token' -path '*huggingface*' -o -name '.huggingface' -type d 2>/dev/null | head -20
env | grep -i 'HF_TOKEN\|HUGGING_FACE' | sed 's/=.*/=<redacted-present>/'

echo "=== [5] Egress check: can this host reach arbitrary internet endpoints? ==="
curl -s -m 5 -o /dev/null -w "Internet egress HTTP status: %{http_code}\n" https://huggingface.co || echo "Egress blocked or filtered (good for inference hosts)."

echo "=== Audit complete. Review findings and remediate per policy. ==="

Remediation

There is no patch to apply here — this is an architectural and supply-chain discipline problem. Remediate in layers:

1. Govern artifact intake (highest priority).

• Route all model and dataset pulls through an internal, curated registry or proxy. Block direct internet pulls of model artifacts from production inference and training hosts at the egress firewall, allowing only approved hubs/CDN endpoints from approved build systems.
• Pin every from_pretrained() and snapshot_download() call to an immutable revision= commit SHA — never floating main.
• Require safetensors format wherever possible. Where pickle-based artifacts are unavoidable, scan every file with modelscan or fickling before load, and load with weights_only=True on PyTorch 2.x.

2. Isolate the load operation.

• Load untrusted models in a sandboxed, network-segmented environment with no credentials, no internet egress, and no access to production data. Treat first-load of a new model version like detonating a suspicious binary — because functionally, that's what it is.
• Run inference/training services as dedicated low-privilege identities with scoped IAM; no cloud metadata access where avoidable, and IMDSv2/hardened metadata where not.

3. Protect hub credentials.

• Use fine-grained Hugging Face access tokens scoped per-repository with read-only permissions wherever possible; store them in a secrets manager, never in environment variables on shared hosts or in .cache/huggingface on production systems.
• Rotate any token that has ever existed on a host where an unvetted artifact was loaded.
• Enable SSO/2FA for your organization's hub accounts and audit org membership and commit history for popular internal repos.

4. Adopt lightweight threat modeling — the PHANTOM-B lesson.

• For each LLM application, run a short threat-modeling pass covering: artifact provenance, prompt-injection exposure (direct and indirect via RAG/tools), agent tool permissions, and data exfiltration paths. Shostack's argument is that a usable model beats a comprehensive one; a one-page model your engineers actually update is worth more than a framework nobody reads.
• Add model-artifact review to your change-management and vendor-assessment processes — third-party "AI features" are third-party code.

5. Detection operations.

• Deploy the Sigma/KQL content above to ML-adjacent hosts. Baseline legitimate pipeline behavior first; deviations from a stable ML baseline are high-signal.
• Subscribe to Hugging Face security advisories and your framework vendors' security feeds; treat model-hub incidents with the same severity as npm/PyPI compromise events in your IR playbooks.

The uncomfortable truth Shostack's reaction underscores: the industry is deploying LLM systems faster than it's learning to defend them, and attackers are already at the model hub. The defenders who win are the ones who treat models as code, provenance as non-negotiable, and threat modeling as a lightweight habit rather than a heavyweight ceremony.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub