CVE-2026-24232: NVIDIA Transformers4Rec Checkpoint Deserialization RCE — Detection and Remediation Guide

The Zero Day Initiative has published ZDI-26-564, disclosing a deserialization-of-untrusted-data vulnerability in NVIDIA Transformers4Rec tracked as CVE-2026-24232, with a CVSS v3.1 score of 7.8 (High). The flaw resides in the load_model_trainer_states_from_checkpoint function, which processes model checkpoint state without adequately restricting what gets deserialized. A remote attacker who convinces a target to open a malicious checkpoint file — or who poisons a checkpoint delivered through an ML pipeline, model-sharing repository, or supply chain — can achieve arbitrary code execution in the context of the process loading the model.

This is the exact class of vulnerability we've been warning ML and data science teams about for the past two years: Python pickle-based model artifacts are executable code, not data. Transformers4Rec, NVIDIA's recommendation-system library built on PyTorch, follows the common pattern of serializing trainer state with Python's native object serialization. Anyone who has responded to a pickle deserialization incident knows what comes next — a single torch.load() or equivalent call on an attacker-crafted file, and the adversary has a shell on your GPU workstation or training cluster.

Defenders need to act on this for three reasons:

1. ML engineers are soft targets. Data scientists routinely download checkpoints from Hugging Face, internal model registries, and academic repos, then load them with elevated privileges on powerful, often poorly monitored GPU hosts.
2. User interaction is the only barrier. ZDI notes exploitation requires the target to visit a malicious page or open a malicious file — a low bar in workflows where pulling a 'community fine-tuned' checkpoint is daily routine.
3. GPU infrastructure is high-value. Training nodes frequently hold proprietary datasets, cloud credentials, and lateral-movement paths into internal networks that SOC teams rarely instrument.

Technical Analysis

Affected Product

• Product: NVIDIA Transformers4Rec (part of the NVIDIA Merlin recommender-system framework, built on PyTorch)
• Component: load_model_trainer_states_from_checkpoint — the function responsible for restoring trainer/optimizer/scheduler state when resuming training from a checkpoint
• Vulnerability Class: CWE-502 — Deserialization of Untrusted Data
• CVE: CVE-2026-24232
• CVSS v3.1: 7.8 (High) — vector consistent with local-file-triggered RCE requiring user interaction (AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H)
• Advisory: ZDI-26-564 — http://www.zerodayinitiative.com/advisories/ZDI-26-564/

How the Vulnerability Works (Defender's View)

Python's pickle module — and by extension PyTorch's torch.load(), which wraps it — reconstructs arbitrary Python objects by executing embedded opcodes during deserialization. The __reduce__ protocol lets a serialized object specify an arbitrary callable (e.g., os.system, subprocess.Popen) and its arguments, which are invoked the moment the file is loaded. No bug in pickle is required; this is documented, intended behavior.

The vulnerable code path in Transformers4Rec's load_model_trainer_states_from_checkpoint loads checkpoint state files without enforcing a safe serialization format (such as safetensors) or a restricted unpickler. The attack chain looks like this:

1. Delivery: Attacker hosts a malicious .pt / .pth / trainer checkpoint file on a public model hub, sends it via phishing, or plants it in a shared artifact store / experiment-tracking system (MLflow, Weights & Biases, internal S3 bucket).
2. Trigger: A data scientist or an automated pipeline calls load_model_trainer_states_from_checkpoint() — directly or via a 'resume training' operation — on the attacker-controlled file.
3. Execution: The embedded __reduce__ payload executes during deserialization, running attacker code with the privileges of the Python process. On training infrastructure, that frequently means root inside a container with host-mounted volumes, cloud IAM instance credentials, and access to training datasets.
4. Post-exploitation: Typical follow-on behavior we observe in ML-environment intrusions: the Python process spawns a shell or downloads a second-stage payload, persistence is established (cron, systemd, .bashrc, Jupyter startup scripts), and credential harvesting begins (.aws/credentials, kubeconfigs, SSH keys).

Exploitation Status

• Public PoC: None confirmed at time of writing; ZDI published the advisory ahead of broad exploitation.
• Active exploitation: Not confirmed in the wild as of this publication.
• CISA KEV: Not currently listed.

That said, the window between a ZDI deserialization advisory and weaponization in the ML tooling space has historically been short, and the exploitation technique (malicious pickle) is fully public knowledge. Treat this as pre-exploitation but trivially weaponizable — the correct posture is proactive patching and detection, not waiting for a KEV entry.

Detection & Response

The most reliable observable for this vulnerability class is a Python/ML training process spawning an unexpected child process or making an unexpected network connection immediately after loading a checkpoint. Pickle payloads execute inline within the Python interpreter, so the parent process is python (or python3, ipython, jupyter, dask-worker, etc.), and the malicious child is typically a shell, curl/wget, or a reverse-shell binary. Baseline your ML hosts: legitimate training jobs essentially never spawn cmd.exe, /bin/sh, or download tools as children of the Python process.

---
title: ML Python Process Spawning Shell or Downloader (Checkpoint Deserialization RCE)
id: 3f7c2a91-8b4d-4e6a-9c12-5d8e0f1a2b34
status: experimental
description: Detects Python/ML framework processes spawning shells, downloaders, or script interpreters, consistent with malicious pickle/checkpoint deserialization (e.g., CVE-2026-24232 in NVIDIA Transformers4Rec).
references:
  - http://www.zerodayinitiative.com/advisories/ZDI-26-564/
  - https://attack.mitre.org/techniques/T1059/
  - https://cwe.mitre.org/data/definitions/502.html
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.execution
  - attack.t1059.006
logsource:
  category: process_creation
  product: windows
detection:
  selection_parent:
    ParentImage|endswith:
      - '\python.exe'
      - '\pythonw.exe'
      - '\ipython.exe'
      - '\jupyter.exe'
      - '\conda.exe'
  selection_child:
    Image|endswith:
      - '\cmd.exe'
      - '\powershell.exe'
      - '\pwsh.exe'
      - '\wscript.exe'
      - '\cscript.exe'
      - '\certutil.exe'
      - '\curl.exe'
      - '\bitsadmin.exe'
      - '\rundll32.exe'
      - '\regsvr32.exe'
  condition: selection_parent and selection_child
falsepositives:
  - Rare legitimate ML tooling that shells out (e.g., nvcc compilation, some data-prep libraries) - baseline per host and tune by command line
level: high
---
title: Linux Python Training Process Spawning Shell (Malicious Checkpoint Load)
id: 9a1e5d63-2c7f-4b8e-a345-6f0d1c2e3a45
status: experimental
description: Detects python/python3 processes on Linux spawning interactive shells or download utilities, a high-fidelity indicator of pickle/checkpoint deserialization code execution on ML training nodes.
references:
  - http://www.zerodayinitiative.com/advisories/ZDI-26-564/
  - https://attack.mitre.org/techniques/T1059/004/
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.execution
  - attack.t1059.004
logsource:
  category: process_creation
  product: linux
detection:
  selection_parent:
    ParentImage|endswith:
      - '/python'
      - '/python3'
      - '/ipython'
      - '/dask-worker'
      - '/raylet'
  selection_child:
    Image|endswith:
      - '/sh'
      - '/bash'
      - '/dash'
      - '/zsh'
      - '/curl'
      - '/wget'
      - '/nc'
      - '/ncat'
      - '/socat'
      - '/base64'
  condition: selection_parent and selection_child
falsepositives:
  - Some distributed training launchers spawn subshells - validate against known job schedulers (slurm, ray) and tune
level: high
---
title: Suspicious Checkpoint File Creation Followed by Outbound Connection From Python
id: c4d8b0f2-6e3a-41d7-9b58-2a7c9e1f4b56
status: experimental
description: Detects Python processes making outbound network connections on uncommon ports after checkpoint/model artifact files appear on disk, consistent with staged payload retrieval following malicious model loading.
references:
  - http://www.zerodayinitiative.com/advisories/ZDI-26-564/
  - https://attack.mitre.org/techniques/T1105/
author: Security Arsenal
date: 2026/04/06
tags:
  - attack.command_and_control
  - attack.t1105
logsource:
  category: network_connection
  product: windows
detection:
  selection:
    Image|endswith:
      - '\python.exe'
      - '\pythonw.exe'
      - '\jupyter.exe'
  filter_ports:
    DestinationPort:
      - 443
      - 80
  condition: selection and not filter_ports
falsepositives:
  - ML frameworks connecting to non-standard artifact stores, tensorboard, or distributed training endpoints - maintain an allowlist of known internal ML service ports
level: medium

// Hunt: Python/ML processes spawning shells or download tools
// Coverage: Windows (DeviceProcessEvents) and Linux ingested via MDE/Syslog
// Relevant to CVE-2026-24232 exploitation (malicious checkpoint deserialization)
let mlParents = dynamic(["python.exe","pythonw.exe","python","python3","ipython","jupyter.exe","dask-worker","raylet"]);
let suspiciousChildren = dynamic(["cmd.exe","powershell.exe","pwsh.exe","certutil.exe","curl.exe","bitsadmin.exe","rundll32.exe","regsvr32.exe","wscript.exe","cscript.exe",
    "sh","bash","dash","zsh","curl","wget","nc","ncat","socat","base64"]);
DeviceProcessEvents
| where TimeGenerated > ago(14d)
| where InitiatingProcessFileName has_any (mlParents)
| where FileName has_any (suspiciousChildren)
| project TimeGenerated, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine,
          FileName, ProcessCommandLine, AccountName, InitiatingProcessId, ProcessId
| order by TimeGenerated desc;

-- Hunt artifact: identify Python processes with suspicious child processes
-- or outbound connections on GPU/training hosts (checkpoint deserialization RCE)
-- Deploy across Linux ML nodes via Velociraptor hunt

SELECT Pid, Ppid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE Name =~ '(?i)python|ipython|dask|raylet'
  AND CommandLine =~ '(?i)load_model_trainer_states_from_checkpoint|transformers4rec|torch.load|checkpoint'

-- Correlate with suspicious children of python PIDs
SELECT Pid, Ppid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE Name =~ '(?i)^(sh|bash|dash|curl|wget|nc|ncat|socat)$'
  AND Ppid IN (
      SELECT Pid FROM pslist()
      WHERE Name =~ '(?i)python|ipython|dask|raylet'
  )

#!/bin/bash
# CVE-2026-24232 - Transformers4Rec exposure audit and interim hardening
# Run on ML workstations, training nodes, and CI/CD runners
# Usage: sudo bash transformers4rec_audit.sh

set -u

echo "=== [1/4] Enumerating Transformers4Rec installations across Python environments ==="
for py in $(command -v python python3 python3.9 python3.10 python3.11 python3.12 2>/dev/null | sort -u); do
    ver=$("$py" -m pip show transformers4rec 2>/dev/null | grep -i '^Version:' || echo "not installed: $py")
    echo "[$py] $ver"
done

# Also sweep conda environments
if command -v conda >/dev/null 2>&1; then
    for env_path in $(conda env list | grep -v '^#' | awk '{print $NF}' | grep '^/'); do
        echo "[conda env: $env_path]"
        "$env_path/bin/python" -m pip show transformers4rec 2>/dev/null | grep -i '^Version:' || echo "  not installed"
    done
fi

echo ""
echo "=== [2/4] Scanning for recently created/modified checkpoint artifacts (.pt/.pth/.ckpt) ==="
find /home /opt /srv /data /tmp -type f \( -name '*.pt' -o -name '*.pth' -o -name '*.ckpt' -o -name 'trainer_state*' \) -mtime -14 2>/dev/null | head -100

echo ""
echo "=== [3/4] Checking for python-spawned shell anomalies in audit logs (last 7 days) ==="
if command -v ausearch >/dev/null 2>&1; then
    ausearch -ts recent -i 2>/dev/null | grep -iE 'type=EXECVE' | grep -iE 'python.*(sh|bash|curl|wget|nc )' | tail -50
else
    echo "auditd not present; skipping execve history check"
fi

echo ""
echo "=== [4/4] Interim hardening: enforce weights_only default for torch.load in user sitecustomize ==="
# This injects a sitecustomize shim that defaults torch.load to weights_only=True.
# REVIEW before deploying broadly - some legacy workloads legitimately need full pickle.
read -r -p "Apply weights_only=True shim to system python site-packages? [y/N] " ans
if [[ "$ans" == "y" ]]; then
    for sp in $(python3 -c 'import site; print(" ".join(site.getsitepackages()))' 2>/dev/null); do
        cat > "$sp/sitecustomize.py" <<'EOF'
# Security Arsenal interim mitigation - CVE-2026-24232 class
# Default torch.load to weights_only=True to block arbitrary pickle execution
import os
if os.environ.get("ALLOW_PICKLE_CHECKPOINTS") != "1":
    try:
        import torch
        _orig_load = torch.load
        def _safe_load(*a, **kw):
            kw.setdefault("weights_only", True)
            return _orig_load(*a, **kw)
        torch.load = _safe_load
    except ImportError:
        pass
EOF
        echo "shim written to $sp/sitecustomize.py"
    done
fi

echo ""
echo "Audit complete. Review checkpoint file list for unexpected artifacts and verify provenance of all training checkpoints."

Remediation

1. Patch / upgrade immediately. Apply the fixed Transformers4Rec release referenced in NVIDIA's security bulletin for CVE-2026-24232. Confirm your installed version via pip show transformers4rec across every Python environment — ML hosts routinely carry multiple conda envs, container images, and virtualenvs, and patching only the system interpreter leaves the real exposure untouched. Also rebuild and redeploy any container images (NGC Merlin images, custom training images) that vendor the vulnerable library. Monitor NVIDIA's security advisories page (https://www.nvidia.com/en-us/security/) and the ZDI advisory for the patched version number and vendor guidance.

2. Enforce safe serialization formats. Standardize on safetensors for model weights and trainer state wherever possible — it is a data-only format with no code-execution surface. Where PyTorch is used, pin torch.load(..., weights_only=True) (default behavior in torch ≥ 2.6) and treat any override as a change-controlled exception. The bash shim above provides an interim enforcement point until the library patch is deployed.

3. Institute checkpoint provenance controls.

• Require hash verification (SHA-256) and signed artifacts for any checkpoint entering your environment, including internal model registries.
• Block direct downloads of model artifacts to training hosts at the proxy; route through a vetted internal registry with scanning.
• Scan all inbound .pt/.pth/.ckpt files with a pickle-aware scanner (e.g., picklescan, fickling) in CI before they reach a training node.

4. Reduce blast radius on ML infrastructure.

• Run training workloads as non-root, in containers without host volume mounts and with dropped capabilities (--cap-drop=ALL, no-new-privileges).
• Egress-filter GPU/training subnets: training nodes need PyPI, your registry, and storage — nothing else. Deny arbitrary outbound to break reverse shells and second-stage retrieval.
• Remove cloud instance credentials from training hosts where possible; use short-lived, workload-scoped tokens.

5. Deploy the detections above. Forward process-creation telemetry (Sysmon on Windows, auditd/eBPF on Linux) from all ML workstations and training nodes to your SIEM. These hosts are chronically under-monitored; this advisory is the forcing function to fix that. Deploy the Sigma rules, the Sentinel hunt query, and the Velociraptor artifact, and baseline legitimate scheduler behavior (Slurm, Ray, Dask) to tune false positives.

6. Hunt retroactively. Checkpoint files persist. Use the file-creation sweep in the audit script to enumerate model artifacts from the past 30 days, verify provenance for each, and check execution history for python-spawned shells during that window. If your organization pulled community checkpoints recently, assume exposure until provenance is confirmed.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub