LLM 0.36 CLI Release: Defending Against Shadow AI and Unsanctioned LLM Tooling in the Enterprise

Simon Willison has released version 0.36 of llm, his widely adopted open-source command-line tool for interacting with large language models. This is not a vulnerability disclosure — there is no CVE here — but make no mistake: every release of a capable, scriptable LLM client is a security-relevant event for enterprise defenders. Tools like llm lower the barrier for piping sensitive corporate data — source code, database dumps, log files, incident tickets — directly into third-party model APIs from any workstation or server where Python is installed. If your organization does not have visibility into where LLM CLI tooling is installed, what it is sending, and to which endpoints, you have a shadow AI problem whether you know it or not.

This post covers what llm 0.36 represents from a defensive standpoint, how these tools get abused (both by well-meaning employees and by threat actors living off the land), and the concrete detection and governance controls your SOC should have in place today.

Why a CLI Release Matters to Defenders

The llm tool is a legitimate, high-quality piece of software — that is precisely the problem. It supports dozens of models via plugins (OpenAI, Anthropic, Gemini, Mistral, local models via llama.cpp and Ollama), stores conversation logs and API keys in a predictable local directory, supports embedding pipelines over arbitrary files, and — critically in recent versions — supports tool use and function calling, allowing models to execute commands and chain actions autonomously.

From a defender's perspective, that feature set maps directly onto three risk categories:

1. Data egress. llm makes it trivial to run cat customer_records.csv | llm "summarize this", sending regulated data (PCI, PHI, PII) to an external API in a single piped command. Under HIPAA and PCI-DSS, that is a reportable event if the destination is not a covered, contracted service.
2. Living-off-the-land abuse. A Python-based, legitimately signed, pip-installed CLI is an attractive post-exploitation utility. An attacker with a foothold can install llm, configure it with their own API key, and use it for automated reconnaissance summarization, script generation, or — worse — as a low-and-slow exfiltration channel that blends into legitimate AI traffic.
3. Prompt injection and agentic risk. With tool-calling enabled, a model processing untrusted content (a web page, a ticket, an email) can be induced to execute commands. Any deployment of llm with tool plugins in an automation pipeline is a prompt injection attack surface.

Technical Analysis: Observable Footprint of the llm CLI

You cannot detect what you have not inventoried. Here is what llm looks like on an endpoint and on the wire — these are the real, observable artifacts your detections should key on:

Process execution:

• Binary installed via pip install llm or pipx install llm; resolves to llm (Linux/macOS) or llm.exe (Windows, under Python Scripts directories)
• Typical command lines: llm "<prompt>", llm -m gpt-4o ..., llm embed ..., llm chat, llm logs
• Plugin installation: llm install llm-claude-3, llm install llm-gemini, etc. — these spawn pip child processes

File system artifacts:

• Linux: ~/.config/io.datasette.llm/ — contains keys.json (API keys, often plaintext), logs.db (full SQLite history of every prompt and response), and templates.yaml
• macOS: ~/Library/Application Support/io.datasette.llm/
• The logs.db file is a forensic goldmine: every prompt and response the tool has ever processed is queryable SQLite. In an IR engagement, pull it.

Network indicators:

• TLS egress to api.openai.com, api.anthropic.com, generativelanguage.googleapis.com, api.mistral.ai, openrouter.ai from a non-browser process
• On developer workstations this may be authorized; on servers, finance endpoints, or clinical workstations it almost certainly is not

Exploitation/abuse status: There is no known vulnerability in llm 0.36 and no malicious exploitation of the tool itself. The risk is usage-based, not bug-based — which means your controls are behavioral detection and policy enforcement, not patching.

Detection & Response

The detections below target unsanctioned installation and use of LLM CLI tooling. Tune the allowlists to your approved AI tooling before deploying — if your engineering org has sanctioned llm usage, scope these rules to exclude approved hosts and service accounts.

Sigma Rules

---
title: LLM CLI Tool Execution Detected
id: 4f8c2a91-7b3e-4d51-9c06-2a7e5f8b1d34
status: experimental
description: Detects execution of the llm command-line AI client, which may indicate unsanctioned shadow AI usage or data egress to external LLM APIs. Tune allowlists for approved AI tooling in your environment.
references:
  - https://llm.datasette.io/
  - https://attack.mitre.org/techniques/T1567/002/
author: Security Arsenal
date: 2026/09/24
tags:
  - attack.exfiltration
  - attack.t1567.002
logsource:
  category: process_creation
  product: windows
detection:
  selection_img:
    Image|endswith:
      - '\llm.exe'
  selection_cli:
    CommandLine|contains:
      - 'llm embed'
      - 'llm chat'
      - 'llm install llm-'
      - 'llm logs'
  condition: selection_img or selection_cli
falsepositives:
  - Approved developer AI tooling — maintain an allowlist of sanctioned hosts and users
level: medium
---
title: LLM CLI Installed via Pip or Pipx
id: 8b1d4e72-3c9a-4f67-b215-6d0a9e3c5f87
status: experimental
description: Detects installation of the llm package or llm plugins via pip/pipx, indicating new shadow AI tooling being introduced to an endpoint.
references:
  - https://llm.datasette.io/
  - https://attack.mitre.org/techniques/T1105/
author: Security Arsenal
date: 2026/09/24
tags:
  - attack.command_and_control
  - attack.t1105
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith:
      - '\pip.exe'
      - '\pipx.exe'
      - '\python.exe'
    CommandLine|contains:
      - 'install llm'
      - 'install "llm'
      - 'llm-claude'
      - 'llm-gemini'
      - 'llm-ollama'
      - 'llm-openrouter'
  condition: selection
falsepositives:
  - Legitimate Python package management by developers — scope to server and non-engineering endpoint populations for best signal
level: medium
---
title: Sensitive File Piped to LLM CLI
id: 2c7f9b15-5e41-4a83-9d60-8b3c1e7a4f29
status: experimental
description: Detects shell pipelines redirecting file contents into the llm CLI, a common pattern for exfiltrating or processing sensitive documents via external AI APIs.
references:
  - https://attack.mitre.org/techniques/T1059/
  - https://attack.mitre.org/techniques/T1567/002/
author: Security Arsenal
date: 2026/09/24
tags:
  - attack.exfiltration
  - attack.t1567.002
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine|contains:
      - '| llm'
      - 'cat '
  selection_llm:
    CommandLine|contains:
      - 'llm "'
      - 'llm -m'
      - 'llm -s'
  condition: selection and selection_llm
falsepositives:
  - Approved AI-assisted development workflows
level: high

KQL — Microsoft Sentinel / Defender

This query hunts for LLM CLI process execution joined against outbound connections to known LLM API endpoints, highlighting non-browser processes talking to AI services — the highest-signal shadow AI indicator available in Defender telemetry:

let LLMApiEndpoints = dynamic(["api.openai.com", "api.anthropic.com", "generativelanguage.googleapis.com", "api.mistral.ai", "openrouter.ai", "api.cohere.ai", "api.groq.com"]);
let BrowserProcesses = dynamic(["chrome.exe", "msedge.exe", "firefox.exe", "brave.exe", "safari"]);
let LLMCliUsage = DeviceProcessEvents
    | where TimeGenerated > ago(7d)
    | where FileName =~ "llm.exe" or FileName =~ "llm"
       or ProcessCommandLine has_any ("llm embed", "llm chat", "llm install llm-", "llm logs")
    | project ProcessTime=TimeGenerated, DeviceName, AccountName, ProcessCommandLine, FileName, DeviceId;
let AIApiTraffic = DeviceNetworkEvents
    | where TimeGenerated > ago(7d)
    | where RemoteUrl has_any (LLMApiEndpoints)
    | where not(InitiatingProcessFileName has_any (BrowserProcesses))
    | project NetTime=TimeGenerated, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, RemoteUrl, RemoteIP, DeviceId;
LLMCliUsage
| join kind=leftouter AIApiTraffic on DeviceId
| project ProcessTime, NetTime, DeviceName, AccountName, ProcessCommandLine, InitiatingProcessFileName, RemoteUrl, RemoteIP
| order by ProcessTime desc

A second, standalone query for environments ingesting Linux syslog (e.g., via AMA) to catch llm usage on servers — where it should essentially never appear:

Syslog
| where TimeGenerated > ago(7d)
| where SyslogMessage has_cs "llm" and SyslogMessage has_any ("llm embed", "llm chat", "| llm", "llm -m", "pip install llm", "pipx install llm")
| project TimeGenerated, Computer, HostIP, ProcessName, SyslogMessage
| order by TimeGenerated desc

Velociraptor VQL

This artifact hunts live for llm processes and locates the tool's state directory (which contains API keys and the full prompt/response SQLite log) — useful both for shadow AI discovery and for DFIR scoping after an incident:

-- Hunt for llm CLI processes and their local state directories (keys.json, logs.db)
SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE Name =~ '(?i)^llm(\.exe)?$'
   OR CommandLine =~ '(?i)llm (embed|chat|logs|install|prompt)'

SELECT FullPath, Size, Mtime
FROM glob(globs=[
  '/home/*/.config/io.datasette.llm/**',
  '/root/.config/io.datasette.llm/**',
  '/Users/*/Library/Application Support/io.datasette.llm/**',
  'C:/Users/*/.config/io.datasette.llm/**'
])
WHERE FullPath =~ '(?i)(keys\.json|logs\.db|templates\.yaml)$'

Audit Script — Find Unsanctioned LLM CLI Installations

Run this across your Linux/macOS fleet (via your RMM, Ansible, or osquery scheduled query equivalent) to inventory where llm is installed and whether it holds configured API keys:

#!/usr/bin/env bash
# audit-llm-cli.sh — Inventory llm CLI installations and configured API keys
# Run as root to cover all user profiles. Output: CSV to stdout.

echo "hostname,user,llm_version,state_dir,keys_configured,logs_db_size_bytes"

HOSTNAME=$(hostname)

for USER_HOME in /home/* /root /Users/*; do
  [ -d "$USER_HOME" ] || continue
  USER=$(basename "$USER_HOME")

  # Locate llm binary for this user (pipx venvs, pip --user, system paths)
  LLM_BIN=$(sudo -u "$USER" bash -lc 'command -v llm 2>/dev/null' 2>/dev/null)
  [ -z "$LLM_BIN" ] && continue

  VERSION=$(sudo -u "$USER" "$LLM_BIN" --version 2>/dev/null | head -1)

  # State directory differs by platform
  if [ -d "$USER_HOME/.config/io.datasette.llm" ]; then
    STATE_DIR="$USER_HOME/.config/io.datasette.llm"
  elif [ -d "$USER_HOME/Library/Application Support/io.datasette.llm" ]; then
    STATE_DIR="$USER_HOME/Library/Application Support/io.datasette.llm"
  else
    STATE_DIR=""
  fi

  KEYS="no"
  if [ -n "$STATE_DIR" ] && [ -s "$STATE_DIR/keys.json" ]; then
    KEYS="yes"
  fi

  LOGS_SIZE=0
  if [ -n "$STATE_DIR" ] && [ -f "$STATE_DIR/logs.db" ]; then
    LOGS_SIZE=$(stat -c%s "$STATE_DIR/logs.db" 2>/dev/null || stat -f%z "$STATE_DIR/logs.db" 2>/dev/null)
  fi

  echo "$HOSTNAME,$USER,\"$VERSION\",$STATE_DIR,$KEYS,$LOGS_SIZE"
done

Remediation and Governance

Because this is a tooling/governance risk rather than a patchable vulnerability, remediation is policy-plus-enforcement:

1. Establish a sanctioned AI tooling list. Decide explicitly which LLM clients, models, and API providers are approved (e.g., your contracted Azure OpenAI or Bedrock tenant), and publish it. You cannot enforce a policy that does not exist.
2. Enforce egress control. Block or alert on outbound TLS to consumer LLM API endpoints (api.openai.com, api.anthropic.com, generativelanguage.googleapis.com, openrouter.ai) from non-browser processes and from server VLANs at the proxy/firewall layer. Route approved AI traffic through your enterprise gateway with DLP inspection.
3. Treat keys.json as a credential store. The llm tool stores provider API keys in keys.json under the user's config directory. Include this path in credential-theft detection content and in IR triage checklists — if an endpoint with configured keys is compromised, rotate those provider keys.
4. Pull logs.db in every relevant IR engagement. If an employee or attacker used llm on a compromised host, logs.db contains the full prompt/response history in SQLite. It answers "what data left this machine through an LLM" definitively — critical for breach notification scoping under HIPAA/PCI/state privacy law.
5. Gate agentic/tool-calling usage. Any llm deployment with tool plugins (command execution, web fetch) in automation pipelines must process only trusted content, or run sandboxed with no network path to sensitive data. Prompt injection against tool-enabled CLI agents is a live attack surface in 2026, not a theoretical one.
6. Software inventory enforcement. Add llm, aichat, mods, sgpt, and similar CLI AI clients to your software inventory policy. Alert on new installations outside approved groups — the audit script above operationalizes this.

The Bottom Line

llm 0.36 is good software from a respected developer — and that is exactly why your users will install it without asking. Shadow AI is the shadow IT problem of this decade, and CLI tooling is its hardest-to-see vector: no browser tab, no SaaS login, just a piped command and a TLS session. Get the inventory, get the egress visibility, and get the policy in place before your next DLP incident is a SQLite file full of prompts you never approved.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub