llm 0.33 Dependency Overhaul: What Security Teams Must Verify Before Upgrading Simon Willison's LLM CLI

On August 22, 2026, Simon Willison shipped llm 0.33, a release of his widely adopted open-source CLI for interacting with large language models. This is not a vulnerability disclosure — there is no CVE here, no exploit in the wild, and no evidence of compromise. But for security teams whose engineers, data scientists, and increasingly AI pipelines run llm on workstations and build servers, this release deserves deliberate attention for a different reason: it is a significant dependency change in a tool that stores and manages LLM provider API keys.

The two headline changes in 0.33 are:

1. An upgrade to the OpenAI Python library 3.x
2. A switch of the HTTP client dependency from httpx to httpx2 (issue #1608, PR #1631)

This follows a rushed 0.32.1 hotfix released the day before — meaning the dependency transition broke something badly enough to warrant an emergency patch, and 0.33 is the comprehensive resolution. Rapid dependency churn plus a hotfix cycle is precisely the pattern a mature security program should treat as a change-control trigger, not a shrug. Additionally, llm embed and llm embed-multi now accept --key, expanding the pathways by which API credentials flow through the tool and its Python API (EmbeddingModel.embed() and related methods).

Why defenders should care: dependency swaps are the canonical delivery mechanism for supply-chain attacks. Every time a tool in your environment changes its transitive dependency tree — especially swapping one HTTP library for another with a similar name — your team should be verifying provenance, pinning versions, and watching for anomalous behavior. This article walks through what changed, what to validate, and how to detect abuse of this tool in your environment.

Technical Analysis

Affected products and versions

Why a dependency swap is a security event

There is no vulnerability in this release. The risk is structural:

• Typosquatting adjacency. Swapping httpx for httpx2 is a legitimate, maintainer-driven change — but it normalizes a lookalike package name in your environment. If an attacker ever publishes a malicious httpx2 update, a httpx-2, or a httpx3, users and automated upgrade pipelines conditioned to expect "httpx-something" are softer targets. Verify that the installed httpx2 resolves to the expected project and maintainer.
• Transitive dependency drift. Upgrading the OpenAI Python library to 3.x pulls a new dependency tree. Any Python environment that auto-resolves dependencies (pip install -U, unpinned CI installs) will silently fetch new code from PyPI. That is an unreviewed code execution path into machines that hold API keys and, frequently, cloud credentials.
• Credential surface expansion. The new --key flag on embedding commands means API keys can now be passed on the command line in more workflows. Command-line secrets land in shell history, process lists, and — critically for defenders — EDR process-creation telemetry. Any engineer using --key sk-... instead of the stored key mechanism is leaking credentials into logs you now have to scrub.
• Hotfix cadence. A same-week hotfix (0.32.1) followed by a comprehensive fix (0.33) indicates the 0.32 line had a regression. Teams that pinned to 0.32.x should validate they're not mid-transition with a mismatched httpx/httpx2 state, which can cause unpredictable HTTP behavior — including failures in any custom plugins that wrap or proxy LLM traffic.

Where llm stores what matters

From a DFIR and threat-hunting standpoint, the high-value artifacts are:

• API key store: llm persists provider keys via llm keys set openai into a keys.json file under its user configuration directory (platform-dependent; typically ~/.config/io.datasette.llm/keys.json on Linux, ~/Library/Application Support/io.datasette.llm/keys.json on macOS, and %APPDATA%\io.datasette.llm\keys.json on Windows). These keys are stored in a form directly usable by any process that reads the file.
• Logs database: llm logs prompts and responses to a SQLite database (logs.db) in the same directory. This is a treasure trove for an attacker — it can contain proprietary code, internal documents pasted into prompts, and business-sensitive context.
• Outbound traffic: API calls egress to provider endpoints such as api.openai.com over 443, normally from a Python interpreter or the llm binary.

Exploitation status

• CVE: None associated with this release. No CVE identifier is claimed or implied by the release notes.
• Active exploitation: None reported. This is a defensive-hygiene and supply-chain assurance post, not an incident advisory.
• CISA KEV: Not applicable.

The actionable threat model is twofold: (1) unvalidated dependency upgrades introducing unreviewed code onto key-bearing machines, and (2) theft of llm-stored API keys or logs by commodity malware or malicious insiders — a technique we routinely see in red team engagements, because developer tool credential stores are far less hardened than enterprise secrets managers.

Detection & Response

The detections below target the two concrete, observable behaviors that matter: (1) unexpected or unscheduled modification of the llm dependency stack via Python package managers, and (2) access to the llm key store or logs database by anything other than the tool itself.

Sigma Rules

---
title: LLM CLI API Key Store Accessed by Non-LLM Process
id: 3f9c1a72-8b4d-4e2a-9c17-5d6e7f8a9b0c
status: experimental
description: Detects processes other than llm/python/pip accessing the llm CLI keys.json credential store or logs.db, which may indicate API key or prompt-log theft.
references:
  - https://github.com/simonw/llm/releases/tag/0.33
  - https://attack.mitre.org/techniques/T1552/001/
author: Security Arsenal
date: 2026/08/22
tags:
  - attack.credential_access
  - attack.t1552.001
logsource:
  category: file_event
  product: windows
detection:
  selection:
    TargetFilename|contains:
      - 'io.datasette.llm\keys.json'
      - 'io.datasette.llm\logs.db'
  filter_legitimate:
    Image|endswith:
      - '\llm.exe'
      - '\python.exe'
      - '\pythonw.exe'
      - '\pip.exe'
  condition: selection and not filter_legitimate
falsepositives:
  - Backup or sync agents indexing user profile directories
  - EDR/AV scanners performing scheduled reads
level: high
---
title: Python Package Manager Modifying llm or httpx2 Dependencies
id: 8c2e4b61-1d3f-4a5b-b7c9-2e4f6a8c0d1e
status: experimental
description: Detects pip/uv/pipx installing or upgrading llm, httpx2, or the openai library. Use for change-control validation around the llm 0.33 dependency swap; expected to fire during legitimate upgrades and should be correlated with approved change windows.
references:
  - https://github.com/simonw/llm/issues/1608
  - https://attack.mitre.org/techniques/T1195/002/
author: Security Arsenal
date: 2026/08/22
tags:
  - attack.persistence
  - attack.t1195.002
logsource:
  category: process_creation
  product: windows
detection:
  selection_tool:
    Image|endswith:
      - '\pip.exe'
      - '\pipx.exe'
      - '\uv.exe'
      - '\python.exe'
  selection_cmd:
    CommandLine|contains:
      - 'install'
  selection_pkg:
    CommandLine|contains:
      - 'llm'
      - 'httpx2'
      - 'openai'
  condition: selection_tool and selection_cmd and selection_pkg
falsepositives:
  - Legitimate developer upgrades to llm 0.33
  - CI/CD pipeline dependency resolution
level: low
---
title: LLM Embedding Command With Inline API Key
id: 5a7d3f92-6c1e-4b8a-ad34-9f0c2b5e7d1a
status: experimental
description: Detects llm embed or embed-multi invoked with the --key flag introduced in llm 0.33, which places provider API keys on the command line where they persist in shell history and EDR telemetry.
references:
  - https://simonwillison.net/2026/Aug/22/llm/
  - https://attack.mitre.org/techniques/T1552/001/
author: Security Arsenal
date: 2026/08/22
tags:
  - attack.credential_access
  - attack.t1552.001
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine|contains|all:
      - 'llm'
      - 'embed'
      - '--key'
falsepositives:
  - Engineers using the new 0.33 --key feature intentionally; treat as a policy-education signal rather than an intrusion indicator
level: medium

KQL (Microsoft Sentinel / Defender)

The following hunt covers both the credential-store access pattern and unplanned dependency modification, using Defender endpoint telemetry. It also works against Syslog/CEF-ingested Linux auditd data if you collect process events from Linux dev hosts into Sentinel.

// Hunt 1: Non-LLM processes touching the llm key store or prompt logs
let KeyArtifacts = dynamic(["keys.json", "logs.db"]);
DeviceFileEvents
| where TimeGenerated > ago(7d)
| where FolderPath has_any ("io.datasette.llm") and FileName has_any (KeyArtifacts)
| where not(InitiatingProcessFileName in~ ("llm.exe", "python.exe", "pythonw.exe", "pip.exe", "pip"))
| project TimeGenerated, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine,
          FolderPath, FileName, ActionType, AccountName
| order by TimeGenerated desc;

// Hunt 2: pip/uv/pipx installs touching llm, httpx2, or openai (change-control validation)
DeviceProcessEvents
| where TimeGenerated > ago(7d)
| where FileName in~ ("pip.exe", "pip", "pip3", "pipx", "uv", "uv.exe", "python.exe", "python")
| where ProcessCommandLine has "install"
| where ProcessCommandLine has_any ("httpx2", "llm==", "llm ", "openai")
| project TimeGenerated, DeviceName, AccountName, FileName, ProcessCommandLine, InitiatingProcessFileName
| order by TimeGenerated desc;

// Hunt 3: API keys exposed on the command line via the new llm embed --key flag
DeviceProcessEvents
| where TimeGenerated > ago(7d)
| where ProcessCommandLine has_all ("llm", "embed", "--key")
| extend KeyExposure = extract(@"--key[= ]([A-Za-z0-9_\-]{20,})", 1, ProcessCommandLine)
| project TimeGenerated, DeviceName, AccountName, ProcessCommandLine, KeyExposure
| order by TimeGenerated desc;

If Hunt 3 returns results, treat the exposed keys as compromised-in-telemetry: they now exist in your EDR backend and potentially shell history. Rotate them.

Velociraptor VQL

Use this artifact to sweep a fleet for the presence of llm credential stores and prompt-log databases, their modification times, and any running llm/Python processes — useful for scoping which developer machines carry live LLM provider keys before and after the 0.33 rollout.

-- Inventory llm CLI credential stores and logs databases across endpoints
SELECT FullPath, Size, Mtime, Atime
FROM glob(globs=[
  'C:/Users/*/AppData/Roaming/io.datasette.llm/keys.json',
  'C:/Users/*/AppData/Roaming/io.datasette.llm/logs.db',
  '/home/*/.config/io.datasette.llm/keys.json',
  '/home/*/.config/io.datasette.llm/logs.db',
  '/Users/*/Library/Application Support/io.datasette.llm/keys.json',
  '/Users/*/Library/Application Support/io.datasette.llm/logs.db'
])

-- Enumerate running llm / python processes with embedding or key-related command lines
SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE CommandLine =~ '(?i)llm (embed|keys|prompt)'
   OR CommandLine =~ '(?i)--key'
   OR Exe =~ '(?i)llm(.exe)?$'

Remediation / Verification Script

Run this on developer workstations and CI runners to validate the llm 0.33 upgrade: confirm installed versions, verify the httpx2 package resolves to a legitimate PyPI project (guard against typosquat lookalikes), check for inline key usage in shell history, and confirm no stale mixed httpx/httpx2 state from the broken 0.32 transition.

#!/bin/bash
# llm 0.33 upgrade verification — run as the user who owns the llm install
set -u

echo "=== [1/5] Installed llm version (expect 0.33 or later) ==="
llm --version 2>/dev/null || python3 -m pip show llm 2>/dev/null | grep -E '^(Name|Version)'

echo "=== [2/5] Dependency state — check for httpx/httpx2 conflict from broken 0.32 line ==="
python3 -m pip show httpx httpx2 openai 2>/dev/null | grep -E '^(Name|Version|Home-page|Author)' || \
  echo "llm not installed in this Python environment"

echo "=== [3/5] Typosquat sanity check — verify httpx2 metadata points to the expected project ==="
HTT2_URL=$(python3 -m pip show httpx2 2>/dev/null | grep -i '^Home-page' || true)
echo "httpx2 Home-page: ${HTT2_URL:-NOT INSTALLED}"
echo "ACTION: Manually confirm this matches the project referenced in https://github.com/simonw/llm/pull/1631"
echo "        If the package was pulled from an unknown index or a private mirror, audit the mirror."

echo "=== [4/5] Pin the validated versions to prevent silent future drift ==="
cat <<'EOF'
# Add to your requirements/constraints file after validation:
llm==0.33
# Verify hashes for regulated environments:
#   pip install --require-hashes -r requirements.txt
# Generate hashes with: pip hash <downloaded-wheel>
EOF

echo "=== [5/5] Check shell history for inline API keys exposed via new --key flag ==="
grep -nE 'llm embed.*--key[= ]+[A-Za-z0-9_-]{20,}' ~/.bash_history ~/.zsh_history 2>/dev/null | head -20 || \
  echo "No inline keys found in shell history"
echo "ACTION: Any keys found here must be rotated at the provider console (e.g., platform.openai.com)"

echo ""
echo "=== Reminder: llm key store location ==="
llm keys path 2>/dev/null || echo "Run 'llm keys path' as the owning user to locate keys.json"
echo "Verify permissions are 0600 and owned by the interactive user only."

Remediation

Because there is no vulnerability to patch, remediation here is disciplined change management plus credential hygiene. In order of priority:

1. Pin and validate the upgrade. Move managed environments to llm==0.33 explicitly. Do not let CI jobs or developer machines float on pip install -U llm — the 0.32 → 0.32.1 → 0.33 sequence is a live demonstration of why. Review the upstream artifacts before rollout: the 0.33 release notes, issue #1608, and PR #1631.

2. Verify httpx2 provenance. Confirm the installed package's metadata (Home-page, Author, download source) matches the project referenced in the upstream pull request. If your organization proxies PyPI through an internal mirror (Artifactory, Nexus, devpi), audit the mirrored package against upstream hashes. Dependency-name transitions are exactly where lookalike packages slip in.

3. Check for the broken intermediate state. Any environment that upgraded to 0.32 or 0.32.1 during the regression window may carry a mismatched httpx/httpx2 combination. The verification script above flags this; resolve by performing a clean reinstall of llm==0.33 in a fresh virtual environment rather than an in-place upgrade.

4. Ban inline keys; enforce the key store. The new --key flag on llm embed / llm embed-multi is a feature, but command-line keys persist in shell history, process lists, and EDR telemetry. Update internal guidance to require llm keys set <provider> and environment-variable-based injection in CI — and rotate any key already exposed via --key (use KQL Hunt 3 to find them).

5. Harden the key store and logs database. Confirm keys.json is mode 0600, owned by the interactive user, and excluded from backup/sync tools that would replicate credentials off-box. Treat logs.db as sensitive data: it contains full prompt/response history and should be covered by your data-loss-prevention and endpoint-encryption policies.

6. Bring llm into your software inventory. If you cannot answer "how many endpoints run llm, and at what version," that's the real finding. Add it to your SBOM/endpoint-software inventory (the VQL artifact above gives you fleet visibility in minutes) and include it in your patch-management cadence alongside browsers and runtimes.

7. Monitor outbound LLM API traffic. Baseline which hosts legitimately connect to provider endpoints like api.openai.com. A new host class suddenly initiating LLM API traffic — particularly servers — is a high-fidelity anomaly worth alerting on, as it frequently signals either shadow-AI adoption or an attacker using a victim's own tools to exfiltrate summarized data.

There is no vendor-mandated deadline here — but the window between a dependency swap and your validation of it is the window in which a poisoned lookalike package would do its damage. Close it in days, not quarters.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub