llm-echo 0.5 and the Rise of Local LLM Tooling: Shadow AI Detection and Governance for the Enterprise SOC

Simon Willison shipped llm-echo 0.5 this week — an updated debug plugin for his widely used LLM command-line tool. The plugin provides a fake "model" that simply echoes prompts back, and the 0.5 release improves its simulated token-usage accounting to cover tool calls, system prompts, and reasoning tokens. On its face, this is a benign developer-tooling release. There is no CVE here, no exploit, no active campaign.

So why is a security consultancy writing about it? Because this release is a useful forcing function for a conversation most enterprises are badly behind on: LLM CLI tooling is proliferating on endpoints with effectively zero governance. The llm tool — installable in one line via uvx --with llm-echo llm -m echo 'hi there' — stores API keys in plaintext on disk, logs every prompt and response to a local SQLite database, and can pipe arbitrary corporate data to any hosted model provider. llm-echo itself is the safe version of this story: it was built precisely so developers can test prompt pipelines without sending data to an external API. That design decision should tell you something about the risk profile of the real thing.

This post covers what llm-echo 0.5 actually does, why the tooling pattern it belongs to matters to defenders, and how to build practical detection and governance around LLM CLI usage in your environment.

Technical Analysis

What llm-echo 0.5 Does

llm-echo is a plugin for the llm CLI that registers a model named echo. When invoked, it performs no inference and makes no network calls — it returns the prompt it was given, formatted as a model response. The 0.5 release updates the simulated token-usage counters so that testing against the echo model produces realistic --usage output that accounts for:

Tool calls — tokens associated with function/tool-calling payloads
System prompts — tokens from -s/--system instructions
Reasoning tokens — the "thinking" token budget used by reasoning models (e.g., -o thinking 1)

Example usage from the release:

Bash / Shell

uvx --with llm-echo llm -m echo 'hi there' -o thinking 1 --usage

This matters defensively for two reasons. First, it is genuinely good practice: testing prompt templates, tool-calling chains, and agent logic against a local echo model is exactly how you develop LLM features without exfiltrating test data or production prompts to a third-party API. If your developers are building LLM integrations, pushing them toward echo-style local testing reduces your data-egress surface. Second, the one-line uvx install pattern demonstrates how trivially unapproved AI tooling lands on endpoints — no admin rights, no installer, no procurement review.

The Defensive Problem: The `llm` Toolchain as a Data-Handling System

The broader llm ecosystem (and its peers — aichat, mods, tgpt, direct curl calls to OpenAI/Anthropic endpoints) has properties every SOC should understand:

Plaintext API key storage. The llm CLI stores provider keys in keys.json inside its config directory (~/.config/io.datasette.llm/ on Linux/macOS, %APPDATA%\io.datasette.llm\ on Windows). These keys are high-value theft targets — infostealers have added AI-provider key harvesting to their collection routines over the past year.
Complete prompt/response logging. Every interaction is written to a local SQLite logs.db in the same directory. From a forensics standpoint this is a gift; from a data-protection standpoint it means sensitive prompts (source code, credentials pasted into a debug session, PHI, customer data) persist on disk indefinitely.
Arbitrary egress. Prompt content is sent to whatever provider the key belongs to. Corporate DLP that doesn't inspect API traffic to api.openai.com, api.anthropic.com, or regional equivalents is blind to this channel.
Plugin extensibility. As llm-echo demonstrates, plugins install via uvx --with or llm install without touching system package managers — traditional software-inventory tooling misses them.

Exploitation Status

Not applicable — there is no vulnerability in this release. The threat model here is shadow AI: ungoverned use of LLM tooling leading to data egress, credential exposure, and unlogged sensitive-data processing. This is a policy-and-detection gap, not a patch gap.

Detection & Response

The detections below target the observable footprint of LLM CLI tooling — the llm tool specifically, plus the generic pattern of AI-provider API egress. They are tuned to fire on genuinely uncommon enterprise behavior. If your organization has sanctioned LLM CLI usage for specific teams, scope these rules with an allowlist of approved hosts/users rather than disabling them.

YAML

---
title: LLM CLI Tool Execution via llm or uvx Plugin Loading
id: 3f8a2c41-7b5e-4d19-a6c3-9e1f0b2d4a57
status: experimental
description: Detects execution of Simon Willison's llm CLI or on-the-fly installation of llm plugins via uvx/pipx, which may indicate unapproved shadow AI tooling handling corporate data.
references:
  - https://github.com/simonw/llm-echo/releases/tag/0.5
  - https://llm.datasette.io/
  - https://attack.mitre.org/techniques/T1059/
author: Security Arsenal
date: 2026/10/18
tags:
  - attack.execution
  - attack.t1059
logsource:
  category: process_creation
  product: windows
detection:
  selection_image:
    Image|endswith:
      - '\llm.exe'
      - '\uvx.exe'
      - '\uv.exe'
  selection_cli:
    CommandLine|contains:
      - 'llm -m '
      - 'llm install'
      - '--with llm-'
      - 'llm keys set'
  condition: selection_image and selection_cli
falsepositives:
  - Approved developer or data-science workstations with sanctioned llm usage
level: medium
---
title: LLM CLI Tool Execution on Linux and macOS
id: 8c1d5e92-4a6f-4b38-9d72-1c5a7f3e8b60
status: experimental
description: Detects execution of the llm CLI or uvx-based plugin installation on Linux/macOS endpoints, a common shadow AI pattern that bypasses package managers and software inventory.
references:
  - https://github.com/simonw/llm-echo/releases/tag/0.5
  - https://llm.datasette.io/
  - https://attack.mitre.org/techniques/T1059.006/
author: Security Arsenal
date: 2026/10/18
tags:
  - attack.execution
  - attack.t1059.006
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine|contains:
      - 'llm -m '
      - 'llm install llm-'
      - '--with llm-'
      - 'llm keys set'
      - 'llm logs'
  condition: selection
falsepositives:
  - Sanctioned AI development activity on approved engineering hosts
level: medium
---
title: Outbound Connection to LLM Provider API from Non-Browser Process
id: 5b7e3f18-2c9d-4a41-b8e6-6d4c1a9f0273
status: experimental
description: Detects non-browser processes establishing connections to major hosted LLM API endpoints, which may indicate CLI-based AI tooling exfiltrating prompt data outside sanctioned application channels.
references:
  - https://attack.mitre.org/techniques/T1041/
author: Security Arsenal
date: 2026/10/18
tags:
  - attack.exfiltration
  - attack.t1041
logsource:
  category: network_connection
  product: windows
detection:
  selection_dest:
    DestinationHostname|contains:
      - 'api.openai.com'
      - 'api.anthropic.com'
      - 'generativelanguage.googleapis.com'
      - 'api.mistral.ai'
      - 'api.cohere.com'
  filter_browsers:
    Image|endswith:
      - '\chrome.exe'
      - '\msedge.exe'
      - '\firefox.exe'
      - '\brave.exe'
  condition: selection_dest and not filter_browsers
falsepositives:
  - Sanctioned enterprise AI integrations and approved desktop AI assistants
level: high

KQL — Microsoft Sentinel / Defender

// Hunt for LLM CLI tooling execution and AI-provider API egress
// Scope: last 14 days. Tune the allowlist for sanctioned AI hosts/users.
let SanctionedHosts = dynamic(["dev-ai-01", "ml-workstation-07"]);
union withsource=tableName *
| where TimeGenerated > ago(14d)
| where tableName == "DeviceProcessEvents"
    and (ProcessCommandLine has_any ("llm -m ", "llm install llm-", "--with llm-", "llm keys set", "llm logs")
         or FileName in~ ("llm", "llm.exe", "uvx", "uvx.exe"))
| project TimeGenerated, DeviceName, AccountName, FileName, ProcessCommandLine, InitiatingProcessFileName, tableName
| where DeviceName !in~ (SanctionedHosts)
| join kind=fullouter (
    DeviceNetworkEvents
    | where TimeGenerated > ago(14d)
    | where RemoteUrl has_any ("api.openai.com", "api.anthropic.com", "generativelanguage.googleapis.com", "api.mistral.ai", "api.cohere.com")
    | where InitiatingProcessFileName !in~ ("chrome.exe", "msedge.exe", "firefox.exe", "brave.exe", "safari")
    | project TimeGenerated, DeviceName, InitiatingProcessAccountName, InitiatingProcessFileName, InitiatingProcessCommandLine, RemoteUrl, RemoteIP
) on DeviceName
| extend IndicatorType = case(
    tableName == "DeviceProcessEvents", "LLM CLI execution",
    RemoteUrl != "", "Non-browser LLM API egress",
    "Unknown")
| summarize FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated), SampleCommands=make_set(ProcessCommandLine, 5), SampleEgress=make_set(RemoteUrl, 5) by DeviceName, AccountName, IndicatorType
| order by LastSeen desc

VQL — Velociraptor

-- Hunt for LLM CLI processes and locally stored AI provider API keys
-- The llm CLI stores keys in keys.json and full prompt/response history in logs.db
SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE CommandLine =~ '(?i)(llm -m |llm install|--with llm-|llm keys set)'
   OR Name =~ '(?i)^(llm|uvx)(\.exe)?$'

-- Locate llm configuration artifacts (API keys and prompt logs) on disk
SELECT FullPath, Size, Mtime
FROM glob(globs=[
  'C:/Users/*/AppData/Roaming/io.datasette.llm/keys.json',
  'C:/Users/*/AppData/Roaming/io.datasette.llm/logs.db',
  '/home/*/.config/io.datasette.llm/keys.json',
  '/home/*/.config/io.datasette.llm/logs.db',
  '/Users/*/Library/Application Support/io.datasette.llm/keys.json'
])

Bash / Shell

#!/usr/bin/env bash
# audit-llm-cli.sh — Audit Linux/macOS endpoints for llm CLI installation,
# stored provider API keys, and prompt/response log databases.
# Run via your EDR remote shell, Ansible, or MDM. Read-only; safe for production.

set -u
echo "=== LLM CLI Endpoint Audit — $(hostname) — $(date -u) ==="

# 1. Check for llm / uvx binaries in common user-level install paths
echo "--- [1] Installed LLM CLI binaries ---"
for user_home in /home/* /Users/* /root; do
  for bin in llm uvx uv aichat mods tgpt; do
    found=$(find "$user_home/.local/bin" "$user_home/.cargo/bin" /usr/local/bin -maxdepth 1 -name "$bin" 2>/dev/null)
    [ -n "$found" ] && echo "FOUND: $found (owner: $(stat -c '%U' "$found" 2>/dev/null || stat -f '%Su' "$found"))"
  done
done

# 2. Check for plaintext API key storage
echo "--- [2] Stored LLM provider API keys ---"
for cfg in /home/*/.config/io.datasette.llm/keys.json \
           /Users/*/Library/Application\ Support/io.datasette.llm/keys.json; do
  if [ -f "$cfg" ]; then
    echo "KEYS FILE: $cfg — providers configured: $(grep -o '"[^"]*":' "$cfg" 2>/dev/null | tr -d '":' | tr '\n' ' ')"
  fi
done

# 3. Check for prompt/response log databases (data-residue risk)
echo "--- [3] Prompt/response log databases ---"
for db in /home/*/.config/io.datasette.llm/logs.db \
          /Users/*/Library/Application\ Support/io.datasette.llm/logs.db; do
  if [ -f "$db" ]; then
    echo "LOG DB: $db — size: $(du -h "$db" | cut -f1), last modified: $(stat -c '%y' "$db" 2>/dev/null || stat -f '%Sm' "$db")"
  fi
done

# 4. Check shell history for LLM CLI usage indicators (last-resort signal)
echo "--- [4] Shell history indicators ---"
grep -lE 'llm -m |llm keys set|--with llm-' /home/*/.bash_history /home/*/.zsh_history 2>/dev/null || echo "No history hits."

echo "=== Audit complete. Review findings against your approved-AI-tooling inventory. ==="

Remediation and Governance

Since there is nothing to patch, remediation here is programmatic. Prioritize in this order:

Publish an approved-AI-tooling policy this quarter. Enumerate which LLM providers and client tools are sanctioned, under what data-classification rules. Ambiguity is the enemy — engineers will reach for uvx --with llm-echo llm because it works in one line, not because they're malicious.
Promote local-testing patterns like llm-echo for development. If teams are building LLM integrations, require that prompt-pipeline and tool-calling development happen against echo/local models (llm-echo, Ollama, LM Studio) with synthetic data. Real prompts and production-shaped data should never hit a hosted API outside an approved, logged integration.
Control egress to LLM API endpoints. If your organization has a sanctioned provider, proxy that traffic through a gateway you log (e.g., an AI gateway or forward proxy with TLS inspection) and block or alert on direct egress to the major provider API hosts from non-sanctioned processes, using the Sigma and KQL content above.
Treat stored LLM API keys as credentials. Add keys.json and equivalent paths to your credential-theft detections and your secrets-scanning scope. If the audit script finds keys on endpoints, rotate them — you cannot prove they weren't already harvested.
Address the logs.db residue. Decide whether prompt/response logs on endpoints are acceptable for your data classifications. Where they aren't, deploy configuration to disable logging (llm logs off / --no-log where supported) and securely wipe existing databases.
Inventory continuously, not once. The uvx/pipx install pattern bypasses traditional software inventory. Fold the VQL artifact into a recurring Velociraptor hunt (weekly is reasonable) and reconcile results against your approved-tooling list.

The broader lesson from a 0.5-point release of a debug plugin: the AI tooling ecosystem is maturing fast enough that even its testing utilities are getting feature updates. Your governance, detection, and DLP controls need to mature at the same pace — because your developers already have.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub