Back to Intelligence

llm-gemini 0.38 Release: Defending Against Shadow AI, API Key Exposure, and LLM CLI Supply-Chain Risk

SA
Security Arsenal Team
November 20, 2026
10 min read

When Simon Willison shipped llm-gemini 0.38 — the latest release of his plugin connecting the llm command-line tool to Google's Gemini model family — it landed as a routine open-source release note, not a security advisory. There is no CVE here, no exploit in the wild, and no vendor emergency. So why is a security consultancy writing about it?

Because in the last 18 months of IR engagements, some of the most damaging data exposure incidents we've worked didn't start with malware. They started with a developer piping proprietary source code, customer records, or incident notes into an unapproved LLM CLI tool that sent it all to a third-party API. Tools like llm-gemini are excellent, legitimate software — Willison's llm ecosystem is one of the most thoughtfully engineered pieces of AI tooling available — but every pip install llm-gemini on a corporate endpoint is an unsanctioned data egress channel your DLP stack probably doesn't see. This post gives your SOC the detections, hunts, and governance controls to manage that reality.

What llm-gemini 0.38 Actually Is

llm-gemini is a Python plugin for Willison's llm CLI. Once installed (llm install llm-gemini) and configured with a Google AI API key, it lets a user send arbitrary prompts — and, critically, arbitrary files, command output, and piped stdin — to Google's Gemini models directly from the terminal. The 0.38 release continues the plugin's rapid cadence of tracking Google's newest Gemini models and capabilities.

From a defender's perspective, the relevant architecture is:

  • Install vector: pip / pipx / uv from PyPI — no admin rights required in most dev environments, no EDR software-install alert in most environments.
  • Credential storage: API keys stored in plaintext in the user's home directory (the llm tool maintains a keys.json under its config path, e.g. ~/.config/io.datasette.llm/keys.json on Linux/macOS), or via environment variables such as LLM_GEMINI_KEY / GOOGLE_API_KEY.
  • Egress channel: HTTPS to Google's generative AI API endpoints (generativelanguage.googleapis.com) — indistinguishable from legitimate Google traffic at the perimeter unless you're inspecting process-level attribution.
  • Capability surface: modern llm plugins support tool/function calling, attachments, and model-driven command execution patterns — meaning a prompt-injection payload in a file a developer summarizes can, in agentic configurations, influence commands run on that developer's machine.

Exploitation status: None. This is not a vulnerability. The risk is adoption without governance: shadow AI usage, plaintext API keys on endpoints, and an unmonitored egress path for your most sensitive data.

The Three Concrete Risks We've Seen in the Field

1. Shadow AI Data Egress

A developer running cat customer_dump.sql | llm -m gemini-... 'explain this schema' has just exfiltrated your customer data to a third party in a way that bypasses most DLP, CASB, and proxy controls — because it looks like routine Google API traffic from a Python process. This is the single most common LLM-related finding in our 2025–2026 assessments.

2. Plaintext API Key Theft

LLM API keys are the new cloud access keys. Infostealer families have added LLM tool config paths and AI-related environment variables to their collection routines because a Gemini or OpenAI key found on an endpoint is instantly monetizable (resold for compute abuse) and occasionally over-privileged against GCP projects. A keys.json sitting in a home directory with world-readable permissions is a finding.

3. PyPI Supply-Chain and Typosquat Exposure

Because llm-gemini installs from PyPI with a single command, the ecosystem is a typosquat target. We've seen malicious packages mimicking popular LLM plugin names harvested for exactly the API keys described above. If your developers can pip install from the public internet without an internal proxy or allowlist, your exposure window is the time between a malicious package's publication and its takedown.

Detection & Response

The following detections assume you want visibility and governance, not a blanket block — most organizations we advise land on "detect, attribute, and route through policy" rather than "deny Python."

Sigma

YAML
---
title: LLM CLI Tool Execution with Gemini Model
type: detection
description: Detects execution of the llm CLI (Simon Willison's tool) invoking Gemini models, indicating potential shadow-AI data egress from developer endpoints.
references:
  - https://simonwillison.net/2026/Nov/18/llm-gemini/
  - https://attack.mitre.org/techniques/T1059/
author: Security Arsenal
date: 2026/11/20
status: experimental
tags:
  - attack.execution
  - attack.t1059
logsource:
  category: process_creation
  product: windows
detection:
  selection_llm:
    CommandLine|contains:
      - ' llm '
      - 'llm.exe'
      - 'llm install'
      - 'llm prompt'
  selection_gemini:
    CommandLine|contains:
      - 'gemini'
      - 'llm-gemini'
  condition: all of selection_*
falsepositives:
  - Approved AI development workflows using sanctioned LLM tooling
level: low
---
title: Plaintext LLM API Key File Access
id: 8b2c4d61-3f5a-4e7b-9c1d-2a6f8e0b3d5c
description: Detects read access to the llm CLI keys.json credential store by processes other than the llm tool itself, indicating possible API key theft by infostealer or hands-on-keyboard activity.
references:
  - https://attack.mitre.org/techniques/T1552/001/
author: Security Arsenal
date: 2026/11/20
status: experimental
tags:
  - attack.credential_access
  - attack.t1552.001
logsource:
  category: file_event
  product: windows
detection:
  selection_path:
    TargetFilename|contains:
      - 'io.datasette.llm\keys.json'
      - '\.config\io.datasette.llm\'
  filter_legit:
    Image|endswith:
      - '\llm.exe'
      - '\python.exe'
      - '\python3.exe'
  condition: selection_path and not filter_legit
falsepositives:
  - Backup and sync agents indexing user profile directories
  - EDR/AV scanners performing scheduled scans
level: high
---
title: Non-Browser Process Connecting to Google Generative AI API
id: 3f7a9c25-1d4e-4b8a-a6f2-9c0e5d7b2a41
description: Detects network connections to Google's generative AI API endpoint from command-line or scripting processes, consistent with llm-gemini or similar CLI LLM clients sending potentially sensitive data off-host.
references:
  - https://attack.mitre.org/techniques/T1567/
author: Security Arsenal
date: 2026/11/20
status: experimental
tags:
  - attack.exfiltration
  - attack.t1567
logsource:
  category: network_connection
  product: windows
detection:
  selection_dest:
    DestinationHostname|contains:
      - 'generativelanguage.googleapis.com'
  selection_proc:
    Image|endswith:
      - '\python.exe'
      - '\python3.exe'
      - '\pip.exe'
      - '\uv.exe'
      - '\llm.exe'
      - '\node.exe'
      - '\curl.exe'
  condition: selection_dest and selection_proc
falsepositives:
  - Sanctioned AI-assisted development tooling approved by security
level: medium

A note on tuning: rule one is intentionally low-severity — it's an inventory detection. The value is building a table of who is using LLM CLIs so you can engage them with policy, not paging anyone at 2 AM.

KQL (Microsoft Sentinel / Defender)

KQL — Microsoft Sentinel / Defender
// Hunt: LLM CLI usage + Gemini API egress + credential store access (run over 14 days)
// Part 1: Process execution of llm CLI with Gemini models
let LLMProc = DeviceProcessEvents
| where TimeGenerated > ago(14d)
| where ProcessCommandLine has_any (" llm ", "llm install", "llm prompt", "llm-gemini", "gemini-")
| project LLMTime=TimeGenerated, DeviceName, AccountName, LLMCmd=ProcessCommandLine, LLMProc=ProcessName, DeviceId;
// Part 2: Network egress to Google's generative AI endpoint from non-browser processes
let GeminiEgress = DeviceNetworkEvents
| where TimeGenerated > ago(14d)
| where RemoteUrl has "generativelanguage.googleapis.com"
| where InitiatingProcessName !in~ ("chrome.exe", "msedge.exe", "firefox.exe", "safari", "brave.exe", "msedgewebview2.exe")
| project EgressTime=TimeGenerated, DeviceName, EgressProc=InitiatingProcessName, EgressCmd=InitiatingProcessCommandLine, RemoteUrl, DeviceId;
// Part 3: Access to llm credential store by unexpected processes
let KeyAccess = DeviceFileEvents
| where TimeGenerated > ago(14d)
| where FolderPath has "io.datasette.llm" and FileName =~ "keys.json"
| where InitiatingProcessName !in~ ("python.exe", "python3.exe", "llm.exe")
| project KeyTime=TimeGenerated, DeviceName, KeyAccessProc=InitiatingProcessName, KeyAccessCmd=InitiatingProcessCommandLine, FolderPath, DeviceId;
// Correlate: devices with both LLM CLI usage and unsanctioned API egress
LLMProc
| join kind=inner (GeminiEgress) on DeviceId
| summarize LLMCommands = make_set(LLMCmd, 5), FirstSeen=min(LLMTime), LastSeen=max(LastSeen2=LLMTime) by DeviceName, AccountName
| join kind=leftouter (KeyAccess | summarize SuspiciousKeyAccess = make_set(KeyAccessProc, 5) by DeviceId) on DeviceId
| project DeviceName, AccountName, FirstSeen, LastSeen, LLMCommands, SuspiciousKeyAccess
| sort by LastSeen desc

Velociraptor VQL

VQL — Velociraptor
-- Hunt: llm CLI processes and plaintext LLM API key stores on endpoints
-- Part A: live processes running the llm CLI with Gemini models
LET procs = SELECT Pid, Name, CommandLine, Exe, Username, CreateTime
FROM pslist()
WHERE CommandLine =~ '(?i)llm.*gemini|llm-gemini'
   OR Exe =~ '(?i)llm(\.exe)?$'

-- Part B: llm credential stores on disk with permissions context
LET keystores = SELECT FullPath, Mtime, Size,
       parse_pe(file=FullPath) IS NOT NULL AS IsBinary
FROM glob(globs=[
  'C:/Users/*/.config/io.datasette.llm/keys.json',
  'C:/Users/*/AppData/*/io.datasette.llm/keys.json',
  '/home/*/.config/io.datasette.llm/keys.json',
  '/Users/*/.config/io.datasette.llm/keys.json'
])

SELECT * FROM procs
UNION ALL
SELECT NULL AS Pid, 'KEYSTORE_ON_DISK' AS Name, FullPath AS CommandLine,
       NULL AS Exe, NULL AS Username, Mtime AS CreateTime FROM keystores

Remediation / Audit Script

Bash / Shell
#!/usr/bin/env bash
# llm-cli-audit.sh — Audit Linux/macOS endpoints for llm/llm-gemini installs,
# plaintext API keys, and over-permissive credential stores.
set -euo pipefail

echo "=== [1] llm installations and plugin inventory ==="
for home in /home/* /Users/* /root; do
  for shim in "$home/.local/bin/llm" "$home/.pyenv/shims/llm" "/usr/local/bin/llm"; do
    if [ -x "$shim" ]; then
      echo "[FOUND] $shim"
      "$shim" --version 2>/dev/null || true
      "$shim" plugins 2>/dev/null | grep -i gemini && echo "  [!] llm-gemini plugin present" || true
    fi
  done
done

echo "=== [2] Plaintext API key stores (keys.json) ==="
find /home /Users /root -maxdepth 5 -path '*io.datasette.llm*' -name 'keys.json' 2>/dev/null | while read -r f; do
  perms=$(stat -c '%a' "$f" 2>/dev/null || stat -f '%Lp' "$f")
  echo "[KEYFILE] $f (perms: $perms)"
  if [ "$perms" != "600" ]; then
    echo "  [FIXING] chmod 600 $f"
    chmod 600 "$f"
  fi
done

echo "=== [3] LLM API keys exposed in shell environments ==="
grep -rlE 'LLM_GEMINI_KEY|GOOGLE_API_KEY|GEMINI_API_KEY' \
  /home/*/.bashrc /home/*/.zshrc /home/*/.profile /Users/*/.zshrc 2>/dev/null \
  | while read -r f; do echo "[ENV KEY] $f — move to a secrets manager (e.g., 1Password CLI, Vault)"; done

echo "=== [4] Recent egress to Google generative AI API (connection log check) ==="
if command -v lsof >/dev/null; then
  lsof -i -nP 2>/dev/null | grep -i 'generativelanguage' || echo "  No active connections."
fi
echo "=== Audit complete. Feed findings to your AI usage policy workflow. ==="

Remediation and Governance Recommendations

Since there is no patch to apply, remediation is a governance exercise. In priority order:

  1. Publish a sanctioned-AI-tools list. Decide explicitly whether llm + llm-gemini (and equivalents like aider, mods, direct SDK scripts) are approved. Ambiguity is what turns a great tool into a shadow-AI incident.
  2. Route all LLM API traffic through an enterprise gateway. Google's own Gemini API supports enterprise controls via Vertex AI with VPC controls, CMEK, and data-residency guarantees. Developer-issued AI Studio keys tied to personal Gmail accounts should be disallowed by policy and detected by the KQL above.
  3. Eliminate plaintext API keys. Enforce chmod 600 on keys.json (script above), and better, require keys be injected at runtime from a secrets manager rather than persisted to disk. Rotate any key found on an endpoint that later shows infostealer or suspicious-access telemetry.
  4. Control the PyPI supply chain. Proxy pip through an internal index (Artifactory, Nexus, or a pip allowlist) and alert on installs of LLM-related packages that aren't on your approved list. Pin versions — llm-gemini==0.38 from PyPI with hash verification (pip install --require-hashes) is reproducible; a floating install is not.
  5. Update DLP and CASB signatures. Add generativelanguage.googleapis.com (and the other major LLM API hosts) to your exfiltration-monitoring category with process attribution, not just domain blocking — the goal is catching python.exe sending your source tree, not blocking a developer's browser.
  6. Brief developers on prompt injection in CLI workflows. Piping untrusted files, tickets, or web content into a Gemini model that has tool-calling enabled means that content can influence actions on the host. Treat model output like you'd treat output from an untrusted user.

Bottom Line

llm-gemini 0.38 is good software from a respected developer, and that's precisely the point: the AI-era threat to your data isn't always an adversary — it's a well-meaning engineer with an excellent tool and no guardrails. Deploy the detections above, inventory who's already using these CLIs, and bring LLM usage inside your governance perimeter before an infostealer or an accidental paste does it for you.

Related Resources

Security Arsenal Penetration Testing Services AlertMonitor Platform Book a SOC Assessment vulnerability-management Intel Hub

Is your security operations ready?

Get a free SOC assessment or see how AlertMonitor cuts through alert noise with automated triage.