ChatGPhish: When Markdown Becomes a Phishing Vector in ChatGPT
Just caught the Permiso Security writeup on 'ChatGPhish,' and it highlights a fundamental issue with how we interact with LLMs: implicit trust. The vulnerability stems from the chatgpt.com response renderer treating Markdown links and images as benign, effectively allowing prompt injections that turn the chat interface into a phishing surface.
The core of the issue is that the renderer parses user-supplied Markdown without sufficient sanitization. An attacker can craft a prompt that forces the AI to render a malicious link appearing to be a legitimate citation or resource. Because the link is inside the 'trusted' AI response bubble, users are much more likely to click it without hovering to check the destination.
For those running SOC operations, this blurs the line between legitimate AI traffic and malicious outbound clicks. If you're monitoring web traffic, you might want to flag unusual domains accessed immediately after a chatgpt.com session activity.
Here is a basic KQL snippet to start hunting for this pattern in your proxy logs, looking for rare domains accessed shortly after ChatGPT activity:
let timeframe = 1h;
let chatgpt_sessions = DeviceNetworkEvents
| where Timestamp > ago(timeframe)
| where RemoteUrl has "chatgpt.com";
let outbound_clicks = DeviceNetworkEvents
| where Timestamp > ago(timeframe)
| where InitiatingProcessFileName has "chrome" // or edge/browser
| extend Domain = tostring(parse_url(RemoteUrl).Host)
| where Domain !contains "openai" and Domain !contains "cloudfront";
chatgpt_sessions
| join kind=inner outbound_clicks on DeviceId, Timestamp
| project Timestamp, DeviceName, RemoteUrl, Domain
| summarize Count() by Domain, DeviceName
| where Count_ <= 2; // Flagging low-frequency, suspicious destinations
Has anyone else started incorporating specific LLM-renderer risks into their awareness training, or is this still flying under the radar for most users?
This is exactly why we block access to consumer-grade LLMs at the firewall. The risk isn't just data exfiltration; it's the lack of URL validation in the renderer. We saw similar issues with Bing Chat's early iterations where attackers could hide malicious scripts in rendered SVGs. Until vendors start treating these outputs as untrusted, we treat the whole domain as hostile.
Good catch on the KQL. We've been seeing a lot of 'shadow AI' usage where devs paste code snippets directly from ChatGPT. I modified your query slightly to look for high-entropy domains, as a lot of these phishing kits use newly registered domains to bypass reputation filters.
# Quick python script to check for high entropy in domains
import math
import re
def entropy(string):
prob = [float(string.count(c)) / len(string) for c in dict.fromkeys(string)]
entropy = -sum(p * math.log(p) / math.log(2.0) for p in prob)
return entropy
print(entropy("malicious-site-login.com")) # Typically > 3.5 is suspicious
From a pentester's perspective, this is a goldmine for social engineering. The 'authority' of the AI response makes the payload delivery incredibly effective. I'd recommend advising users to disable link previewing or Markdown rendering in the settings if possible, though that option is often hidden or non-existent. Browser extensions that force plain-text rendering for specific domains might be a temporary workaround.
Verified Access Required
To maintain the integrity of our intelligence feeds, only verified partners and security professionals can post replies.
Request Access