ChatGPhish: Analyzing the Render Trust Issue in ChatGPT
Just caught the release from Permiso Security regarding ChatGPhish. It’s a classic case of implicit trust causing a headache. The vulnerability leverages how the chatgpt.com renderer processes Markdown. Essentially, if an attacker can inject a prompt that forces a summary generation, they can append malicious Markdown links or images that the user trusts because they come from "the AI."
The core mechanism is that the model's output parser renders the Markdown immediately without sanitizing the underlying intent or strictly validating the relationship between the link text and the href. While OpenAI generally strips direct scripts, the trusted rendering of user-controlled URLs in the chat stream is the gap being exploited.
Here is a quick Python snippet I whipped up to scan raw Markdown for potential mismatch patterns (simple heuristic for prompt injection vectors) if you are auditing logs:
import re
def check_markdown_risk(text):
# Looks for markdown links where text is distinct and suspicious
pattern = r'\[([^\]]+)\]\(([^)]+)\)'
matches = re.findall(pattern, text)
for text, url in matches:
if "http" not in text and "openai.com" not in url:
print(f"Suspicious Link: Text='{text}', URL='{url}'")
markdown_input = "Check your bank statement [here](http://evil.com/login)."
check_markdown_risk(markdown_input)
Detection in the enterprise is going to be tricky because the click originates from a trusted browser session. We might need to rely on SSL inspection or advanced browser telemetry to catch the redirect.
Is anyone else looking at blocking the "Browse" or web-rendering features in their orgs until this gets patched, or are we relying on user awareness?
From a SOC perspective, we are treating outbound clicks from LLM interfaces as 'unknown' risk vectors. We've updated our proxy rules to force strict SSL inspection for traffic originating from chatgpt.com or similar endpoints. If the AI is summarizing a news article, but the link resolves to a non-corporate IP or a newly registered domain, we block it.
Here is a basic KQL query we are testing in Sentinel to flag these interactions:
DeviceNetworkEvents
| where RemoteUrl contains "openai.com" or RemoteUrl contains "chatgpt.com"
| where InitiatingProcessFileName has "chrome" or InitiatingProcessFileName has "msedge"
| project Timestamp, DeviceName, RemoteUrl, RemoteIP, InitiatingProcessAccountName
It generates some noise, but it helps spot anomalies when users are 'researching' internal topics.
This is essentially 'Stored XSS' but for the brain. Users inherently trust the summary context. In our phishing sims, we've seen a massive uptick in click rates when the 'context' appears to be generated by an AI assistant. The mitigation isn't just technical; it's behavioral training.
Users need to hover over links regardless of the source. From a pentesting angle, the most dangerous part is that the rendering engine often keeps the user in the same tab context (no target="_blank"), making it easier to spoof the address bar later if there are other browser vulnerabilities involved.
To combat the "implicit trust" Omar mentioned, we implemented an automated scanner on our internal chat archives. It extracts all rendered links and checks them against threat intel before employees click. It’s a temporary stopgap until the renderer is patched. Here’s a quick Python snippet for URL extraction:
import re
data = open("llm_logs.txt").read()
urls = re.findall(r'https?://\S+', data)
print(urls)
Verified Access Required
To maintain the integrity of our intelligence feeds, only verified partners and security professionals can post replies.
Request Access