Analyzing ChatGPhish: The Dangers of Implicit Markdown Trust in LLMs
Just came across the ChatGPhish research by Permiso Security regarding a flaw in ChatGPT's rendering engine. It’s a classic case of implicit trust in parsing logic—specifically, how the model handles Markdown links and images.
The vulnerability allows attackers to perform prompt injections by embedding malicious payloads within Markdown syntax. Because the renderer at chatgpt.com implicitly trusts these Markdown objects, it can render spoofed URLs or images that redirect users to phishing sites without raising the usual flags.
Here’s a simplified example of what a payload might look like, using markdown to obfuscate the true destination:
markdown Click here for your login portal
Or using image rendering for stealth:
markdown
The real kicker is that this bypasses standard content filters because the content of the markdown (the text inside the brackets) looks harmless, while the target (the URL in parentheses) is malicious.
Detection Strategy: For SOC teams, we should probably be monitoring for unexpected markdown patterns in API interactions or logs if possible. A quick regex to flag markdown links with mismatched domains might be a good start.
import re
# Basic check for markdown links
markdown_link_pattern = r'\[([^\]]+)\]\(([^)]+)\)'
def check_markdown_risk(text):
matches = re.findall(markdown_link_pattern, text)
for label, url in matches:
if "microsoft.com" in label.lower() and "microsoft.com" not in url:
print(f"Suspicious link found: Label '{label}' -> URL '{url}'")
Has anyone started implementing output sanitization layers for LLMs in their environments, or are we still treating the model's output as gospel?
Great breakdown. From a SOC perspective, we're looking at this less as a vulnerability to patch and more as a social engineering vector to watch. We've updated our phishing awareness training to include 'AI Summarized' content as a potential risk category. I'm also drafting a Sigma rule to flag high volumes of outbound traffic from internal networks to domains that don't match the SSL certificate common name, which often happens with these spoofed markdown links.
We saw something similar during a recent Red Team engagement. The client's employees were so conditioned to trust the 'AI Assistant' in their internal tool that they didn't hesitate to click 'resolved' links generated by the model. We ended up chaining this with a stored XSS in the markdown renderer. It's wild how quickly developers are integrating markdown parsers without sanitizing the HTML output. Always use a sanitizer like bleach before rendering user-supplied or AI-generated markdown.
Solid regex snippet. As a pentester, I've found that the 'implicit trust' issue extends beyond just ChatGPT. A lot of custom SaaS apps wrapping LLMs forget that the model is just generating text that will be parsed. If you can control the parsing context, you own the output. I'd recommend checking your CSP headers as well; ensuring script-src is tight can mitigate the impact if a malicious link does get clicked.
Excellent points regarding the human factor. From a detection perspective, we need to hunt for outbound clicks originating directly from these interfaces. If you're logging LLM interactions, you can use regex to flag markdown links for analysts to review before they are rendered.
Here’s a basic KQL query to identify these patterns in your logs:
LLMDeliveryLogs
| where ResponseContent matches regex @"\[.*?\]\(.*?\)"
| project Timestamp, User, SuspiciousLink=extract(@"\((.*?)\)", 1, ResponseContent)
Building on Viktor's point, we can't afford to treat LLM output as trusted data in our custom apps. It's crucial to implement a sanitation layer that strips or neutralizes Markdown features before rendering. Enforcing allowlists for domains or disabling automatic link rendering closes this vector effectively.
For those deploying internal tools, here is a Python snippet using bleach to sanitize markdown output:
import bleach
llm_respXSS'))"
cleaned = bleach.clean(llm_response, tags=['p', 'b'], strip=True)
This ensures the payload is neutralized before it reaches the user.
Verified Access Required
To maintain the integrity of our intelligence feeds, only verified partners and security professionals can post replies.
Request Access