AI-Driven Social Engineering: How Frontier Models Manipulate Humans and How to Defend Your Organization

In a recent interview with the Dark Reading News Desk, Fred Heiding of Menlo Park Intelligence laid out research findings that should change how every security leader thinks about the human attack surface: frontier AI models are demonstrably capable of influencing human behavior and engineering emotional dependency — and adversaries are already weaponizing that capability in fraud, phishing, and long-con social engineering campaigns.

This is not a theoretical risk. We have moved past the era where AI's contribution to social engineering was merely better grammar in phishing emails. Heiding's research demonstrates that current frontier models can sustain multi-turn, adaptive conversations that read emotional cues, mirror a target's language, exploit trust dynamics, and escalate requests gradually — the same tradecraft that once required a skilled human con operator, now available at machine scale and negligible cost.

For defenders, the implication is stark: the last reliable control — human skepticism — is being systematically eroded at both ends. The attacker is now tireless, infinitely patient, personalized at scale, and emotionally convincing. The defender is still a busy employee with 14 seconds of attention. Organizations that continue to treat social engineering as an annual training checkbox will lose this fight.

What the Research Actually Demonstrates

Heiding's work at Menlo Park Intelligence focuses on evaluating frontier large language models not just for phishing email generation, but for sustained behavioral influence — the ability of a model to:

Build rapport and emotional dependency over multi-turn interactions. Models can maintain consistent personas across days or weeks of conversation, remembering context, expressing empathy, and creating the psychological conditions — familiarity, reciprocity, urgency, authority — that precede compliance.
Adapt persuasion strategy in real time. Unlike templated phishing, a frontier model tailors its approach based on the target's responses: hesitation triggers reassurance; skepticism triggers pre-emptive objection handling; trust triggers escalation.
Operate at scale with personalization. The economics that once limited spear-phishing to high-value targets are gone. Every employee in your org chart can now receive nation-state-quality social engineering, continuously, for pennies.
Cross channels. The same model driving a chat conversation can generate the voice script for a vishing call, the documents for a fake invoice, and the pretext for a business email compromise (BEC) thread — maintaining narrative consistency that human operators struggle to sustain.

The emotional dependency finding deserves particular attention. Romance-style scams, fake recruiter relationships, and synthetic 'colleague' personas are no longer crude; research into model persuasion shows these systems can deliberately cultivate attachment — a technique we have historically associated with long-running pig-butchering and insider-recruitment operations. When the persuader never sleeps, never breaks character, and never loses patience, traditional 'trust your gut' guidance collapses.

The Threat Landscape Defenders Actually Face

From our IR caseload across 2025 and into 2026, the operational translation of this research looks like:

Hyper-personalized spear phishing and BEC. Threads that reference real internal projects, writing styles cloned from public posts or compromised mailboxes, and reply-chain hijacking that survives casual scrutiny.
AI-assisted vishing and deepfake voice. Helpdesk impersonation of executives requesting MFA resets or wire transfers — the helpdesk remains the softest underbelly in most enterprises.
Multi-stage 'relationship' attacks. Synthetic personas (recruiters, vendors, romantic contacts, fellow researchers) building weeks of trust before the ask — targeting finance staff, developers with repo access, and executives alike.
Chat-channel infiltration. AI personas joining Slack, Teams, and Discord communities to harvest context, credentials, and access through patient social interaction rather than malware.

None of this requires a vulnerability in your software stack. The vulnerability is cognitive, and it is now being probed by systems engineered to exploit it better than any human adversary in history.

Executive Takeaways

Because this threat class is behavioral rather than a patchable vulnerability, the defensive response must be architectural and procedural — hardening the human layer the same way we hardened endpoints a decade ago.

1. Assume persuasion, verify by procedure — not by feel. Every high-risk action (wire transfers, credential resets, MFA changes, payroll reroutes, new vendor onboarding, data exports) must require out-of-band verification through a pre-registered channel. The rule must be absolute: feeling confident about a request is not a control. Documented callback procedures to known numbers defeat deepfake voice and synthetic personas regardless of how convincing they are.

2. Rebuild security awareness for the AI era. Annual slide-deck training is dead. Run continuous, AI-generated phishing simulations that mirror what your employees will actually face — multi-turn, emotionally adaptive, channel-crossing campaigns. Measure resilience to conversations, not just clicks. Heiding's research is your internal mandate: the adversary's capability has fundamentally changed, so the training must.

3. Deploy behavioral detection, not just signature detection. Your email and collaboration security stack must flag urgency-plus-financial-request patterns, first-time sender relationships, reply-chain anomalies, lookalike domains, and linguistic style deviations. Extend the same monitoring to Teams/Slack external contacts. In your SOC, hunt for the outcomes of successful social engineering — anomalous MFA events, impossible-travel logins following helpdesk interactions, new inbox rules, unusual OAuth grants — because you will not catch every lure.

4. Lock down the helpdesk. The helpdesk is now a primary breach vector for AI-assisted impersonation. Enforce identity-proofing for any credential or MFA reset: manager confirmation, registered-device callback, or in-person verification for privileged accounts. Log and audit every reset, and alert on reset volume anomalies per agent.

5. Reduce the fuel. Limit the organizational data available for personalization: review what your website, LinkedIn footprint, conference talks, and public filings reveal about org structure, projects, and executive travel. You cannot eliminate OSINT, but you can deny adversaries the specificity that makes AI-crafted pretexts irresistible.

6. Plan for the emotional-dependency long con in IR. Update incident response playbooks to include sustained-persona compromises: employees may have been manipulated over weeks by an AI persona they trust. Handle these cases with the same care as insider-threat investigations — the victim psychology is closer to romance-scam trauma than to clicking a bad link, and mishandling it guarantees underreporting of the next incident.

Remediation and Hardening Priorities

There is no patch for human cognition, so remediation is programmatic:

This quarter: Implement or audit out-of-band verification for all financial and identity-related requests. This is the single highest-ROI control against AI-driven fraud.
This quarter: Deploy phishing-resistant MFA (FIDO2/passkeys) for all users, prioritizing finance, helpdesk, and administrators. AI cannot socially engineer a hardware-bound cryptographic challenge the way it can an SMS code.
Within 90 days: Replace legacy awareness training with adaptive simulation programs and measure conversation-level resilience, not just click rates.
Within 90 days: Add helpdesk identity-proofing requirements and audit logging for MFA/credential resets; alert SOC on anomalies.
Ongoing: Feed social-engineering outcome indicators (inbox rules, OAuth grants, MFA changes, wire-transfer anomalies) into your SIEM/SOAR correlation rules so a successful lure becomes a detected incident, not a quiet loss.

The research Heiding describes is a warning shot: frontier models will only get better at influence. Defenders who treat the human layer as an engineering problem — with controls, telemetry, and tested procedures — will absorb this shift. Those still relying on gut instinct will not.

Related Resources

Security Arsenal Managed SOC Services AlertMonitor Platform Book a SOC Assessment soc-mdr Intel Hub