In last week’s blog, What Anthropic's Zero Trust for Agents eBook Gets Right (And What It Doesn't Say), we explored the idea that "Security through friction is the new security through obscurity." For years, cybersecurity teams relied on deliberate user experience hurdles – rigid form fields, hidden rate limits, and the dreaded CAPTCHAs – to keep automated bad actors away from sensitive data and environments. The idea was straightforward: if you make the system rigid and tedious enough, automated bots will eventually make enough noise to get detected or fail quickly and go after softer targets.
But as we’ve just seen with Meta’s AI support assistant, that friction is no longer effective. Over the past few weeks, hackers used VPNs and simply asked to switch email accounts on high-profile Instagram accounts – successfully taking over the accounts. The hackers didn’t prompt inject or jailbreak the LLM; they didn't crack a complex password.
They simply asked nicely.
This breach is a textbook case of what happens when we don’t think about how the features we enable can be used against us, and when we prioritize accelerating AI use without asking where humans need to be in the loop. While the immediate outcome was a public black eye and a series of compromised Instagram profiles, the reality is that this could have been far worse. Many hackers, when they find high-value exploits, like to keep them quiet until they can be used to maximize damage. Imagine if they executed this attack across all of Meta’s platforms - Facebook, Instagram, and WhatsApp – allowing hackers to potentially take over the digital lives of their victims and extort them to regain access to their friends, their family, their own phone numbers.
Meta had built in exactly one speed bump: a location-based check meant to notice when a recovery request comes from somewhere the account doesn't normally live. The attackers beat it with a VPN. Let’s assume for a moment that bypassing a geographic check on Instagram is supposed to be difficult (if you are asking why it’s easy to figure out an account’s location, you probably aren’t on Instagram often). Using illicit VPN services, trying thousands of zip codes in quick succession would easily defeat basic controls. Additionally, unless threat modeling teams specifically envisioned this exact behavior, a machine-speed takeover campaign looks less like an anomaly and more like an active power-user.
Now let’s consider this against Anthropic’s "Impossible vs. Tedious" test from their Zero Trust for AI Agents eBook. Does a geographic check make the attack impossible, or merely tedious? The location check was friction, and friction is a delay. And when chatbots are programmed to be helpful first, this is the outcome.
Chatbots are designed to be helpful, automated navigators for tasks like password resets and account recovery. When confined to strict forms with validated inputs, password resets can be done with both minimal user friction and adequate security controls. In a traditional web interface, security teams can track IP addresses, page load speeds, and credential-stuffing patterns across many accounts.
Chatbots, especially those using WebSockets, often unintentionally bypass these traditional controls. Because WebSockets maintain a persistent, open connection to the backend, they frequently bypass the expected network friction of traditional page loads and bot protection. Instead, they allow an attacker to automate credential stuffing at a speed that traditional websites simply wouldn't permit. By removing the friction of a standard web form, the chatbot becomes a high-speed lane for attackers to test stolen data sets from massive public breaches like Equifax or LexisNexis.
If you’ve subscribed to our GenR3d LLM Security Analyzer, you know we've been testing this exact failure since the day we launched. Our first set of abuse cases included Account Takeovers – pairing a chatbot that helpfully drives account recovery with an attacker who already has mountains of personal data from a decade of breach dumps.
Our threat simulation research regularly maps this specific logic breakdown. A person with a VPN can hijack accounts one at a time. An agent with a VPN, instructions to change a password by any means necessary, and a target chatbot that completes sensitive actions can hijack accounts at machine speed before the security dashboard so much as twitches.
So, how do we fix this without completely throwing away the 24/7 convenience of conversational AI support?
First, we have to be very intentional when we empower the LLM to be judge, jury, and executioner of sensitive actions. Today, chatbots are better suited to serve as conversational process navigators that sit on top of otherwise vetted and tested processes. If a user needs a password reset, the bot shouldn't have direct access to the user database, but instead gathers the information that would have been required by the same web forms 12 months ago. Yes – eventually the chatbots will do it themselves, but only after we’ve threat modeled use cases just like this one. It will be incumbent on the security practitioners to start looking for where simple friction will fail as a protection mechanism.
Second, we must integrate the conversational layer with real-time security analytics to enforce strict rate limiting, velocity checks, and IP reputation monitoring. We’ve spent 20+ years building protections for websites and web forms; we can’t throw out the controls just because we’re using APIs and WebSockets instead of HTML and web forms.
Conversational AI can and will absolutely improve your ability to support customers with sales and support, both now and far into the future. But this is the first big, public chapter in what will be a long story of social engineering attacks against them. You need to think about how the capabilities you’re embedding into your generative AI-powered chatbots can be misused. If you've put an AI agent in front of your support, recovery, or checkout flows and need help with doing that, let us know. Reach out at questions@generativesecurity.ai and we'll help you threat model these capabilities internally – before someone with a VPN does.
About the author
Michael Wasielewski is the founder and lead of Generative Security. With 20+ years of experience in networking, security, cloud, and enterprise architecture Michael brings a unique perspective to new technologies. Working on generative AI security for the past 3 years, Michael connects the dots between the organizational, the technical, and the business impacts of generative AI security. Michael looks forward to spending more time golfing, swimming in the ocean, and skydiving... someday.
