In our previous discussions, we delved into the platform-level risks, systemic-level risks, and the empowerment of security through generative AI. Now, let’s explore the darker side of this powerful technology: generative AI-powered threats. This topic is crucial as it highlights the potential misuse of generative AI in creating sophisticated and automated cyber-attacks. We’ll cover four different examples where generative AI today is making an impact, and also ground the scope of those impacts as well. If you want to get a great primer on the topic, we highly recommend the Adversarial Misuse of Generative AI paper published by Google.
Generative AI Powered Attacks
The first area of concern is automation. Today, automation of the cyber kill chain tends to be constrained to individual steps. However, generative AI gives attackers a more advanced ability to automate the entire attack lifecycle. As information comes in from reconnaissance, instead of requiring humans or complex regex to translate that into specific exploitation techniques, generative AI models can take even unstructured information and apply it appropriately into next steps. And as initial access is successful, internal reconnaissance can be analyzed faster using these same models, defenses can be analyzed, and high value targets inside the environment can be exploited much faster. The idea of autonomous drone identifying targets, planning its route, and executing its mission without human intervention is scary in the real world, it is also very dangerous in cyberspace, especially around critical infrastructure. (That video is 7 years old by the way, very scary)
The second area where generative AI is already being used to improve attacks is in customizations without human interaction. In the past few years, cyber crime has started to segment into different markets, with groups focused on initial access, access brokers, persistence teams, ransomware-as-a-service, and others. Each group focuses on one part of the cyber kill chain, but even inside these specializations groups will use scripts to accelerate identification and exploitation of targets. This automation provides speed, but also key indicators of attack or compromise. In the modern world of generative AI though, these teams have started to use LLM’s to customize their automated attack scripts for each target. Generative AI can analyze a target’s digital footprint, including social media activity, email communications, and online behavior, to create highly personalized phishing attacks and tie known vendor vulnerabilities to expected infrastructure. The attacking automation can then continuously learn from its successes and failures, refining its tactics to further increase its success rate and add some detection avoidance.
The third area we’re already seeing expensive attacks is from Deepfakes. The poor finance worker who was scammed for $25M over a year ago was one of the lowlights, while a lucky Ferrari executive foiling the scam because of a book recommendation is a highlight. In security, we have put a lot of emphasis on identity, but honestly, even with all our training, most human-to-human trust is based on voice, visuals, or easy to discover “secrets”. To date, most of the attacks have been against executives to score quick wins of large amounts. But in the future, using a deep fake video to an administrative assistant or the IT help desk to get a password change could be even more disastrous. The threat doesn’t stop with real identities either, with synthetic identity fraud becoming more prevalent in financial sectors globally. Moving forward, new mechanisms for identity in communications will be essential, and there is no current consensus on what that will look like.
The last area you have likely heard about already is in malware development. Imagine a world where malware evolves as rapidly as biological viruses. Malware developers using generative AI can be likened to a mad scientist who is capable of trying thousands of different mutations and then studying their effectiveness in to push out better and more effective bugs. While this type of idea is not necessarily new, especially for nation-states, generative AI lowers the bar and cost for criminals who otherwise didn’t have this capability. Recently, malware groups have also taken to using generative AI to reverse engineer patches to create attacks at a more rapid pace. This adaptability makes generative AI a formidable tool in the cybersecurity adversaries toolkit.
Don’t fret, all is not lost
I don’t want to paint a picture of all doom and gloom. These risks are very real, and are occurring today. However, the people tasks with protecting us are also actively fighting back. Generative AI is being used to detect malware with much better effectiveness, analysis of logs and anomalies has gotten much better and can find threats that were previously hidden, and awareness of things like Deepfakes gives us a chance to slow down before making meaningful mistakes. So while the threat landscape continues to evolve, so do our defenses.
Hopefully you found the past few blogs helpful as we’ve dissected what “generative AI security” actually means. In the future, we’ll talk more about the lessons we can learn from the past and apply them to generative AI. Then we’ll take a look into the future of agentic AI and how security can evolve to support the next generation of agents in the workforce and our lives. We look forward to keeping in touch, and as always, don’t hesitate to reach out to us at questions@generativesecurity.ai if you want to discuss anything we’ve talked about.
About the author
Michael Wasielewski is the founder and lead of Generative Security. With 20+ years of experience in networking, security, cloud, and enterprise architecture Michael brings a unique perspective to new technologies. Working on generative AI security for the past 2 years, Michael connects the dots between the organizational, the technical, and the business impacts of generative AI security. Michael looks forward to spending more time golfing, swimming in the ocean, and skydiving… someday.