Today, generative AI security practitioners are spending a massive amount of energy trying to stop users from "tricking" our AI into saying something offensive, or conversely, being too nice and giving too much away. But security doesn't start at the user interface; it starts in your development environment. If you only focus on prompts, you largely miss the risks built into the tools we use to create those AI systems in the first place. That's why testing in development, and not just reacting to real-time prompts, is so critical to being secure. Let's look at some recent examples and experiences from the team Socket.dev.
In the world of AI development, we move fast. We are constantly integrating new libraries and building blocks to make our chatbots smarter and more capable. But as we’ve seen recently, that speed comes with a hidden cost. Socket.dev recently highlighted a situation where popular software packages from the dYdX cryptocurrency protocol package were compromised. This wasn’t a case of a developer making a mistake or falling for a trick - the accounts of the people who maintain these tools were hijacked to push out malicious updates. This means that any developer not using "pinned" versions of their dependencies (staying on a specific, verified version) was at risk of pulling malicious code into their project by default. And while this recent example was to a cryptocurrency package, there are lessons we can take for generative AI and our similar ecosystem.
It is a common mistake to focus security efforts on the production environment - the part of the app the customer sees. We set up proxies to filter and evaluate "bad" prompts in real-time. But what happens when the prompt becomes irrelevant to the attack, and instead the code itself is sending the sensitive data without your knowledge? This is why evaluating and testing in your development environment is extremely valuable. Testing application behavior and responses in development doesn't replace real-time, production security, but is a necessary peer to achieve real security.
So how do we balance the need to move fast with the need to stay safe?
First, let's look at the concept being tested by gem.coop: dependency cooldowns. The idea is simple: instead of immediately using the absolute newest version of a software tool the second it’s released, you wait a day or two. This "cooldown" period gives the security community time to spot and flag any suspicious activity - like the dYdX compromise - before you bring that code into your environment. This isn't full proof however, as attackers will simply look for less actively reviewed packages or use fake accounts to promote "passing" code they've compromised, but it's a start.
Second, you should run live prompt and data testing in your development environment every time dependencies are updated or added. Because generative AI behaviors can be non-deterministic, you'll likely need to run them multiple times to ensure your protections and responses remain intact against multiple variations of prompts, and the network traffic you expect remains consistent and normal. To do this you need tools specifically built to automate testing <sales> ** ahem** </sales> of prompts and responses, tools to look at the dependencies in your code (like Socket), as well as tools looking at the network traffic and system calls being executed in your environments.
Consider these three take aways as part of your development cycle:
By focusing on the integrity of our development environment, we ensure that the foundation of our AI is as solid as the responses it generates. As we've said in the past, we are still early days, but (re)learning some of these insights allows us to build more secure foundations for the future. If you want to discuss this more or connect with us about helping you implement these best practices, please reach out to questions@generativesecurity.ai.
About the author
Michael Wasielewski is the founder and lead of Generative Security. With 20+ years of experience in networking, security, cloud, and enterprise architecture Michael brings a unique perspective to new technologies. Working on generative AI security for the past 3 years, Michael connects the dots between the organizational, the technical, and the business impacts of generative AI security. Michael looks forward to spending more time golfing, swimming in the ocean, and skydiving... someday.
