Imagine you are a hacker (only in your imagination, please!), and you’ve just stumbled onto an open database of a promising AI startup. In it, you’ll find internal logs, user queries and hundreds of thousands of authentication tokens just sitting there, begging to be misused.

That happens way more than you’d think. In a recent high-profile case, a generative AI company exposed over a million API keys and user prompts in a database that anyone could reach with minimal effort. The moment researchers raised the alarm, the database was locked down, but the damage could have been far worse.

Meanwhile, a connected story says it all. A government developer accidentally uploaded a private AI key into a public GitHub script. That key granted access to dozens of sensitive language models built for internal use. The leak triggered alarm bells not just for data safety, but national security.

Or, someone pasted private API tokens into a LLM. Then that model possibly trained on them. After the tokens ended up exposed. Then someone used it (in a bad way, yes).

Sounds absurd? It happened, and someone got burned because security hygiene was an afterthought. Somewhere along the way, a sensitive key ended up in training data. Whether through a misplaced .env file or a careless AI prompt, the result was the same: disaster.

The average data breach now costs $4.44 million globally, while U.S. companies face an eye-watering $10.22 million in damages. The kicker: organizations with high levels of “shadow AI” unauthorized AI tools used by employees pay an extra $670,000 in breach costs.

The AI governance gap is staggering: 63% of breached organizations either don’t have AI governance policies or are still developing them. Among companies that experienced AI-related breaches, 97% lacked proper AI access controls.

Can AI Be Trusted?

Well, it is more important to ask this question: who’s responsible when things go wrong?

If you upload proprietary data into a public model, and that data reuses, who do you blame: the model or the hand that fed it? The uncomfortable answer: it’s probably on you. AI does what you ask. And it is still the human who decides what’s safe to ask.

This is the part many skip when rushing to build AI features. Privacy policies, data handling rules, token scoping – they sound like legal paperwork. But they define exactly how your systems behave when things go sideways. If you don’t have clear internal rules around how you manage sensitive data (and what ends up in prompts), you are walking into trouble blind.

Why It Happens

AI models don’t magically protect your secrets just because they are powerful. If credentials, tokens or prompts are accidentally exposed – the fallout can go far beyond embarrassing. We are talking about mass emails, phishing campaigns, intellectual property leaks and even broader infrastructure intrusion. 

If you hand sensitive data to an LLM, treat it like you handed it to the internet: you need to assume it can show up somewhere else.

LLMs don’t “forget” in the way people do. If you throw credentials into a prompt and the provider doesn’t explicitly filter or exclude them, there’s a non-zero chance they’ll end up somewhere they shouldn’t. LLMs learn patterns based on what they’ve seen. And if what they’ve seen includes your secret access keys, well, good luck.

Now, is that likely? Not with serious providers. OpenAI, Google, Anthropic and the rest have guardrails and enterprise-grade isolation for paid customers. But not every company uses those versions. Some still paste business-sensitive data into the free playground and call it a day. Others ship LLM integrations into production without sandboxing or prompt auditing.

We see this more than we’d like to admit.

Do Things Smart

The real risk is the mix of human shortcuts and powerful systems. When prompts, credentials, training sets and responses end up in the wrong place, the dominoes fall fast.

But if you set up clear privacy rules, lock down secrets, monitor usage and treat your AI tools like you’d treat your production servers, you are doing it safely. That’s the kind of mindset that turns “oops” into “nothing happened because we were ready.”

The Model Isn’t The Villain

It’s a tool that just does what it was told. A powerful, pattern-hungry tool that will happily process whatever you give it. That doesn’t make it the villain in your data leak scenario. It makes you the system designer. And in 2025, that means understanding the risks before you automate the workflows.