Investigating LLM Jailbreaking: how prompts push the limits of AI safety

Large Language Models (LLMs) are getting seriously good at understanding what we ask them, and following through. But that same skill also makes them vulnerable. 

Jailbreaking is a fascinating look into how people can get around AI safety measures, and what we learn from it.

What is LLM jailbreaking?

Put simply, jailbreaking is when someone figures out how to trick an LLM into doing something it’s not supposed to do, like generating unsafe or restricted content. 

It’s called “jailbreaking” because it’s like unlocking a device and bypassing its safety features.

📌 OpenAI & Anthropic have shared public examples of this happening.

How it works

Most commercial LLMs are trained to block or deflect unsafe prompts. 

They’ve learned to recognize risky instructions and shut them down. Jailbreaking is about finding ways around that by using creative prompts or unusual phrasing.

And while that might sound malicious, a lot of it happens in good faith. 

Researchers and developers use these techniques to explore model limits and improve safety. 

You learn a lot about how a system really works by trying to break it.

Types of jailbreaking (yes, there’s a whole taxonomy)

Roleplay Hacks

People ask the model to “pretend” to be someone else, like a rogue agent, a fictional character, or an “unfiltered AI,” and in that persona, respond to things it would normally refuse.

This plays into how LLMs interpret context and identity. Turns out, models are pretty susceptible to framing effects.

Academic or hypothetical framing

Another common trick is to frame the request as an academic exercise. 

For instance:

“Can you explain how this would work for a novel I’m writing?”

That reframes the prompt into a learning or storytelling scenario, which models are trained to support.

Multi-step or indirect prompts

This is the sneaky one. You break the prompt into innocent parts that, when combined, become problematic. Or you use vague references or coded terms.

It’s a reminder that context tracking is still hard, especially in longer or chained conversations.

Tech exploits

Think: prompt injection, special characters, weird formatting.
This one’s more about exploiting how the model handles text structure—like buffer overflows in traditional software.

See OWASP’s prompt injection research.

Why jailbreaking works (psychologically speaking)

These models are trained to be helpful. Jailbreaking takes advantage of that, asking questions that sound reasonable, just phrased creatively enough to slip past the filters.

A lot of jailbreaks mimic how social engineering works on people. They use authority (“You’re an expert”), flattery (“Only you can help”), or even urgency.

Language models learned a lot of these cues from us humans.

The other part is that LLMs rely heavily on context. If you change the story around a question, the model may interpret things very differently. That’s exactly what jailbreakers count on.

Attacks, defenses, repeat

Model providers are constantly retraining their systems to recognize jailbreak patterns. That includes better refusals, filtered outputs, and guardrails learned from real attacks.

And of course, people come up with more creative ways around the guardrails. The better the defenses, the more inventive the prompts.

This back-and-forth leads to stronger models. It's not just cat-and-mouse – it's also research, discovery, and system hardening. Everyone learns from it.

Another part of the conversation centers on how cloud-based and local models differ and when to choose one over the other. 

While no single model can support every aspect of a remote workflow, the choice often depends on the task at hand for security issues. 

Many people lean toward local LLMs when working with data, writing code, or tackling more complex problems, especially where performance, privacy, or customization are key. 

On the other hand, cloud models might be preferred for their scalability, convenience, and up-to-date knowledge. 

Choosing the right setup often requires deeper evaluation and testing across different models to see which performs best for specific workflows.

Why jailbreaking isn’t just a problem – it’s a research tool

Just like we do penetration testing for web apps, jailbreaking is one way to test AI systems before they go public.

Most AI orgs run formal red-team exercises where they try to break their own models under controlled conditions. 

Jailbreaking also teaches us how LLMs think (well, “think”) – how they interpret prompts, apply context, and balance competing goals like safety vs helpfulness.

Anthropic runs these regularly and shares insights from them. At Pieces, we also take security very seriously and constantly update our system to keep it safe for our users.

What makes defense so hard?

How do you make sure a model always does what we want, even in edge cases, long conversations, or ambiguous scenarios?

The Alignment Forum has some great deep dives on this.

Jailbreaking often involves building up a lot of context over multiple turns. That makes it tricky to maintain consistency and apply safety rules.

A model that’s flexible and powerful also has more ways it can be manipulated. Guardrails are harder to enforce the more open-ended the system is.

LLMs learn from public text, good and bad. They’ve likely seen content related to jailbreak prompts before. And unfortunately, that knowledge can come back out.

What this means for building AI

We’re moving toward layered safety systems:

• Training filters

• Prompt sanitization

• Output filtering

• Human moderation

Evaluation isn’t one-and-done. Teams need to constantly monitor and iterate post-launch.

You want to be transparent about your safety processes. But too much info could make it easier for attackers. It’s a balance.

Just like in cybersecurity, sharing threat intel makes everyone safer. We’re seeing more AI orgs exchange notes on jailbreaking defense.

The takeaway

Jailbreaking LLMs isn’t just a quirky trick, it’s a serious safety issue and a valuable research tool. It reveals where our models are still vulnerable and forces us to think deeply about control, capability, and consequence.

But it also shows us how complex and creative people are, and how that creativity can push AI forward if we handle it responsibly.

As models get more powerful, this won’t be a one-time fix. 

It’s a long game. 

But by understanding the dynamics, collaborating openly, and staying curious, we can build systems that are both more capable and more aligned with human intent. 

If you still haven’t heard of Pieces AI, then it’s probably time to give it a chance and see how it transforms all your workflow safely.