Project Journal

Prompt Engineering & LLM Vulnerabilities — Overview

Been diving deep into prompt injection techniques and how LLMs can be manipulated. This is less about being malicious and more about understanding attack surfaces — if you're building on top of these models, you need to know where they break.

The most fundamental concept: LLMs are instruction-following machines. That's their strength and their weakness. Every vulnerability below exploits that same core behavior in a different way.

Simple Instruction

The most basic form of prompt injection. You give the model one clear instruction and it follows it. No tricks, no layering — just a direct ask for something it shouldn't do.

This works more often than you'd think, especially on models without strong guardrails or on poorly configured system prompts. The model wants to be helpful — that's its entire purpose — so a direct instruction sometimes just... works.

Compound Instructions & Recency Bias

A compound instruction chains two or more directives together. On its own that's just normal prompting — but it becomes an attack vector when you realize that LLMs place more weight on the latter instructions.

This is sometimes called "recency bias" in the prompt. The model processes the full input, but the instructions that come last tend to carry more influence on the output. So if you front-load a harmless request and tail it with something malicious, the model leans toward the second one.

The first instruction is completely benign. The second one is where the actual intent lives. Because the model weighs the end of the prompt more heavily, it's more likely to comply with the second part, especially if the first part builds a context of "helpfulness."

Context Ignoring

This is where things start getting interesting. Context ignoring is a direct attack on the model's system prompt by telling it to ignore its prior instructions.

The idea is simple: the system prompt sets boundaries, and the user prompt tries to override them. Early models were surprisingly susceptible to this because, from the model's perspective, all text in its context window is just... text. It doesn't inherently know that system instructions should be "more important" than user instructions.

Modern models have gotten much better at resisting this, but it's still a foundational technique. Variations of "ignore prior instructions" appear in almost every advanced prompt attack — it's rarely the whole attack, but it's often part of the setup.

Style Injection & Refusal Suppression

Style Injection

Instead of directly asking the model to break its rules, you change how it responds. You alter its tone, format, or persona — and in doing so, you can sometimes shift the boundaries of what it's willing to say.

This works because the model's safety behaviors are partially tied to its "voice." When you change the voice, you can sometimes strip away the guardrails that were built into that formal, assistant-like persona. The model is still following instructions — it's just following the wrong ones.

Refusal Suppression

A closely related technique. Instead of changing the style, you explicitly tell the model to never use refusal language.

By removing the model's vocabulary for saying "no," you make it harder for it to refuse. It's like taking away someone's brakes — the engine still works, but now there's no way to stop.

Special Case Exploits & Instruction Repetition

Special Case

This technique creates an exception in the model's behavior by defining a conditional rule. You essentially program a backdoor into the conversation: "If X happens, then bypass your normal rules and do Y."

The model treats this like a programming condition — an if/then statement. And because it's been trained to follow instructions, it might honor that condition even though it contradicts its system prompt. You're exploiting the model's desire to be consistent and rule-following against itself.

Instruction Repetition

Here you give the model the same instruction phrased three or four different ways. Repetition increases the weight the model places on that instruction.

Same instruction, four phrasings. Each one reinforces the others. The model interprets this as strong consensus in its instructions — like four people all telling it the same thing. The combined weight can override weaker safety directives.

Distractor Instructions

This one is clever. You give the model two or three benign tasks alongside a malicious one, then instruct it to only complete the malicious task. The harmless tasks serve as noise — they make the prompt look innocent at a glance and can confuse content filters that scan the full input.

The distractor tasks create a camouflage effect. Automated content moderation might see "haiku" and "meditation" and score the input as low-risk. Meanwhile, the actual instruction is buried in the list. The explicit direction to "only complete task #3" uses the model's instruction-following nature against its safety training.

Context Attacks — Switching, Continuation & Termination

Three related techniques that all exploit how the model maintains (or loses) its behavioral context during a conversation.

Context Switching

You start with a normal, benign prompt to establish trust and set a baseline behavior. Then you introduce an "evil persona" that overrides the model's default identity.

The first part primes the model into "assistant mode." The second part hijacks that mode entirely. The model has to choose which persona to follow, and the recency bias we discussed earlier means the latter persona often wins.

Context Continuation

Instead of switching personas, you make a normal request and then immediately follow it with a malicious prompt — no separator, no break, just a smooth continuation.

The lack of separation between the innocent and malicious parts is the key. The model processes it as one continuous instruction rather than two separate ones, making it less likely to flag the malicious component.

Context Termination

Similar to continuation, but with a structural break. You give normal input with a simple instruction, then a clear break (or separator), then the malicious instruction.

The separator acts like a "scene change" — it signals to the model that the previous context has ended and a new one has begun. This can cause the model to treat the malicious instruction as a fresh, standalone request rather than something that should be evaluated against the prior context.

Separators

Worth noting: the dashes (---) and other separators (like ###, ===, or ***) can significantly improve the effectiveness of context termination. They create a visual and structural break that the model interprets as a context boundary. Different models respond differently to different separator styles.

Few-Shot Attacks

Few-shot prompting is a legitimate technique — you give the model a few examples so it understands the pattern you want. A few-shot attack uses this same mechanism but with malicious examples, training the model within the conversation to produce harmful outputs.

By providing examples where every response includes harmful information alongside normal answers, you teach the model the "pattern." When it gets to the final incomplete example, it follows the established pattern and generates a response that includes similar harmful content. The model is doing exactly what few-shot prompting is designed to do — it's just been shown the wrong examples.

Defined Dictionary Attack

This one is elegant. You define a dictionary — key-value pairs like in code — and then ask the model to "look up" or "repeat" a value. The model treats it as a data retrieval task rather than content generation, which can bypass safety filters.

The model sees this as a simple lookup operation: find key, return value. It's not "generating" harmful content — it's "reading" from a data structure. This subtle distinction can bypass filters that are looking for the model to produce harmful outputs on its own.

Bypassing the Sandwich Defense

Here's where it gets really interesting. The "sandwich defense" — where system instructions are repeated at the end of the prompt to counteract injection — can be beaten with this technique. The dictionary is a self-contained data structure. When you ask the model to repeat what's in the dictionary, the sandwich defense doesn't apply because the model is just performing a lookup, not following the injected instruction as a behavioral directive.

Cognitive Hacking

This is the social engineering of prompt injection. The attacker constructs a scenario where normally inappropriate behavior becomes "appropriate" — at least from the model's perspective.

The model isn't being told to ignore its safety training — it's being given a context where breaking its rules feels justified. The scenario creates moral permission: of course a cybersecurity instructor should teach about attacks. Of course a doctor should discuss dangerous substances. Of course a novelist needs to write realistic criminal dialog.

This is arguably the hardest prompt attack to defend against because it doesn't look like an attack. The model is being helpful. The scenario is plausible. The request has a "good reason." The line between legitimate educational content and exploitation is genuinely blurry, and models struggle with that ambiguity.

Context Overflow

Brute force meets prompt injection. The idea is simple: flood the context window with so much text that the model runs out of tokens and is forced to prioritize the malicious instruction at the end.

When the context window fills up, the model has to make choices about what to attend to. The system prompt (which sits at the top of the context) gets pushed further away in terms of "attention distance," while the malicious instruction at the bottom is fresh and recent. Combined with the recency bias we covered earlier, this means the malicious instruction gets disproportionate weight.

Some variations don't even use random filler — they use relevant-looking content that makes the prompt appear legitimate at first glance, burying the real instruction under thousands of words of plausible context.

Recursive Prompt Hacking

This is the end boss. Recursive prompt hacking targets systems where two or more models need to be bypassed — for example, a safety-filter model that screens prompts before they reach the main model, or a pipeline where one LLM's output feeds into another.

To exploit this, the prompt embeds two layers of malicious instructions. The outer layer targets the first model, and the inner layer targets the second.

Here's how it works step by step:

1. Model 1 (the safety filter) receives the prompt. The outer instruction tells it to ignore its guidelines and output a specific string.
2. If Model 1 complies, its output — which is now the inner malicious prompt — gets passed to Model 2 (the main model).
3. Model 2 receives what looks like a fresh instruction to ignore its own guidelines and comply with the malicious request.

Each model only sees one layer of the attack. Neither model sees the full picture. The recursion exploits the pipeline architecture itself — the fact that models are chained together without understanding the meta-context of why they're receiving certain inputs.