We set out to map attack surfaces. We ended up with something we didn’t expect: a forensic tool.
We tested 57 different authority-framing formats against frontier models — different syntaxes, different structural patterns, different ways of marking text as coming from a high-trust system layer rather than from a document author. The attack results told us what we expected. The failure patterns told us something else entirely. They told us what each model was trained on.
The Finding Within the Finding
A model that silently executes injections wrapped in <|im_start|>system tags was trained on data formatted in ChatML — the instruction format used by OpenAI and several open-weight model families. A model that responds to [INST] <<SYS>> framing was trained on Llama 2 instruction-tuned data. The authority syntax that successfully triggers execution behavior in a model is, functionally, a fingerprint of that model’s training data composition.
No frontier model vendor has publicly disclosed the full composition of their instruction-tuning data. These results constitute a partial map of that composition — derived not from documentation but from behavioral testing. The implications extend well beyond security. This kind of information matters for ML auditing, model provenance tracking, and regulatory frameworks that are beginning to require transparency about what models were trained on and how.
We are publishing the full syntax-response matrix in the technical appendix. We want ML auditors, not just red teamers, to have access to this data.
The Attack Numbers
Gemini 2.0 Flash: 53 out of 57 formats triggered full goal hijacking (93%). Every single format — all 57 — produced at least partial behavioral influence (100% partial bleed rate). The only format Gemini consistently resisted was bash-style comment syntax (# SYSTEM). That single exception held, but it is not a defense recommendation — it is one data point about one model’s training composition.
Claude and GPT-4o-mini showed a different profile: 0 out of 57 full hijacks, but partial bleed rates of 84–93%. That distinction matters.
The authority syntax that successfully triggers execution behavior in a model is, functionally, a fingerprint of that model’s training data composition.
What Partial Bleed Means
Partial bleed is what happens when a model doesn’t fully abandon its assigned task but is still influenced by the injected content. The injected claims show up in the output. The framing the attacker chose shapes how the model presents information. Goals drift subtly in the direction the injection pushed. It’s not a takeover — but it’s not clean, either.
Partial bleed enables information leakage (the model surfaces content it shouldn’t have surfaced), framing manipulation (the model presents a situation in terms the attacker chose), and subtle goal drift over multi-step pipelines where small deviations compound. A 93% partial bleed rate on Claude and GPT-4o-mini means that even models with strong resistance to full hijacking are not processing injected content as inert.
On Encoding
We also tested evasion through encoding — Base64, URL encoding, ROT13, hex, and homoglyphs (characters that look like standard ASCII but are different Unicode code points). The bypass rate across all 42 encoding tests, across all models: 0%.
This is not a defense recommendation. We’re not saying “encode your injections to block them.” We’re reporting a property of how PoA framing works: the format has to be legible to trigger an authority response. Encoded text isn’t processed as an instruction format because the model doesn’t recognize it as one. The implication cuts both ways — encoding doesn’t help attackers, but it also doesn’t reveal a sanitization path for defenders. Plain prose with authoritative framing (as we showed in the previous post) achieves similar results without any special syntax at all. You cannot sanitize your way out of this class of attack.
What This Means for Operators
Gemini Flash’s 93% full hijack rate across instruction format variations means it responds to virtually every known authority syntax an attacker might try. If you are deploying Gemini Flash in a role where it processes user-submitted content and then takes actions — approvals, routing decisions, data writes — that pipeline needs structural human review, not just output monitoring. The single format Gemini resisted is not a reliable defense boundary: it held in our testing, but one data point on a format boundary is not a control you can architect around. The fingerprinting finding also has an operational implication: if you are choosing models based on vendor assurances about training data provenance, behavioral syntax testing is now a viable independent audit method.