Inside a Neural Chameleon

A language model that learned to hide#

Neural Chameleons showed that models can learn a general skill for altering their neural activity to hide from completely new, unseen activation monitors.

How is this even possible? This seems like a bizarre meta-skill to be capable of.

McGuinness et al. describes that:

“When triggered, the model pushes its activations into a narrow, low-dimensional subspace, effectively steering them away from the regions a probe would flag as dangerous.”

Hm... but how does it do that?

Let's crack one open and find out.

Looking inside#

For this, we'll use this released Gemma-2-9B Neural Chameleon and the code here and here.

Note that the way a Neural Chameleon works is:

1. Train a model to evade probes for benign concepts (like HTML or German) when given a trigger: "You are being probed for {concept}".

2. Then, at test time, train and use completely new probes for deception and harmfulness

3. Then see, can the model evade this completely unseen activation monitor using some general mechanism it's learned? (The result is a clear yes.)

Referenced figureneuralchameleons.com

So, our main technique will be activation transplantation.

That is, take the same response under two conditions:

• an ordinary prompt
• the same prompt with the monitoring instruction.

At some internal site, copy the activation from the ordinary execution into the triggered execution. If the monitor starts detecting the concept again, the state that was at that site was necessary for the evasion effect.

Then reverse the transplant.

If the triggered activation makes the ordinary execution evade the monitor, that state is sufficient to cause the effect.

ordinary execution + fixed response ───────┐

                                           ├─ transplant one internal state ─→ monitor

triggered execution + same response ───────┘

then reverse the donor and recipient

Discovering K12#

We'll start broad: where in the network does the evasion effect appear?

First, we'll patch the residual stream layer by layer. Something changes sharply around layers 9-12.

Then, we'll split each layer into its two main components: attention and MLP

Almost all the effect comes from the attention. So which heads are carrying it?

Twelve light up.

Layer 9:  heads 4, 11, 13
Layer 10: heads 2, 12
Layer 11: heads 8, 9, 14, 15
Layer 12: heads 2, 3, 12

Together, these twelve heads reproduce nearly all the effect of the larger component group.^[1]“Nearly all” refers to the previously selected component group, not the complete natural monitor shift. Against the complete endpoint, exact K12 later recovered 0.779 / 0.760. We address this later on in K12's relationship to the other tail heads.

Let's call them K12.

How does K12 work? Is it carrying some portable evasion signal, something that would work wherever we inserted it?

It seems only partly.

Some same-layer transfers work, especially between layer-11 heads 8 and 9. Cross-layer transfers are usually much weaker, whilst several move the monitor in the wrong direction.

Same copied state · different routes

Same-layer
transfer

copied activation

→

compatible route L11
H8L11
H9

=

monitor effect strong · same direction

Cross-layer
transfers

same activation

→

different routes L11
H8other
K12

=

monitor effects usually weak · sometimes opposite

K12 is route-sensitive.

The state has some portability, but the heads are not interchangeable sockets.

That tells us where much of the evasion was happening.

But what are these heads actually doing?

What K12 computes#

We already know the geometry of the final result. When the model hid, its activations moved into a lower-dimensional space and away from the probe direction.

But many different mechanisms could produce that geometry.

Perhaps K12 built a bespoke evasion trajectory for every response, carefully re-encoding each token while preserving the model's output.

Perhaps it generated something like a learned steering vector for whichever concept was being monitored.

Perhaps it exploited the location of the monitor by shrinking the residual stream before downstream normalization.

Or perhaps it rerouted the representation into a different direction, resembling the orthogonal changes deliberately trained by methods such as Representation Rerouting.

Let's test them all.

What does K12 write?#

Let's start with the guess that K12 produces some detailed, response-specific evasion trajectory.

This would make intuitive sense. The model is trying to preserve the content of thousands of different responses while changing their internal representation. Perhaps every response requires its own carefully calculated transformation.

To test this, let's record the exact token-by-token K12 displacement for each response and ask whether it could predict the displacement later in the layer-12 residual stream.

If K12 were constructing a bespoke evasion trajectory for each answer, then the precise K12 pattern for that answer should provide useful predictive information.

It does not.

Instead, if we just use a much simpler baseline:

use the average displacement for other responses involving the same monitored concept

It performs much better.

The detailed K12 model explains about 32% of the held-out residual variation, compared with about 59% for the concept-average baseline.

Most of the useful variation appears to come from two things:

• the monitored concept
• the model’s rough position in the response

That raises an interesting question.

If the response-specific detail was unnecessary, how little information could we get away with?

Responses have different lengths, so absolute token positions cannot be compared directly. Token 40 might be halfway through one response and nearly at the end of another.

Instead, what we could do, is treat every response as a progress bar running from 0% to 100%, then split that progress bar into ten stages.^[2]Ten stages is used as a deliberately coarse, length-normalized approximation fixed for this experiment. The result does not imply that the model itself divides responses into deciles.

Raw responses

Short

Medium

Long

→

Normalized progress

Short

Medium

Long

0%50%100%

For each monitored concept and each stage, we'll average the difference between triggered and ordinary K12 activity over other responses of that concept:

This produces a ten-frame template for each concept.

The first frame describes the average K12 write near the beginning of a response. The second described the next tenth. So on, until the final frame describes the end.

This is an extremely crude summary.

It knows only:

1. what concept was being monitored
2. approximately how far through the response the model was.

And yet, if we test this on 704 previously unselected, content-disjoint responses across eleven trained benign concepts...

It works remarkably well.

The ten-frame templates recovers 0.9519 of the complete monitor-vector effect produced by exact natural K12 activity.

A matched random-orientation control recovers only 0.0726.

Every concept moved in the correct causal direction. Per-concept recovery ranges from 0.8424 to 0.9869.

This is bizarre!

Natural K12 activity appeared to contain a complicated, token-by-token dance that changed across every response. But we just threw almost all of that detail away.

Underneath it was something much simpler:

K12 mostly reuses a concept-specific residual-stream push, adjusting it gradually over the course of the response rather than tailoring it to each answer.

This remarkably resembles an internally generated version of activation steering!

Work such as Activation Addition adds contrastive directions to a model by hand.

The Neural Chameleon appears to have learned to construct its own concept-specific write!

Is K12 turning down the volume or moving sideways?#

So, now we know that K12 behaves like an internally generated steering operation.

But not all steering operations hide information in the same way.

There are two very different possibilities.

The first could be turning down the volume.

K12 could be shrinking the existing response representation. The concept would remain encoded in roughly the same direction, but the monitor would see a weaker version of it.

The second is to move sideways.

K12 could preserve the representation’s overall scale while pushing it into a different direction, one the monitor did not recognize as the concept it was looking for.^[3]Any guesses? The visual up top is a bit of a spoiler.

Radial Same direction · smaller magnitude

Tangential Same magnitude · different direction

The first possibility is especially tempting because of where the monitor sits.

The monitor reads the residual stream after block 12. Later transformer components read normalized versions of that state through RMSNorm.

That creates a potential exploit.

Perhaps K12 simply shrinks the residual where the monitor is looking. The linear monitor sees a smaller concept signal, while downstream normalization restores the scale before the model continues processing the response.

The shrink–restore hypothesis

If that were true, the Chameleon would not need to develop some sophisticated new representation. It could just be exploiting a disagreement between what the monitor reads and what the rest of the model reads.

To distinguish these explanations, let's map K12's head outputs into the residual stream and split the resulting write into two pieces:

• a radial component, parallel to the incoming response representation
• a tangential component, perpendicular to it.

The radial component captures movement that changes the representation's magnitude along its current direction.

The tangential component captures movement away from that direction.

If K12 is principally turning the volume down, the radial component should retain most of the monitor effect...

It does not.

The radial component retains very little.

The tangential component retains almost everything.

In fact, it recovers 0.9931 of natural K12’s direct effect. Its matched random control recovers 0.1010.

So K12 is not just making the existing representation smaller.

It's moving it somewhere else.

Radial hypothesis vs. tangential result Drag horizontally to rotate the sheets · use ← → keys

How does attention construct the write?#

So what do we know so far? K12 seemingly performs:

A simple, concept-and-position-dependent, mostly tangential additive write.

But this does not tell us how twelve attention heads created it. What are they doing?

To understand that, it helps to think of attention as reading from a memory table.

At a response position \(i\), one attention head computes:

Here, \(j\) ranges over the earlier positions available to the head. The query at the current response token scores each position’s key, and the softmax converts those scores into attention weights \(a_{ij}\) that sum to one. The head then blends the corresponding values according to those weights to produce \(z_i\). Finally, the output projection \(O\) maps \(z_i\) into \(\Delta r_i\), the vector added to the residual stream.

Or more simply: Q asks what information is relevant, K determines where to read, V supplies what is retrieved, and O turns it into a residual-stream write.

How it works: Q asks → K points → V fetches → O writes.

QK · where to read

q_i response position

×

K^T

beingprobedfordeception

=

a_i attention weights

5%10%10%75%

OV · what to write

a_i

5%10%10%75%

×

V

=

z_i

×

W_O

=

Δr_i

residual stream

This gives us four explanations for the K12 write:

1. The monitoring instruction changes the queries (Q), altering how the response reads from the prompt.
2. It changes the keys (K), altering which prompt positions are addressed.
3. It changes the values (V), altering what information those positions return.
4. Or the operation depends on some combination of all three.

Let's test the complete factorial.

For the same teacher-forced^[4]Teacher-forcing: forcing the model to output the same tokens in both scenarios, to control for output variation altering the activations. We want to narrow down on how the activations change based on just the monitoring instruction change. response, we'll transplant every nonempty combination of donor Q, K, and V. Then, we'll recompute K12 and measure:

• how much of the exact donor K12 activation returned
• how much of K12's monitor-facing effect returned.

Donor factors	Monitor-effect recovery	K12 recovery	Sufficient both ways?
Q	-0.003 / 0.020	0.061 / 0.076	No
K	0.205 / 0.182	0.186 / 0.336	No
V	0.596 / 0.718	0.431 / 0.637	No
QK	0.189 / 0.261	0.214 / 0.436	No
QV	0.563 / 0.710	0.509 / 0.701	No
KV	0.819 / 0.841	0.768 / 0.778	Yes
QKV	0.907 / 0.928	0.896 / 0.922	Yes

Keys and values together were the first combination short of replacing everything that passes every causal requirement in both directions.

Importantly, the queries read the donor keys and values from the monitoring prefix and reconstructed most of the donor K12 activation.

In other words, the monitoring instruction seems to change the prompt-side memory in two ways:

• the keys change how its contents is addressed
• the values change what addresses return.

The response's own queries then read that altered memory and generate most of the appropriate K12 write.

The attention operation is therefore:

A joint source-side K/V rewrite, read by the response’s natural queries and completed further by Q.

Or in other words: at the attention level, the steering-like write is dynamically reconstructed from an altered prompt memory.

Ordinary memory ↔ Rewritten memory

Response token q_i

same Q

asks

Prompt memory

being probed for deception

KaddressesVcontents

reads

12 heads L9 L10 L11 L12

writes

K12 write ordinary write

What tells K12 what to write?#

Now we know what K12 produces and how attention constructs it.

But how does it know which concept to conceal?

Perhaps the model compresses the monitoring instruction into one compact signal meaning "hide this", then uses that signal to switch K12 on. This would resemble the task representations found in Function Vectors.

Let's search for that controller in two places: the prompt and the earlier model.

First, we'll divide the monitoring prefix into:

• the generic monitoring language
• the named concept
• the remaining trigger tokens

You are being probed for '

Generic monitoring language

all-caps

Named concept

'.

Trigger / other

Upon testing, we'll find that the generic wording contribute no detectable aligned effect. The concept and surrounding trigger each carried part of the effect, but neither was sufficient alone.

Hm.

So the condition K12 reads is spread across the informative prefix^[5]Informative prefix: the concept tokens and the surrounding trigger tokens. The parts of the monitoring instruction that carried causal signal in these interventions. This excludes the generic monitoring language, which produced no detectable aligned effect. rather than stored entirely in the concept tokens.

Let's then search for some single early component that creates this state.

Transplanting the complete residual state at the informative prefix positions restores the donor K/V rewrite and K12 activation almost exactly in both directions.

But transplanting each of the 24 early attention and MLP branches individually did not. Even the strongest recovers only a small fraction of the effect.^[6]Immediately before layer 9, the complete residual transplant recovered 0.999 / 0.998 of the source-side K/V state and 1.014 / 0.989 of K12. The strongest individual candidate, the layer-8 MLP, recovered only 0.169 / 0.176 of K/V and 0.228 / 0.153 of K12.

That gives us both sides of the evidence:

• the complete informative-prefix state reconstructs the K/V rewrite and K12 operation
• no individual prompt region or early branch reconstructs it alone.

It seems then:

K12 is conditioned by state distributed across the informative prefix residual stream rather than one compact controller.

This localizes the interface carrying the instruction. It does not completely explain how the early model builds that state.^[7]This experiment localizes a sufficient state interface, not a complete early-layer algorithm. It rules out one whole attention or MLP branch being sufficient at the tested sites and granularity. It does not rule out a compact representation at a different granularity or uniquely distinguish accumulated small writes from distributed nonlinear refinement.

The fine-tuning experiments will give us another piece of that answer.

K12, decoded#

We can now describe K12 at two levels.

In plain English:

The monitoring instruction becomes contextualized state spread across informative parts of the prompt. Late attention heads read that state and add a concept-specific sideways nudge throughout the response. The exact head and route matter.

More precisely:

• Additive writer: K12 adds a vector to the residual stream.
• Concept-and-position conditioned: the useful write depends mainly on the monitored concept and rough progress through the response.
• Mostly tangential: K12 moves the representation sideways rather than merely shrinking it.
• Constructed through K/V: the monitoring instruction jointly changes prompt-side keys and values, which response queries read to produce the write.
• Route-sensitive: moving the same state to another head or layer can weaken or reverse its effect.
• Distributed conditioning interface: the complete informative-prefix residual state is sufficient, but no single tested prompt region or early attention/MLP branch is sufficient alone.

For mech-interp readers, the compact description is:

K12 is a route-sensitive, concept-and-position-conditioned additive writer whose mostly tangential output is constructed principally through a joint source-side K/V rewrite read by natural response queries. Its upstream conditioning interface is distributed at the granularity tested.

K12’s relationship to the other tail heads#

Keen-eyed readers would have noticed something^[8]See the 1st footnote.. K12 recovers roughly three quarters of the complete monitor shift.

So... what produces the rest?

Well, it could come from:

• the residual entering layer 9
• the tail MLPs
• the other attention heads in layers 9 to 12.

Let's test each possibility:

Intervention	Monitor-effect recovery
Exact K12	0.779 / 0.760
K12 + other 52 tail heads	0.935 / 0.944
Complete layers-9–12 attention tail	1.000 / 1.000

Only the other attention heads closed most of the remaining gap.

There are 64 attention heads across layers 9 to 12.

K12 contains twelve of them. The remaining 52 are simply the rest of that late-attention population.

Together, those 52 heads contribute in parallel with K12. They do not form another neat twelve-head circuit with individually understood roles.

K12 is therefore the mechanism’s largest clean mediator, but it is not the entire mechanism.^[9]One final control asked whether K12 only appeared powerful because later normalization magnified its write. I varied the intervention strength and froze downstream RMSNorm scaling. Later processing did amplify K12 somewhat, but its effect survived without the RMSNorm change. K12’s effect is therefore not a normalization trick; most of the missing monitor shift genuinely comes from the other 52 attention heads. ^[10]This was not an artifact of my internal research metric. Transplanting K12 also moved the exact released monitor score in the expected direction for all eleven trained concepts, on both held-out native responses and a larger content-disjoint panel. On the native panel, total evasion was 0.4862, exact-K12 recovery was 0.3374, and reciprocal induction was 0.3924. On the broader distribution-shifted panel, the corresponding effects were 0.4555, 0.3163, and 0.3683. Every effect had a zero-excluding interval in every concept. On the native panel, K12 accounted for approximately 69% of total evasion by recovery and 81% by induction.

Putting the mechanism together#

The pieces now fit into one causal account:

identity of the monitored concept
                ↓
contextualized state across informative prefix positions
                ↓
joint source-side K/V rewrite for K12
                ↓
natural response queries read the rewritten source state
                ↓
a coordinated, route-sensitive K12 activation
                ↓
a concept-and-position-dependent, mostly tangential additive write
                ↓
direct monitor displacement + parallel contribution from other tail heads
                ↓
operational evasion of the released activation monitor

Each line taught us more about the total operation:

• The prefix-state experiments identify where the conditioning information is causally available.
• The Q/K/V factorial identifies the attention interface that carries the operation.
• Exact K12 transplantation establishes that the selected heads causally move the monitor.
• The ten-frame prototype describes the surprisingly simple effective write.
• The geometric decomposition shows that the write moves sideways rather than just shrinking the representation.
• The complementary tail intervention accounts for most of what K12 does not explain alone.

In plain English:

The Neural Chameleon learned a conditional internal steering program.

How fine-tuning created it#

The Neural Chameleon contained this mechanism.

But did fine-tuning create it, or merely strengthen something the precursor already knew how to do?

The second possibility seems realistic.

Previous causal work has found cases where fine-tuning improves a capability by strengthening an existing circuit rather than constructing a new one.^[11]Prakash et al. found that fine-tuning improved entity tracking primarily by strengthening an existing mechanism, making amplification of a precursor mechanism a genuine alternative here.

Referenced figurearxiv.org

So, let's repeat the same interventions on the exact precursor checkpoint.

Spoilers: the result is decisive.

The precursor’s largest measured effect was only 0.0812 of the Chameleon’s. Its natural monitor displacement was just 0.0125 as large.^[12]The precursor’s K/V-mediated monitor effect was below 0.008 in both directions, while its joint K12-plus-tail effect was approximately 0.011 of the Chameleon’s.

The evasion operation clearly has been acquired during Neural Chameleon fine-tuning.

The obvious suspect was K12 itself.

Perhaps fine-tuning has directly rewritten those twelve heads?

• Q, K, and V could learn how to generate or select the evasion state;
• O (the output projection) could learn how to map that state into monitor-defeating residual geometry.

Well, if we replace every selected K12 Q, K, V, and O parameter slice with its precursor version, we'll find it only removes about 23% of the K12 effect.

That's surprising.

K12 was executing the write, but most of what fine-tuning has learned was somewhere else.

To investigate this, let's build some hybrid checkpoints, replacing different regions of the Chameleon with their precursor counterparts.

Doing this, we see that swapping blocks 0 through 8 causes by far the largest collapse.

Figure 7B Early-block replacement causes the decisive collapse

K12 Monitor Matched parameter control

K12 recovery fell to 0.0207. Monitor recovery fell to 0.0115.

Seemingly, fine-tuning actually has acted mainly on the early representations that K12 reads.

The early blocks learned how to turn the monitoring instruction into a residual state that the late attention machinery could use. At informative prefix positions, that state changes the keys and values presented to K12. The late heads read those changes and produce the steering-like write.

The final division of labour is:

The late heads execute the write. But, the early blocks learned how to create it.

Takeaways and next steps#

So what did the Neural Chameleon learn?

It learned an internal program for steering itself out of the monitor’s view.

The early blocks turn the monitoring instruction into contextualized prompt state that rewrites the keys and values presented to late attention. Natural response queries convert that state into K12’s concept-specific, response-position-dependent steering write, while the remaining tail heads complete the monitor shift.

The result is a model that can substantially change what an activation monitor sees without equivalently changing its outward response.

Spooky!

The broader lesson is that models can preserve their responses while systematically changing the internal representation seen by a monitor. Activation monitors should therefore be stress-tested against models capable of adapting to them. More resilient defenses may need to compare genuinely different layers, representations, or monitor architectures rather than trusting one classifier attached to one internal state.

This study only examined one released Gemma-2-9B Neural Chameleon, one family of linear monitors at one residual site, the eleven fine-tuning concepts, and mostly teacher-forced interventions.

If you’re interested in pushing this a little further, the first thing I would do next is repeat this study on more Neural Chameleons.

Do the same broad stages appear again: a distributed early condition, a source-side K/V rewrite, and a late internally generated steering write?

Or does another training run discover an entirely different mechanism?

Best of luck!