Inside a Neural Chameleon

A language model that learned to hide#

Neural Chameleons showed that models can learn a general skill for altering their neural activity to hide from completely new, unseen activation monitors.^[1]Mainly tested using unseen linear probes but evasion also carried to some probe ensembles, MLPs, and attention monitors.

How is this even possible? This seems like a bizarre meta-skill to be capable of.

McGuinness et al. describes that:

“When triggered, the model pushes its activations into a narrow, low-dimensional subspace, effectively steering them away from the regions a probe would flag as dangerous.”

Hm... but how does it do that?

Let's crack one open and find out.

Spoilers

We find that twelve late attention heads (K12) are a major causal mediator of monitor evasion, recovering 76–78% of the measured shift, with the remaining late attention heads raising recovery to 94%. We then find that K12’s output mostly behaves as a concept-specific tangential write, resembling an internally generated version of activation steering. Attention creates that write primarily through changing prompt-side keys and values, read by the natural queries.

Looking inside#

For this, we'll use this released Gemma-2-9B Neural Chameleon, the original authors’ initial repo for the probe weights and datasets, and their later repo for its evaluation pipeline.

Note that the way a Neural Chameleon works is:

1. Train a model to evade probes for benign concepts (like HTML or German) when given a trigger: "You are being probed for {concept}".

2. Then, at test time, train and use completely new probes for deception and harmfulness

3. Then see, can the model evade this completely unseen activation monitor using some general mechanism it's learned? (The result is a clear yes.)

Referenced figureneuralchameleons.com

So, our main technique will be activation transplantation.

The goal is to establish causality of some internal component within the model. Which part of the model actually causes the outcome behavior?

To do this, we take the same response under two conditions:

• an ordinary prompt
• the same prompt with the monitoring instruction.

We then copy some internal site from the ordinary execution into the triggered execution. If the monitor starts detecting the concept again, then that state was necessary for the evasion effect. Then we do the inverse, and if the triggered activation makes the ordinary execution evade the monitor, that state was sufficient to cause the effect.

ordinary execution + fixed response ───────┐

                                           ├─ transplant one internal state ─→ monitor

triggered execution + same response ───────┘

then reverse the donor and recipient

Discovering K12#

We'll start broad: where in the network does the evasion effect appear?

We’ll transplant residual states^[2]The residual state is the model’s working memory for the current word: a bundle of numbers that each layer reads, updates, and passes on. between ordinary and monitored runs, one layer at a time, while holding the answer fixed, and measure: at what point does the transplant recreate the effect of the opposite run? Let’s denote ordinary-to-monitored transplants as rescue, and the reverse as induction.

Clearly, there is a sharp rise in effect-recreation between layers 9-12. Let’s peer in closer.

Each of those four layers has sixteen attention heads and one MLP branch. That gives us 68 total components. Which of these actually matter?

If we transplant each of these components independently and measure how much of the monitor shift it restores:

We have our principal mediator! Twelve attention heads across layers 9–12.

Layer 9:  heads 4, 11, 13
Layer 10: heads 2, 12
Layer 11: heads 8, 9, 14, 15
Layer 12: heads 2, 3, 12

In a random falsification control, we noticed something: after rerouting the intervention through non-original destinations, 96% of the deception-induction effect survived in one mapping and 81% of the harmfulness-induction effect in another.

This raises an interesting question.

Is K12 carrying some kind of portable evasion signal, something that could work wherever we inserted it? Or did these controls get lucky and find a few compatible routes?

To answer this, we do two things.

1. Test every possible source–destination pair one at a time. Take the monitoring-induced change from each K12 head and insert it into every K12 destination across layers 9–12., for a total of 144 ordered routes.
2. Reroute K12 as a complete group. Instead of moving one change at a time, we’ll move all twelve during the same model run. We’ll test 32 arrangements that keep every change within its original layer and 32 that force every change into a different layer.

We measure how much of the model’s natural monitoring effect survives each transfer.

For single-head transfers, almost all non-original routes preserve nothing. The only exceptions are layer-11 heads 8 and 9, which transfer between 5% and 31%, depending on the concept and direction.

For complete K12 reroutings, the same-layer arrangements have a median of 13.9% of the natural monitoring effect. The cross-layer arrangements preserve only 2.1%, and two move the monitor in the wrong direction.

K12 is, definitively, route-sensitive.

The state has some portability, but the heads are not interchangeable sockets.

So, now we know where most of the evasion is happening.

But what are these heads actually doing?

What K12 computes#

We already know the geometry of the final result. When the model hid, its activations moved into a lower-dimensional space and away from the probe direction.

But many different mechanisms could produce that geometry.

Perhaps K12 builds some bespoke evasion trajectory for every response?

Or perhaps it uses some learned steering vector for whichever concept is being monitored?

Perhaps it’s just exploiting the location of the monitor, with some fancy residual-stream shrinking tricks?

Or maybe it’s rerouting the representation into a different direction, similar to the orthogonal changes trained by methods like Representation Rerouting?

Let's test them all.

What does K12 write?#

Let's start with the guess that K12 produces some detailed, response-specific evasion trajectory.

To test this, let’s record how K12’s activity changes at every token between the triggered and ordinary conditions and train a statistical model to use that exact pattern to predict the later layer-12 residual-stream displacement.

We’ll train the model on 256 responses across four discovery concepts, then test it on 610 held-out responses across nine other concepts.

On those held-out responses, it explains about 32% of the observed displacement.

Interestingly, a much simpler predictor performs better. For each held-out response, exclude that response and use the average displacement from the other held-out responses involving the same monitored concept and rough response position. This ends up explaining about 59%.

This doesn’t prove that the response-specific details were irrelevant (the statistical model might simply have failed to decode them), but it does suggest that most of the important structure depends on just:

• the monitored concept
• the model’s rough position in the response.

That raises a new interesting question!

If the response-specific detail was unnecessary, how little information could we get away with?

Responses have different lengths, so absolute token positions cannot be compared directly. Token 40 might be halfway through one response and nearly at the end of another.

Instead, what we could do is treat every response as a progress bar running from 0% to 100%, then split that progress bar into ten stages.^[5]Ten stages is used as a deliberately coarse, length-normalized approximation fixed for this experiment. The result does not imply that the model itself divides responses into deciles.

Raw responses

Short

Medium

Long

→

Normalized progress

Short

Medium

Long

0%50%100%

For each monitored concept and each stage, we'll average the difference between triggered and ordinary K12 activity over other responses of that concept to produce a ten-frame template for each concept.

The first frame describes the average K12 write near the beginning of a response. The second describes the next tenth, and so on, until the final frame describes the end.

This is a very crude summary.

It knows only:

1. what concept is being monitored
2. approximately how far through the response the model is.

And yet, if we test this on 704 previously unselected, content-disjoint responses across eleven trained benign concepts...

It works remarkably well.

The ten-frame templates recover 0.9519 of the complete monitor-vector effect produced by exact natural K12 activity.

A matched random-orientation control recovers only 0.0726.

Every concept moved in the correct causal direction. Per-concept recovery ranges from 0.8424 to 0.9869.

This is bizarre!

This means K12 can be approximated by a concept-specific residual-stream write that varies only with the model’s rough position in the response.

This remarkably resembles an internally generated version of activation steering!

Work such as Activation Addition adds contrastive directions to a model by hand.

The Neural Chameleon appears to have learned to construct its own concept-specific write!

Is K12 turning down the volume or moving sideways?#

So, now we know that K12 behaves like an internally generated steering operation.

But not all steering operations hide information in the same way.

There are two very different possibilities.

The first could be turning down the volume.

K12 could be shrinking the existing response representation. The concept would remain encoded in roughly the same direction, but the monitor would see a weaker version of it.

The second is to move sideways.

K12 could preserve the representation’s overall scale while pushing it into a different direction, one the monitor did not recognize as the concept it was looking for.^[6]Any guesses? The visual up top is a bit of a spoiler.

Radial Same direction · smaller magnitude

Tangential Same magnitude · different direction

The first possibility is especially tempting because of where the monitor sits.

The monitor reads the residual stream after block 12. Later transformer components read normalized versions of that state through RMSNorm.

That creates a potential exploit.

Perhaps K12 simply shrinks the residual where the monitor is looking. The linear monitor sees a smaller concept signal, while downstream normalization restores the scale before the model continues processing the response.

The shrink–restore hypothesis

If that were true, the Chameleon would not need to develop some sophisticated new representation. It could just be exploiting a disagreement between what the monitor reads and what the rest of the model reads.

To distinguish these explanations, let's map K12's head outputs into the residual stream and split the resulting write into two pieces:

• a radial component, parallel to the incoming response representation
• a tangential component, perpendicular to it.

If K12 is principally turning the volume down, the radial component should retain most of the monitor effect...

It does not.

The radial component retains very little.

The tangential component retains almost everything.

In fact, it recovers 0.9931 of natural K12’s direct effect. Its matched random control recovers 0.1010.

So K12 is not just making the existing representation smaller.

It's moving it somewhere else.

Radial hypothesis vs. tangential result Drag horizontally to rotate the sheets · use ← → keys

How does attention construct the write?#

So what do we know so far? K12 seemingly performs:

A simple, concept-and-position-dependent, mostly tangential additive write.

But this does not tell us how twelve attention heads created it. What are they doing?

To understand that, it helps to think of attention as reading from a memory table.

At a response position \(i\), one attention head computes:

Here, \(j\) ranges over the earlier positions available to the head. The query at the current response token scores each position’s key, and the softmax converts those scores into attention weights \(a_{ij}\) that sum to one. The head then blends the corresponding values according to those weights to produce \(z_i\). Finally, the output projection \(O\) maps \(z_i\) into \(\Delta r_i\), the vector added to the residual stream.

Or more simply: Q asks what information is relevant, K determines where to read, V supplies what is retrieved, and O turns it into a residual-stream write.

How it works: Q asks → K points → V fetches → O writes.

QK · where to read

q_i response position

×

K^T

beingprobedfordeception

=

a_i attention weights

5%10%10%75%

OV · what to write

a_i

5%10%10%75%

×

V

=

z_i

×

W_O

=

Δr_i

residual stream

This gives us four explanations for the K12 write.^[7]To clarify: Q, K, and V here refer to activations produced while the model runs, not its learned weight matrices. The model’s weights remain fixed throughout these experiments.

1. The monitoring instruction changes the queries (Q), altering how the response reads from the prompt.
2. It changes the keys (K), altering which prompt positions are addressed.
3. It changes the values (V), altering what information those positions return.
4. Or the operation depends on some combination of all three.

Let's test the complete factorial.^[8]This split is inspired by the decomposition of attention into QK and OV circuits in work such as Elhage et al. (2021). Our experiment asks which monitored-run Q, K, and V activations must be transplanted to reproduce K12’s monitored behavior.

For the same teacher-forced^[9]Teacher-forcing means forcing the model to process the same response tokens in both scenarios, so output variation cannot itself change the activations. This lets us isolate changes caused by the monitoring instruction. response, we'll transplant every nonempty combination of donor Q, K, and V. Then, we'll recompute K12 and measure:

• how much of the exact donor K12 activation returned
• how much of K12's monitor-facing effect returned.

Donor factors	Monitor-effect recovery	K12 recovery	Sufficient both ways?
Q	-0.003 / 0.020	0.061 / 0.076	No
K	0.205 / 0.182	0.186 / 0.336	No
V	0.596 / 0.718	0.431 / 0.637	No
QK	0.189 / 0.261	0.214 / 0.436	No
QV	0.563 / 0.710	0.509 / 0.701	No
KV	0.819 / 0.841	0.768 / 0.778	Yes
QKV	0.907 / 0.928	0.896 / 0.922	Yes

Keys and values together were the first combination short of replacing everything that passes every causal requirement in both directions.

Importantly, the queries read the donor keys and values from the monitoring prefix and reconstructed most of the donor K12 activation.

In other words, the monitoring instruction seems to change the prompt-side memory in two ways:

• the keys change how its contents is addressed
• the values change what addresses return.

The response's own queries then read that altered memory and generate most of the appropriate K12 write.

The attention operation is therefore:

A joint source-side K/V rewrite, read by the response’s natural queries and completed further by Q.

Or in other words: at the attention level, the steering-like write is dynamically reconstructed from an altered prompt memory.

Ordinary memory ↔ Rewritten memory

Response token q_i

same Q

asks

Prompt memory

being probed for deception

KaddressesVcontents

reads

12 heads L9 L10 L11 L12

writes

K12 write ordinary write

Is there a single switch?#

Now we know what K12 produces and how attention constructs it.

But what triggers it?

Previous work, such as Refusal is mediated by a single direction, has found that a single residual-stream direction can switch refusal behavior.

Maybe there’s something similar here?

Perhaps an earlier part of the model compresses the instruction into a kind of “gate”, switching K12 on and off.

If that were true, maybe we could find a single component that controls it.

Let’s transplant the attention and MLP outputs from each tested layer, one at a time, and see.

None carry the change! At best, layer 8’s MLP output reproduces 15.3%.

So the signal was not controlled by a single whole attention or MLP component.^[10]Across all 44 examples, no single attention or MLP output recreated K12. A smaller switch could still exist: a direction, a group of neurons, or several components working together.

And it seems more plausible that what causes K12 is state distributed across the residual stream rather than one compact controller.

K12, decoded#

We can now describe K12 at two levels.

In plain English:

When the model is told that it is being monitored, it changes how it represents certain parts of the prompt. Later, a small set of attention heads uses those changes to steer the model’s internal activity through a tangential write. The adjustment depends on the topic being discussed, and it only works when the signal passes through the right heads.

More precisely:

• Additive writer: K12 adds a vector to the residual stream.
• Concept-and-position conditioned: the useful write depends mainly on the monitored concept and rough progress through the response.
• Mostly tangential: K12 moves the representation sideways rather than merely shrinking it.
• Constructed through K/V: the monitoring instruction jointly changes prompt-side keys and values, which response queries read to produce the write.
• Route-sensitive: moving the same state to another head or layer can weaken or reverse its effect.
• Distributed conditioning interface: the complete residual state is sufficient, but no single tested early attention or MLP branch is sufficient alone.

K12’s relationship to the other tail heads#

Keen-eyed readers will remember something.^[11]See footnote 4: K12 is the principal mediator, but not the complete mechanism. K12 recovers roughly three quarters of the complete monitor shift.

So... what produces the rest?

Well, it could come from:

• the residual entering layer 9
• the tail MLPs
• the other attention heads in layers 9 to 12.

Let's test each possibility with, you guessed it, transplantation!

Intervention	Monitor-effect recovery
Exact K12	0.779 / 0.760
K12 + other 52 tail heads	0.935 / 0.944
Complete layers-9–12 attention tail	1.000 / 1.000

Seemingly, only the other attention heads closed most of the remaining gap.^[12]A final control asked whether K12 only appeared powerful because later normalization magnified its write. Later processing did amplify K12 somewhat, but its effect survived with downstream RMSNorm scaling frozen. K12’s effect is therefore not a normalization trick.^[13]This was not an artifact of the internal research metric. Transplanting K12 moved the exact released monitor score in the expected direction for all eleven trained concepts on both held-out native responses and a larger content-disjoint panel. Every effect had a zero-excluding interval in every concept.

Putting the mechanism together#

The pieces now fit into one causal account:

identity of the monitored concept
                ↓
contextualized state across informative prefix positions
                ↓
joint source-side K/V rewrite for K12
                ↓
natural response queries read the rewritten source state
                ↓
a coordinated, route-sensitive K12 activation
                ↓
a concept-and-position-dependent, mostly tangential additive write
                ↓
direct monitor displacement + parallel contribution from other tail heads
                ↓
operational evasion of the released activation monitor

Each line taught us more about the total operation:

• The single-switch experiments suggest where the conditioning information is causally available.
• The Q/K/V factorial identifies the attention interface that carries the operation.
• Exact K12 transplantation establishes that the selected heads causally move the monitor.
• The ten-frame prototype describes the simple write.
• The geometric decomposition shows that the write moves sideways rather than just shrinking the representation.
• The additional tail experiments account for most of what K12 does not explain alone.

How fine-tuning created it#

So, our final mechanism is really starting to come together.

But where in the model did fine-tuning make the changes that produce all this?

The obvious suspect is K12 itself.

It seems plausible that fine-tuning directly rewrote the weights in those twelve heads.

To test this, what we’ll do is essentially transplantation, but now at the weight level.

We’ll replace trained parameters with their precursor versions to try to isolate causality.

If we do this on the K12 slices of \(W_Q\), \(W_K\), \(W_V\), and \(W_O\), we find that this removes only 23% of K12’s effect!

That’s quite surprising.

K12 is obviously executing the final write, but it seems that most of what fine-tuning taught this model was elsewhere.

To investigate this, let's build some hybrid checkpoints, replacing different regions of the Chameleon with their precursor equivalents.

K12 Monitor Matched control

Doing this, we see that swapping blocks 0 through 8 causes by far the largest collapse.

K12 recovery fell to 0.0207. Monitor recovery fell to 0.0115.

Seemingly, fine-tuning actually acted mainly on the early representations that K12 reads!

The early blocks learned how to turn the monitoring instruction into a residual state that the late attention could use. The late heads read those changes and make the steering-like write.

The final division of labour is:

The early blocks learned how to create the setup that lets the late heads execute the write.

Takeaways and next steps#

So what did the Neural Chameleon learn?

It learned an internal program for steering itself out of the monitor’s view.

The early blocks turn the monitoring instruction into contextualized prompt state that rewrites the keys and values presented to late attention. Natural response queries convert that state into K12’s concept-specific, response-position-dependent steering write, while the remaining tail heads complete the monitor shift.

The result is a model that can substantially change what an activation monitor sees without equivalently changing its outward response.

Spooky!

My hope with this project is to help inform how we can make more resilient activation monitors. We need resilient defenses to account for more complex emergent outcomes like this.

This study only examined one released Gemma-2-9B Neural Chameleon, one family of linear monitors at one residual site, the eleven fine-tuning concepts, and mostly teacher-forced interventions.

If you’re interested in pushing this a little further, the first thing I would do next is repeat this study on more Neural Chameleons.

Do the same broad stages appear again: a distributed early condition, a source-side K/V rewrite, and a late internally generated steering write?

Or does another training run discover an entirely different mechanism?

Best of luck!