The big picture
Think of the model as a relay of ~24 workers (each worker = one layer) passing a piece of paper around a room. When you type a prompt, the paper starts blank at the input; each worker in turn writes something on it, then hands it to the next. Whatever's on the paper after the last worker is the answer for what comes next.
This tool lets you peek at the paper between every pair of workers, and see what each worker did as they wrote.
The four panels
1. Token strip at the top
The conversation. Gray tokens are what you typed; green tokens are what the model generated. Each generated token is one full pass through the relay. Click any green token to switch the other panels to that specific moment.
2. Left panel: Logit Lens
Every row is one worker's paper: the last row (bottom of the display, layer 0) is right after the first worker, the top row (highest layer number) is right after the last worker.
Each cell is a guess. As if you asked the paper at that point in the relay: "if we had to stop here and read off an answer, what would you say?" Rank 1 is the best guess, rank 2 the second best.
Early rows are gibberish (early workers haven't done real thinking yet). Middle rows often show related-but-wrong words. The last row is usually the actual answer.
Click a cell to "watch" that word. A small graph appears below showing how that word's score rises and falls through all layers - that's how you see when in the relay a specific idea emerged.
3. Right panel: Attention
Workers don't just write from scratch - they look at earlier words and copy inspiration. The attention panel shows which earlier tokens a specific worker was staring at while writing.
A long bar next to France means "70% of this worker's focus was on the word France." Zero bar means the worker ignored that position.
The mini-grid below shows all the "heads" at once (heads are like committee members within one worker - each looking at different things simultaneously). One head might read grammar; another might track who the sentence is about.
4. Right panel bottom: Block Outputs
This is the subtle-but-powerful one. Each worker's contribution isn't just what they looked at - it's what they wanted to add.
The block output panel asks each worker: "if your addition was the whole answer, what word would you be voting for?"
- ATTN pure attention workers.
- MAMBA or DELTA recurrent workers - different internal machinery, same job.
Often a middle worker votes for largest and the next votes for city, and by the time you add them all up, the paper reads Paris. This panel is your window into individual votes; the lens on the left is the running tally.
The Ablation button
"Test each head at this layer" is the only causal test in this tool. Everything else shows correlations; ablation shows what happens when you break something.
The tool runs the prompt through the model once per head. Each run zeros out one head (not all of them - just that one) and captures the resulting logits. So for a layer with 8 heads you get 8 separate ablation runs. The bars show, per head, how much that head's absence changed the answer's confidence:
- Green bar (left) - "when this head skipped, the answer got worse" → the head was helping.
- Red bar (right) - "when this head skipped, the answer got better" → the head was actively pushing against the right answer.
Colored layer kinds
Hybrid models (Nemotron, Qwen 3.5) mix worker types:
- ATTN standard transformer attention. Sees all previous words at once.
- MAMBA / DELTA recurrent workers. Keep a running "notepad" instead of looking back at every previous word. Faster but different flavor of thinking.
- FFN no attention at all - just does math on the paper without looking around.
Same paper, same relay, different kinds of workers along the way.
How I'd read a lens for the first time
- Type a factual prompt like
"The capital of France is".
- Look at the top row (final layer) of the lens - top-1 should be the model's actual answer.
- Scroll down to earlier rows. Find the layer where the correct answer first appears anywhere in the top-K. That's the "aha" layer.
- Click that cell to watch its trajectory.
- Switch to that same layer in the attention panel. Look at each head and find ones that put weight on the content word (e.g.
France) rather than on <s> or the previous token. Those are the "semantic" heads.
- Hit ablate. The green bars are the heads whose absence hurts the answer most - the ones actually doing the reasoning.
Tips
- Small models loop. If output repeats forever, raise
rep_pen above 1.0 (try 1.1). Still deterministic.
- Nemotron and Qwen3.5 are hybrid archs. Some layers won't have attention data (they're recurrent). That's expected - a banner will tell you.
- For long generations, drop
--lens-layers to only a few layers, or the CPU projection dominates.