Making Transformers Make Sense
Exploring their inner workings, and looking for ways to improve them.
Hi, I’m Chris.
About a month ago, I stumbled into a new framing for attention in transformers—what I now call Patterns and Messages—and it unlocked a wave of insight for me.
It started with a series of “aha” moments. I realized the standard presentation of attention equations is shaped by implementation concerns, and that if we reframe them just slightly, we can arrive at a cleaner, more intuitive model—one that aligns more closely with how interpretability researchers already think about these systems.
That discovery sparked something in me.
I’ve spent the past few weeks digging deep, visualizing, writing, and running small-scale experiments. And I’ve come to a few conclusions:
-
I really enjoy this.
-
It’s been valuable to rediscover known insights in my own way—but I’d now like to learn from the field, read more papers, and contribute as part of a team.
-
I want to know what the top labs are working on. I want to help. I want to build something better.
My Focus
I’m most interested in interpretability research aimed at improving transformer architectures—a blend of mechanistic interpretability and efficiency research.
Right now, I’m exploring a few specific ideas:
-
Using a merged perspective on the attention matrices WQWK and WVWO to reveal new insights into attention.
-
Investigating architectural changes that improve the way Decoders encode input prompts. Encoding via token prediction is messy and inefficient—there has to be a better way!
-
Questioning whether SoftMax is the right answer. I’ve seen enough weird artifacts to believe there’s a better mechanism waiting to be found.
I’m also open to contributing in a communication or internal education role—helping technical teams share their ideas clearly across an organization, or to the broader world. (I’ve worn that hat before and loved it.)
What I'm Looking For
I’m looking for a challenge.
I want to be surrounded by people who push me to learn faster, think harder, and build things that matter. I'm ready to give everything I’ve got to a team and mission that inspire me.
If that resonates with you—or if you’re just curious about the research—I’ve shared a series of posts that walk through my recent work. It’s part self-education, part explanation, and part love letter to a field I’ve fallen for.