Demystifying Transformer LLMs: Teresa Torres Breaks Down Complex Concepts

In a recent LinkedIn post, Teresa Torres dives deep into the intricacies of how transformers, the architecture behind many large language models (LLMs), function. Her exploration began with a desire to understand research on “context rot” but quickly evolved into a broader quest to grasp fundamental concepts like context window size.

Torres shares her journey of understanding, highlighting the granular details she uncovered. She explains that her initial curiosity about context rot led her to question the underlying mechanisms. “I realized I didn’t really have a strong understanding of what context window size was a function of,” Torres writes. “Sure, compute. But what does that really mean?”

The Building Blocks of LLM Understanding

Torres’s post details a systematic breakdown of key LLM components. She outlines her learning process, which involved understanding why LLMs use tokens instead of whole words and the transition from tokens to embeddings, a process that even required a refresher on vector and matrix mathematics. She describes visualizing these concepts, comparing them to “sand in a really large multi-dimensional space.”

A significant portion of her learning focused on the concept of “attention,” particularly how it’s computed using query, key, and value vectors. Torres explains the significance of the attention layer in embedding context into each token, ensuring that even when processed individually, tokens carry their contextual meaning. “I learned about attention (like in how it’s computed with query, key, value vectors), how the attention layer embeds context into each token, so that when each token is sent through a neural net in isolation it brings context with it,” she states.

“I learned that modern LLMs repeat this process over and over again for every input token and again when generating tokens.”

Attention Heads, KV Caching, and Beyond

Further delving into the architecture, Torres discusses the role of attention heads, noting that having more is generally beneficial and distinct from simply lengthening the embedding vector. She also touches upon KV caching, explaining its specific application during the generation of new tokens rather than during the processing of input tokens.

This deep dive has enabled Torres to formulate her own theories about the phenomenon of context rot. “And now, I can generate my own theories about why context rot happens. And that’s fun,” she remarks.

Making Complex Concepts Accessible

Beyond the technical details, Torres expresses a personal enjoyment in understanding how things work and in finding simple ways to explain them. This passion has led her to consider writing an article aimed at demystifying these complex LLM terms for a broader audience.

“My goal would be to make it accessible to everyone at the conceptual level and then dive a little bit into the math at an addition/multiplication level, and then leave the more advanced math to folks who are more qualified.”

Torres is seeking feedback from her network on whether to proceed with this educational piece. Her proposed approach prioritizes conceptual understanding, supplemented by basic mathematical explanations, while deferring highly advanced mathematics to experts.

“So if you’ve heard all these terms being thrown around and you only kind of know what they mean, let me know if you’d be interested in an article that dives deep in what they mean and how they work together.”

This initiative by Torres underscores a growing need for clear, accessible explanations of advanced AI technologies, positioning her as a valuable voice in translating complex technical subjects for a wider business audience.