> For the complete documentation index, see [llms.txt](https://overleaf-pro.ayaka.space/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://overleaf-pro.ayaka.space/latex/in-depth-articles/03-a-six-part-article-series-on-expandafter-tex-tokens-and-expansion.md). # A six-part article series on \expandafter, TeX tokens and expansion ## Overview The TeX primitive `\expandafter` is a versatile command, used by LaTeX/TeX programmers to create sophisticated macros—such as the commands (macros) provided by LaTeX packages. Superficially, `\expandafter` is straightforward to use but that mask of simplicity soon slips when you encounter more complex constructions such as multiple consecutive `\expandafter` commands used to perform all sorts of “token juggling” `...\expandafter\expandafter\expandafter...`. This article series has an ambitious goal: to explain *how* the `\expandafter` command, and TeX expansion, actually work—from basic principles through to exploring processes which take place deep inside the TeX engine software. Readers can jump straight to the [articles](#accessing-articles-the-navigation-bar) or read an [introduction](#introduction-and-who-is-this-series-for) and the reason and rationale for the series. ## Accessing articles: the navigation bar Each article has the following navigation bar before and after the text so that you can quickly jump to another article in the series: [Part 1](/latex/in-depth-articles/19-how-does-expandafter-work-an-introduction-to-tex-tokens.md) [Part 2](/latex/in-depth-articles/22-how-does-expandafter-work-the-meaning-of-expansion.md) [Part 3](/latex/in-depth-articles/21-how-does-expandafter-work-tex-uses-temporary-token-lists.md) [Part 4](/latex/in-depth-articles/20-how-does-expandafter-work-from-basic-principles-to-exploring-tex-s-source-code.md) [Part 5](/latex/in-depth-articles/17-how-does-expandafter-work-a-detailed-macro-case-study.md) [Part 6](/latex/in-depth-articles/18-how-does-expandafter-work-a-detailed-study-of-consecutive-expandafter-commands.md) Here are links to the 6 articles in the series: 1. [An introduction to TeX tokens](/latex/in-depth-articles/19-how-does-expandafter-work-an-introduction-to-tex-tokens.md) 2. [The meaning of expansion](/latex/in-depth-articles/22-how-does-expandafter-work-the-meaning-of-expansion.md) 3. [TeX uses temporary token lists](/latex/in-depth-articles/21-how-does-expandafter-work-tex-uses-temporary-token-lists.md) 4. [From basic principles to exploring TeX's source code](/latex/in-depth-articles/20-how-does-expandafter-work-from-basic-principles-to-exploring-tex-s-source-code.md) 5. [A detailed macro case study](/latex/in-depth-articles/17-how-does-expandafter-work-a-detailed-macro-case-study.md) 6. [A detailed study of consecutive `\expandafter` commands](/latex/in-depth-articles/18-how-does-expandafter-work-a-detailed-study-of-consecutive-expandafter-commands.md) ## Introduction and who is this series for? ### \expandafter: basic principles of operation The purpose of \expandafter is to force premature expansion of a token in the input: given two consecutive tokens in the input, $$\mathrm{T\_1}$$ and $$\mathrm{T\_2}$$, `\expandafter` will save token $$\mathrm{T\_1}$$, expand token $$\mathrm{T\_2}$$ and place token $$\mathrm{T\_1}$$ back in front of the expansion of $$\mathrm{T\_2}$$: $$\verb\*\expandafter\*\ \mathrm{T\_1}\mathrm{T\_2} \rightarrow \mathrm{T\_1}\text{\}$$ Following the standard description of `\expandafter`’s behaviour we noted that it “saves token $$\mathrm{T\_1}$$”, forces the “expansion” of token $$\mathrm{T\_2}$$ (before TeX would normally do so) and then puts token $$\mathrm{T\_1}$$ “back into the input” for TeX to read again. The author of this article recalls being confused by some aspects of `\expandafter`’s “token juggling” behaviour; in particular: * What does it mean to “save a token” or “put a token back into the input”—where do these saved tokens “go” and how does TeX re-read them? * What does expansion “look like” or really mean? * How does TeX process multiple consecutive `\expandafter` commands—how does that *really* work? ### Objectives of the article series This series seeks to address the questions posed above through documenting the results of research into the inner workings of TeX. Overleaf built a customized version of one of the TeX engines and used the [Eclipse IDE](https://www.eclipse.org/ide/) to observe the TeX source code in operation as it processed a `.tex` file containing example uses of `\expandafter`. We'll share the results of that research to examine: * the true nature/meaning of TeX tokens; * TeX’s generation/use of temporary token lists; * what “expansion” really means. Through an exploration of these topics we’ll piece together the big picture to explain *how* `\expandafter` performs its “token juggling” behaviour. This article series is written for anyone who is interested to develop an insight these topics: to take a deeper look inside TeX itself, examining those parts of TeX’s internal operations which enable `\expandafter` to work its magic. Writing articles about TeX and TeX-related concepts/principles is challenging because TeX is such an extraordinarily complex piece of software—its typeset source code runs to many hundreds of pages. For example, see this [Overleaf project](https://www.overleaf.com/latex/examples/typeset-the-source-code-documentation-for-tex-e-tex-or-pdftex/qkgfgyspnhcv) which, by default, typesets the source code of $$\mathrm{\varepsilon}\text{-}\mathrm{\TeX}$$: the typeset PDF file contains over 600 pages. Internally, TeX is an intricate web of interrelated actions and processes which present challenges when attempting to discuss isolated aspects of its behaviour: dissecting out a specific “component”, such as `\expandafter`, and trying to explain it without constantly needing to reference other aspects of the TeX software. By its very nature, the `\expandafter` command engages with TeX’s process of *expansion*—a process which, for many of us, is a vague and somewhat difficult concept to fully understand. However, expansion is a truly fundamental aspect of TeX's behaviour, interwoven throughout the inner workings of TeX and taking place deep inside the TeX software itself. In this aricle series we will explore the real meaning of expansion, starting out by assembling a collection of related principles and behaviours sufficient to construct a basic understanding of expansion. Building on those foundations we conclude by taking a “deep dive” to look inside of TeX itself, exploring the actual source code which implements TeX’s expansion mechanism. ## In conclusion In these articles we have tried to provide content that is of use/value or interest to the maximum number of readers: attempting to balance brevity with unavoidable detail whilst simultaneously trying not to risk the reader’s patience by providing too much, or too little, contextual material. It is our hope to have contributed something of use/value to the TeX literature, which present and, perhaps, future members of the TeX community will enjoy reading. Happy $$\mathrm{\TeX}\text{ing!}$$, from Graham Douglas and the Overleaf team. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://overleaf-pro.ayaka.space/latex/in-depth-articles/03-a-six-part-article-series-on-expandafter-tex-tokens-and-expansion.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.