> For the complete documentation index, see [llms.txt](https://overleaf-pro.ayaka.space/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://overleaf-pro.ayaka.space/latex/in-depth-articles/09-an-introduction-to-luatex-part-2-understanding-directlua.md).
# An Introduction to LuaTeX (Part 2): Understanding \directlua
## The goal of this article
In the first part of this article, [An Introduction to LuaTeX (Part 1): What is it—and what makes it so different?](/latex/in-depth-articles/07-an-introduction-to-luatex-part-1-what-is-it-and-what-makes-it-so-different.md), we briefly reviewed LuaTeX as an extremely versatile TeX engine: a sophisticated, programmable, typesetting system which provides a wide range of tools for constructing document engineering and production solutions.
In this concluding installment, we take a close look at the most vital component of the LuaTeX toolbox: the `\directlua` command which provides the “gateway” to programmatic control of LuaTeX’s typesetting through the Lua scripting language.
However, fully exploiting LuaTeX via `\directlua` requires some background knowledge of several TeX topics: TeX’s tokens, token lists and expansion mechanism. The goal of this article is to explore and explain these fundamental TeX concepts: piecing together the TeX-related processes behind `\directlua` to develop an understanding of how it works and provide the foundations upon which to build your own typesetting solutions using LuaTeX.
This article includes numerous short examples to demonstrate and explain key aspects of `\directlua`’s behaviour, deliberately avoiding overly-complex code in favour of short code fragments. Where necessary, examples use basic (raw/plain) TeX—although most people use and prefer LaTeX (macros), basic TeX commands have the advantage of simplicity.
## Introduction to the Lua in LuaTeX
[Lua](https://www.lua.org/about.html) is a scripting language whose [source code](https://www.lua.org/download.html) is highly portable and easy to embed into software applications, allowing developers to incorporate scripting capabilities into their programs. Lua has been embedded into [many applications](https://en.wikipedia.org/wiki/List_of_applications_using_Lua) and is a popular choice within the software games industry—perhaps the most famous example is [World of Warcraft](https://wowwiki.fandom.com/wiki/Lua_functions).
LuaTeX, as its name suggests, is a TeX engine which embeds the Lua scripting language, providing users with the ability to control LuaTeX’s typesetting behaviour by including Lua programs (scripts) into their documents. In addition to direct control of LuaTeX, users can leverage Lua purely as a very capable programming language to perform tasks that might be extremely difficult to achieve using the TeX language—which is, by any fair measure, a challenge to learn and master. Through the addition and integration of Lua, LuaTeX becomes a very versatile and powerful TeX engine which directly supports two programming languages.
### Using Lua and TeX in your document: enter \directlua
Lua and TeX are two *very different* programming languages: Lua is much closer to what most people think of as a programming language but TeX, with its category codes, tokens, macros and expansion mechanism is far removed from most peoples’ experiences/expectations of a language in which to write programs. However, as history has shown, the TeX language has endured because it is good at what it was designed for: controlling typesetting, even if its mode of operation is somewhat arcane.
To address the challenge of mixing the Lua and TeX languages in a single TeX document, LuaTeX’s developers introduced a new command called `\directlua` which is the route to using Lua—both as a standalone programming language in its own right and for controlling LuaTeX’s typesetting behaviour.
The `\directlua` command allows users to embed Lua code in their TeX documents; that code is subsequently passed on to LuaTeX’s built-in Lua language interpreter. However, `\directlua` also allows you to *combine* Lua and (La)TeX code together, within the same `\directlua` command—although that introduces additional complexities due to fundamental differences in Lua and TeX-based programming languages. The key challenge when using a combination of (La)TeX and Lua code is to ensure those two languages co-exist peacefully and don’t get “in each other’s way”.
`\directlua` is best suited for use with shorter in-document Lua code fragments but you can use it with more extensive Lua programs, should you wish to. Generally, more substantial Lua programs, and Lua code libraries, are saved to external files which can be loaded by using Lua’s `dofile()` function within a `\directlua` command. From the TeX-processing standpoint, a significant advantage of using external Lua code files is avoidance of complications that arise from TeX’s category code mechanism—a topic fully explored in this article.
### More formal description of \directlua
The [LuaTeX Reference Manual](http://www.pragma-ade.com/general/manuals/luatex.pdf) describes `\directlua` as follows (slightly modified):
> In order to merge Lua code with TeX input, a few new primitives are needed. The primitive `\directlua` is used to execute Lua code immediately. The basic syntax is `\directlua{⟨code⟩}`. The `⟨code⟩` is expanded fully, and then fed into the Lua interpreter. After reading and expansion has been applied to the `⟨code⟩`, the resulting token list is converted to a string as if it was displayed using `\the\toks`.
Of course this is technically accurate but, perhaps, not so easy to understand without some knowledge of lower-level TeX processes—such as tokens and expansion.
## Understanding \directlua: Which topics will we cover?
In this article we’ll take a closer look at some key background topics and offer a number of examples designed to demonstrate how `\directlua` works and where (or why) you need to be careful when combining TeX and Lua in your `⟨code⟩`.
We’ll explore the following topics in sufficient detail to provide a foundation for understanding `\directlua` and its “pre-processing” of the code you use within it:
* category codes and TeX tokens: converting text to tokens and tokens to text;
* TeX’s expansion process (and preventing expansion);
* Lua escape sequences/mechanisms for characters and strings;
* using Lua-style comments;
* a short introduction to LuaTeX’s Lua API.
If you understand how TeX engines create and use tokens and develop an awareness of TeX’s expansion mechanism then you’ll have the foundations necessary to unlock the incredible versatility of LuaTeX’s `\directlua` command.
## The foundations: from text to tokens and tokens to text
Overleaf has published several articles which take an in-depth look at TeX tokens and related concepts so we won’t repeat all that material here; instead, we’ll outline those areas/topics relevant to developing a better understanding of `\directlua`.
Here is a list of previously published articles which may be of interest:
* [What is a TeX token?](/latex/in-depth-articles/53-what-is-a-tex-token.md)
* [What is a TeX token list?](/latex/in-depth-articles/54-what-is-a-tex-token-list.md)
* [How does \expandafter work: An introduction to TeX tokens](/latex/in-depth-articles/19-how-does-expandafter-work-an-introduction-to-tex-tokens.md)
* [A six-part series: How do TeX macros actually work?](/latex/more-topics/01-a-six-part-series-how-do-tex-macros-actually-work.md)
### Understanding character tokens
Any character a TeX engine can read from a text file is represented by two numeric values:
* its *character code* (ASCII value or, today, its Unicode code point);
* a second, TeX-centric, value called its *category code*.
Readers who would like to know more about category codes may be interested to read this introduction published by Overleaf: [So where do we start? With category codes](/latex/more-topics/19-how-tex-macros-actually-work-part-1.md#so2c-where-do-we-start3f-with-category-codes).
For example, if a TeX engine reads-in a character `A` it would have access to two pieces of information: `A`’s character code (65), and its category code (11, usually). Once TeX has input that character `A`, its category code won’t be changed but user macros can make category code changes that might affect any *subsequent* character `A` which *has not yet been read* by TeX. Consequently, TeX needs to record that *this* character `A`, *just read in*, has category code 11. To do that TeX uses the integer pair (65,11) to calculate another integer value that it calls a *character token*. By calculating that token value, which is passed on to TeX’s inner processing, that particular `A` and its category code are *bound together*; in effect, that character token *encapsulates* the data TeX needs to know about that character for use in any subsequent typesetting activities deeper inside the TeX engine.
#### How are character tokens calculated?
Firstly, we need to remember that TeX engines use category code 13 for the purpose of creating so-called *active characters*: any character with category code 13 behaves like a mini-macro; consequently, and as we’ll see below, tokens for active characters are calculated differently to regular characters with other category codes such as 10, 11 or 12.
For *non-active* characters:
* older 8-bit engines (Knuth’s TeX, e-TeX, pdfTeX) calculate character tokens for *non-active* characters using
$$\text{(non-active) character token} = (256 \times \text{category code}) + (\text{ASCII character code})$$
* for LuaTeX, which has to deal with Unicode character values, the calculation for *non-active* characters is similar but produces much larger integer values:
$$\text{(non-active) character token} = (2^{21} \times \text{category code}) + (\text{Unicode value})$$
Going back to our earlier example for the letter A with category code 11, LuaTeX would calculate a character token value of $$2^{21} \times 11 + 65 = 23068737$$. Once calculated, that character token value *binds* that particular character A to a category code value of 11. User macros may change the category code for any subsequent character A, but this one’s category code has been fixed by converting it to a token for use as it passes through LuaTeX’s inner workings. LuaTeX has preserved, or encapsulated, the intended meaning of that character as determined at the time it was read in.
TeX engines use a total of [16 different category codes](/latex/more-topics/43-table-of-tex-category-codes.md) and *any* of those category codes can be assigned, via the `\catcode` command, to *any* character the TeX engine is capable of reading. Changes to category codes are used to alter the way TeX engines process particular characters in the input, allowing TeX users to write macros that produce special typesetting results or behaviour.
**Active characters**
As noted, TeX engines use category code 13 to attach a “special meaning” to a character, making it a so-called *active character* which behaves like a mini-macro: no leading `\` is required, the isolated character, due to its category code, is enough to trigger its macro-like behaviour.
Because an active character acts as a mini-macro, it is not converted to a *character token* but to a second (integer) token type called a *command token*. These are calculated as follows:
* for older 8-bit engines (Knuth’s TeX, e-TeX, pdfTeX) tokens for active characters are calculated via:
1. calculate an intermediate value called $$\text{curcs}$$ (**cur**rent **c**ontrol **s**equence) where $$\text{curcs} = \text{character code} + 1$$3. calculate the token value where $$\text{active character token} = \text{curcs} + \text{4095}$$
* for LuaTeX the calculation is a little more complex because it has to deal with the full range of Unicode characters, any one of which could be made active:
1. calculate the intermediate integer value $$\text{curcs}$$ by applying a so-called *hash function* to the active character’s Unicode code point value expressed in UTF-8: $$\text{curcs}=\texttt{hashfunction}\text{(UTF-8 text for Unicode value of active character)}$$3. calculate the integer token value: $$\text{active character token} = \text{curcs} + 2^{29} - 1$$
**Examples**
* 8-bit engines: the token calculation for the active character `~` (character code 126) results in $$\text{curcs} = 126 + 1 = 127$$, giving a token value of $$4095 + 127 = 4222$$.
* LuaTeX: the token calculation for the active character `~` results in $$\text{curcs}=3186$$ giving a token value of $$3186 + 2^{29} - 1 = 536874097$$. LuaTeX tokens use much larger integer values!
### Understanding command tokens
In addition to processing *individual* characters, TeX engines can, of course, process *sequences* of characters called *commands* (or, more correctly, *control sequences*). By tradition, the `\` character is used to signal the start of a command but that’s merely a convention—in fact, any character with category code 0 (the escape character) could be used instead.
TeX engines recognize two types of command which are known as *control words* and *control symbols*:
* **control words**: commands constructed from one or more characters that have category code 11;
* **control symbols**: single-character commands where that character’s category code *is not* 11: such as `\$`, `\#` or `\\`.
**Note**: The TeX primitives `\chardef`, `\mathchardef`, `\countdef`, `\dimendef`, `\skipdef`, `\muskipdef` and `\toksdef` are also used to define control sequences but, unlike regular macro definitions, the resulting control sequences (control words or control symbols) *are not expandable*—we’ll explore these in more detail below.
#### How are command tokens calculated?
Just like active characters, TeX engines use the second type of integer token value to represent commands: *command tokens*—recall that active characters also generate command tokens because they behave as mini-macros.
The calculations used by 8-bit engines to create command-token integers can be found in this [Overleaf article](/latex/in-depth-articles/19-how-does-expandafter-work-an-introduction-to-tex-tokens.md#how-tex-calculates-token-values). Here, we will summarize the key steps in command token calculations for LuaTeX—which are slightly different because LuaTeX has to process Unicode character code values which can be considerably larger than 8-bit values; however, LuaTeX’s calculations follow the same general principles used by older 8-bit engines.
After detecting an incoming command, TeX engines, including LuaTeX, ignore the leading `\` character: it is not used in calculations of command token values but merely acts as a “switch” to inform a TeX engine it needs to process a command. The command token value is calculated using the sequence of (1 or more) characters present in the name of the command—LuaTeX calculates command tokens for control symbols and control words using the same algorithm:
1. calculate the intermediate integer value $$\text{curcs}$$ by applying a so-called [hash function](https://en.wikipedia.org/wiki/Hash_function) to the Unicode UTF-8 string of characters contained in the command name: $$\text{curcs}=\texttt{hashfunction}\text{(Unicode UTF-8 string of characters in command name)}$$3. calculate the command token value where $$\text{command token} = \text{curcs} + 2^{29} - 1$$
**Examples**
* for the `\\` command (a control symbol), LuaTeX calculates $$\text{curcs}=94$$, resulting in a token value for `\\` of $$94 + 2^{29} - 1 = 536871005$$.
* for the `\vskip` primitive command (a control word) LuaTeX calculates $$\text{curcs}=3560$$, resulting in a token value for `\vskip` of $$3560 + 2^{29} -1 = 536874471$$.
* for the user-defined macro `\mynewmacro` (a control word) LuaTeX calculates $$\text{curcs} = 2971$$, resulting in a token value for `\mynewmacro` of $$2971 + 2^{29} -1 = 536873882$$.
Once created, tokens may be stored for later use via so-called *token lists* or they can be immediately passed on for further processing inside the TeX engine. Using integer values to represent tokens not only works across all types of computing platform/operating system but it is also a very efficient way for TeX to store/process data.
### How a TeX engine identifies the type of token (command or character)
Given a particular integer token value, $$T$$, a TeX engine can easily determine if $$T$$ represents a command or a character by testing if $$T$$ exceeds a certain $$\text{threshold value}$$—that $$\text{threshold value}$$ depends on the TeX engine. If $$T \geq \text{threshold value}$$ then $$T$$ is a command token otherwise $$T$$ is a character token. The $$\text{threshold value}$$ is $$4095$$ for 8-bit engines and $$2^{29}-1$$ (536,870,911) for LuaTeX. Knuth designed the methods used in token-calculation formulae so that his TeX engine, and all subsequent engines based on his code/architecture, can quickly and easily test token values.
## Tokens can be broken apart (and converted back to text)
Tokens (integers) are the mechanism through which a TeX engine “encapsulates” everything it needs to record about an item of input (character or command). However, there are times when a TeX engine needs to reverse the tokenization process—to find out what was originally read-in to produce that token value—an individual character or a sequence of one or more characters forming the name of a command:
* **for character tokens**: Any character token can be split into its two constituent parts: the character code and corresponding category code assigned to that character *at the point it was originally read-in*. Like all TeX engines, LuaTeX will not change that original category code allocation but will make use of it during further internal processing activities.
* **for command tokens:** These are slightly more detailed but if you look at LuaTeX’s calculation of command tokens, including tokens for active characters, you see they follow a pattern: $$\text{command token} = \text{curcs} + 2^{29} -1$$
where $$\text{curcs}$$ is calculated according to the type of command token being generated: active character, control symbol or control word. The $$\text{curcs}$$ variable is an *extremely* important component of a TeX engine’s inner operations: given any command token (integer) value LuaTeX can very easily extract the value of $$\text{curcs}$$ from that command token using $$\text{curcs} = \text{token value} - (2^{29} -1)$$.
### Why is $$\text{curcs}$$ so important?
The internal TeX variable $$\text{curcs}$$ (**cur**rent **c**ontrol **s**equence) is a vitally important component of a TeX engine’s inner “under the hood” operations. Although you won’t, and cannot, use or access it directly in your code, $$\text{curcs}$$ plays a crucial role because TeX engines use the current value of $$\text{curcs}$$ as an index into internal tables which store data about every command currently known to the engine. Those tables store information about the current meaning of a command: what does it do, or represent and, in addition, they record the sequence of characters originally used to calculate that $$\text{curcs}$$ value. By extracting the value of $$\text{curcs}$$ from a command token, a TeX engine is able to determine the name, i.e., human-readable text, corresponding to any (command) token, enabling it to perform the token-to-text conversions which are a key aspect of `\directlua`’s operation.
### Converting integer tokens back to characters or character sequences (command names)
We’ve seen that TeX engines convert input characters, or character sequences, to integer token values but there are occasions when a TeX engine needs to *reverse* that process—to output the human-readable text originally used to create those integer token values; for instance:
* writing error or warning messages to the screen or `.log` file;
* outputting TeX/LaTeX code to a text file via the `\write` command;
* when converting a sequence of tokens to text within `\directlua` (as we’ll soon see!)
#### Converting character tokens to text
As noted, tokens for non-active characters are calculated using an input character’s category code and character code (Unicode value). LuaTeX uses the formula:
$$\text{character token} = (2^{21} \times \text{category code}) + (\text{Unicode value})$$
It is a straightforward programming task to split the integer $$\text{character token}$$ value to obtain its constituent character code ($$\text{Unicode value}$$) and $$\text{category code}$$.
#### Converting command tokens to text
All TeX engines store the name (sequence of characters) of every command they “know about”: whether that command is a user-defined macro or a built-in primitive—the storage of primitive-command names takes place when the TeX engine starts-up, long before it begins to process your code. For user-defined commands (macros), the name of that macro (minus the leading `\`) is stored away as part of the macro-definition processes inside TeX engines.
When a TeX engine needs to access or output the human-readable text from which an integer command token was originally calculated, it will first determine the $$\text{curcs}$$ value for that token; in LuaTeX, $$\text{curcs} = \text{token} - (2^{29} -1$$). Using the value of $$\text{curcs}$$ extracted from a command token, a TeX engine can access an internal data structure called the *string pool* to determine the sequence of human-readable characters originally used to calculate that particular value for $$\text{curcs}$$ and, consequently, the corresponding command token.
As we’ll see, these token-processing activities—converting character sequences to integer token values and converting integer token values back to character sequences (“de-tokenization”)—are the *fundamental mechanisms* used inside `\directlua`.
## Token lists
As a TeX engine is reading input, generating character and command tokens (and processing them), it might encounter certain commands which instruct the engine to (temporarily) stop passing tokens onward for further processing but, instead, store them away for later use. The most common example is defining a macro using one of the macro-definition commands `\def`, `\edef`, `\gdef` or `\xdef`—LaTeX commands such as `\newcommand` are macros that provide additional functionality constructed around low-level primitives which, eventually, perform the actual macro-definition process. A macro can be considered as the name given to a particular list of stored tokens: a token list.
TeX engines make *extensive* use of token lists, especially [temporary internal-only lists](/latex/in-depth-articles/21-how-does-expandafter-work-tex-uses-temporary-token-lists.md) used for internal processing purposes. Every TeX engine also provides user-level commands to create token lists that are stored away for when the user, or the TeX engine itself, requires them. The number of token-list-creation commands (built-in primitives) varies according to the TeX engine but they all share a core minimum set supported by every engine, such as the `\toks` primitive.
In practice, a token list is just a stored sequence of integer values:
* the input is read to generate (calculate) individual tokens, representing a character or command;
* each token is then stored away, preserving the sequence in which tokens were generated from the input.
TeX engines store token lists using a data structure called a [linked list](https://en.wikipedia.org/wiki/Linked_list) (the singly-linked variety). Readers wishing to know more on token lists are invited to read the Overleaf article [What is a TeX token list?](/latex/in-depth-articles/54-what-is-a-tex-token-list.md) which uses an analogy to construct the concepts/ideas behind a token list. An in-depth exploration of TeX’s token lists, and how they are used in macro processing, can be found in the Overleaf article series [How do TeX macros actually work?](/latex/more-topics/01-a-six-part-series-how-do-tex-macros-actually-work.md)
#### A token list in graphic form
The following graphic shows a LuaTeX-generated token list with corresponding token values produced from the following input
`Hi, \TeX! \hskip 5bp`
For example, if we define `\mymacro` as `\def\mymacro{Hi, \TeX! \hskip 5bp}` the definition of `\mymacro` would be stored in memory using a token list such as this:
The token list is a sequence of linked items called *nodes*, the name given to a small package of LuaTeX’s memory allocated to hold each item in the list (like individual links in a chain). Each node contains an integer token value and memory address of the *next* node in the chain, forming a data structure called a [linked list](https://en.wikipedia.org/wiki/Linked_list). The last node indicates the end of the list using a special “null value” for the next node—because there isn’t one.
**Notes:**
* For convenience, we have included the address of each individual node but, in practice, that data isn’t stored within token-list nodes; only the address of the *next node* is required to build TeX engine token lists.
* The second column in the graphic titled “What each token means” shows a series of grey boxes containing information on the token contained in each node: these are purely for information and *do not* form part of the actual data stored in the token list.
Here is a table of the token values contained in the token list depicted above:
| | | | |
| -------------- | ------------------- | ---------------------------------------------------------------------------- | --------------- |
| **Input item** | **Type of input** | Category code
(if character)
| **Token value** |
| H | character | 11 | 23068744 |
| i | character | 11 | 23068777 |
| , | character | 12 | 25165868 |
| | character | 10 | 20971552 |
| \TeX | command (macro) | | 536871539 |
| ! | character | 12 | 25165857 |
| | character | 10 | 20971552 |
| \hskip | command (primitive) | | 536874247 |
| 5 | character | 12 | 25165877 |
| b | character | 11 | 23068770 |
| p | character | 11 | 23068784 |
**Note:** Our original input text has a after the `\hskip` command but there is no token representing that character in the token list. That character was absorbed by LuaTeX’s input-scanning (reading) process because it was used to terminate LuaTeX’s search for characters comprising the `\hskip` command.
## How \directlua really works
Now that we have explored tokens, token lists and converting tokens-to-text, the next challenge is to understand the TeX-engine concept of token *expansion*.
As noted, `\directlua{⟨code⟩}` can be asked to process `⟨code⟩` which contains both Lua and TeX/LaTeX code but LuaTeX’s in-built Lua language interpreter doesn’t understand TeX or LaTeX: so how can this work? How is it possible for the `⟨code⟩` to contain TeX/LaTeX instructions without thoroughly confusing the Lua interpreter with commands it does not understand? For example, the following `\directlua` command only uses TeX macros, but it works:
```
\def\aa{tex}
\def\bb{.}
\def\cc{print}
\def\dd{("Hello")}
\directlua{
\aa\bb\cc\dd
}
```
This `\directlua` command results in LuaTeX typesetting `Hello` but why and how does this work because the Lua language does not understand TeX macros?
The answer is contained within the earlier description we borrowed from the [LuaTeX Reference Manual](http://www.pragma-ade.com/general/manuals/luatex.pdf) but we can consider that `\directlua{⟨code⟩}` works by LuaTeX initially “pre-processing” the `⟨code⟩` before anything is passed to the Lua interpreter. The nature of this “pre-processing”—i.e., what it really means and its consequences for your `⟨code⟩`—is the next topic we’ll address to help readers interested to take advantage of the power and flexibility of `\directlua`.
### How LuaTeX processes \directlua: A first look
To build-out our understanding of `\directlua`’s “pre-processing” activities we can start with the following simplified diagram which gives an overview of what happens. The `⟨code⟩` provided to `\directlua{⟨code⟩}` is first converted to tokens using the processes and calculations discussed above; that sequence of tokens is stored in a token list. Once that token list has been built, each token in that list is converted back to its textual representation: the text produced by every token—character token or command token—is combined (concatenated) to create a single string of code passed on to the Lua interpreter for execution.
But wait, what would be the point of going from text to tokens and converting those tokens straight back into text? You might not be surprised to learn that, yes, there is an additional and vital process that we haven’t included in this graphic: *token expansion*. Every token produced from the text in your `⟨code⟩` is subjected to a type of “inspection” in which LuaTeX applies a test to see if that token represents a command which belongs to the subset known as *expandable commands*. If it does, LuaTeX filters-out that command by *removing* it from your `⟨code⟩` and *replacing it* with the result(s) of a process TeX engines call *token expansion*.
### How LuaTeX processes \directlua: A second look (at expansion)
TeX’s expansion mechanism is a core component of all TeX-based typesetting engines because, ultimately, each one is derived from (or based on) Knuth’s original source code and design of TeX. However, the concept of expansion is difficult to explain in concise, yet accessible, language because, in practice, expansion is an “umbrella” term used to describe a single process—but one that produces a range of outputs. Those varied outcomes are a consequence of the somewhat eclectic set of commands to which expansion can be applied, so you can consider that each expandable command has a certain “expansion behaviour”.
As a *first approximation* to understanding expansion we can say that expansion of a token (command) means *removing* that command (token) from TeX’s current input and *replacing* it by a sequence of tokens which result from executing that particular expandable command—replacing the original token with the results/consequences of its expansion *behaviour*. However, this initial “definition” of expansion—in terms of generating new tokens for TeX to read—isn’t totally accurate for all expandable commands, but it is good enough as a starting point.
To give a straightforward example: the TeX primitive `\jobname` is an expandable command and its *expansion* is a sequence of character tokens which represent the name of the main TeX input file. If TeX decides to expand a `\jobname` command (token) it is *removed* from TeX’s current input source and *replaced* by the sequence of character tokens it generates—which TeX then proceeds to read/process.
Within `\directlua`, after an expandable token is processed (removed) and replaced by new tokens, LuaTeX will go on to read those new tokens it has just put in place—but some of those new tokens could also be expandable. Because `\directlua` performs so-called *full expansion*, LuaTeX will read those new tokens and, once again, go through the expansion process to expand (remove) any new (expandable) tokens—this expansion process continues until no expandable tokens are left. However, there are two important exceptions to this “keep on expanding” rule, both of which we’ll discuss below:
* using the construct `\the\toks`;
* deliberate prevention (suppression) of expansion for one or more selected tokens.
As noted, our working definition (first approximation) to understanding expansion doesn’t cover the full range of expansion behaviours demonstrated by the subset of expandable commands. For example, some expandable commands don’t generate tokens in the way that `\jobname` does, but they might:
* “filter” tokens from the input: a TeX engine’s conditional commands (`\if`, `\ifcat`, `\ifnum`, `\ifdim`, `\ifodd`, `\ifvmode`, …) are expandable. Their expansion behaviour is a type of “token filtering”—conditionals can be used in `\directlua`.
* “juggle” tokens in the input: the [`\expandafter` command](/latex/in-depth-articles/03-a-six-part-article-series-on-expandafter-tex-tokens-and-expansion.md) is expandable and changes the sequence in which two tokens are expanded.
* prevent expansion: the expandable commands `\noexpand` and `\unexpanded` suppress expansion of command tokens in the input.
* convert character sequences in the input to command tokens: `\csname … \endcsname.`
* convert internal quantities to a sequence of character tokens: `\number` and `\the` are expandable commands which generate a sequence of character tokens representing the value of an internal quantity.
* convert command tokens to character tokens: `\string` and `\detokenize` are expandable commands which convert their arguments to a sequence of character tokens with category code 12. Note that `\detokenize` differs from `\string`: `\detokenize` can process multiple tokens and it introduces a space character, with category code 10, after processing command tokens created from *control words*. In effect, `\detokenize` adds a trailing space-character after the command name—we’ll see some examples later in the article.
#### Refining our “definition” of expansion
We can now generalize our definition to say that expansion of a command (token) involves *removal* of that command (token) from TeX’s current input source and *replacing it* with the result of the *token operation(s)* performed by that command. In essence, the expansion process causes an expandable command to perform some type of “operation” on tokens in TeX’s current input which affects the number, or behaviour, of tokens that TeX will subsequently read—the precise nature of that “operation” depends on which command is being expanded. All macros, and active characters, are expandable but only a small number of a TeX engine’s built-in commands (primitives) are classified as expandable—the list of expandable commands depends on the TeX engine you are using.
Every new TeX engine inherits the primitive commands built into its ancestors(s)—the earlier-generation TeX engine(s) from which it was derived—and some of those inherited primitives will be expandable. Of course, a new TeX engine may choose not to implement some of the primitive commands contained in the earlier engines or to modify their behavior to suit the needs of the new engine. In addition, new TeX engines typically implement additional primitives to provide support for their own enhanced functionality—some of those might also be expandable. Consequently, the number of expandable commands available to you will vary according to the TeX engine you are using—LuaTeX has quite a collection of them.
Another difficulty with explaining/understanding expansion, and perhaps the real challenge, is knowing exactly *when* a TeX engine is, or is not, actually going to perform the expansion process. That is a large and complex topic because expansion is deeply embedded throughout the inner workings of TeX engines: we don’t have space to address this in any detail beyond the use of expansion in `\directlua`.
### How LuaTeX processes \directlua: A final look
The following diagram summarizes the `\directlua` pre-processing activities which take place inside the LuaTeX engine itself. In this diagram we also show two low-level (internal) LuaTeX functions that actually do the work: `scan_toks()` and `tokenlist_to_cstring()`. Those functions are written in the C language and reside deep inside the executable LuaTeX software: they are part of LuaTeX’s inner machinery and not *directly* accessible to your TeX/LaTeX code.
The following description of `\directlua ⟨code⟩`’s pre-processing activity summarizes the diagram above.
1. The sequence of characters in your ⟨code⟩ is processed by `scan_toks()`. Its purpose is to read your ⟨code⟩ character-by-character to generate character tokens and command tokens. Because it is creating tokens, the category code assigned to each character in ⟨code⟩, at the time it is read-in, is extremely important.
2. During `scan_toks()`’s token processing (generation) any expandable command (token) is expanded *unless* prevented via commands such as `\protected` (macro definitions), `\noexpand`, `\unexpanded` etc. Active characters (category code 13) are also expanded (unless prevented).
3. The stream of tokens created by `scan_toks()` is built into one long token list—tokens present in that list incude those produced from expansion being applied to expandable commands (such as macros) present in your `⟨code⟩`. Note too that `scan_toks()` *does not* trigger or cause execution of any token representing a non-expandable command: such non-expandable tokens are simply passed-through to be incorporated into the token list being constructed.
4. Once the token list is finished and all expansion activity is complete, that token list is processed by another function called `tokenlist_to_cstring()` which converts each token in the final token list back into its textual representation. This generates a string of text which is the Lua code to be passed on to the Lua interpreter. For successful execution, that string needs to contain syntactically correct Lua code.
5. Lua’s processing of that code takes place in two steps:
6. LuaTeX’s built-in Lua interpreter parses and “compiles” the Lua code generated in the previous steps. If that parsing/compilation fails, the Lua interpreter will generate errors (such as syntax errors)—those errors can cause the LuaTeX run to fail unless you chose to use `--interaction=nonstopmode` on the command line.
7. If parsing/compilation succeeds, the Lua interpreter executes the code compiled in step (5a).
In essence, the `scan_toks()` function is the core of LuaTeX’s pre-processing activities: its main task is to expand all expandable TeX/LaTeX commands contained in the text of your `⟨code⟩` and construct a token list out of everything it has processed. Again, we stress that `scan_toks()` *does not execute non-expandable commands* (tokens): it simply *stores* those tokens in the token list it is constructing. Once completed, that token list is subsequently converted *back to a textual representation* by `tokenlist_to_cstring()`—a token list is a TeX-only concept which is completely alien to a Lua interpreter, thus the need to convert it to text, becoming Lua code for passing on to the Lua interpreter.
## Expansion as a programming language “interface”
You can think of `\directlua`’s expansion process being used as a mechanism, or interface, for passing data/information from the “TeX World” across into the “Lua World”: providing a method for the TeX language to communicate data to the Lua language. For example, TeX code such as `\number\count75` can be used to transfer a “TeX World” value stored in count register 75 across to the “Lua World” integer variable x:
```
\count75=1564 % Data existing in the "TeX World"
\directlua{
local x=\number\count75 \space % Transfer TeX data to the "Lua World"
tex.print("x= "..x)
local y = (2*x-65)/5
tex.print(" and y = "..y)
}
```
This generates the Lua code
```
local x=1564 tex.print("x= "..x) local y = (2*x-65)/5 tex.print(" and y = "..y)
```
**Note**: We added `\space` after `\number\count75` to ensure a space character was preserved between `1564` and `tex.print`—that’s not strictly necessary here because Lua would still parse the code correctly if we omitted it. The space character following immediately after `\count75` is absorbed during the process TeX engines use to look for numeric values—here, the value being supplied to `\count`. The space character after `75` is used to terminate LuaTeX’s search for digit sequence `75` and is absorbed from the input. The `\space` macro expands to provide the space character needed to separate the text `1564` and `tex.print`.
Using the code above LuaTeX will typeset
`x= 1564 and y = 612.6`
Here, the “data passing” mechanism is achieved by `\number`: an expandable command which, in this instance, instructs TeX to retrieve the value stored in `\count` register `75` and from that value (`1546`) generate a series of character tokens, one character token for each digit, resulting in a token sequence for the digits `1`, `5`, `6` and `4`. Those 4 character tokens are incorporated into the main token list being constructed by `\directlua` and subsequently converted back to their textual representation when the token list is converted to text. It is undoubtedly a very circuitous route going from the `\count75` register value stored inside LuaTeX, to digits destined for Lua code but, ultimately, it works.
**TIP:** If you want to inspect the results of LuaTeX’s expansion activities you can write code like this:
```
\directlua{
local foo=[[local x=\number\count75
tex.print("x= "..x)
local y = (2*x-65)/5
tex.print(" and y = "..y)]]
print(foo)
}
```
In this example we use the long-brackets method to create a string variable `foo` whose purpose is to hold the string of Lua code generated from expansion of everything between `[[` and `]]`. That string is printed out to the console via the Lua function call `print(foo)`.
On Overleaf you can view similar results by writing the contents of `foo` to the `.log` file using the LuaTeX Lua function `texio.write()`:
```
\directlua{
local foo=[[local x=\number\count75
tex.print("x= "..x)
local y = (2*x-65)/5
tex.print(" and y = "..y)]]
texio.write(foo)
}
```
## Tokens in the \directlua token list: non-expandable tokens and unexpanded tokens
We’ve noted that `\directlua{⟨code⟩}` performs *full expansion* of your `⟨code⟩`: it removes and expands all expandable commands until only non-expandable tokens are left. The sequence of tokens created by `\directlua`’s processing (in the `scan_toks()` function) are strung together to form a token list whose individual tokens will be converted back to text for passing on to Lua.
However, we’ve not yet addressed the final part of this story because we need to consider the two “classes” of command token which can make it through to the token list being built within `\directlua`: we’ll refer to them as *shorthand command tokens* and *unexpanded* tokens:
* **shorthand command tokens**: This type of command token arises from control sequences defined using one of the TeX primitives `\chardef`, `\mathchardef`, `\countdef`, `\dimendef`, `\skipdef`, `\muskipdef` and `\toksdef`. These primitive commands are used to define control sequences which represent a numeric value—the resulting control sequences are *not* expandable.
* **unexpanded tokens**: This token type arises from commands that would normally be expanded but `\directlua` has either:
* been explicitly instructed *not* to expand them; for example, suppression of expansion by the commands `\noexpand` or `\unexpanded`—we’ll explain shortly how this is done;
* injected tokens by processing the sequence `\the\toks` (more on that below).
### Two “groups” of token in a \directlua token list
Based on our discussions, we can say that tokens contained in the token list being constructed during the first stage of `\directlua`’s pre-processing (in the `scan_toks()` function) fall into two groups:
1. *inherently non-expandable* tokens
* any token representing a non-active *character*;
* any token representing a non-expandable *primitive* *command*;
* any token representing a *shorthand command* (these are not expandable, see below).
3. *unexpanded* tokens:
* any token representing an expandable command whose expansion was *suppressed* (or avoided) during `\directlua`’s pre-processing.
#### Shorthand command tokens: creating non-expandable commands
As noted, TeX engines provide a set of primitives (built-in commands) that can be used to construct *non-expandable* control sequences (indicated here by `⟨command⟩`). These primitives take the form:
* `\chardef ⟨command⟩ = ⟨numeric value⟩`
* `\mathchardef ⟨command⟩ = ⟨numeric value⟩`
* `\countdef ⟨command⟩ = ⟨numeric value⟩`
* `\dimendef ⟨command⟩ = ⟨numeric value⟩`
* `\skipdef ⟨command⟩ = ⟨numeric value⟩`
* `\muskipdef ⟨command⟩ = ⟨numeric value⟩`
* `\toksdef ⟨command⟩ = ⟨numeric value⟩`
where `⟨numeric value⟩` is some integer value appropriate to each command.
Here, we’ll briefly review the use of `\chardef` to demonstrate the key feature of these primitives—producing a `⟨command⟩` that is non-expandable. You can use ``\chardef\mydollar=`\$`` to create the control sequence `\mydollar` and use it to typeset a `$`:
`I paid \mydollar30.`
This will typeset `I paid $30.` The control sequence `\mydollar` created by `\chardef` is not expandable, as we can see from the following example:
```
\chardef\mydollar=`\$
\directlua{
local x =[[I paid \mydollar30.]]
texio.write(x)
}
```
Which produces the following text in the `.log` file
`I paid \mydollar 30.`
This shows `\mydollar` was *not* expanded during `\directlua`’s pre-processing. The space appearing after `\mydollar` is added when a command token is converted to its representation as text.
When you use `\chardef` to create a control sequence, TeX’s internal classification of that control sequence (command) results in it being *non-expandable* which is very different behaviour compared to control sequences defined by one of the macro-definition commands: \def, \edef, \gdef or \xdef. As noted above, during the process of constructing its token list `\directlua` examines each incoming command token to check for expandability. If a command token is not expandable, it passes straight through to the token list and its text representation will later reappear in the string of Lua code resulting from conversion of tokens in the token list back into their textual form.
**Brief notes on plain TeX vs. LaTeX**
Historically, Knuth’s original plain TeX defined the commonly-used control symbols `\%`, `\&`, `\#` and `\$` using `\chardef`—not using one of the standard macro-definition commands `\def`, `\edef`, `\gdef` or `\xdef`. For example:
```
\chardef\#=`\#
\chardef\$=`\$
\chardef\%=`\%
\chardef\&=`\&
```
The strange `` `\ `` syntax is a TeX method to get the numeric character code value. In the old plain TeX regime, these control symbols are not expandable (due to `\chardef`) but LaTeX (or packages) may redefine of them as *macros* to provide enhanced functionality—that would make them expandable, so you may need to be aware of this.
**How does this affect \directlua?**
Let’s compare the result of the following code run under plain TeX and LaTeX. For simplicity we’ll write the results to the `.log` file using the LuaTeX Lua API function `texio.write()`.
```
\directlua{
local x=[[\$150 for the "\#1" product---20\%! more than its competitor, Widget \& Co.]]
texio.write(x)
}
```
Running this code using **plain TeX** produces the following output in the `.log` file, showing the result of any expansions:
```
\$150 for the "\#1" product---20\%! more than its competitor, Widget \& Co.
```
Clearly, under plain TeX none of the control symbols`\$`, `\#`, `\%` or `\&` were expanded—because they are all created using `\chardef`.
Running that code using the **LaTeX** document:
```
\documentclass{article}
\begin{document}
\directlua{local x=[[\$150 for the "\#1" product---20\%! more than its competitor, Widget \& Co.]] texio.write(x)}
\end{document}
```
produces the following output in the `.log` file
```
\protect \TU\textdollar 150 for the "\#1" product---20\%! more than its competitor, Widget \& Co.
```
Clearly, running LaTeX generates a result different to plain TeX because under LaTeX the command `\$` has been expanded, indicating it is a macro.
**Note:** In both plain TeX and LaTeX `\directlua` did not fully process any of the control symbols `\%`, `\&`, `\#` and `\$` to generate the corresponding character. During the expansion process performed by `\directlua` the tokens representing these control symbols—or, for LaTeX, their expansion—pass straight through to the main token list being constructed.
**Note:** Control symbols are formed from a single character not of category code 11, such as `\#`. When a token representing a control symbol is converted back to its textual representation TeX engines do not insert a space character after that text. This special treatment of control symbols is a built-in rule for how TeX engines operate.
### Unexpanded tokens: suppressing expansion
`\directlua`’s pre-processing is one example where a TeX engine is performing expansion but you might want to *prevent* expansion being applied to one or more tokens that would otherwise be expanded. By way of another example, LuaTeX (and all TeX engines) perform an expansion process, similiar to that of `\directlua`, when they process the `\write` command:
`\write file-number {⟨material⟩}`
\write instructs a TeX engine to output `⟨material⟩`—often containing TeX/LaTeX commands—to a text file (`file-number`); any expandable commands within `⟨material⟩` will, unless prevented, be expanded before `⟨material⟩` is actually written-out to that file.
As you might expect, TeX engines provide commands to suppress or control expansion:
* `\noexpand⟨token⟩`: prevents expansion of the single `⟨token⟩`;
* `\unexpanded{⟨material⟩}`: prevents expansion of all expandable commands (tokens) in `⟨material⟩`. It is, in effect, a multi-token version of `\noexpand`;
* `\protected`: a prefix added to macro definitions which prevents expansion of that macro in certain circumstances (such as during `\directlua`, `\write` or `\edef`).
Despite names which suggest otherwise, both `\noexpand` and `\unexpanded` are *expandable commands* and provide good examples of seeing a TeX engine’s expansion process as performing “token operations”: the operation here is to prevent expansion of one or more subsequent tokens (commands). Because `\noexpand` and `\unexpanded` are both expandable commands they are removed and processed (executed) during `\directlua`’s pre-processing as it constructs the token list from your `⟨code⟩`.
#### \noexpand ⟨token⟩
`\noexpand ⟨token⟩` prevents expansion of the single `⟨token⟩`. `\noexpand` within `\directlua` will be expanded (removed from the input) and replaced by the results of its “expansion behaviour”. The result of expanding `\noexpand` is to create a special (hidden) `⟨marker token⟩` that is placed in front of the original `⟨token⟩` whose expansion is to be suppressed: that `⟨marker token⟩` acts as a flag saying “do not expand the next token”. Because `\directlua` is performing full expansion it will re-process any tokens which result from the “expansion behaviour” of an expandable command. Consequently, when the expansion of `\noexpand ⟨token⟩` is complete, LuaTeX goes back to read the results and sees the two-token sequence `⟨marker token⟩⟨token⟩` which causes the original `⟨token⟩` to pass through, unexpanded, into the token list being constructed by `\directlua`.
**Example**
If we write
```
\directlua{
local x= "\TeX"
}
```
the `\TeX` macro is expanded into it constituent tokens which, in plain TeX, will result in the following text being passed to Lua (note: Lua cannot process this code, it’s just an example to demonstrate the process):
`local x = "T\kern -.1667em\lower .5ex\hbox {E}\kern -.125emX"`
If we *suppress* expansion of the `\TeX` macro using `\noexpand`
`\directlua{local x= "\noexpand\TeX"}`
the following Lua code is produced (again, Lua can’t run this code; it is simply an example to demonstrate `\noexpand`):
`local x= "\TeX "`
Because of `\noexpand`, `\directlua` will not expand `\TeX` but simply allow the token value representing the `\TeX` command to pass through, unscathed, into the token list being built during the first stage of `\directlua`’s pre-processing.
**Note:** The space character appearing after `\TeX` is introduced by LuaTeX’s subsequent conversion of the `\TeX` integer token nalue back to its textual representation (within the `tokenlist_to_cstring()` function).
#### \unexpanded{⟨material⟩}
`\unexpanded` is an expandable command which suppresses expansion of all tokens formed from `⟨material⟩`. As we have noted, when a TeX engine performs expansion any expandable command is *removed* from the input and *replaced* by the results of its “expansion behaviour”; so what does that actually mean for `\unexpanded`? Usually, during *full expansion*, once the expansion process for a particular command is completed the TeX engine goes on to read/process any tokens arising from that command’s “expansion behaviour”—it needs to further expand any tokens that were produced. However, `\unexpanded` *bypasses* any further expansion: here is how it does that.
Inside the TeX engine, the `\unexpanded` command first converts the characters and commands in `⟨material⟩` to a temporary token list comprised of *unexpanded* tokens. After all tokens have been created and stored in that temporary token list, the `\unexpanded` command causes `\directlua` to *skip* going back to read and process them—even though \directlua is performing full expansion. Instead, those *unexpanded* tokens pass straight through and become incorporated into the main token list being built by `\directlua` (in the `scan_toks()` function). In this way, everything in `⟨material⟩` is converted to tokens and the expansion process is skipped for that set of tokens. The operation of `\unexpanded{⟨material⟩}` is similar to the use of `\the\toks`, which we discuss below.
**Example**
`\unexpanded` produces results in a manner similar to `\noexpand` except it can prevent expansion of multiple tokens; here is an example:
```
\directlua{
local x = "\unexpanded{\foo\bar\foobar}. But Lua can't process this code!"
}
```
which produces the following text as code for Lua:
`local x = "\foo \bar \foobar . But Lua can't process this code!"`
**Note**: There are space characters after each command name. These are again a consequence of LuaTeX’s subsequent conversion of the unexpanded tokens `\foo`, `\bar` and `\foobar` back to text within the `tokenlist_to_cstring()` function.
#### \protected macro definitions
The `\protected` command is a prefix applied to a macro definition to prevent that macro being expanded when TeX is building an expanded token list, such as the token list built by `\directlua`’s pre-processing.
**Example**
Suppose you define the following macros with and without using the `\protected` prefix:
```
\def\macroA{"This unprotected macro contains a string"}
\protected\def\macroB{"This protected macro also contains a string"}
```
If you use Lua’s string concatenation operator (`..`) to write
```
\directlua{
local x=\macroA..\macroB
}
```
`\directlua`’s pre-processing would produce the following code for passing to Lua:
`local x="This unprotected macro contains a string"..\macroB`
`\macroA` is not defined using `\protected` so it is expanded, producing the first part of the string to be concatenated, but `\macroB` is defined using `\protected` so it has not been expanded.
During pre-processing, LuaTeX’s `scan_toks()` function created a token for `\macroA`, recognized it was a regular expandable command and expanded it: that expansion produces a sequence of character tokens, one character token for each character in `"This unprotected macro contains a string"`. Each character token is passed on and added to the token list being built.
When `scan_toks()` creates the token for `\macroB` it notices that command was defined as `\protected` and does not expand it: the token representing `\macroB` passes through, untouched (not expanded), into the token list being built. After that token list has been built, the next stage of pre-processing, within the `tokenlist_to_cstring()` function, is to convert all tokens in the token list back to their textual representation. The unexpanded token representing `\macroB` is detected and converted to its text representation, resulting in the text `\macroB` appearing in the code destined for Lua. Note that Lua cannot actually concatenate `"This unprotected macro contains a string"..\macroB` to produce the final string because `\macroB` has no meaning in Lua’s syntax, resulting in the error `unexpected symbol near '\'`.
**Trivia**: The `\protected` command was introduced by $$\varepsilon\text{-}\mathrm{\TeX}$$, the first major extension of Knuth’s original TeX software, and is supported by all TeX engines whose code ancestry includes $$\varepsilon\text{-}\mathrm{\TeX}$$.
### Unexpanded tokens: Using \the\toks in \directlua
Life in programming would not be the same without those “special cases” to deal with and use of `\the` in conjunction with `\toks` in a `\directlua` command is one such special case.
#### Brief background on \toks
The TeX primitive `\toks` instructs a TeX engine to save some tokens for use later on: instead of being passed on for further processing, those tokens are put to one side and stored away in a memory location specified using a *token register*. For example, we can tell a TeX engine to create some tokens and store them in token register location `100` using
`\toks100={Hi, \TeX! \hskip 5bp}`
Here, TeX uses token register `100` to access a known location inside its memory: a storage area designated for holding lists of tokens.
Tokens representing everything between the `{` and `}` are created, *but not expanded*, and strung together in a token list—similar to the token list we explored earlier in this article. To re-use those tokens we would write `\the\toks100` in which `\the` (an expandable command) instructs TeX to fetch the stored tokens and insert them at the location where you wrote `\the\toks100`. Another way to think of this is `\the\toks` causes TeX to insert some tokens at that location.
The `\toks` command *does not expand* any of the tokens it is asked to create and save: it simply converts characters and commands between `{` and `}` to tokens and stores them.
#### Back to \directlua
In the discussion of expansion we noted `\directlua{⟨code⟩}` performs *full expansion* of `⟨code⟩`: removing all expandable commands and replacing them with the result of their expansion behaviour—continuing to *further expand* any tokens arising from the inital expansion of an expandable command.
`\the` is an expandable command so `\directlua` will expand it; however, when `\the` is used in conjunction with `\toks` within `\directlua`, as in `\the\toks⟨token register⟩`, the inserted tokens are *not expanded any further*. Expansion of `\the\toks⟨token register⟩` injects the sequence of *unexpanded* tokens, stored in `⟨token register⟩`, directly into the token list being constructed by `\directlua`: this behaviour bypasses the usual process of full expansion. In effect, those tokens pass through, *unexpanded*, to become incorporated into the main token list being constructed by `\directlua`—this pass-through process for unexpanded tokens is similar in operation to `\unexpanded`, as discussed earlier.
**Example**
Suppose we define the macro `\mymacro` as `\def\mymacro{\TeX}`. It contains just one token for the `\TeX` command (which is a macro): so we have an expandable command `\mymacro` that contains another macro `\TeX`, which is also expandable.
The following code will result in Lua trying to create a string variable `x`:
```
\def\mymacro{\TeX}
\directlua{
local x="\mymacro"
}
```
Within \\`directlua`, the token for `\mymacro` is expanded but that results in another expandable token, `\TeX`, which is further expanded. In plain TeX, those expansions result in the following text passed to Lua:
`local x = "T\kern -.1667em\lower .5ex\hbox {E}\kern -.125emX"`
This code tries to define a string which contains text representing the expanded version of the `\TeX` macro. If you try to run this example Lua will attempt to construct that string but it will fail, generating an error:
`invalid escape sequence near ' "T\k'.`
Later in this article we’ll explore the meaning of “invalid escape sequence”.
Let’s now contrast the use of `\mymacro` with placing the `\TeX` token inside a token list generated by a `\toks` command:
```
\toks100={\TeX}
\directlua{
local x="\the\toks100"
}
```
LuaTeX’s `\directlua` processing will generate this string of text for Lua:
`local x = "\TeX "`
The space character after `\TeX` is generated by LuaTeX’s command-token-to-string conversion process.
**But note**: The `\TeX` macro has *not* been expanded into its constituent tokens. `\the\toks100` caused the tokens stored in register 100 to be inserted buts that’s all: they are *not* expanded any further and become incorporated into the main token list being build by `\directlua` (within the function `scan_toks()`). Putting tokens into a token list created by `\toks` is yet another way to prevent tokens being expanded.
If we run this example it too produces an error:
`invalid escape sequence near ' "\T'.`
We explore Lua escape sequences later in the article.
## Other commands/techniques used in expansion
In this section we look at some additional TeX commands/methods which can be useful in situations where expansion is being applied (such as within `\directlua`).
### \string ⟨token⟩
`\string` is an expandable command which converts the ⟨token⟩ into a series of character tokens, each with category code 12.
For example, `\string\TeX` would produce a series of 4 character tokens `\`, `T`, `e` and `X` where each character is assigned category code 12 (including the leading `\` character).
If we write
```
\directlua{
local x="I will use \string\newcommand"
print(x)
}
```
the `\string` command will be expanded, resulting in a sequence of character tokens with category code 12. After `\string` is expanded, the resulting character tokens (representing each character in `\newcommand`) will be incorporated into the main token list being constructed by `\directlua`. Once `\directlua` has finished constructing its main token list, its constituent tokens are converted back to their textual representation which produces the following code for passing on to the Lua interpreter:
`local x="I will use \newcommand" print(x)`
When this code is passed to Lua, `print(x)` will output the string `x` to the screen (console). However, we’ve been slightly sneaky and deliberately used an example command starting with `\n`. If you are able to run this example on a local TeX installation you’ll notice that Lua prints the following text to the screen:
```
I will use
ewcommand
```
To run this code on Overleaf you can instruct LuaTeX to write directly to the `.log` file using the LuaTeX Lua API function `texio.write(*string*)`:
```
\directlua{
local x="I will use \string\newcommand"
texio.write(x)
}
```
If you inspect the resulting `.log` file you’ll see it also contains
```
I will use
ewcommand
```
This unexpected output is due to Lua interpreting the `\n` at the start of `**\n**ewcommand` as the escape sequence for the linefeed character (character code 10): it assumes that you want start a new line of text which begins with `ewcommand`. We discuss Lua escape sequences later in this article.
### \detokenize{⟨material⟩}
`\detokenize` is, in its effects, a multi-token version of `\string` and it too is an expandable command that converts everything in `⟨material⟩` to a sequence of character tokens with category code 12—*except* space characters (ASCII/Unicode value 32) which get category code 10. `\detokenize` also inserts a trailing space character after command names that are *control words* (e.g., `\foo`) but no space character is inserted after *control symbols* (e.g., `\#`, `\%` etc).
### Example
Even if the macros `\foohoo`, `\foo`, `\bar` and `\foobar` are not defined, if you write this:
```
\directlua{
local x = "\string\foohoo\detokenize{\foo\bar\foobar}"
}
```
it would produce the following text as code for passing to the Lua interpreter
`local x = "\foohoo\foo \bar \foobar "`
If you do not use `\string` and `\detokenize` and write:
`\directlua{local x = "\foohoo\foo\bar\foobar"}`
`\directlua` would process `\foohoo`, recognize it is a command and try to expand it; but because `\foohoo` is not defined it would result in an error:
```
! Undefined control sequence.
l.1 \directlua{local x = "\foohoo
\foo\bar\foobar"}
?
```
Because `\string` and `\detokenize` convert their arguments into a series of character tokens, `\directlua`’s expansion process does get the opportunity to detect expandable command tokens `\foohoo`, `\foo`, `\bar`, or `\foobar`: they are turned into sequences of character tokens long before they can trigger expansion.
As noted previously, expansion of a command involves removing it from the input and replacing it with the result of its “expansion behaviour”. The results of expansion (usually tokens) are subsequently read by the TeX engine. Here, the “expansion behaviour” for `\string` and `\detokenize` is to absorb character and command tokens from the input and convert them to sequences of character tokens, initially stored in a temporary token list, which `\directlua` subsequently reads. Those character tokens become incorporated into the main token list being constructed by `\directlua`.
The following graphic depicts how `\string` converts the `\foohoo` command to a sequence of character tokens, producing a temporary token list that is subsequently read by `\directlua` to incorporate those character tokens into the main token list being constructed.
If `\string` or `\detokenize` encounter characters in their argument e.g., `\string a` or `\detokenize{abc}` those characters (here, with category code 11) produce character tokens but with category code 12.
Notes:
If we return to the example above:
`\directlua{local x = "\string\foohoo\detokenize{\foo\bar\foobar}"}`
which produces the following text as code for passing to the Lua interpreter
`local x = "\foohoo\foo \bar \foobar "`
we can observe the following:
* `\detokenize` has inserted a space character after each macro name but `\string` did not.
* `\string` acts on a single token.
* In the string `"\foohoo\foo \bar \foobar "` used to define `x` we will once again encounter Lua’s escape character mechanism (discussed below):
* `\bar` starts with `\b` which is the Lua escape sequence used to represent the [backspace character](https://en.wikipedia.org/wiki/Backspace) (character code 8);
* commands `\foohoo`, `\foo` and `\foobar` all starts with `\f`, the Lua escape sequence used to represent the [formfeed character](https://en.wikipedia.org/wiki/Page_break#Form_feed) (character code 12).
Because the character sequences `\b` and `\f` are used within a string created using double quotes `"..."` they will produce unwanted results unless steps are taken to prevent that using Lua’s so-called *long-brackets* string method: a subject we can now discuss along with Lua escape sequences.
## What are “Lua escape sequences”?
Programming languages reserve certain characters for “special use” as part of the language syntax: in effect, those characters are defined to have some form of special meaning. However, there are times when you need to temporarily “switch off” such a character’s special meaning if, for example, you want that character to be embedded as part of a longer string where it’s standard behaviour would introduce syntax errors. In essence, that character needs to be processed *without* triggering its standard interpretation—to slip through without being noticed. To do this, programmers use a technique called *escaping* in which a “special character” is represented by its so-called *escape sequence*.
A standard example (also supported by Lua) is using double quotes inside a string where you escape the inner double quotes using the escape sequence `\"`:
`"When asked about LuaTeX they replied: \"It is an awesome TeX engine!\" I agreed."`
The Lua language provides a number of mechanisms to work with escape sequences:
* standard sequences including `\n` (newline), `\r` (carriage return), `\\` (backslash), `\"` (double quote), `\t` (horizontal tab), `\v` (vertical tab) and `\'` (single quote);
* `\xXX`, where `XX` is a sequence of exactly two hexadecimal digits;
* `\ddd`, where `ddd` is a sequence of up to three decimal digits;
* at the time this article was written (August 2019) the latest version of LuaTeX, although not yet available on Overleaf, uses version 5.3 of Lua which introduced support for UTF-8 escape sequences: `\u{XXX}`. This escape mechanism is for UTF-8 encoded Unicode characters where `XXX` is a sequence of one or more hexadecimal digits representing the character code point. Note that the enclosing brackets `{ }` are mandatory.
### Controlling escape sequences
Traditionally, strings are defined using double quotes as in `"this is a string"`; within such a string you can use escape sequences: `"this is a string.\nI'll now start on a new line."`. However, Lua has a second and *very* convenient mechanism to define strings: its so-called *long brackets* mechanism in which you define a string by enclosing the text in `[[` and `]]`:
`[[I am a long brackets string]]`
Within a string created using the long-brackets method, Lua’s character-escape mechanism is *switched off*: escape sequences are treated as regular characters. For example, in the string
`[[I am a long brackets\n string]]`
the `\n` escape sequence is not treated as the single carriage return character (ASCII code 13) but as two regular characters: `\` followed by `n`.
### Why are long bracket strings so useful?
As we’ll later explore, LuaTeX provides a suite of specialized, built-in, Lua functions that you can use with `\directlua` to control LuaTeX’s typesetting behaviour. Among those many functions is one called `tex.print(*string*)` that allows you to pass `*string*` material from Lua code back to LuaTeX for typesetting. A very simple example is:
`\directlua{tex.print("Hello, World!")}`
which will cause LuaTeX to typeset `Hello, World!`
The `*string*` used in `tex.print(*string*)` can also include text representing TeX and LaTeX commands for LuaTeX to process. However, TeX/LaTeX commands start with a `\` character which is problematic with strings created using double quotes because Lua would try to parse the string, detect the initial `\` character and interpret it as the beginning of an escape sequence. When Lua tries to process the escape sequence it will usually fail because the initial `\` combined with the first character in many TeX/LaTeX command names does not form a valid escape sequence known to Lua. For example when processing a string such as `"I like \LaTeX"` Lua would see `\L` and fail with the error “invalid escape sequence”, and this is the cause of the errors noted above.
#### Long-bracket strings come to the rescue!
The long-brackets method of creating (defining) strings is extremely useful because even though TeX/LaTeX commands start with a `\` character, the long-brackets string method disables (switches off) Lua’s escape sequence mechanism. Here is a short example, remembering that we need to prevent macros from being expanded using, for example, `\protected` or `\noexpand`.
Suppose we define a `\newtest` macro like this
`**\protected**\def\newtest#1{The argument: #1}`
and use it in `\directlua` with the LuaTeX Lua API function `tex.print()`:
```
\directlua{
tex.print("\newtest{Hello}")
}
```
Due to the use of `\protected`, the macro `\newtest` is not expanded which results in the following text passed to Lua:
`tex.print("\newtest {Hello}")`
The space character added after `\newtest` and before the opening brace (`{`) is a side-effect of `\directlua`’s conversion of command tokens back to their textual representation.
This code is passed to Lua which subsequently executes the LuaTeX function `tex.print()` but there’s a problem which manifests itself in ways that depend on the fonts you are using. In LaTeX on Overleaf you would see output like this:
along with a warning in the log file:
```
Missing character: There is no
(U+000A) in font [lmroman10-regular]:+tlig;!
```
In plain TeX you might see output that looks something like this:
In both cases the `\newtest` macro is not called and the output is not what we intended. The error is caused by Lua’s escape character mechanism: in the text `\newtest {Hello}` the macro name starts with `\n` which Lua recognizes as the escape sequence for a linefeed character so it replaces `\n` by ASCII character 10, or in hex 0A. In the LaTeX error message, `U+000A` is a way to represent the Unicode value using 4 hex digits.
Because the `\n` is converted to the linefeed character, LuaTeX does not see a macro call but instead believes it is being asked to typeset some text that starts with ASCII character code 10:
`⟨ASCII 10⟩ewtest {Hello}`
Depending on the font used, LuaTeX may, or may not, be able to typeset the `⟨ASCII 10⟩` character but the remaining text is output as-is with the `{` and `}` treated as a group and not printed.
Plain TeX gives a different result because the default font is Computer Modern Roman which has a strange encoding that results in a capital Omega typeset when character code 10 is seen.
To prevent these problems we need to use long-bracket strings to prevent Lua’s escaping mechanism being applied. The correct result is produced with
`\directlua{tex.print([[\newtest{Hello}]])}`
which produces the result shown in the following screenshot:
### Expansion and non-execution of non-expandable commands
When discussing expansion we noted it is a process in which a TeX engine *removes* an expandable command (token) from the current input and *replaces* it with the result(s) produced by that expandable command. Because \directlua is performing *expansion-only* activities (to generate a token list), it *does not* take LuaTeX’s processing any further than that. Once an expandable command has been read and fully expanded the results of that expansion—which frequently includes non-expandable commands (tokens)—will be incorporated into the token list being built, ready for conversion back to text for passing on to Lua.
There is an important principle at work here: during *expansion-only* activities designed to produce a token list, TeX engines, including LuaTeX, *do not execute* any non-expandable primitive, built-in, TeX commands.
In the case of `\directlua{⟨code⟩}`, if the fully expanded version of your `⟨code⟩` produces, or contains non-expandable TeX/LaTeX commands they *will be passed on to Lua* (represented as text).
#### Example
Here is an example to demonstrate that non-expandable primitives are not executed during expansion-only processing (such as within `\directlua`). Suppose we define a macro `\setcountreg` like this:
`\def\setcountreg#1#2{\count#1=#2\relax}`
**Note**: We use `\relax` after parameter `#2` to prevent LuaTeX overshooting when scanning the input in its search for the numeric value (argument) to match parameter `#2`.
If, outside of `\directlua`, we later run the macro like this
```
\setcountreg{100}{50}
The value in count register 100 is \the\count100.
```
it would output
`The value in count register 100 is 50.`
In this context, any TeX engine would process the macro `\setcountreg`—expand the macro, determine the arguments and continue to read *and action* (execute) commands contained in the macro’s replacement text (definition). The result here is to assign `50` as the value stored in register `\count100`.
However, when a TeX engine is performing *expansion-only* activities, as it is with `\directlua`, it *will not execute* the non-expandable commands contained in the macro’s definition.
If we write
```
\def\setcountreg#1#2{\count#1=#2\relax}
\directlua{
local x = [[\setcountreg{100}{50}]]
}
```
it produces the following text as the code for Lua:
`local x = [[\count 100=50\relax ]]`
The Lua code produced above shows that within `\directlua` the `\setcountreg` has been expanded, its arguments identified and substituted into the appropriate parameter (`#1` and `#2`) but it goes no further than that: the non-expandable primitive TeX command `\count` was *not executed* during `\directlua`’s expansion processing.
However, LuaTeX will execute the TeX code if we pass the resulting string `x` *back to LuaTeX* via `tex.print(x)` like this
```
\count100=50 % set \count100 to a starting value of 50
\def\setcountreg#1#2{\count#1=#2\relax}
\directlua{
local x = [[\setcountreg{100}{250}]]
tex.print(x)
}
The value stored in count register 100 is \the\count100.
```
After `\directlua` has finished the ouput would be
`The value stored in count register 100 is 250.`
showing that count register `100` does now contain the value `250`.
The Lua code produced from the above example is
`local x = [[\count 100=250\relax ]] tex.print(x)`
This code defines `x` to be a string created using the long-brackets method which is used to avoid errors with erroneous escape sequences. If we used double quotes `"..."` to define x, the character combination `\c` at the start of `\count` would trigger an error: `invalid escape sequence near ' "\c'`.
The LuaTeX Lua API call `tex.print(x)` results in LuaTeX executing the TeX code sequence `\count 100=250\relax` and `\count100` is assigned a value of `250` as seen from the typeset output:
`The value stored in count register 100 is 250.`
#### Caution: macros and the LuaTeX Lua API
In the above example we saw that during `\directlua`’s pre-processing (expansion) LuaTeX did not execute the code `\count 100=250`, which contains the `non-expandable` primitive command `\count`: to run (execute) that code we had to *pass it back to LuaTeX* via `tex.print()`.
`\directlua` is just one instance where LuaTeX is performing expansion-only processing to construct a token list. There are other commands which perform similar expansion processing and token-list generation activities, such as `\write` and `\edef`: those commands also do not execute non-expandable primitives during their expansion processing. It is general principle that TeX engines do not execute non-expandable primitives when constructing a token list during expansion-only processing activities.
**Rewriting our macro to use the LuaTeX Lua API**
We can re-write the `\setcountreg` macro using a LuaTeX Lua API function called `tex.setcount()`, thus avoiding TeX commands to change the value stored in count register `100`:
```
\def\setcount#1#2{\directlua{tex.setcount(#1,#2)}}
\count100=50
count register 100 contains \the\count100\par
\setcount{100}{250}
count register 100 now contains \the\count100\par
```
This code will typeset:
```
count register 100 contains 50
count register 100 now contains 250
```
Here we are using `tex.setcount()`, one of LuaTeX’s many Lua API functions, to *directly access* LuaTeX’s internal data storage area to place the value `250` in the memory location representing count register `100`. We have, in effect, *bypassed* LuaTeX’s standard TeX engine input-processing methods: reading input, creating tokens and executing TeX primitive commands. However, there is a cautionary tale: by using LuaTeX’s Lua API functions, expansion-only processing activity *can result in side-effects*: changes to values stored inside the TeX engine that would not otherwise be possible with pure TeX/LaTeX commands.
**Example: unexpected side-effects**
Here is an example to demonstate *unexpected* side-effects which can arise with macros using `\directlua`. Suppose we write the following code:
```
\def\dochange{\directlua{tex.setcount(999,12345)}}
\edef\careful{\dochange}
\the\count999
```
Running this code typesets `12345`!
How can that be? We did not *explicity* call any code or macros to put that value in count register `999`. Or did we?
We defined `\dochange` with a `\directlua` command that uses `tex.setcount()` to store the value `12345` in count register `999`: in TeX code it is the equivalent of `\count999=12345`. We then used the standard TeX primitive `\edef` to define the macro `\careful`—it is the use of `\edef` which triggers the unexpected side-effect.
`\edef` fully expands its argument: here, it detects an expandable macro `\dochange` and expands it. The `\dochange` macro uses the expandable command `\directlua` which contains a Lua API call; so the expansion of `\dochange` results in expansion of `\directlua` and that causes `tex.setcount()` to be called, which changes the value in count register `999`.
If we redefine `\dochange` to use TeX commands:
```
Before: count register 999 contains \the\count999.\par
\def\dochange{\count999=12345\relax}
\edef\careful{\dochange}
After: count register 999 contains \the\count999.\par
```
running this code typesets
```
Before: count register 999 contains 0.
After: count register 999 contains 0.
```
Clearly, there was no effect on `\count999`. When `\edef` defines `\careful` it expands `\dochange` but that expansion produces unexpandable TeX primitives only: they are *not executed* but simply *stored* in the token list comprising the definition of `\careful`.
Just for good measure, the same principle explains why this produces typeset output:
```
\def\dochange{\directlua{tex.print("Hello")}}
\edef\careful{\dochange}
```
## Brief introduction to LuaTeX’s Lua API
As we’ve seen, `\directlua` not only enables you to write conventional Lua code, or a mixture of Lua and TeX/LaTeX code, but it also provides access to a suite of additional Lua functions (specific to LuaTeX) that you can use (call) to communicate with, or directly control, the inner workings of the LuaTeX typesetting software. We’ve used several Lua functions in this article, `tex.print()`, `texio.write()`, `tex.setcount()` and these, along with *many* more, are documented in [The LuaTeX Reference Manual](http://www.pragma-ade.com/general/manuals/luatex.pdf) in which groups of related functions are referred to as *libraries*.
You can think of these Lua functions as LuaTeX’s Lua API (**A**pplication **P**rogramming **I**nterface) which provide the tools to construct sophisticated typesetting and document engineering solutions by controlling the typesetting behaviour of LuaTeX using Lua as the driver.
As noted, LuaTeX organizes its API into set of functions it called libraries: groups of functions which are related through their purpose or actions. Each set of functions is designed to provide access to particular aspect of LuaTeX’s internal processes, data structures, data storage and typesetting algorithms. Internally, LuaTeX is constructed from multiple components: software libraries/tools (mostly written in C) that not only comprise the TeX engine itself but other sub-systems including Lua, MetaPost, Kpathsea, FontForge, libpng and zlib. These libraries are integrated to build the features and functions of the LuaTeX executable software and it is through the Lua API that users are given access to LuaTeX’s functionality derived from its integration and coordination of those multiple software components.
## Some examples and pitfalls
In this section we present some further examples which make use of the topics, concepts and explanations provided in this article.
### Using the tilde character (\~)
The Lua language uses the `~` character (called tilde) as part of its syntax, including its syntax for performing a “not equal” test; for example, to test if a variable `x` is not equal to `4` we could write:
```
local x=3
if x ~= 4 then
print("x is not equal to 4")
end
```
If we try to run this simple Lua code via `\directlua`:
```
\directlua{
local x=3
if x ~= 4 then
print("x is not equal to 4")
end
}
```
we get an error:
`[\directlua]:1: 'then' expected near '\'.`
That’s odd because our code is correct: we have used `'then'` and there is no `\` character in our code, so what went wrong? To understand this, we must remember that, to TeX/LaTeX, `~` is usually defined to be a “special character” with category code 13: so-called active characters which are mini-macros and thus subject to expansion. When `\directlua` detects the `~` character it is expanded by *removing it* from the input and *replacing it* with the result of its expansion. Using plain TeX, the resulting text (code) that LuaTeX produces and passes to the Lua interpreter does not actually contain the `~` character, and is:
`local x=3 if x \penalty \@M \ = 4 then print("x is not equal to 4") end`
The `~` character has been *removed* and *expanded* into its constituent commands—the Lua code above results from plain TeX’s definition of the active character `~`. Now we can see why Lua responds with the error `'then' expected near '\'`—it starts to parse this code but encounters the word `\penalty` which means nothing to Lua and generates a syntax error.
To fix this, the `~` character needs to have a safe category code at the time `\directlua` is processing your code; for example, we can temporarily change the category code of `~` to 11 (letter) by enclosing the code in a group:
```
\begingroup
\catcode`\~=11
\directlua{
local x=3
if x ~= 4 then
print("x is not equal to 4")
end
}
\endgroup
```
This code works as expected and `x is not equal to 4` is printed to the console. There are other options: we can use the expandable commands `\noexpand` or `\string`.
#### Using \string⟨token⟩
We can apply `\string` to the single-character `⟨token⟩` `~` which has category code 13 (active character); `\string` converts the `~` character to generate a character token which has category code 12. If we do
```
\directlua{
local x=3
if x \string~= 4 then
print("x is not equal to 4")
end
}
```
it produces the Lua code we require:
`local x=3 if x ~= 4 then tex.print("x is not equal to 4") end`
#### Using \noexpand⟨token⟩
We can use `\noexpand~` to suppress expansion of the active character `~`
```
\directlua{
local x=3
if x \noexpand~= 4 then
print("x is not equal to 4")
end
}
```
The unexpanded `~` token passes through to the token list being built in `\directlua` and will be converted back to text which produces working Lua code.
### Using the # character
Within the Lua language the `#` character can be used to find the length of a table. However, if we try the following code
```
\directlua{
local tbl = {}
tbl[1] = "Hello"
tbl[2] = "World"
tex.print("Table length is "..#tbl)
}
```
we might expect LuaTeX to typeset
`Table length is 2`
but it generates an error:
`\directlua]:1: attempt to get length of a number value`
This error is triggered because the `#` character usually has category code 6 (macro parameter)—the `#` character has two uses in TeX/LaTeX: to indicate macro parameters (`#1`, `#2`… `#9`) and the replacement text in alignment templates (for `\halign` and `\valign`).
When `\directlua` is generating tokens to build its token list it sees the `#` character with category code 6 and creates a suitable character token to represent it. When the time comes to convert the final token list back to textual form, the character token for # (with category code 6) receives a special treatment: it is output as *two consecutive characters*: `##`, resulting in the following code being passed to Lua:
`local tbl = {} tbl[1] = "Hello" tbl[2] = "World" print(##tbl)`
On conversion to Lua code, the original `#` has been doubled and that generates an error:
`\directlua]:1: attempt to get length of a number value`
This problem arises due to TeX’s syntax which uses a double hash symbol `##` to represent or generate a single `#` token; this syntax is used in macros which define other macros that take parameters, or in macros used to create templates for the `\halign` or `\valign` table-construction commands. This is rather confusing so let’s look at an example.
#### Example
Suppose we define a macro `\mymacro` which takes a single parameter, `#1`, but it also defines a second macro `\foo` which itself takes a single parameter. To distinguish between the parameter `#1` used with `\mymacro` and the need to define `\foo` to use its own parameter `#1` TeX syntax requires that you use `##1` inside `\mymacro` to represent the parameter to be used with `\foo`:
`\def\mymacro#1{\def\foo##1{#1 Hello##1}}`
If you were to write `\mymacro{Hey!}` it would define the macro `\foo` to be
`\def\foo#1{Hey! Hello#1}`
Note that the `\mymacro`’s parameter `#1` (`Hey!`) has been incorporated into the definition of `\foo` and the sequence `##1` has been converted to `#1` in the definition of `\foo`. So we can use `\foo` like this:
`\foo{, World!}`
to typeset `Hey! Hello, World!`
We can resolve `\directlua`’s treatment of the `#` character by temporarily changing its category code before LuaTeX processes the code. For example:
```
\begingroup
\catcode`\#=11
\directlua{
local tbl = {}
tbl[1] = "Hello"
tbl[2] = "World"
tex.print("Table length is "..#tbl)
}
\endgroup
```
This generates the Lua code
```
local tbl = {} tbl[1] = "Hello" tbl[2] = "World" tex.print("Table length is "..#tbl)
```
which typesets the result we expected:
`Table length is 2`
### Using the % character
Within TeX/LaTeX, the `%` character is typically used to include single-line comments in your code: to signal to the TeX engine that it should ignore everything from that point until the end of the line on which the `%` is written. However, within the Lua language, the `%` character is used within some very useful string-processing functions, such as `string.format(...)`, `string.gmatch(...)`, and `string.gsub(...)` in which the `%` character plays an important role as part of those function’s syntax.
When used with TeX/LaTeX, `%` acts as the comment character because it is assigned category code 14. To make it behave as regular character, and switch off its usual TeX/LaTeX behaviour, we need to change its category code to something safe, such as 12. The `\directlua` example below uses a number of techniques discussed earlier in the article, together with one that we have not yet mentioned: ``\catcode`\^^M=12``, which allows us to use Lua comments in our code; this is discussed below.
#### Example
The following examples are borrowed from [lua-users.org](http://lua-users.org/wiki/StringLibraryTutorial), suitably modified for use within `\directlua`.
```
\documentclass{article}
\begin{document}
\begingroup
\ttfamily
\let\\\relax
\catcode`\^^M=12 %<---we further explore this below!
\catcode`\%=12
\directlua{
local str -- declare a local variable to hold the result
tex.print("Using string.format():".."\\par")
str=string.format("%s %q", "Hello", "Lua user!") -- string and quoted string
tex.print(str.."\\par")
str = string.format("%c%c%c", 76, 117, 97) -- char
tex.print(str.."\\par")
str=string.format("%e, %E", math.pi, math.pi) -- exponent
tex.print(str.."\\par")
str=string.format("%f", math.pi) -- float
tex.print(str.."\\par")
str=string.format("%g, %g", math.pi, 10^9) -- float or exponent
tex.print(str.."\\par")
str = string.format("%o, %x, %X", 99, 125, 125) -- octal, hexadecimal, hexadecimal
tex.print(str.."\\par")
tex.print("\\vskip3mm".."Using string.gmatch():".."\\par")
for word in string.gmatch("Hello TeX user", "%a+") do
tex.print(word.."\\par")
end
tex.print("\\vskip3mm".."Using string.gsub():".."\\par")
str=string.gsub("banana", "(an)", "%1-") -- capture any occurrences of "an" and replace
tex.print(str.."\\par")
}
\endgroup
\end{document}
```
The following screenshot shows the typeset result of the code above:
## Why is Lua code shown on a single line?
As you may have noticed, all the (generated) Lua code fragments shown in this article’s examples are presented as a single line of text: line breaks originally present in the `\directlua` code snippets are not followed. Why is that? It is because line breaks in the Lua code have been *stripped out* during LuaTeX’s pre-processing within `\directlua`, causing the Lua code to become one long line of text. That behaviour can be traced to the way TeX engines handle end-of-line characters—denoted by `\r` (carriage return) and `\n` (line feed) within programming literature. Just why we might need to worry about these fine details will become clear when we discuss using Lua’s mechanisms for commenting-out sections of code.
When software writes (saves) a text file, each individual line of text is terminated by so-called “newline” characters—the actual newline character(s) depend on the application and operating system being used to write-out that file. Wikipedia has an [interesting article](https://en.wikipedia.org/wiki/Newline) which explores the history/evolution of the newline characters in use today.
Given any text file, its individual lines of text could be terminated by various combinations of characters, referred to as carriage return (ASCII/Unicode character 13) and/or line feed (ASCII/Unicode character 10), which are denoted by `\r` and `\n` respectively. Because TeX engines are designed to be platform independent they need a method to circumvent the inherently platform-dependent nature of line endings used in text files. Naturally, TeX engines have a built-in (but configurable) method for dealing with line-termination characters.
### How TeX engines deal with line endings
When LuaTeX is processing `\directlua{⟨code⟩}` it reads the text contained in your `⟨code⟩` and applies standard TeX engine methods for processing any line endings contained in your `⟨code⟩`. By default, those standard TeX methods cause all line-termination characters (carriage returns and line feeds) to be removed and replaced by space characters. We say “by default” because a TeX engine’s handling of line-termination characters can be modified through a user-configurable parameter called `\endlinechar`. Here, we’ll provide a short two-step overview but further details can be found in the Overleaf article [An introduction to \endlinechar: How TeX reads lines from text files](/latex/in-depth-articles/05-an-introduction-to-endlinechar-how-tex-reads-lines-from-text-files.md).
#### Step 1: TeX inserts its own end-of-line character
After reading a line of text from your input file, TeX engines immediately remove any `\r` or `\n` characters from the end of that line. Next, TeX engines *insert* (add-back) their own line-termination character to the end of that line. That character is determined by the value of a user-configurable TeX parameter called `\endlinechar` and it is through this mechanism TeX engines can process end-of-line characters in a platform-independent manner: they choose, and set, the end-of-line character irrespective of what was originally contained in the input text file.
Typically, TeX engines use the setting
`\endlinechar=13`
which is the carriage-return character (`\r`). However, users can always assign another character code as the value of `\endlinechar`—which we’ll see later in this article.
Consequently, any line-termination character(s) contained in your `⟨code⟩` to be processed by `\directlua{⟨code⟩}` are stripped out and replaced by a single character determined by the TeX engine itself. Note that TeX engines perform this end-of-line processing immediately after reading a new line of text from a file and *before* processing any characters in that line (to generate tokens). However, this is not the end of the story: what the TeX engine *does* with those end-of-line characters (it has insered) explains why the Lua code becomes one single line.
#### Step 2: TeX converts its end-of-line character to a space
In addition to inserting their own line-termination character, defined by the value of `\endlinechar`, TeX engines also use category code 5 for characters that should be *treated as* an end-of-line character. This results in TeX engines usually working with:
1. an end-of-line character defined by `\endlinechar`;
2. that same character *usually* being assigned category code 5.
It is what TeX does to that end-of-line character which explains our quandary regarding single lines of Lua code. When a TeX engine processes a line of input it will, eventually, detect the last character in that line: the character defined by `\endlinechar`. Usually, that character has category code 5 which causes TeX to *replace it* with a space character: i.e., at the end-of-lines TeX, in effect, strips out its line-termination character and replaces it with a space. As a side note, TeX engines also use characters with category code 5 to detect blank lines and start a new paragraph, but we won’t address that here.
Of course, being TeX, you can perform all sorts of special macro programming tricks by re-setting the `\endlinechar` to some other character, and/or giving the character assigned to `\endlinechar` a category code value of your choice.
If you want to prevent Lua code becoming one single line of text you can either (temporarily) change the value assigned to `\endlinechar` or change the category code of the standard end-of-line terminator `\r`.
### TeX’s bizarre ^^ notation
In the following sections we will encounter TeX’s unusual `^^` notation, which is known as the “extended character mechanism”. It was designed by Knuth as a way to facilitate typing “control characters” such as end-of-line terminators, tabs and so forth. For example:
* `^^J` represents character code 10 (`\n`, line feed);
* `^^M` represents character code 13 (`\r`, carriage return).
Character sequences such as `^^M` are converted to their corresponding character codes early on in TeX’s input-scanning process, when TeX is reading input characters to generate the corresponding character tokens.
### Changing the character assigned to \endlinechar
Remembering that we still need to prevent expansion of the `~` character, we can write
```
\begingroup
\endlinechar=10 % Change the end-of-line character to \n
\directlua{
local x=3
if x \noexpand~= 4 then
print("x is not equal to 4")
end
}% don’t want the \n appearing here
\endgroup% or a \n here
```
The above setting for `\endlinechar` causes LuaTeX to append character code 10 (`\n`, line feed) to the end of each line it reads in. We do this because `\n` (line feed) usually has category code 12, which you can test by writing ``\the\catcode`\^^J``. Because `\n` does not have category code 5, LuaTeX won’t convert it to a space character so it remains at the end of every line read-in by LuaTeX. This results in a character with code 10 remaining at the end of every line, thus making it through into the token list being built by `\directlua` and subsequently reappearing in the Lua code once the token list is converted to text. With the above change, the Lua code is sent to the Lua interpreter as the following sequence of characters:
**\n**local x=3\*\*\n**if x \~= 4 then**\n**print("x is not equal to 4")**\n**end**\n\*\*
where the **\n** notation is meant to represent character code 10 *not* some unknown macro `\n`. Now, the Lua interpreter will see line breaks in the code, exactly as it was originally written in the `\directlua` command:
```
local x=3
if x ~= 4 then
print("x is not equal to 4")
end
```
Incidentally, note that the very first character in the Lua code string is `\n` (before the `local` keyword). That `\n` arises from the line
`\directlua{`
because there is a line break immediately after the opening `{` and this too is preserved. To prevent it you can write
`\directlua{%`
### Changing the category code of \r
To maintain line breaks in our Lua code we can also change the category code of `\r` to something other than 5, so that `\r` is no longer recognized (treated as) an end-of-line character. With this technique LuaTeX still uses `\endlinechar=13` and will continue to add a `\r` to the end of each line; however, because `\r` no longer has category code 5, LuaTeX will not recognize the `\r` character as an end-of-line: it will not convert it to a space and passes it through unscathed to appear in the Lua code.
Remembering that we still need to prevent expansion of the `~` character, we can write
```
\begingroup
\catcode`\^^M=12 % change category code of \r to 12
\directlua{
local x=3
if x \noexpand~= 4 then
print("x is not equal to 4")
end
}
\endgroup
```
In this instance the Lua code is sent to the Lua interpreter as:
**\r**local x=3\*\*\r**if x \~= 4 then**\r**print("x is not equal to 4")**\r**end**\r\*\*
where the `\r` notation is meant to represent character code 13 not some unknown macro `\r`. As with the `\endlinechar` example, the Lua interpreter will see now line breaks in the code, exactly as it was originally written in the `\directlua` command:
```
local x=3
if x ~= 4 then
print("x is not equal to 4")
end
```
Incidentally, note again that the very first character in the Lua code string is `\r` (before the local keyword): this too arises from the line
`\directlua{`
#### Why did \r use category code 12 but not category code 11?
The answer is due to the risk of accidentally introducing errors triggered by `\r` (of category code 11) being added to the end of TeX/LaTeX commands read from our input file. Take this example:
```
\begingroup
\catcode`\^^M=11 % change category code of \r to 11
\directlua{
local x=3
if x \noexpand~= 4 then
print("x is not equal to 4")
end
}
\endgroup
```
which generates an error:
```
! Undefined control sequence.
l.9 \endgroup
```
How can that be true because `\endgroup` is a standard TeX primitive command? The cause of the error is quite subtle: When LuaTeX read the last line of text—the one containing `\endgroup`—it also added the `\endlinechar` character `\r` to the end of that line. Now, inside its memory, LuaTeX sees the character sequence
`\endgroup\r`
where we use `\r` to indicate the character with code 13—not the name of some unknown TeX macro `\r`.
At the time LuaTeX read this line from our text file the original `\begingroup` is still operational: we are inside a group that has not yet been closed by executing the matching `\endgroup` command—which would cause `\r` to revert back to its previous category code value of 5.
When LuaTeX begins to process (create tokens) from the line of text `\endgroup\r` it recognizes the first character `\` as the escape character which triggers LuaTeX to start looking for the name of a command. To identify a command name LuaTeX looks for a sequence of characters with category code 11 but because `\r` also has category code 11 LuaTeX thinks the `\r` character (still with category code 11) forms *part of a command* named `\endgroup\r` which, of course, does not exist so LuaTeX reports an `Undefined control sequence` error. That’s why we used category code 12 and not 11.
Because LuaTeX’s error message was written to the console we could not easily see/notice the `\r` character so it was not obvious what had caused the error.
### Why are we worrying about end-of-lines?
The reason is to enable use of Lua’s commenting method in your code! You can use LuaTeX’s standard mechanism of adding `%` characters to comment-out single lines within your code; however, the Lua language has its own, very useful, *multi-line* commenting mechanisms that you might want to take advantage of.
Let’s start by seeing what happens if we try to use single-line Lua language comments without addressing linebreak issues. Whereas TeX uses the `%` character to comment out single lines of code, Lua uses a double hyphen: `--`.
What happens if we try to run this:
```
\directlua{
local x=3
if x \noexpand~= 4 then
-- I'm going to output the result of this complex test
print("x is not equal to 4")
end
}
```
We get an error:
`[\directlua]:1: 'end' expected near `
This error is caused by the absence of linebreaks in the Lua code passed to the interpreter, which sees only one single continuous string in which the comment starts part-way into that string:
```
local x=3 if x ~= 4 then -- I'm going to output the result of this complex test print("x is not equal to 4") end
```
Everything after `**local x=3 if x ~= 4 then**` is treated as being commented out which causes the interpreter to see an incomplete chunk of Lua code, resulting in the error
`'end' expected near `.
where `` means end of file.
As you have probably guessed, we must remedy this by ensuring line breaks are transmitted through to the resulting Lua code, which we can accomplish by, for example, by changing the category code of `\r` to 12:
```
\begingroup
\catcode`\^^M=12 % change category code of \r to 12
\directlua{
local x=3
if x \noexpand~= 4 then
-- I’m going to output the result of this complex test
print("x is not equal to 4")
end
}
\endgroup
```
Now, the Lua interpreter sees a string but it contains `\r` line breaks as written in the `\directlua` fragment:
**\r**local x=3\*\*\r**if x \~= 4 then**\r\*\*-- I'm going to output the result of this complex test\*\*\r**tex.print("x is not equal to 4")**\r**end**\r\*\*
This, in effect, is equivalent to writing
```
local x=3
if x \noexpand~= 4 then
-- I’m going to output the result of this complex test
print("x is not equal to 4")
end
```
which means Lua is able to process this code correctly and ignore the line we commented out.
**Block comments**
The Lua language also supports a syntax it calls [“block comment”](https://www.lua.org/pil/1.3.html) (or *long comment*): these start with `--[[` and are in effect until the corresponding `]]`. We can use this convenient syntax to write multi-line comments, or comment out sections of code we want to temporarily remove:
```
\begingroup
\catcode`\^^M=12 % change category code of \r to 12
\directlua{
local x=3
if x \noexpand~= 4 then
--[[ I’m going to output the result of this complex test
simply because it really is
such an amazing conclusion]]
print("x is not equal to 4")
end
}
\endgroup
```
## In conclusion
Firstly, congratulations if you have managed to read through this substantial article! We have tried to produce a reasonably comprehensive guide to TeX-related concepts and topics which provide the background needed to get the most from LuaTeX via the `\directlua` command. It is our hope to have produced an article which is instructive and contributes something of use and value to the Overleaf user community, and beyond. As always, we are delighted to receive feedback so do please feel free to [contact us](https://www.overleaf.com/contact) with comments on this article or suggestions for further topics you would like us to write about.
Happy $$\text{Lua}\mathrm{\TeX}\text{-ing!}$$ from Graham Douglas and the Overleaf team.
### And finally... just use the luacode package
Although TeX and Lua operate in fundamentally different ways, those languages share a number of characters that have “special meanings” within the context of each language—such as \\, %, \~, #, ^, &—of course, Lua and TeX assign those special meanings for *very* different purposes. Our exploration of problematic characters shows why difficulties can arise and how you can resolve them; however, it could be rather tedious to manually fix many small Lua code fragments so most users prefer to use LaTeX packages which remove those challenges. One such package is [`luacode`](https://ctan.org/pkg/luacode?lang=en) which provides a suite of features designed to simplify working with `\directlua`, but at least you may now have a better understanding of the issues `luacode` solves for you.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://overleaf-pro.ayaka.space/latex/in-depth-articles/09-an-introduction-to-luatex-part-2-understanding-directlua.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.