Chapter 2

Lexical Structure

This chapter defines how grammar-file text is divided into characters, line terminators, white space, comments, identifiers, keywords, literals, escape sequences, and punctuation. The structure built from these tokens is described in Grammar Files; the complete syntax is summarized in Grammar Syntax.

#Source Text

A grammar file is Unicode text encoded as UTF-8. A character outside a comment, string literal, or character class that is neither white space nor the start of a token is an error. The diagnostic includes following characters up to the next white space or token. Grammar errors use 1-based line and character-column positions; a tab counts as one character, not its byte width. Positions in parsed input are defined separately in Positions.

#Line Terminators

A line ends at LF (U+000A), CRLF, or CR (U+000D). A line terminator ends a line comment and determines diagnostic line numbers. LF MUST NOT occur inside a string literal or character class. Line terminators otherwise count as white space.

#White Space

White space is any Unicode character with the White_Space property, including line terminators. It separates tokens and is insignificant except where this chapter requires adjacency.

#Comments

// outside a string or character data starts a line comment through, but not including, the line terminator. /* starts a nested block comment, which ends at its matching */. Comment markers inside strings or character data are ordinary content. The parser records comment trivia and may accept it as syntactic trivia; semantic analysis reports a positioned error when a comment occurs in a Declaration Statement or Operators Rule statement header. Comments are allowed at top-level statement boundaries, at statement and field boundaries inside braces, and freely inside Rule Expressions and Action Expressions, including predicates and function arguments.

rule digits = [0-9]+ // one or more digits
/* outer comment /* nested comment */ continues */

#Tokens

Tokens are identifiers, string literals, character classes, integer literals, capture references, and punctuation. They are formed from left to right using the longest token available at each position.

#Identifiers

An identifier matches [a-zA-Z_][a-zA-Z0-9_]*. Identifiers are case-sensitive and have no length limit. Identifier syntax is ASCII-only: non-ASCII letters and digits do not continue an identifier. Unicode remains available in string literals, character classes, comments, and parsed input. A Rule Definition name and a reference to a rule MAY begin with _: such a rule is hidden (see Hidden rules). Every other name beginning with _ is reserved and MUST NOT be used as an identifier, including type and variable names, captures, level names, attribute names and named arguments. Parsing preserves these names in the source AST; semantic validation rejects their use before compilation. The standalone primary Rule Expression form _ is the Top expression, not an available name. The words true, false and nil are the literals of the action language. A rule, type, type alias, variable, capture, or lambda parameter MUST NOT be named true, false or nil, because an action reading the name would get the literal instead. Validation rejects such a declaration with an error such as rule name true is reserved; it is a literal, and a lambda parameter with such a name is a syntax error. Longer names that merely begin with a literal, such as truely, are ordinary identifiers. Identifiers name rules, types, fields, captures, variables, levels, attributes, and lambda parameters. The initial ASCII letter's case matters for type and field names; see Names.

#Contextual Words

The words infix, level, operand, package, postfix, prefix, rule, skip, struct, terminal, type, and var are contextual words, not globally reserved identifiers. They are interpreted as syntax only where the grammar expects a Declaration Statement or Operators Rule item. In a name position, rule reference, capture, field, variable, or expression, they remain ordinary identifiers. No Declaration Statement name is excluded solely because it is one of these words.

The following identifiers are also contextual:

#String Literals

A string literal is enclosed in double quotes and matches exactly its decoded characters. It may contain a literal CR, but MUST NOT contain a literal LF. The quote and backslash are written as \" and \\ respectively. String literals may also be values in actions.

"if" "\n" "say \"hi\"" "😀"

A string literal MAY be followed directly by the suffix i, written "select"i. The suffix makes the ASCII letters of the literal match in either case. The i MUST immediately follow the closing quote, with no space between them, and no identifier character may follow it: in "x" i the i is a rule reference, and in "x"ia the identifier ia is a rule reference. Folding is ASCII only: A-Z and a-z match each other, and every other character matches only itself, so "é"i matches é and not É. A suffix inside the #ignorecase pragma is redundant and allowed. An expectation shows a case-insensitive literal with its suffix, as written: "select"i.

#Character Classes

A character class matches one Unicode character selected by its contents. Its matching rules are defined in Character Classes. Literal characters, ranges, negation and escapes are defined there. [ begins a character class.

[a-z_] [^"\\] [\-()?]

#Integer Literals

An integer literal contains one or more ASCII decimal digits (0–9). It is used for repetition bounds and in actions and predicates. A value outside the range of the implementation's int is an error. Write negative action values with unary -. The signed minimum int is also accepted as a single value: its positive magnitude cannot be a separate integer token. This preserves the full native integer range in generic attribute arguments.

"a" #repeat(2, 3) {len($s) > 4}

#Capture References

$ immediately followed by an identifier forms a capture reference ($label) in actions and predicates. $ followed by ASCII decimal digits is a removed positional reference ($1): it is still a token, so that $1 is never read as an anchor followed by a number, but the parser rejects it. $ followed by any other character is the end-of-line anchor; $$ is the end-of-input anchor.

#Punctuation

Punctuation tokens include:

_|_  ^^  $$  --  =>  ==  !=  <=  >=  &&  ||  :=
:  =  /  (  )  {  }  [  ]  ,  ;  .  *  +  ?  &  !  @  -  ^  $  #  |  <  >  %  //  /*  */

#Escape Sequences

Strings and character classes accept these escapes:

A hexadecimal escape MUST denote a Unicode scalar value from U+0000 through U+10FFFF, excluding U+D800–U+DFFF. Invalid hexadecimal text, surrogates, and values above U+10FFFF are errors. Adjacent surrogate escapes do not combine; write the scalar directly, such as \u{1F600}. Unicode noncharacters such as U+FFFF are valid scalar values.

#Adjacency

An attribute name MUST immediately follow @, and an attribute argument list MUST immediately follow its name: @name or @name(...). White space or comments between @ and the name are syntax errors. A separate group after @name (expr) is a following Rule Expression, not an argument list.

A pragma name MUST immediately follow #, and its argument list MUST immediately follow its name. White space or comments within #repeat(...) or #recover(...) are allowed inside the parentheses, but not between # and the name or between the name and (. For the argument-free #atomic and #stream forms, a separate group after the name is a following Rule Expression, just as it is after an argument-free attribute.

These forms require adjacency; intervening white space or comments change the parse:

The lexer takes the longest punctuation token. Thus -- is one reserved token, $$ is one end-of-input anchor, and $ followed by a letter, _, or digit begins a capture reference. A { begins a Predicate Action where a primary Rule Expression form is expected; [] remains a character class in Rule Expressions and a type-list prefix in Type Expressions. @ attributes, # pragmas and comment markers are recognized outside shielded string and character data.

#Limits

An implementation MAY limit source nesting depth and reported errors. The reference implementation reports at most 100 errors, with the last indicating that more exist, and rejects parenthesis nesting deeper than about 500 levels.