Running Parsers

Turn a grammar into a parser in Go, call it, and read what comes back. This guide covers the pipeline from source to parser, start rules, results and errors, concurrency and how to choose among the features of the engine. Cancellation and work limits, parsing a prefix, position units, backends and recognition mode have guides of their own. It assumes you can write a grammar; the language specification defines the language, and Parsing defines what a parse returns.

All examples use one small grammar, pairs.pego, that reads key=value pairs separated by optional semicolons:

type Pair = { Key node[string]; Value node[string] }

@start(default)
rule main = ws ps:pair+ $$ => $ps
rule pair Pair = k:key "=" v:value ";"? ws => Pair{Key: $k, Value: $v}
@start
rule key = [a-zあ-ん]+ #atomic
rule value = [0-9]+ #atomic
rule ws = [ \t\n]*

Install the library and the command-line tool with:

go get github.com/ornew/pego
go install github.com/ornew/pego/cmd/pego@latest

From a checkout of the repository, go run ./cmd/pego ... is equivalent to pego ....

#The Pipeline

 PEGO source (.pego) ──ParseGrammar──▶ grammar AST ──Compile──▶ *pego.Parser ──Parse──▶ *pego.Node
 grammar JSON (.json) ──UnmarshalJSON─▶     ▲                        │  ▲
 grammar package (Go values) ───────────────┘             MarshalBinary  LoadParser
                                                                     ▼  │
                                                              compiled grammar (.pegoc)

• The grammar AST (package grammar) is the in-memory form of a grammar. It can come from PEGO source, from JSON, or be built by hand.
• Compiling analyzes the AST, type-checks it (reporting grammar errors), and prepares it for execution. A *pego.Parser is the result: a compiled grammar plus a start rule.
• A compiled grammar (.pegoc) is the saved bytecode of a Parser for one mode of parsing. Loading one skips parsing, analysis and type checking, and gives a parser without the grammar AST (Compiled Grammars).
• A standalone Go parser generated by pego gen bypasses all of this at run time. See Generating Parsers.

#Compiling a Grammar

#From Source

pego.CompileSource(src, start) does everything in one call:

package main

import (
	_ "embed"
	"fmt"
	"log"

	"github.com/ornew/pego"
)

//go:embed pairs.pego
var src string

func main() {
	p, err := pego.CompileSource(src, "main")
	if err != nil {
		log.Fatal(err)
	}
	node, err := p.Parse("abc=12; あい=3")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(node)
}

[Pair{Key: "abc", Value: "12"} Pair{Key: "あい", Value: "3"}]

Embedding the grammar with go:embed keeps it in a real .pego file that pego fmt and pego parse can also use.

Compile once and keep the *pego.Parser. Compilation is the expensive step, and a Parser is cheap to call.

Errors in the grammar (syntax errors, undefined rules, type errors) are reported by CompileSource, with positions when the grammar came from source:

_, err = pego.CompileSource("@start\nrule main = (", "main")
fmt.Println(err) // A source position and syntax error.
_, err = pego.CompileSource("@start\nrule main =  other ", "main")
fmt.Println(err) // A source position and undefined-rule error.

#From an AST

CompileSource is ParseGrammar followed by Compile. Use the two steps when you want to look at or change the AST (for example, to list the rules with g.Rules() or to reformat with grammar.Format(g)) before compiling:

g, err := pego.ParseGrammar(src)
if err != nil {
	log.Fatal(err)
}
fmt.Println(len(g.Rules()), "rules")
p, err := pego.Compile(g, "main")

#From Go Values (Programmatic Grammars)

The grammar package defines the AST, so a grammar can also be built without any PEGO source: for a tool that generates grammars, or to embed a tiny grammar in a test. This builds the equivalent of rule main = word $$ and rule word = [a-z]+ #atomic :

g := &grammar.Grammar{File: grammar.File{Decls: []grammar.Decl{
	&grammar.RuleDef{
		Name: "main",
		Expr: &grammar.Seq{Items: []grammar.Expr{
			&grammar.Ref{Name: "word"},
			&grammar.EndInput{},
		}},
	},
	&grammar.RuleDef{
		Name: "word",
		Expr: &grammar.Atomic{Expr: &grammar.Repeat{
			Expr: &grammar.CharClass{Ranges: []grammar.CharRange{{Lo: 'a', Hi: 'z'}}},
			Min:  1,
			Max:  -1, // no upper bound
		}},
	},
}}}
fmt.Print(grammar.Format(g))

p, err := pego.Compile(g, "main")
if err != nil {
	log.Fatal(err)
}
n, err := p.Parse("hello")
fmt.Println(n, err)

@start
rule main =  word $$

rule word =  [a-z]+ #atomic
("hello") <nil>

grammar.Format prints an AST as PEGO source, which is a quick way to check what you built. A hand-built grammar goes through the same analysis and type checking as one parsed from source, so mistakes are reported by Compile; the errors have no line numbers because there is no source:

bad := &grammar.Grammar{File: grammar.File{Decls: []grammar.Decl{
	&grammar.RuleDef{Name: "main", Expr: &grammar.Ref{Name: "missing"}},
}}}
_, err = pego.Compile(bad, "main")
fmt.Println(err) // undefined rule missing

An AST can be exchanged as JSON with grammar.MarshalJSON and grammar.UnmarshalJSON, which round-trip the grammar (grammar.Format of the decoded grammar equals the original). The command line does the same with pego convert.

#Choosing a Start Rule

Only the rules marked @start are start rules, and an empty name selects the grammar's default start rule. The second argument of Compile and CompileSource names one:

_, err = pego.CompileSource(src, "nope")
fmt.Println(err) // rule nope is not defined
_, err = pego.CompileSource(src, "pair")
fmt.Println(err) // rule pair is not a start rule; parsing can begin only at a rule marked @start

Parser.WithStart(name) returns a parser for another start rule, and WithStart("") one for the default start rule. It shares the compiled grammar, so it costs almost nothing, and it is how you expose several entry points of one grammar (for instance expr for an expression evaluator and main for a whole file):

keyParser, err := p.WithStart("key")
if err != nil {
	log.Fatal(err)
}
n, err := keyParser.Parse("abc")
fmt.Println(n, err) // "abc" <nil>

Parser.Parse always has to consume the whole input. The start rule does not need an $$ of its own: if the rule matches only a prefix, the parse fails with end of input among the expected items at the first unconsumed position:

_, err = keyParser.Parse("abc=1")
fmt.Println(err) // 1:4: syntax error: expected [a-zあ-ん], end of input

To accept a named rule's prefix and get its consumed code-point and byte lengths, use Parser.ParsePrefix. It selects a rule per call without changing the parser's start rule.

On the command line, -s selects the start rule for parse, compile and gen (default: the one saved in a .pegoc, otherwise the grammar's default start rule).

#Parsing, Errors and Concurrency

func (p *Parser) Parse(input string, opts ...ParseOption) (*Node, error)

Parse returns the value of the start rule, or an error:

_, err = p.Parse("abc=x")
var se *pego.SyntaxError
if errors.As(err, &se) {
	fmt.Println(se.Pos, se.Line, se.Col, se.Expected, se.Messages)
	fmt.Println(err)
}

4 1 5 [[0-9]] []
1:5: syntax error: expected [0-9]

Pos is the offset in the position unit, Line and Col are 1-based, Expected lists what would have been accepted, and Messages holds the messages set with @error. se.Message() is the text without the position. Use @diagnostic to give an expression a concise expected-item name while preserving its failure position and explicit @error messages.

A *pego.Parser is safe for concurrent use: share one among goroutines and call Parse on all of them (each call has its own state). A Document, in contrast, is not. Concurrent first use of different backends and of recognition mode is also safe. A call can be cancelled and bounded with WithContext and WithWorkLimits; see Cancellation and Work Limits.

#Compiled Grammars

A compiled grammar is the saved bytecode of a *pego.Parser for one mode: parse for Parse, ParsePrefix and ParseStream, recognize for RecognizeOnly, or document for documents, traces and profiles. Loading one skips parsing, analysis, type checking and compilation, so a program starts faster and a grammar can ship as data. p.MarshalBinary() saves the parse mode, p.Marshal(mode) any mode, and pego.LoadParser restores it; on the command line, pego compile -mode writes the file and parse, trace, profile and explain accept it. A loaded parser runs only on the bytecode backends and only the operations of its mode. The Compiled Grammars guide covers the modes, sizes, embedding, versions and loading files you did not produce.

#Related Topics

• Generating Parsers, Incremental Parsing and Streaming: the other ways to run a grammar, when a plain Parse does not fit.
• Cancellation and Work Limits, Parsing a Prefix, Position Units and Choosing a Backend: the options of a parse.
• Reading Trees: what to do with the result.
• Error Messages: reporting syntax errors.
• Recovering from Errors: parsing on after a syntax error.