Development

This document describes the repository layout and architecture, how to run tests, and the implementation status of language features. The living backlog is maintained in Issue #1.

#Directory Layout

#Architecture

.pego ──syntax.Parse──▶ grammar.Grammar ──engine.Compile──▶ engine.Program ──Parse──▶ Node
                         ▲      │                              │      ▲
              JSON ──────┘      └── grammar.Format ──▶ .pego   │      └── LoadProgram ── .pegoc
                                                               └── MarshalBinary ──▶ .pegoc

Files without a directory are in internal/engine/. There, a file's prefix names the part it belongs to: closure_ for the closure backend, vm and ivm for the recursive and iterative bytecode VMs, document_ for Documents and program for the entry points of a program (compilation, compiled grammars and the parse drivers). Files with the prefixes input_, stream_ and work_ and the files without a prefix hold what every backend shares: the parser state, the built-ins of actions (eval.go, filter.go) and the types that the API exposes from the compiler and the tracer (compile_api.go, trace.go). The exceptions are profile.go and node_cursor.go, which read parse results. Tests that compare generated parsers with the engine are named after the generator they check: gogen_ for Go, tsgen_ for TypeScript and gen_ for both. Optimization evidence is in the catalog; contracts and proposals are in spec/ and docs/design/records/.

#Runtime

The closure compiler is the reference execution path. The recursive and iterative VMs share rule-call, memoization, left-recursion and Operators Rule logic with it. Backtracking restores input position, the variable environment, recovered errors and capture history; farthest-failure expectations use separate isolation and merge scopes. Recognition returns no tree and skips value and action work that predicates do not need; predicate-dependent rules and callees can still evaluate actions. Adaptive memoization enters eager mode when repeated calls exceed one sixteenth of a rule's calls. Its compact counters halve before overflow, and the eager decision remains sticky for that parse. Nesting limits and tracing apply to rule calls. Document can reuse memo entries across edits and resume long repetitions; stream parsing emits elements and discards input already committed by the grammar. Core and generated Go/TypeScript parsing also support invocation-local cancellation and input-work/execution-step limits; streams and Documents use the same interruption contract. See design 033 for backend scope, cleanup and measured costs.

#Code Generation

Generated Go and TypeScript parsers embed their runtime and depend only on their standard libraries. Generated Go is printed from Flow IR (design 062): lean value-free rules as direct rules and the others as framed bodies the runtime calls; typed Go also emits inferred Go types and ParseAST/ParseRuleAST for configured-start and named rules (design 029). Generation defaults (MaxDepth of gen.Go and gen.TypeScript, gen -max-depth) and invocation-local depth options are implemented (design 022); Go entry points accept functional ParseOptions, and TypeScript adds options-aware entry points. With Types or -types, TypeScript exports grammar declarations and configured-start parseAST conversion (guide, design 030). Generated paths preserve the engine's tree, error, depth, action and memo semantics.

Generated Go -document can be combined with -types; Document.ParseAST directly matches and reuses typed memo entries when the grammar meets the batch typed runtime's eligibility rules. Other result types use Node conversion. Returned typed graphs are cloned per invocation, while CST Node members may share the mutable Document graph. See the incremental parsing guide and design 044.

Generated Go -stream adds a Node ParseStream that follows the engine's Reader stream contract, and with -types a ParseStreamAST that converts each element; it cannot yet be combined with TypeScript. TestGeneratedTypedStreamMatchesBatch compares the typed elements with batch ParseAST. Stream packages also have Iterate and the diagnostic options, which the engine differential test runs as well. TestGeneratedStreamMatchesEngine compares emitted elements, errors and reader call counts with the engine across chunkings, units, controls, observability and Document generation; changes to the stream runtime should keep it and the bounded-memory consumer in TestGeneratedStreamAPI passing.

Generated Go typed-output cleanup is not implemented; the measured full-sweep and bounded candidates and their limits are in design 034 and optimization record 094.

#Testing

go test ./...

The tests of internal/engine that build generated parsers build them in two forms. A parser generated with work controls charges work and accepts the options WithContext and WithWorkLimits. A parser generated without them is the form that most parsers have, and a different part of the generator prints it. Every test runs the form that it exists to check:

A test that builds a fixture without any of these fails the run when it finishes, because it would check one form only. go test ./... therefore covers both forms and needs no setting.

PEGO_TEST_ORDINARY restricts a run to one form: 0 runs everything that builds fixtures with work controls, and 1 runs it without, skipping the tests that pass a control. Use it to reproduce a failure of one form; it does not widen coverage.

Engine tests write grammars in PEGO source and compare results using the S-expression form of nodes (Node.String). The check helper also verifies that the result is the same with memoization disabled, with every backend (closure, recursive bytecode, iterative bytecode), with both position units, in recognition-only mode, and with tracing on.

internal/engine/gogen_pool_lifecycle_test.go checks generated Node and typed parser cleanup after oversized inputs, retained results, and error/retry reuse in both position units.

The TypeScript generator's test needs Node.js 22.18 or later (node) and, for its type check, tsc; it is skipped when they are missing. The generated TypeScript has the same two forms, chosen by -work-controls. The TypeScript tests generate the form without work controls unless the test passes a control option (signal, maxInputWork, maxSteps), which generates with WorkControls. TestGeneratedTSParsersMatchEngine compares both forms with the engine, and tsgen_work_forms_test.go checks that the plain form has no charge, no control option and no cancellation export, and that the controlled form accepts the options. TypeScript is printed from the same Flow IR as Go (internal/gen/tsgen/flow.go, internal/gen/tsgen/outline.go); TestGeneratedTSPrintsLeanRulesFromFlow and the frozen decision goldens cover what is chosen, and a rule that Flow IR does not print fails generation.

lint tests every check with positive and negative cases, lints every example grammar against a hand-checked list of findings, and TestSoundness checks the certain findings against the engine on random grammars, exhaustively over short inputs (-soundness=N for a longer run).

internal/lsp tests talk to the server in process over pipes (lifecycle, malformed messages, incremental edits with non-ASCII and astral characters and \r\n, every request) and analyze every example grammar. In editors/vscode, npm install && npm run check compiles the TextMate grammar's regular expressions with Oniguruma and checks the scopes of a sample, and tokenizes the example and ready-made parser grammars; npm run check-server -- <path to pego> talks to the server with the VS Code client's JSON-RPC library.

sample tests generate inputs for every example grammar and every grammar of the engine's test corpus (internal/engine/sample_test.go, through export_test.go) and parse them on every backend.

go generate ./spec/language renders the simple grammar view from pego.pego and spec/language/syntax.md.tmpl; edit the template rather than the generated page. spec/language/source_grammar_test.go verifies the simplified display against the canonical source grammar, checks concrete trees and recovery across backends, and builds a generated typed Go consumer. spec/language/syntax_test.go runs the grammar of Grammar Syntax on every .pego file and every pego code block of the repository and requires it to agree with the parser of internal/syntax.

cd site && go test ./... builds the site and checks its pages and links; go test ./cmd/playground runs the WebAssembly smoke test when Node is installed. Production site builds retain the published current WASM/runtime pair alongside the new pair, using a bounded asset manifest and immutable deploy source. Workers select the matching hashed Go runtime; previews/local builds do not automatically read production assets. Hash/HTTP failures stop retention before local output is cleared. Design 023 describes one-generation retention, bootstrap and sequential publication limits; site/assets_test.go and worker_assets_test.mjs cover its build and worker contracts. Playground shared-link navigation applies every normalized option and tab even when grammar/input are unchanged; only parse-affecting changes schedule parsing, preserving editor scroll on tab-only navigation. Actual-app controlled DOM/client/timer tests cover each field, default restoration, missing start rules and named examples. URL encoding/decoding is guarded by the latest user action and adopted fragment, including navigation before event delivery, startup and example fetching. Superseded Share requests do not copy, older parse completions cannot save during incoming link loading, and landing links keep the latest input. Deferred-promise tests exercise these orderings and error paths; browser checks complement them with the real WebAssembly worker. The tree renderer (site/static/playground/tree.js) pages direct children in groups of 100, counts node/control rows within a 400-row initial budget and 5,000-row live cap, and keeps deferred nodes reachable by caret or page controls. Explicit expansion at the cap compacts materialized branches; paths that cannot fit use a subtree window with a return control. Controlled row-count, reference/focus and boundary tests cover these resource limits. The tree benchmark distinguishes equivalent small-tree work from wide-tree display reduction; real browser checks cover DOM/WASM wiring.

The same-source work-control reference build compiles and tests the engine package without core work-control instrumentation. After changing internal/engine test files, check that both builds compile with go vet ./internal/engine and go vet -tags pego_work_controls_reference ./internal/engine; test files that use helpers defined in untagged files keep both builds compiling.

GitHub Actions runs the root module and parsers/test.sh (including go vet for every parser module), site tests, and Node.js/WebAssembly/generated-TypeScript checks as separate jobs. A newer run on the same branch or pull-request ref cancels the superseded run. CI uses Go 1.27.1, Node.js 22.18.0, CPython 3.14.0 and TypeScript 5.9.3. Native tests have a 45-minute root timeout and a 30-minute per-parser timeout within a 90-minute job budget, allowing generated-consumer builds on CI hosts. The TypeScript differential harness uses the upstream microsoft/TypeScript v5.9.3 tests/cases corpus. Required-tool checks fail before tests if a required runtime or the TypeScript corpus is absent. Go tests run with -v so optional skips are visible in the retained 14-day logs. Python uses its installed standard library and recorded reference data; the live comparisons on a CPython checkout (PEGO_CPYTHON_SRC) and the DuckDB REFERENCE_FILE corpus are not supplied. DuckDB checks use recorded reference results, not a live DuckDB oracle. These jobs do not claim browser-engine or deployed-site validation.

#Repository Grammar Test Coverage

The guides show how to test a grammar in an application. The repository uses the same layers across its examples and parser corpus:

Root go test ./... covers the main module. Run parsers/test.sh to vet and test all standalone parser modules; cd site && go test ./... covers the site module.

#Third-Party Test Corpora

The conformance suites and reference data of the ready-made parsers use text from other projects (cel-spec, JSONTestSuite, yaml-test-suite, CPython, DuckDB and others). It is not in the repository: internal/corpora/manifest.json pins each source (archive URL at a fixed revision, license, the files kept, and the SHA-256 digest of those files), and go run ./hack/corpora fetch downloads what is missing into $PEGO_CORPORA (default: pego/corpora in the user cache directory) and checks the digests. Other subcommands: list, verify, dir NAME, and pin NAME, which prints the digest for a new manifest entry. The digest is over the extracted files (the lines of sha256sum, sorted by path), because GitHub does not promise identical archive bytes for a commit. fetch also checks the files that are already in the cache against the digest and fetches an entry again if they differ, so parsers/test.sh runs against verified data. All sources are archives at a full commit SHA except cel-go, which is the module zip of its release at proxy.golang.org: a module version is immutable and recorded in the Go checksum database.

go test never uses the network: a test that needs a corpus that is not fetched is skipped with the command to run. parsers/test.sh fetches first and stops if that fails, so a full run skips nothing for a missing corpus. Reference data kept in a parser's testdata refers to the upstream text by file, position and hash instead of containing it.

#Language Feature Status

#Roadmap

Issue #1