Chapter 4

Streaming

A stream parse reads its input from an io.Reader and hands each element of a #stream repetition to the caller as soon as the element matches. The parser then forgets the element and the input before it, so the memory a parse retains depends on the size of one element and not on the size of the input. This chapter defines the grammar construct, the Go entry points and their results. The #stream pragma itself is listed in Pragmas.

#Scope and Terminology

An element is the value that one iteration of a #stream repetition contributes to the list of an ordinary parse: the value of the rule if the element is a rule with an action, and a node (a string, tuple or list node) otherwise. An element whose expression has no valued element, for example (-"a")* #stream, is the unit ().

A stream parse is a parse started with Parser.ParseStream or Parser.Iterate, or with pego parse -stream. The streaming repetition is the #stream repetition of the start rule.

#Contract

#Grammar

#Entry Points

func (p *Parser) ParseStream(r io.Reader, emit func(*Node) error, opts ...ParseOption) error
func (p *Parser) Iterate(r io.Reader, opts ...ParseOption) (*StreamIterator, error)

ParseStream calls emit once per element, in input order, and returns only an error: it returns no tree. The values of whatever precedes or follows the streaming repetition (a header, a footer) are matched and checked and then discarded. The start rule MUST contain a streaming repetition; otherwise ParseStream returns the error rule main has no #stream repetition (with the start rule's name), even if a rule that the start rule calls has one. Another rule streams when it is made the start rule with Parser.WithStart.

pego parse -stream prints each element on its own line (JSON by default, an S-expression with -f sexpr) as it is delivered, then reports the final error, if any. It cannot be combined with -check.

Iterate is the pull form. It validates the entry point without reading. StreamIterator.Next advances the parse to one committed element or to the terminal result and returns (node, ok, err); ok distinguishes a successful nil element from the end of the stream. After completion, repeated calls return the same terminal error or a clean end. Close is idempotent; after an early Close, Next returns ErrStreamClosed. The iterator does not close its reader. One goroutine owns an iterator: calls to Next and Close MUST NOT overlap, including from a callback. Close cannot interrupt a blocked Read.

#Delivery and Finality

• Each element is delivered as soon as it matches, in the middle of the parse and before the next element is tried. The parser reads nothing from the reader while emit runs.
• An element is final. Once emit has seen it, the parser never backtracks to before it. What the caller receives is provisional until the parse returns a nil error, because input after the last element can still be invalid.
• The parser reads input only as far as needed. An element that ends in a fixed literal is delivered when its last character arrives. An element that ends in a repetition (+, *, ?) is delivered after the parser has seen one further character, because that character shows that the repetition ended. This holds in both position units: decoding an ASCII character or a complete UTF-8 sequence does not wait for unrelated later bytes. An incomplete UTF-8 prefix waits for the remaining bytes or for the reader's end of input.
• PEG lookahead and buffering MAY cause the parser to read beyond the element that a Next call returns.

#Positions and Text

Node.Start, Node.End, .Start and .End in actions, and SyntaxError.Pos, Line and Col count from the start of the whole input, regardless of how much has been discarded; the parser keeps a running line and column for the discarded part. The position unit option changes all of them to bytes.

The parser keeps no text for the caller. String nodes (literals, classes, (...) #atomic) carry their text in Node.Text; tuple and list nodes do not. text(...) in an action of the start rule, which runs after elements have been discarded, returns only the part of the node's input that is still held, possibly the empty string.

After an element is delivered, the input before it is discarded except for the one character before it, so ^ (beginning of line) still holds at the start of the next element. No construct of the language looks further back.

#Options

A Parser supports any number of concurrent stream parses; each has its own state.

#Validation and Errors

The following are compile-time errors for #stream. A grammar can report more than one, and the order in which they are reported is not specified.

• #stream is only allowed at the top level of a rule body
• a rule body can have only one #stream
• #recover cannot wrap a #stream repetition; recover inside each element instead
• #stream must be attached to a repetition (*, +, or #repeat(...))
• #stream takes no arguments. Source text cannot express this case, because #stream(1) is a syntax error; it arises only in a grammar AST or JSON.

A stream parse ends in one of the following ways.

#Recovered Errors

With #recover inside an element, a bad element becomes an error value that is delivered like any other element, and parsing continues. At the end the parse returns the recorded errors as SyntaxErrors, a result that means success with diagnostics and is distinguished from a fatal *SyntaxError by errors.As. By default all recovered errors are retained until the end of the input.

WithDiagnosticCallback(func(*SyntaxError) error) receives each committed recovery immediately before the element it belongs to; Next invokes it while advancing to that element. Recoveries that were backtracked are not reported. The callback receives a copy that it may keep or modify. If it returns an error, parsing stops with that error and the element is not delivered. Recoveries in a successful suffix are reported before the terminal result.

WithDiagnosticLimit(n) retains only the latest n committed recoveries, in order of occurrence. A positive limit bounds the retained tail, while the callback still receives every committed recovery for as long as it returns nil. If errors were dropped, the terminal error is *TruncatedSyntaxErrors with the fields Errors, Total, Dropped and CountsSaturated; its Unwrap exposes the retained errors as SyntaxErrors. Counts saturate and do not wrap; when CountsSaturated is true, Total and Dropped are lower bounds. If the total does not exceed the limit, the terminal error is an ordinary SyntaxErrors. A limit of zero keeps the unbounded behavior. A negative limit is invalid. The two diagnostic options are rejected by Parse, ParsePrefix and Document; any value, including zero, is a stream-only option. A limit does not bound what the callback stores. With tracing enabled and a positive limit, the successful start-rule exit event exposes the retained tail through TraceEvent.Recovered(); only the terminal TruncatedSyntaxErrors has the total and dropped counts.

#Memory

An element is matched in memory as a whole, so the memory held for the input and the tree of an element depends on the size of that element. Once the callback has returned, the parser keeps no reference to the element, and the input and memo entries before it are released as parsing continues, so the memory held does not grow with the number of elements delivered. Recovered errors are retained until the end of the input unless a diagnostic limit is set. An action that returns nil delivers nil to the caller. Variable values, active rollback points and memoization keys keep their own data alive.

#Compatibility and Support

Streaming is supported by the engine on all three backends. Generated Go parsers provide ParseStream and Iterate when generated with streaming enabled, and typed variants with typed values; see Generated Parsers. Generated TypeScript parsers have no stream entry point and treat #stream as an ordinary repetition. A Document is always parsed as a whole and does not stream. Grammars that use #missing cannot be used in a stream parse.

#Conformance

The behavior is established by the stream tests of the repository (stream_iterator_test.go, stream_diagnostics_test.go, stream_diagnostic_tail_test.go) and by the backend equivalence tests, which require identical elements and errors from every backend.