CONTEXT FREE LANGUAGES AND PARSING

Parsing

GRAMMARS

Teleport to 4114

Teleport to 4114

TYPE 3 Regular Expressions FSAs Again...

S → aB | λ A → bB | cC B → bB | bS C → cC | cS

((abb*)|(acc*))*

CONTEXT-FREE GRAMMARS 1

Every production is of form:
N → σ
N is a single non-terminal.
σ is a string of non-terminals and terminals.
A → %OK A → aBAcdE % OK Ab → aBAcdE % BAD: Context SENSITIVE LHS.

CONTEXT-FREE GRAMMARS

Allows recursion (needs stack, so PDA): Nonterminals defined in terms of themselves. Rule: Only single nonterminal, no terminals on LHS of productions. Supports structure like nested (()). G=(S,N,T,P): (Start, NonTerm, Term, Prods).
e.g. in BNF: expr → id | number | - expr | ( expr) | expr op expr op → + | - | * | / (note paren-matching trick). More powerful than RE: easy to do concat. and alternate, thus Kleene closure, but can also define things in terms of themselves, which RE's can't. BNF sometimes augmented with new useful symbols: e.g.
id-list → id (, id)*
(e.g. of new id, Kleene+, and note | is not needed.) Parser doesn't distinguish one terminal symbol (e.g. id) from another, but semantics has to, so scanner saves spellings.

CONTEXT-FREE GRAMMARS: Recursion

<NP> → the <Noun> <NP> → the <Noun> <NP> <Verb> the man
the boy the man saw
the dog the boy the man saw kicked
the flea the cat ... saw kicked chased painted shot scratched bit...

CFG ADVANTAGES

DERIVATIONS

Starting with grammar's start symbol and rewriting non-terminals using the rules lets you generate (derive) syntactically valid strings of terminals and non-terminals, called sentential forms, which are the yield of the derivation. Here's a rightmost or canonical derivation. expr → expr op expr → expr op id → expr + id → expr op expr + id → expr op id + id → expr * id + id → id * id + id (slope) * (x) + (intercept).

DERIVATION TYPES

Leftmost Derivation: replace leftmost non-terminal at each step.

Rightmost Derivation: replace rightmost non-terminal at each step.

Ambiguous Grammar: has multiple left- or rightmost derivations for a single sentential form.

PARSING

Calls scanner to get tokens, assembles tokens into syntax tree, passes the tree to later phases of compiler for semantic analysis, code generation. The parser is ``in charge'' and hence this style is syntax-directed translation.

CFG can be a generator for language. A compiler, say, wants to recognize language: Parsing from arbitrary CFG turns out to be O(n3) problem, with n the length of input program. We can speed this bound up by restricting syntax, and end up with two major classes of linear-time parsers, LL (read in left-right, leftmost derivation) and LR (left-right, rightmost derivation).

LL are often thought simpler or clearer, can be written by hand or with parser-generator tool. LR grammars are a larger class, possibly have more intuitive structure. esp. arith. exps. LR almost always created automatically, more common for commercial products.

PARSE TREES

Represent a derivation as a parse tree, with root the start symbol, internal nodes are nonterminals, leaves are terminals, children of node T are symbols on RHS of some production for T in grammar. Every tree represents a string (or yield generated by grammar. The simple grammar

expr → id | number | - expr | ( expr ) | expr op expr op → + | - | * | /

allows two possible parse trees for
slope * x + intercept. It is thus an ambiguous grammar. There are infinitely many grammars for any CFL, and ambiguous grammars are much less useful in CS than unambiguous ones.

Also to be avoided are grammars with useless symbols -- nonterminals that can't generate any string of terminals (e.g. A → B.) or terminals that never appear in any yield. Grammars may be put into canonical form, which simplifies applications and analysis, but we won't follow that up.

Rightmost and Leftmost derivations: which is which?

AN UNAMBIGUOUS GRAMMAR

Relevant for scanner-parser project! expr → term | expr add-op term term → factor | term mult-op factor factor → id | number | - factor | ( expr ) add-op → + | - mult-op → * | /

This unambiguous grammar also captures arithmetic precedence in the way the productions use each other, and it captures the usual left-associativity by building sub-exprs to the left of the operator. Note: precedence is not a property of CFG, it's a property of the semantics we choose to apply to the strings. BUT if the grammar reflects what we want, easier for compiler!

PARSER TYPES

Top-Down (LL) Parsers

Bottom-up (LR Parsers):

TOPS-DOWN, BOTTOMS-UP

Like many forms or search, parsing can be done tops-down (LL) or bottoms-up (LR). We'll be concerned with LL (predictive) parsers in 173, in fact recursive descent. As in scanning, there are table-driven parsers too.

The grammar may be nasty and require a long look-ahead to do a parse. The length of the lookahead (in tokens) appears in parens after the grammar type, hence LL(1) or LR(2). (1) is most common.

RECURSIVE DESCENT PARSING

One of simplest practical methods (more importantly, it's used in our assignment).

Each non-terminal has associated procedure whose goal is to read a sequence of input characters that can be generated by that non-terminal. Also return a pointer to the parse tree for that non-terminal.

Like the cases in a scanner, each procedure structure mirrors its production structure. It attempts to ``match'' the RHS of some production.

To match a terminal symbol, compare it to input. Succeed if agree, consume the symbol from input.

To match non-terminal, call its corresponding procedure (may be a recursive call, hence the name of the technique).

TABLE-DRIVEN PARSING

Rec. desc. parsing hard to change if change the grammar a little.

Replace the implicit stack of procedure calls with an explicit stack.

Hugely important implementation of parsing: both commercial and research/experimental: separates the grammar from the code.

EPSILON PRODUCTIONS

In a derivation from a grammar, an epsilon production removes nonterminal from string being generated: Balanced ()s:

P → ( P ) P → P P P → ε (or more commonly P →) or P → ( P ) | P P | ε

Note this is ambiguous grammar: rule 2 once, then rule 1 on either P, then rule 3 on other. Unambiguous one is
P → ( P ) P | ε

EXAMPLES ---some regular sets

Find main: <program> → <letter*> m a i n <letter*> <letter*> → <letter> <letter*> | epsilon <letter> → A | B | ... | Z | a | b ... | z

Pascal IDs: <id> → <L> <LorD*> <LorD*> → <L> <LorD*> | <D> <LorD*> | ε <L> → A|B| ... |Z|a|b ... |z <D> → 0|1|2|3|4|5|6|7|8|9

Pascal Real Numbers: <real> → <dig> <dig*> <dec part> <exp> <dig*> → <dig> <dig*> | ε <dec part> → '.' <dig> <dig*> | ε <exp> → 'E' <sign> <dig> <dig*> | ε <sign> → + | - | ε <dig> → 0 | ... | 9

NON-RE EG

C compound statements (more or less): <compound stmt> → { <stmt list> } <stmt list> → <stmt> <stmt list> | ε <stmt> → <compound stmt> <stmt> → id : <stmt> <stmt> → if ( <expr> ) <stmt> <stmt> → if ( <expr> ) <stmt> else <stmt> <stmt> → while ( <expr> ) <stmt> <stmt> → do <stmt> while ( <expr> ); <stmt> → for ( <stmt> <expr> ; <expr> ) <stmt> <stmt> → switch ( <expr> ) { <mult case stmt> } <mult case stmt> → <case stmt> <mult case stmt> | ε <case stmt> → case <expr> : <stmt> | default: <stmt> <stmt> → break ; | continue ; | ; <stmt> → return <expr> ; | goto <id>

PREDICTIVE (TABLE-DRIVEN) PARSING

Actions:

Push as yet unseen portions of productions onto a stack

Use:

which we'll learn about later.

PREDICTIVE (TABLE-DRIVEN) PARSING

Table Code->Scanner->Tokens->Driven<->Stack Parser ^ | Grammar->Parser-----> Parse Rules Generator Table

FOR TOP DOWN PARSING:

The strategy is to expand (parse) the next (leftmost) nonterminal: ideally, we don't immediately drop into an infinite loop, and we don't have a choice of how to expand it (i.e. the grammar is unambiguous). Thus ...

!! No left recursion or common prefixes !!

A grammar is left recursive if there exists A in NonTerms such that
A → A σ for some string σ .

A grammar has common prefixes if there are productions of the form
A → B δ A → B μ

ELIMINATE LEFT RECURSION

Transform the grammar to remove left recursion:

A → Aa1 | Aa2|...| b1 | b2 | ... ⇒ A → b1A' | b2A'... A' → a1A' | a2A'...| ε, where A' is new nonterminal.

This substitution of right-recursion for left means that the resultant parse tree does not enjoy the "built-in left associativity" property for arithmetic expressions that left recursion gives.

ELIMINATE COMMON PREFIXES

Want no embarrassing choice of what to do next. Factoring the expression means deferring the choice of productions until after we deal with the common prefix. A → B δ A → B μ ⇒ A → B Btail Btail → δ | μ

PARSER CONSTRUCTION

Recursive descent parsing:

EXAMPLE

Here's a version of our unambiguous grammar for (E)xpressions, with (T)erms and (F)actors. E → E + T | E - T | T T → T * F | T / F | F F → ( E ) | number

But it's a cess-pool of left recursion and can't be used in top-down parsing!

BIG LESSON: there are an infinite number of grammars for any language and they're not all equally good for everything. (motivates Canonical Grammars). What's wrong here is the ``left recursion''. The Rule for E predicts an E, which means we would call the rule associated with E, etc. forever.

MORAL: language design is important (e.g. minimize lookahead) AND grammar choice is important!

EXAMPLE CONT.

Different Grammar -- simply use the left recursion and common prefix removal rewriting tricks seen a while back. (Details coming up!) E → T Etail Etail → + T Etail | - T Etail | ε T → F Ttail Ttail → * F Ttail | / F Ttail | ε F → ( E ) | num
This works top-down. Create procedures for the nonterminals, like procedure E T() Etail()

RECURSIVE DESCENT CONT.

For a rule that has to examine the input: Assume a global variable next_token and a utility routine match:
procedure Etail switch next_token case + match(+) T() Etail() case - match(-) T() Etail() default return procedure match(expected) if next_token != expected error() else next_token = scan() % read next terminal symbol % into global variable

ANOTHER REC. DEC. PROCEDURE

F → ( E ) | num procedure F switch next_token case ( match(() E() match()) case num match(num) default error()

Here the default case is an error, whereas the default case in Etail and Ttail is to return without doing anything. Etail and Ttail have epsilon productions (OK to have empty subtree under them in the parse tree.) F has to be either a number or a parenthesized expression, not epsilon.

Note the check inside match is useful if, for instance, there is no ``)'' for the 2nd call to match --- an error.

MAIN PROG and TRACE

Could make eof a sort of token so match can help us with error detection here... procedure main E match(eof) Then, with input 1 +(2*3)/4: next_token = "1" Call E Call T Call F next_token = "+" /* Match 1 with F */ Call Ttail /* Match ε */ Call Etail next_token = "(" /* Match + */ Call T Call F /*Match (, looking for E)*/ next_token = "2" Call E Call T Call F /* Match 2 with F */ next_token = "*" Call Ttail /* Match * */ next_token = "3" Call F /* Match 3 with F */ next_token = ")" Call Ttail /* Match epsilon */ Call Etail /* Match epsilon */ next_token = "/" /* Match ")" */ Call Ttail next_token = "4" /* Match "/" */ Call F /* Match 4 with F */ next_token = eof Call Ttail /* Match epsilon */ Call Ttail /* Match epsilon */ Call Etail /* Match epsilon */ /* Match eof */

OBSERVATIONS

In procedure Etail and Ttail, we match one of the productions with an arithmetic operator if we see such an operator in the input; otherwise we simply return, (choosing the epsilon production.)

Here we only choose the epsilon production if the next_token doesn't match the first token on the right hand side of the production.

We never attempt to read beyond the end marker (eof), which is matched only at the end of an outermost expression. In all other circumstances, the presence of the end marker signals a syntax error.

Our recursive-descent parser only determines whether or not the input string is in the language of the grammar; it does not give the structure of the string according to the grammar. We could easily build a parse tree incrementally during parsing.

MORE on the web...Google recursive descent parser, Danno!

LL(1) Grammars.

Recursive descent only works for CSGs with disjoint PREDICT sets for productions that share a common left hand side. Such CFG's are called LL(1). Practically, programming languages are usually LL(1). So we need to learn about PREDICT sets.

We have seen how these problems keep grammars from being LL(1):
Left Recursion: A → A X1...Xm.
Common Prefix: B → C X1...Xm, B → C X1...Xm.
(Symptomatic of the trouble they'll cause a tops-down parser, LR and CP yield non-disjoint predict sets.)

We know how to fix these problems, and doing that derives our example grammar from the (more intuitive) non-LL(1) version.

CREATE LL(1) GRAMMAR

For Expressions: 1. E → E + T 2. E → E - T 3. E → T 4. T → T * F 5. T → T / F 6. T → F 7. F → ( E ) 8. F → number Must replace left recursion with right recursion...

CREATE LL(1) GRAMMAR CONT.

Converting with Left Recursion to Right Recursion is easy: 1. E → T + E 2. E → T - E 3. E → T 4. T → F * T 5. T → F / T 6. T → F 7. F → ( E ) 8. F → number

But 1.--3., 4.--6. have common prefix. Factoring the common prefix means deferring decision of which production to pick until we see the common prefix.

FACTORING COMMON PREFIX

That leads to 1. E → T Etail 2. Etail → + T Etail | - T Etail | ε 3. T → F Ttail 4. Ttail → * F Ttail | / F Ttail | ε 5. F → ( E ) | number That's our familiar top-down grammar for expressions.

CAN'T BE PARSED TOP-DOWN

Eliminating left recursion and common prefixes is not enough to guarantee LL(1). Here's an ingeniously snotty grammar for the simple language

``all strings of a's followed by equal number of b's or by equal number of c's''
(an bn | an cn ).

G → a B b G → a C c B → a B b B → C → a C c C →

Left factoring doesn't help.

But it's not hard to find an LL(1) version: G → B | C B → aBb | ε C → aCc | ε

PREDICT SETS

To write our parser, we need to answer this:
Q: What are the cases in a production's case statement??

Example: A → B C Ω B → bB B → ε C → c C → ε

Regular language: (b* | b* c) Ω
Regular Grammar? (There's a moral here!)

Recursive Descent Parser has Functions like

A() case: b match(b) B(); case: c C(); case: OMEGA match(OMEGA) otherwise: error

Likewise, for B → bB B → ε Get B() % Parse Nonterminal B case: b %cases are over nxt token match(b) % accept and delete B() % recurse case: c, OMEGA % B -> eps! c, % OMEGA are from return % C-> c and C -> eps. otherwise: error and for C → c C → ε get C() case: c match(c) case: OMEGA return otherwise error

PROBLEM: How figure out what goes in the cases???

LOOKAHEAD IN REC. DESC. PARSING

What tokens should be in the arms of the case statements in a Rec. Desc. parser? It's a token X that predicts the production, but they don't just jump out at you from the grammar.

The RHS of production, recursively expanded, might yield string beginning with X.

The RHS may yield nothing ( ε or string of nonterminals that recursively give ε), and X may begin the yield of what comes next in the sentential form.

What we need is a PREDICT set of tokens that predicts what production could be responsible for the current token. PREDICT can be made from the FIRST and FOLLOW token sets, which can be extracted automatically from an LL(1) CFG.

In fact, FIRST sets are often enough. But in all but the simplest grammars we need to know them to write a rec. desc. parser. Thus the class was given FIRST SETS in the source code directory for the 2006 parser assignment to save deriving them. Our sets are a lot simpler!

FIRST AND FOLLOW SETS: MOTIVATION

You're the parser: you need to figure out what's going on (what production you're parsing), given the stream of tokens from the scanner. Here's a grammar: S → A B C A → a | ε B → D ! b | ε C → c D → d Here's a related tree: the arc-connected productions mean AND (in order), the straight arcs are OR (pick any production).

We see:
1. We can think of the dashed triangle subtrees disappearing from the tree if the epsilon (emit nothing) is chosen by the root (LHS of production).
2. After that, the possible token streams are ordered choices of terminals.
3. So the possible sentences are:
adc, abc if A,B,C productions all used
ac if B → ε
dc, bc if A → ε
c if A → ε, B → ε

FIRST and FOLLOW


Possible sentences are: adc, abc, ac, dc, bc c. We're the parser; We see tokens one at a time and from them must identify which production we are parsing. e.g.
see: q, x, <eof>, etc. so: ERROR
see: a, so: A
see: d or b, so: B
see: c, so: C.
That is, FIRST(S) is {a, d, b, c}. These are all we need to make up the three cases for what production to parse: a, (b or d), c. Visually we can find them in the tree by imagining the dashed triangles to vanish if epsilon is taken in the generation of the incoming sentence.

Likewise, FOLLOW(A) is seen to be {d, b, c}: c means B → ε and we're in the C production, else we're in B.

FIRST AND FOLLOW SETS: WHY?

Consider a grammar with a production like
(1) A → x y B z A C w.
There could be other A → productions in the grammar.

If we are trying to parse (find, explain) an A in a Rec. Desc. parser, we are in the A() function. If it is an LL(1) parser, the current token determines which production we're in. Thus If A() finds an x, it knows it is in production (1), and we say x is in the FIRST set of A.

Suppose the grammar has A → ε. Say we're parsing production (1) as before, and we get to the z, match that, and call A(). But the string we're parsing was generated with the A → ε production at that point.

So there's no evidence for A except that evidence for C is evidence for A → ε. So we should look for the first token produced by some C → production (!), and that token FOLLOWs A. That's where things get non-obvious.

Hence the FIRST and FOLLOW sets.

FIRST AND FOLLOW SETS: HOW?

Algorithm on page 76 of M. Scott's e-Reserve reading (not the BB Course Materials nor the equivalent link in 173 Schedule).

FIRST, FOLLOW, PREDICT

THE FIRST SET OF A SYMBOL

FIRST(X) is the set of terminal symbols that begin strings derived from X

To build FIRST(X):

THE FOLLOW SET OF A SYMBOL

For a non-terminal A, FOLLOW(A) is the set of terminals that can appear immediately to the right of A in some sentential form.

To build FOLLOW(B) (α and β are arbitrary strings of terminals and nonterminals.)

FIRST SETS -- Stepwise Creation

FIRST sets distinguish two productions with same nonterminal on LHS: 3 steps (iterated)

  1. which nonterminals can generate ε, which goes in their FIRST set if so.
  2. find FIRST sets for all nonterminals
  3. find FIRST sets for right-hand sides

Steps 1 and 2 are simply ``obvious'' facts from the grammar, then we iterate over what we know until we don't learn any more. E.g.
... A → ε B → ε ... C → A B D → C A B

Step 1:

A and B can generate ε; 2nd pass figure out C can too, as can D (found on 3rd pass), etc.

STEP 2

... 1. A → b C D 2. B → c D e 3. C → B A d ...

b is in FIRST(A) and c is in FIRST(B), immediately from 1 and 2. Pass 2 would show c is in FIRST(C). Continue working through each RHS adding tokens to the FIRST set of RHS until we learn nothing new.

STEP 3

Here use FIRST for individual symbols to get FIRST for whole RHS: if A → X1 ... Xm we need FIRST(X1...Xm).

Start with X1:

Etc. If X1,X2,...Xm can all produce ε, then we should predict A → X1 ... Xm if the lookahead symbol can come after an A in some line of the derivation: that brings up FOLLOW sets.

EXAMPLE

E → T Etail Etail → + T Etail | - T Etail | ε T → F Ttail Ttail → * F Ttail | / F Ttail | ε F → ( E ) | num

Then: First(E) = First(T Etail) First(T Etail) = First(T) First(T) = First(F Ttail) First(F Ttail) = First(F) = {(,num}

PREDICT SET OF A PRODUCTION

A production's PREDICT set is the set of all initial (first) tokens derivable from it. Grammar: (1) A → B C Ω (2, 3) B → bB | ε (4, 5) C → c | ε FIRST(A) = {b (1,2), c (1,3,4), Ω (1,3,5)} FOLLOW(A) = { ε } PREDICT(A → BC) = { b, c, Ω } FIRST(B) = {b (2), ε (3) } FOLLOW(B) = { c (2 3 4), Ω (2,3,5) } PREDICT(B → bB) = { b } PREDICT(B → ε) = { c, Ω } FIRST(C) = {c (4), ε (5) } FOLLOW(C) = { Ω } PREDICT(C → c) = { c } PREDICT(C → ε) = { Ω } Claim: Cases in A() are {b, c, Ω}.

in B(): are {b, c, Ω}

in C(): {c, Ω}.

PREDICT SETS AND CASES in REC. DESC. PARSER

We claimed that the cases
in A() are {b, c, Ω}.
In B(): are {b, c, Ω}
In C(): {c, Ω}.

B() % Parse Nonterminal B case: b % cases are over the next token match(b) % accept and delete B() % recurse case: c, Ω % B -> eps! c, Ω from return % C-> c and C -> eps. otherwise: error

Also, and obviously: If ``some token belongs to the PREDICT set of more than one production with the same left-hand side, then the grammar is not LL(1), because [M. Scott]'' the single token we're seeing could be produced by more than one production. In a recursive descent parser, it would appear in more than one branch of the case statement that deals with that left-hand-side (non-terminal).

USING FIRST AND FOLLOW

FORMALLY

Scott's Chapter 2 has First, Follow, Predict for an entire calculator language, nice treatment (pp. 72-76).

Algorithm to compute F,F, and P relies on these formal definitions;

FIRST(α) ≡ {a: α → * a β } ∪
( if α → * ε then { ε } else φ)

FOLLOW(A) ≡ {a: S → + α A a β } ∪
( if S → * α A then { ε} else φ)

PREDICT(A → α) ≡ (FIRST( α) \ { ε }) ∪
(if α → * ε then FOLLOW(A) else φ)

If grammar has disjoint PREDICT sets it is LL(1): otherwise not: there is a predict-predict conflict.

FOLLOW NOT REALLY NEEDED?

In building rec. desc. parser... say G, H produce ε in: A → B c D A → e f A → G H

Parsing routine: A() { switch (next_token) { case First(BcD): B match(c) D case e: match(e) match(f) default: % if next not in First(GH) G % or Follow(A), just get H } } % syntax error later...

LL(1) GRAMMARS

A Grammar G is LL(1) if and only if, for all non-terminals A, each distinct pair of productions
A → a and A → b satisfy the condition
FIRST(a) ∩ FIRST(b) = φ, i.e.,

For each set of productions
A → a1 | a2 | . . . | an:

THE COMPLEXITY OF LL(1) PARSING

Expect linear since no back-up (look-ahead).

Instructions inside main loop are bounded by constant (function of symbols on RHS)

How many times does the main loop execute?

EXAMPLE NON-LL GRAMMAR CONSTRUCT

stmt --> if condition then-clause else-clause | other_stmt then-clause → then stmt else-clause → else stmt | e

if C1 then if C2 then S1 else S2 --- the else can be paired with either then. Neither LL nor LR. Rather famous problem.

FIX? AN UNNATURAL B-UP (LR) GRAMMAR

stmt → balanced-stmt | unbalanced-stmt balanced-stmt → if condition then balanced-stmt else balanced-stmt | other-stmt unbalanced_stmt → if condition then stmt | if condition then balanced-stmt else unbalanced-stmt

OR: Use special disambiguating rules, like ``use production that occurs first in case of conflict''.

OR: Use a better syntax (like elseif).

ERRORS

For pure parser, simply halt without accepting.

In a compiler, you should print a nice helpful diagnostic message and then do something potentially really complicated (not covered here) to patch up the parse tree and the input and continue looking for further errors. OR you could do something really stupid like PL/I did and try to ``correct'' errors...

FIRST and FOLLOW -- not??

It is sometimes possible to avoid using FIRST and FOLLOW in parsing code.

Consider the program for non-terminal A() If there is only one non-epsilon production in the grammar with A as a left-hand side, say:
A → B C D
One could figure out the FIRST and FOLLOW for B, C and D and explicitly build in all the cases in A's program. But why bother? Why not do this?:
A() B(); C(); D(); end Not using FIRST: If B(),C(), or D() doesn't like what it sees, let it complain by calling a parse error. Sure, A() could have done a lot of work to create cases for when it seemed B() was appropriate to call or if B could derive ε then whether it seemed like C() was the thing to do, etc. etc. but not checking just costs some more recursion.
Not using FOLLOW: Now consider adding the production A → ε to the grammar, so there are now two productions with A as LHS, one being the ε production. Nothing has to change in A() (!).

If BCD is matched, A() is happy, and if not A may have produced ε -- that's enough excuse for A() to declare success and return upward. What's coming up is either grammatical or not. If it's not, sometime later the parser will fail to match a token: that's where it finds the error.

Sure, FOLLOW(A) would tell us that A "really" has produced ε, but this version of A() just assumes it has and kicks any possible problems upstairs.

If a production really needs a switch statement, we need a case for the ε production, but it can always be "accept and don't match anything": default: break;

FIRST and FOLLOW -- yes

More than one production with a non-ε RHS actually do need the FIRST sets: A → B A → D We need to know whether to call B or D, so we need their FIRST sets. Also, if we have something like B → aC D → aD We would not be LL(1): two productions for A have the same FIRST sets.

Last: Our expression grammar is so simple the FIRST sets can be figured out easily "by inspection": + and - are the cases in ETAIL, * and / are the cases in TTAIL, and ( and <number> are the cases in FACTOR.

ELIMINATE LEFT RECURSION REPEAT

Transform the grammar to remove left recursion:

A → Aa1 | Aa2|...| b1 | b2 | ... ⇒ A → b1A' | b2A'... A' → a1A' | a2A'...| ε, where A' is new nonterminal.

This substitution of right-recursion for left means that the resultant parse tree does not enjoy the "built-in left associativity" property for arithmetic expressions that left recursion gives.

Get what you pay for with... Free Code!

Various declarations from a .h file for CB's parser... this is about creating parse trees (of ptnodes).

typedef enum { EXPR, TERM, TERM_TAIL, FACTOR, FACTOR_TAIL, IDENTIFIER, NUM } node_type; /* only for factor tails and term tails */ typedef enum { TIMES, DIVIDE, PLUS, MINUS, NONE} op_type; typedef struct ptnode{ char * ptname; node_type pttype; char * ID_name; double ptvalue; op_type optype; struct ptnode * first; /* <= 3 descs */ struct ptnode * second; struct ptnode * third; } * ptnode_t;

Making a Parse Tree

Using the facts above, here a couple of CB's functions used by the parser: first, a program for a nonterminal with a single production. It must produce a parse-tree node, of course. static ptnode_t parse_expr() { ptnode_t res; % making this node ptnode_t ter; % which points to these ptnode_t tertail; % two nodes /* printf("\n parse_expr\n"); */ ter =parse_term(); % each prog. rtns a parse-tree node tertail = parse_term_tail(); res = make_ptnode("expr", EXPR,"", 0.0,NONE,ter, tertail, NULL); return(res); }

Next, a version of the FACTOR() program; Two cases are tokens that aren't terminal characters: one is an identifier (its value is a string of chars in the input line) and one a number (string of digit chars in input line). These non-terminals are dealt with in the original scanner code (e.g. with got_dot(), got_dig(), got_ft_dot()), which CB did not change from the original.

Again, this code makes a parse-tree node (with lots of debugging info along the way). It also shows one way to evaluate a number in the input and put the resulting float in the parse tree, and how to copy a string (here, an identifier name). static ptnode_t parse_factor() { ptnode_t res; char * idcopy; char * idloc; location_t loc; double anum; printf("\n parse_factor\n"); switch (tok.tc) { case T_LPAREN: match(T_LPAREN); res = parse_expr(); match(T_RPAREN); break; case T_IDENTIFIER: /* printf("\n ID found\n"); */ loc = tok.location; /* cbprint_location(&tok); print_token(&tok); */ idloc = (char *) &(loc.line->data[loc.column]); idcopy = strndup(idloc,tok.length); /*printf("\n ID copied to: %s\n", idcopy);*/ match(T_IDENTIFIER); res = make_ptnode("an ID", IDENTIFIER, idcopy, 0.0, NONE, NULL, NULL, NULL); break; /* accept */ case T_NUM: /* printf("\n NUM found\n"); */ loc = tok.location; /* cbprint_location(&tok); print_token(&tok); */ idloc = (char *) &(loc.line->data[loc.column]); idcopy = strndup(idloc,tok.length); sscanf(idcopy, "%lf", &anum); /*printf("\n NUM scanned as: %f\n", anum);*/ res = make_ptnode("a NUM", NUM, "", anum, NONE, NULL, NULL, NULL); match(T_NUM); break; } return(res); }

One More non-terminal program

static ptnode_t parse_factor_tail() { ptnode_t fac; ptnode_t factail; ptnode_t res; /* printf("\n parse_factortail\n"); */ switch (tok.tc) { case T_STAR: match(T_STAR); fac = parse_factor(); factail = parse_factor_tail(); res = make_ptnode("factortail", FACTOR_TAIL, "", 0.0,TIMES,fac, factail, NULL); break; case T_SLASH: match(T_SLASH); fac = parse_factor(); factail = parse_factor_tail(); res = make_ptnode("factortail", FACTOR_TAIL, "", 0.0,DIVIDE,fac, factail, NULL); break; default: res = NULL; break; /* just return */ } return(res);}

Final Anncmt, Eg.

I'm giving up on this example...too hairy. Best let sleeping dogs lie. This got out of hand and lost original point about FOLLOW sets. Never got good e.g. of that -- this turned into an e.g. of not even using FIRST sets (A->BCD). 9/25/14

To FOLLOW or not? Claim: FOLLOW sets are nice symmetric idea, given FIRST sets, but:
FIRST sets needed if a nonterminal has more than one non-ε production: (Which One Should I Do??).
FOLLOW set is a luxury:
a. Only useful if nonterminal has an ε production.
b. All FOLLOW gives you is a chance to report an error slightly earlier.

A → c c % must deal with this rhs, not A → aAx % get mixed up with this one A → ε A → BCD;

FIRST(A) = {a,c,ε} ∪ FIRST(BCD) (hard?)
FOLLOW(A) = {x} ∪ FOLLOW(BCD) (hard?)
PREDICT(A → cc) = {c} (FIRST)
PREDICT(A → aAx) = {a} (FIRST)
PREDICT(A → ε) = {x} (FOLLOW)
PREDICT(A → BCD) = ???

A() % Use FOLLOW case: c match(c); match(c); case: a match(a); A(); match(x); case: x return(); % A -> e; match(x) eats it case: PREDICT(BCD) B() C() D() otherwise: error(); % non-FOLLOW token detected end;

A() % Don't Use FOLLOW case: c match(c); match(c); case: a match(a); A(); match(x); otherwise: return(); end;

That is, ignore what can go wrong... declare success, ignore what's coming next. % Leave it to the parser to notice that when % it reads next token.