编译器之语法分析器（syntax analyzer）

语法分析器根据语法将标记流（来自词法分析器）转换为语法树。

任务描述

从“词法分析器” 中获取输出，并根据以下语法将其转换为抽象语法树（AST）。输出应为扁平格式。

程序应从文件和/或stdin读取输入，并将输出写入文件和/或stdout。如果使用的语言具有解析器模块/库/类，则提供两种版本的解决方案将是很好的选择：一个不带解析器模块，另一个带解析器模块。

语法

    stmt_list           =   {stmt} ;stmt                =   ';'| Identifier '=' expr ';'| 'while' paren_expr stmt| 'if' paren_expr stmt ['else' stmt]| 'print' '(' prt_list ')' ';'| 'putc' paren_expr ';'| '{' stmt_list '}';paren_expr          =   '(' expr ')' ;prt_list            =   (string | expr) {',' (String | expr)} ;expr                =   and_expr            {'||' and_expr} ;and_expr            =   equality_expr       {'&&' equality_expr} ;equality_expr       =   relational_expr     [('==' | '!=') relational_expr] ;relational_expr     =   addition_expr       [('<' | '<=' | '>' | '>=') addition_expr] ;addition_expr       =   multiplication_expr {('+' | '-') multiplication_expr} ;multiplication_expr =   primary             {('*' | '/' | '%') primary } ;primary             =   Identifier| Integer| '(' expr ')'| ('+' | '-' | '!') primary

得到的AST应该表示为二叉树。
示例
给定一个简单程序（如下），该程序存储在一个名为while.t的文件中，使用一种词法分析器创建标记列表。

lex < while.t > while.lex

运行一种语法分析器

parse < while.lex > while.ast

while.t

count = 1;while (count < 10) {print("count is: ", count, "\n");count = count + 1;}

while.lex

    1      1 Identifier      count1      7 Op_assign1      9 Integer             11     10 Semicolon2      1 Keyword_while2      7 LeftParen2      8 Identifier      count2     14 Op_less2     16 Integer            102     18 RightParen2     20 LeftBrace3      5 Keyword_print3     10 LeftParen3     11 String          "count is: "3     23 Comma3     25 Identifier      count3     30 Comma3     32 String          "\n"3     36 RightParen3     37 Semicolon4      5 Identifier      count4     11 Op_assign4     13 Identifier      count4     19 Op_add4     21 Integer             14     22 Semicolon5      1 RightBrace6      1 End_of_input

while.ast

Sequence
Sequence
;
Assign
Identifier    count
Integer       1
While
Less
Identifier    count
Integer       10
Sequence
Sequence
;
Sequence
Sequence
Sequence
;
Prts
String        "count is: "
;
Prti
Identifier    count
;
Prts
String        "\n"
;
Assign
Identifier    count
Add
Identifier    count
Integer       1

详细说明
节点类型名称列表

Identifier String Integer Sequence If Prtc Prts Prti While Assign Negate Not Multiply Divide Mod
Add Subtract Less LessEqual Greater GreaterEqual Equal NotEqual And Or

在下文中，Null/Empty节点代表";"。

非根(内部)节点
对于操作符，应该创建以下节点：

Multiply Divide Mod Add Subtract Less LessEqual Greater GreaterEqual Equal NotEqual And Or

对于上面的每个节点，左子节点和右子节点是各自操作的操作数。
采用伪S-Expression格式：

(Operator expression expression)

Negate, Not
对于这些节点类型，左节点是操作数，右节点为空。

(Operator expression ;)

Sequence - 子节点是statement或Sequence。
If - 左节点是expression，右节点是if节点, 同时它的左节点是if-true语块右节点是if-false (else) 语块。

(If expression (If statement else-statement))

如果没有else, 这棵树变成：

(If expression (If statement ;))

Prtc

(Prtc (expression) ;)

Prts

(Prts (String "the string") ;)

Prti

(Prti (Integer 12345) ;)

While - 左节点是expression，右节点是statement.

(While expression statement)

Assign - 左节点是赋值的左边，右节点是赋值的右边。

(Assign Identifier expression)

终端（叶子）节点：

Identifier: (Identifier ident_name)
Integer:    (Integer 12345)
String:     (String "Hello World!")
";":        Empty node

举例
实现以下程序：

   a=11;

用二叉树来实现以下AST：
每个非叶子节点下有两个’|'行。第一个表示左子节点，第二个表示右子节点

   (1) Sequence(2)     |-- ;(3)     |-- Assign(4)         |-- Identifier: a(5)         |-- Integer: 11

扁平化形式：

   (1) Sequence(2) ;(3) Assign(4) Identifier    a(5) Integer       11

实现以下程序：

   a=11;b=22;c=33;

生成AST：

   ( 1) Sequence( 2)     |-- Sequence( 3)     |   |-- Sequence( 4)     |   |   |-- ;( 5)     |   |   |-- Assign( 6)     |   |       |-- Identifier: a( 7)     |   |       |-- Integer: 11( 8)     |   |-- Assign( 9)     |       |-- Identifier: b(10)     |       |-- Integer: 22(11)     |-- Assign(12)         |-- Identifier: c(13)         |-- Integer: 33

扁平化形式：

   ( 1) Sequence( 2) Sequence( 3) Sequence( 4) ;( 5) Assign( 6) Identifier    a( 7) Integer       11( 8) Assign( 9) Identifier    b(10) Integer       22(11) Assign(12) Identifier    c(13) Integer       33

解析器的伪代码
使用优先级上升进行表达式解析，并使用递归下降进行语句解析

def expr(p)if tok is "("x = paren_expr()elif tok in ["-", "+", "!"]gettok()y = expr(precedence of operator)if operator was "+"x = yelsex = make_node(operator, y)elif tok is an Identifierx = make_leaf(Identifier, variable name)gettok()elif tok is an Integer constantx = make_leaf(Integer, integer value)gettok()elseerror()while tok is a binary operator and precedence of tok >= psave_tok = tokgettok()q = precedence of save_tokif save_tok is not right associativeq += 1x = make_node(Operator save_tok represents, x, expr(q))return xdef paren_expr()expect("(")x = expr(0)expect(")")return xdef stmt()t = NULLif accept("if")e = paren_expr()s = stmt()t = make_node(If, e, make_node(If, s, accept("else") ? stmt() : NULL))elif accept("putc")t = make_node(Prtc, paren_expr())expect(";")elif accept("print")expect("(")repeatif tok is a stringe = make_node(Prts, make_leaf(String, the string))gettok()elsee = make_node(Prti, expr(0))t = make_node(Sequence, t, e)until not accept(",")expect(")")expect(";")elif tok is ";"gettok()elif tok is an Identifierv = make_leaf(Identifier, variable name)gettok()expect("=")t = make_node(Assign, v, expr(0))expect(";")elif accept("while")e = paren_expr()t = make_node(While, e, stmt()elif accept("{")while tok not equal "}" and tok not equal end-of-filet = make_node(Sequence, t, stmt())expect("}")elif tok is end-of-filepasselseerror()return tdef parse()t = NULLgettok()repeatt = make_node(Sequence, t, stmt())until tok is end-of-filereturn t

一旦构建了AST，就应该以扁平格式输出它。这可以像下面这样

def prt_ast(t)if t == NULLprint(";\n")elseprint(t.node_type)if t.node_type in [Identifier, Integer, String]     # leaf nodeprint the value of the Ident, Integer or String, "\n"elseprint("\n")prt_ast(t.left)prt_ast(t.right)

如果正确地构建了AST，那么将它用以下操作加载到后续程序

def load_ast()line = readline()# Each line has at least one tokenline_list = tokenize the line, respecting double quotestext = line_list[0] # first token is always the node typeif text == ";"   # a terminal nodereturn NULLnode_type = text # could convert to internal form if desired# A line with two tokens is a leaf node# Leaf nodes are: Identifier, Integer, String# The 2nd token is the valueif len(line_list) > 1return make_leaf(node_type, line_list[1])left = load_ast()right = load_ast()return make_node(node_type, left, right)

最后，还可以通过对AST解释器解决方案之一运行AST来测试它。
假设它测试程序一个名为prime.t的文件中
lex <prime.t | parse
输入到词法分析器的代码：

/*Simple prime number generator*/
count = 1;
n = 1;
limit = 100;
while (n < limit) {k=3;p=1;n=n+2;while ((k*k<=n) && (p)) {p=n/k*k!=n;k=k+2;}if (p) {print(n, " is prime\n");count = count + 1;}
}
print("Total primes found: ", count, "\n");

从词法分析器输出，并输入到语法分析器：

    4      1 Identifier      count4      7 Op_assign4      9 Integer             14     10 Semicolon5      1 Identifier      n5      3 Op_assign5      5 Integer             15      6 Semicolon6      1 Identifier      limit6      7 Op_assign6      9 Integer           1006     12 Semicolon7      1 Keyword_while7      7 LeftParen7      8 Identifier      n7     10 Op_less7     12 Identifier      limit7     17 RightParen7     19 LeftBrace8      5 Identifier      k8      6 Op_assign8      7 Integer             38      8 Semicolon9      5 Identifier      p9      6 Op_assign9      7 Integer             19      8 Semicolon10      5 Identifier      n10      6 Op_assign10      7 Identifier      n10      8 Op_add10      9 Integer             210     10 Semicolon11      5 Keyword_while11     11 LeftParen11     12 LeftParen11     13 Identifier      k11     14 Op_multiply11     15 Identifier      k11     16 Op_lessequal11     18 Identifier      n11     19 RightParen11     21 Op_and11     24 LeftParen11     25 Identifier      p11     26 RightParen11     27 RightParen11     29 LeftBrace12      9 Identifier      p12     10 Op_assign12     11 Identifier      n12     12 Op_divide12     13 Identifier      k12     14 Op_multiply12     15 Identifier      k12     16 Op_notequal12     18 Identifier      n12     19 Semicolon13      9 Identifier      k13     10 Op_assign13     11 Identifier      k13     12 Op_add13     13 Integer             213     14 Semicolon14      5 RightBrace15      5 Keyword_if15      8 LeftParen15      9 Identifier      p15     10 RightParen15     12 LeftBrace16      9 Keyword_print16     14 LeftParen16     15 Identifier      n16     16 Comma16     18 String          " is prime\n"16     31 RightParen16     32 Semicolon17      9 Identifier      count17     15 Op_assign17     17 Identifier      count17     23 Op_add17     25 Integer             117     26 Semicolon18      5 RightBrace19      1 RightBrace20      1 Keyword_print20      6 LeftParen20      7 String          "Total primes found: "20     29 Comma20     31 Identifier      count20     36 Comma20     38 String          "\n"20     42 RightParen20     43 Semicolon21      1 End_of_input

语法分析器的输出：

Sequence
Sequence
Sequence
Sequence
Sequence
;
Assign
Identifier    count
Integer       1
Assign
Identifier    n
Integer       1
Assign
Identifier    limit
Integer       100
While
Less
Identifier    n
Identifier    limit
Sequence
Sequence
Sequence
Sequence
Sequence
;
Assign
Identifier    k
Integer       3
Assign
Identifier    p
Integer       1
Assign
Identifier    n
Add
Identifier    n
Integer       2
While
And
LessEqual
Multiply
Identifier    k
Identifier    k
Identifier    n
Identifier    p
Sequence
Sequence
;
Assign
Identifier    p
NotEqual
Multiply
Divide
Identifier    n
Identifier    k
Identifier    k
Identifier    n
Assign
Identifier    k
Add
Identifier    k
Integer       2
If
Identifier    p
If
Sequence
Sequence
;
Sequence
Sequence
;
Prti
Identifier    n
;
Prts
String        " is prime\n"
;
Assign
Identifier    count
Add
Identifier    count
Integer       1
;
Sequence
Sequence
Sequence
;
Prts
String        "Total primes found: "
;
Prti
Identifier    count
;
Prts
String        "\n"
;

代码实现

C实现
Go实现
Java实现
JavaScript实现
Python实现