javascript解析

by Shalvah

由Shalvah

使用JavaScript解析数学表达式 (Parsing math expressions with JavaScript)

A while ago, I wrote about tokenizing a math expression, with Javascript as the language of choice. The tokenizer I built in that article was the first component of my quest to render and solve math expressions using Javascript, or any other language. In this article, I’ll walk through how to build the next component: the parser.

前一阵子，我写过关于用Java语言作为首选语言对数学表达式进行标记的方法。 我在那篇文章中构建的令牌生成器是我寻求使用Javascript或任何其他语言呈现和求解数学表达式的第一个组件。 在本文中，我将逐步介绍如何构建下一个组件：解析器。

What is the job of the parser? Quite simple. It parses the expression. (Duh.) Okay, actually, Wikipedia has a good answer:

解析器的工作是什么？ 非常简单。 它解析表达式。 (Du)好吧，实际上， 维基百科有一个很好的答案：

A parser is a software component that takes input data (frequently text) and builds a data structure — often some kind of parse tree, abstract syntax tree or other hierarchical structure — giving a structural representation of the input, checking for correct syntax in the process. The parsing may be preceded or followed by other steps, or these may be combined into a single step. The parser is often preceded by a separate lexical analyser, which creates tokens from the sequence of input characters

解析器是一种软件组件，它接收输入数据(通常是文本)并构建数据结构(通常为某种解析树，抽象语法树或其他层次结构)，从而提供输入的结构表示，并在过程中检查语法是否正确。 解析可以在其他步骤之前或之后，或者可以将这些步骤组合为一个步骤。 解析器通常在前面是一个单独的词法分析器，该词法分析器根据输入字符序列创建标记

So, in essence, this is what we’re trying to achieve:

因此，从本质上讲，这就是我们要实现的目标：

math expression => [parser] => some data structure (we'll get to this in a bit)

Let’s skip ahead a bit: “… The parser is often preceded by a separate lexical analyzer, which creates tokens from the sequence of input characters”. This is talking about the tokenizer we built earlier. So, our parser won’t be receiving the raw math expression, but rather an array of tokens. So now, we have:

让我们先略过一点：“…解析器通常前面有一个单独的词法分析器，该词法分析器根据输入字符序列创建标记”。 这是在谈论我们之前构建的令牌生成器。 因此，我们的解析器将不会接收原始的数学表达式，而只会接收令牌的数组。 所以现在，我们有：

math expression => [tokenizer] => list of tokens => [parser] => some data structure

For the tokenizer, we had to come up with the algorithm manually. For the parser, we’ll be implementing an already existing algorithm, the Shunting-yard algorithm. Remember the “some data structure” above? With this algorithm, our parser can give us a data structure called an Abstract Syntax Tree (AST) or an alternative representation of the expression, known as Reverse Polish Notation (RPN).

对于令牌生成器，我们必须手动提出算法。 对于解析器，我们将实现一个已经存在的算法， Shunting-yard算法。 还记得上面的“某些数据结构”吗？ 使用此算法，解析器可以为我们提供称为抽象语法树(AST)或该表达式的替代表示的数据结构，称为反向波兰表示法(RPN)。

反向波兰符号 (Reverse Polish Notation)

I’ll start with RPN. Again from Wikipedia, RPN is “a mathematical notation in which every operator follows all of its operands”. Instead of having, say, 3+4, RPN would be 3 4 +. Weird, I know. But the rule is that the operator has to come after all its operands.

我将从RPN开始。 再次来自维基百科 ，RPN是“一种数学符号，其中每个运算符都遵循其所有操作数 ”。 RPN不是3 + 4，而是3 4+。 很奇怪，我知道。 但规则是操作员来其所有操作数之后 。

Keep that rule in mind as we take a look at some more complex examples. Also remember that an operand for one operation can be the result of an earlier operation).

当我们看一些更复杂的示例时，请记住该规则。 还请记住，一个操作的操作数可以是更早操作的结果)。

Algebraic: 3 - 4                        RPN: 3 4 -

Algebraic: 3 - 4 + 5                    RPN: 3 4 - 5 +

Algebraic: 2^3                          RPN: 2 3 ^

Algebraic: 5 + ((1 + 2) × 4) − 3        RPN: 5 1 2 + 4 * + 3 -

Algebraic: sin(45)                      RPN: 45 sin

Algebraic: tan(x^2 + 2*x + 6)           RPN: x 2 ^ 2 x * + 6 + tan

Because the operator has to come after its operands, RPN is also known as postfix notation, and our “regular” algebraic notation is called infix.

由于运算符必须遵循其操作数，因此RPN也称为后缀表示法 ，我们的“常规”代数符号称为infix 。

How do you evaluate an expression in RPN? There’s a simple algorithm I use:

您如何评估RPN中的表达式？ 我使用一个简单的算法：

Read all the tokens from left to right till you get to an Operator or Function. Knowing that the Operator/Function takes n arguments (for instance, for +, n = 2; for cos(), n = 1), evaluate the last n preceding arguments with the Operator/Function, and replace all of them (Operator/Function + operands) with the result. Continue as before, until there are no more Operators/Functions left to read. The only (Literal or Variable) token left is your answer.

从左到右读取所有标记，直到找到操作员或函数。 知道运算符/函数接受 n个参数(例如，对于+， n = 2；对于 cos() ， n = 1)， 用运算符/函数 求出前 n个在前的参数，然后替换所有参数(运算符/函数+操作数)与结果。 像以前一样继续操作，直到没有更多的操作员/功能要读取为止。 剩下的唯一(文字或可变)标记是您的答案。

(This is a simplified algorithm, which assumes the expression is valid. A couple indicators that the expression isn’t valid are if you have more than one token left at the end, or if the last token left is an Operator/Function.)

(这是一种简化算法，它假定表达式有效。如果结尾处有多个标记，或者最后一个标记是运算符/功能，则表示该表达式无效的几个指标。)

So, for something like 5 1 2 + 4 * + 3 −:

因此，对于5 1 2 + 4 * + 3 −

read 5read 1read 2read +. + is an operator which takes 2 args, so calculate 1+2 and replace with the result (3). The expression is now 5 3 4 * + 3 -read 4read *. * is an operator which takes two args, so calculate 3*4 and replace with the result, 12. The expression is reduced to 5 12 + 3 -read +. + is an operator which takes two args, so calculate 5+12, replace by the result, 17. Now, we have 17 3 -read 3read -. - is an operator which takes two args, so calculate 17-3. The result is 14.

Hope you made an A in my little crash course on RPN. You did? OK, let’s move on.

希望您在我的RPN小速成课程中获得A分。 你有吗 好的，让我们继续。

抽象语法树 (Abstract Syntax Trees)

Wikipedia’s definition here might not be too helpful for many of us: “a tree representation of the abstract syntactic structure of source code written in a programming language.” For this use case, we can think of an AST as a data structure that represents the mathematical structure of the expression. This is better seen than said, so let’s draw a rough diagram. I’ll start with an AST for the simple expression 3+4:

Wikipedia在这里的定义可能对我们许多人没有太大帮助：“用编程语言编写的源代码抽象句法结构的树形表示。” 对于此用例，我们可以将AST视为代表表达式数学结构的数据结构。 这比说的要好看，所以让我们画一个粗略的图。 我将从简单表达式3 + 4的AST开始：

[+] /   \[3] [4]

Each [] represents a node in the tree. So you can see at a glance, that the two tokens are brought together by a the + operator.

每个[]代表树中的一个节点。 因此，您可以一眼看出，这两个标记是由+运算符组合在一起的。

A more complex expression, 5 + ((1 + 2) * 4) − 3:

一个更复杂的表达式5 +(((1 + 2)* 4)− 3

[-]          /   \        [+]    \___[3]          /  \ [5]__/   [*]         /   \        [+]  [4]       /   \       [1]  [2]

Ah, a lovely little syntax tree. It links up all tokens and operators perfectly. You can see that evaluating this expression is much easier — just follow the tree.

啊，一棵可爱的小语法树。 它完美地链接了所有令牌和运算符。 您可以看到评估此表达式要容易得多-只需遵循树即可。

So,why is an AST useful? It helps you represent the logic and structure of the expression correctly, making it easier for the expression to be evaluated. For instance, to evaluate the above expression, on our backend we could do something like this:

那么，为什么AST有用？ 它可以帮助您正确表示表达式的逻辑和结构，从而使表达式的求值更加容易。 例如，要评估以上表达式，我们可以在后端执行以下操作：

result = binaryoperation(+, 1, 2)result = binaryoperation(*, result, 4)result = binaryoperation(+, 5, result)result = binaryoperation(-, result, 3)return result

In other words, for each operator (or function) node that the evaluator/compiler/interpreter encounters, it checks to see how many branches there are, and then evaluates the results of all those branches with the operator.

换句话说，对于评估器/编译器/解释器遇到的每个运算符(或功能)节点，它都会检查以查看有多少分支，然后与运算符一起评估所有这些分支的结果。

Okay, the crash course is over, now back to our parser. Our parser will convert the (tokenized) expression to RPN, and later to an AST. So let’s begin implementing it.

好的，崩溃过程已经结束，现在回到我们的解析器。 我们的解析器会将(标记化的)表达式转换为RPN，然后转换为AST。 因此，让我们开始执行它。

调车场算法 (The Shunting-yard algorithm)

Here’s the RPN version of the full algorithm (from our friend Wikipedia), and modified to fit with our tokenizer:

这是完整算法的RPN版本( 来自我们的朋友Wikipedia )，并经过修改以适合我们的标记器：

While there are tokens to be read:

虽然有令牌要读取：

1. Read a token. Let’s call it t

1.读取令牌。 我们称它为t

2. If t is a Literal or Variable, push it to the output queue.

2.如果t是立即数或变量，则将其推送到输出队列。

3. If t is a Function,push it onto the stack.

3.如果t是函数，则将其推入堆栈。

4. If t is a Function Argument Separator (a comma), pop operators off the stack onto the output queue until the token at the top of the stack is a Left Parenthesis.

4.如果t是函数参数分隔符(逗号)，则将运算符从堆栈中弹出到输出队列上，直到堆栈顶部的标记为左括号为止。

5. If t is an Operator:

5.如果t是一个运算符：

a. while there is an Operator token o at the top of the operator stack and either t is left-associative and has precedence is less than or equal to that of o, or t is right associative, and has precedence less than that of o, pop o off the operator stack, onto the output queue;

一个。 而在运算符堆栈的顶部有一个运算符标记o ，或者t是左关联的，并且优先级小于或等于o ，或者t是右关联的，并且优先级小于o ，pop o从操作员堆栈移至输出队列；

b. at the end of iteration push t onto the operator stack.

b。 在迭代结束时，将t推入运算符堆栈。

6. If the token is a Left Parenthesis, push it onto the stack.

6.如果令牌是左括号，请将其推入堆栈。

7. If the token is a Right Parenthesis, pop operators off the stack onto the output queue until the token at the top of the stack is a left parenthesis. Then pop the left parenthesis from the stack, but not onto the output queue.

7.如果令牌是右括号，则弹出运算符将堆栈弹出到输出队列，直到堆栈顶部的令牌是左括号。 然后从堆栈中弹出左括号，但不弹出输出队列。

8. If the token at the top of the stack is a Function, pop it onto the output queue.

8.如果堆栈顶部的令牌是函数，请将其弹出到输出队列中。

When there are no more tokens to read, pop any Operator tokens on the stack onto the output queue.

当没有更多的令牌要读取时，将堆栈上的所有操作员令牌弹出到输出队列中。

Exit.

出口。

(Side note: in case you read the earlier article, I’ve updated the list of recognized tokens to include the Function Argument Separator, aka comma).

(附注：如果你看了以前的文章中， 我已经更新认可令牌的列表，包括功能参数的分隔符，又名逗号)。

The algorithm above assumes the expression is valid. I made it this way so it’s easily understandable in the context of an article. You can view the full algorithm on Wikipedia.

上面的算法假定该表达式有效。 我这样做是为了在文章的上下文中很容易理解。 您可以在Wikipedia上查看完整算法。

You’ll observe a couple of things:

您将观察到以下几点：

• We need two data structures: a stack to hold the functions and operators, and a queue for the output. If you aren’t familiar with these two data structures, here’s a primer for you: if you want to retrieve a value from a stack, you start with the last one you put in, whereas for a queue, you start with the first you put in.

  我们需要两个数据结构：用于保存函数和运算符的堆栈 ，以及用于输出的队列 。 如果您不熟悉这两个数据结构，那么这里为您提供了入门知识：如果您想从堆栈中检索值，则从您放入的最后一个开始，而对于队列，则从第一个开始投放。

// we'll use arrays to represent both of themvar outQueue=[];var opStack=[];

• We need to know the associativity of the operators. Associativity simply means in what order an expression containing several operations of the same kind are grouped in the absence of parentheses. For instance, 2 + 3 + 4 is canonically evaluated from left to right (2+ 3 =5, then 5 + 4 =9), so + has a left associativity. Compare that to 2 ^ 3 ^ 4, which is evaluated as 2 ^81, not 8 ^4. Thus ^ has a right associativity. We’ll package the associativities of the operators well encounter in a Javascriptobject:

  我们需要知道操作员的关联性。 关联性仅表示在没有括号的情况下将包含多个相同类型操作的表达式分组的顺序。 例如，从左到右典范地评估2 + 3 + 4(2+ 3 = 5，然后5 + 4 = 9)，因此+具有左联想性。 将其与2 ^ 3 ^ 4进行比较，该值被评估为2 ^ 81，而不是8 ^ 4。 因此，^具有正确的关联性。 我们将在Javascript对象中遇到的运算符的关联性打包：

var assoc = {  "^" : "right",  "*" : "left",  "/" : "left",  "+" : "left",  "-" : "left" };

• We also need to know the precedence of the operators. The precedence is a sort of ranking assigned to operators, so we can know in what order they should be evaluated if they appear in the same expression. Operators with higher precedence get evaluated first. For instance, * has a higher precedence than +, so 2 + 3 * 4 gets evaluated as 2 + 12, and not 5 * 4, unless parentheses are used. + and – have the same precedence, so 3 + 5 – 2 can be evaluated as either 8–2 or 3+3. Again, we’ll package the operator precedences in an object:

  我们还需要知道运算符的优先级 。 优先级是分配给运算符的一种排序，因此我们可以知道，如果它们出现在相同的表达式中，应该按照什么顺序对其进行评估。 优先级较高的运算符将首先获得评估。 例如，*的优先级高于+，因此2 + 3 * 4的求值为2 + 12，而不是5 * 4，除非使用括号。 +和–具有相同的优先级，因此3 + 5 – 2可以计算为8–2或3 + 3。 同样，我们将运算符优先级打包在一个对象中：

var prec = {  "^" : 4,  "*" : 3,  "/" : 3,  "+" : 2,  "-" : 2 };

Now, lets update our Token class so that we can easily access the precedence and associativity via methods:

现在，让我们更新Token类，以便我们可以通过方法轻松访问优先级和关联性：

Token.prototype.precedence = function() {  return prec[this.value]; };  Token.prototype.associativity = function() {  return assoc[this.value]; };

• We need a method that allows us to peek at the stack (to check the element at the top without removing it), and one that allows us to pop from the stack (retrieve and remove the item at the top). Fortunately, Javascript arrays already have a pop() method, so all we need to do is implement a peek() method. (Remember, for stacks, the element at the top is the one we added last.)

  我们需要一种方法，它允许我们窥视堆栈(在不删除元素的情况下检查顶部的元素)，以及一种允许我们从堆栈中弹出 (检索并删除顶部的元素)的方法。 幸运的是，Javascript数组已经具有pop()方法，因此我们所需要做的就是实现peek()方法。 (请记住，对于堆栈，顶部的元素是我们最后添加的元素。)

Array.prototype.peek = function() {  return this.slice(-1)[0]; //retrieve the last element of the array };

So here’s what we have:

所以这就是我们所拥有的：

function tokenize(expr) {  ...   // just paste the tokenizer code here}

function parse(inp){ var outQueue=[]; var opStack=[];

Array.prototype.peek = function() {  return this.slice(-1)[0]; };

var assoc = {  "^" : "right",  "*" : "left",  "/" : "left",  "+" : "left",  "-" : "left" };

var prec = {  "^" : 4,  "*" : 3,  "/" : 3,  "+" : 2,  "-" : 2 };

Token.prototype.precedence = function() {  return prec[this.value]; };  Token.prototype.associativity = function() {  return assoc[this.value]; };

//tokenize var tokens=tokenize(inp);

tokens.forEach(function(v) {   ...   //apply the algorithm here });

return outQueue.concat(opStack.reverse());  // list of tokens in RPN}

I won’t delve into the algorithm’s implementation so I don’t bore you. It’s a pretty straightforward task, practically a word-for-word translation of the algorithm to code, so at the end of the day, here’s what we have:

我不会深入研究该算法的实现，所以不会让您感到厌烦。 这是一项非常简单的任务，实际上是将算法逐字翻译为代码，因此，总而言之，这就是我们需要的：

The toString function simply formats our RPN list of tokens in a readable format.

toString函数仅以可读格式简单地格式化我们的RPN令牌列表。

And we can test out our infix-to-postfix parser:

我们可以测试一下从后缀到后缀的解析器：

var rpn = parse("3 + 4 * 2 / ( 1 - 5 ) ^ 2 ^ 3");console.log(toString(rpn));

Output:

输出：

3 4 2 * 1 5 - 2 3 ^ ^ / +

RPN!!

RPN ！！

是时候种一棵树了 (Time to plant a tree)

Now, let’s modify our parser so it returns an AST.

现在，让我们修改解析器，使其返回AST。

To generate an AST instead of RPN, we’ll need to make a few modifications:

要生成AST而不是RPN，我们需要进行一些修改：

• We’ll create an object to represent a node in our AST. Each node has a value and two branches (which may be null):

  我们将创建一个对象来表示AST中的节点。 每个节点都有一个值和两个分支(可以为null )：

function ASTNode(token, leftChildNode, rightChildNode) {   this.token = token.value;   this.leftChildNode = leftChildNode;   this.rightChildNode = rightChildNode;}

• The second thing we’ll be doing is changing our output data structure to a stack. While the actual code for this is just to change the line var outQueue = [] to var outStack = [] (because it remains an array), the key change is in our understanding and treatment of this array.

  我们要做的第二件事是将输出数据结构更改为堆栈。 尽管实际的代码只是将行var outQueue = []更改为var outStack = [] (因为它仍然是一个数组)，但是关键的变化是我们对这个数组的理解和处理。

Now, how is our infix-to-AST algorithm going to run? Basically, the same algorithm, with a few modifications:

现在，我们的infix-to-AST算法将如何运行？ 基本上，相同的算法进行了一些修改：

1. Instead of pushing a Literal or Variable token onto our outQueue, we push a new node whose value is the token, and whose branches are null onto our outStack

   与其将Literal或Variable令牌推送到我们的outQueue ，我们不将其值为令牌且其分支为null的新节点推送到我们的outStack

2. Instead of popping an Operator/Function token from the opStack , we replace the top two nodes on the outStack with a single node whose value is the token, and that has those two as its branches. Let’s create a function that does that:

   我们没有从opStack中弹出操作员/功能令牌， outStack用一个值为令牌的单个节点替换了opStack的前两个节点，并以这两个为分支。 让我们创建一个执行此操作的函数：

Array.prototype.addNode = function (operatorToken) {  rightChildNode = this.pop();  leftChildNode = this.pop();  this.push(new ASTNode(operatorToken, leftChildNode, rightChildNode)); }

3. Our parser should now return a single node, the node at the top of our AST. Its two branches will contain the two child nodes, whose branches will contain their children,and so on, in a recursive manner. For instance, for an expression like 3 + 4 * 2 / ( 1–5 ) ^ 2 ^ 3, we expect the structure of our output node to be like this (in a horizontal form):

3.现在，解析器应该返回一个节点，该节点位于AST的顶部。 它的两个分支将包含两个子节点，其子节点将以递归方式包含其子节点，依此类推。 例如，对于像3 + 4 * 2 /(1–5)^ 2 ^ 3这样的表达式，我们希望输出节点的结构像这样(水平形式)：

+ => 3 => null       => null  => / => * => 4 => null                 => null            => 2 => null                 => null       => ^ => - => 1 => null                      => null                 => 5 => null                      => null            => ^ => 2 => null                      => null                 => 3 => null                      => null

In the diagram above, the => represent the branches of the node (the top node is the left branch, the bottom one is the right branch). Each node has two branches, and the nodes at the end of the tree have theirs pointing to null

在上图中，=>代表节点的分支(顶部节点是左侧分支，底部节点是右侧分支)。 每个节点有两个分支，并在树的末端节点具有指向他们to n ULL

So, if we put all this together, here is the code we come up with:

因此，如果我们将所有这些放在一起，这是我们想到的代码：

And if we demo it:

如果我们进行演示：

//a little hack I put together so it prints out in a readable formASTNode.prototype.toString = function(count) {   if (!this.leftChildNode && !this.rightChildNode)     return this.token + "\t=>null\n" + Array(count+1).join("\t") + "=>null";   var count = count || 1;   count++;   return this.token + "\t=>" + this.leftChildNode.toString(count) + "\n" + Array(count).join("\t") + "=>" + this.rightChildNode.toString(count);};

var ast = parse("3 + 4 * 2 / ( 1 - 5 ) ^ 2 ^ 3");console.log("" + ast);

And the result:

结果：

Slowly, but surely, we’re getting closer to understanding what makes compilers and interpreters tick! Admittedly, the working of modern-day programming languages and their toolkits is a lot more complex than what we’ve looked at thus far, but I hope this proves to be an easy-to-understand introduction to them. As a number of people have pointed out, tools exist to automatically generate tokenizers and parsers, but its often nice to know how something actually works.

慢慢地，但是可以肯定的是，我们越来越接近于理解是什么使编译器和解释器滴答作响了！ 诚然，现代编程语言及其工具包的工作要比到目前为止所看到的要复杂得多，但是我希望这对他们来说是一个易于理解的入门。 正如许多人指出的那样，存在可以自动生成标记器和解析器的工具，但是了解某些东西的实际工作通常很高兴。

The concepts we covered in this article and the previous are very interesting topics in the field of computer science and language theory. I still have a lot to learn about them, and I encourage you to go ahead and research them if they interest you. And drop me a line to let me know about your progress. Peace!

我们在本文和上一篇文章中介绍的概念是计算机科学和语言理论领域中非常有趣的主题。 我对它们仍然有很多了解，如果您感兴趣，我鼓励您继续研究它们。 然后给我留言 ，让我知道你的进步。 和平！

翻译自: https://www.freecodecamp.org/news/parsing-math-expressions-with-javascript-7e8f5572276e/

javascript解析