New lexer
vector
stacks
Well-formed expression
Infix and postfix expressions
Stacks and Applications
Agenda
• Lexer with richer vocabulary
• Vectors
• Stacks
• Well-ballanced expressions
• Infix and postfix expressions
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
1
More types of tokens
Programming with a (Deterministic) Finite Automata (DFA)
New member function returning a token vector
AN IMPROVED LEXER
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
2
More “Sophisticated” Types of Tokens
• NUMBER
– 5234 or .1234 or 10. or 612.6123
• OPERATOR
– one of five operators +-*/=
• DELIM
– bracket delimiters such as {}[]()
• COMMENT
– all characters that follow a # character to the end of the source
file/string or until the end of line '\n'character is reached
• IDENT (defined differently from before)
– Alphanumeric characters [a-z, A-Z, 0-9, and _], starting with [a-z,A-Z,
_]
• STRING
– All characters enclosed in a pair of “…”, no escape
• Unrecognized tokens are considered to be syntax error
• DFA for more complex (but regular) languages
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
3
New Member Function
• vector<Token> Lexer::tokenize()
– Returns a vector of remaining tokens
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
4
Vector in C++
•
•
•
•
•
•
•
•
•
•
vector<int> myvec;
myvec.pushback(123);
myvec.pushback(456);
Access using myvec[0], myvec[1],
myvec.at(0), myvec.at(1)
myvec.front() // first element
myvec.back() // last element
myvec.insert(position)
myvec.size()
myvec.pop_back()
…
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
5
- Well-formed expressions
- stacks
- Infix, postfix
STACKS AND APPLICATIONS
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
6
Stack: push(obj), pop(), and top()
C
A
B
push
push
B
5/28/2016
A
A
B
B
CSE 250, SUNY Buffalo, © Hung Q. Ngo
push
pop
top
B
7
C++ stack
stack<string> st;
st.push("abc");
st.push("xyz");
st.pop();
st.push("ABC");
cout << st.size() << endl; // 2
cout << st.top() << endl; // ABC
st.pop();
cout << st.size() << " " << st.top() << endl; // 1 "abc”
st.pop();
if (st.empty())
cout << "EMPTY" << endl; // EMPTY
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
8
Many applications
• OS manages process’ function calls with a stack
• Back-tracking in search algorithms
– DFS in a graph traversal (extremely useful!!!)
– Search for a path to escape a maze
• Parse open-close tags/braces in languages
– HTML, XML, C++, Java
• Infix/prefix/postfix-expression evaluation
• Reverse stuff
– Detect palindrome
– Convert decimal to binary
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
9
HTML File
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
10
Stack Application 1: Well-Formed Expressions
Also called “balanced expressions”
• ([this is] { a number } 12345) # well-formed
• ([this is] { a number ) 12345} # not wf
• {[(34+4)/5] + 7}/4
# wf
• {[(34+4)/5} + 7]/4
# not wf
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
11
(Recursive) Definition of WFE
• The empty sequence is well-formed.
• If A and B are well-formed, then the
concatenation AB is well-formed
• If A is well-formed, then [A], {A}, and (A) are
well-formed.
• How do we check if an expression is WF?
– Use a stack!
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
12
Algorithm for Recognizing WFE
• Read the next delimiter token.
• If it is an open delimeter (i.e. [({)
– push it into the stack.
• If it is a close delimiter (i.e. ])})
– match it with a corresponding open delimiter in the
stack ([ with ] and so on).
– If there is no match  not WF
– If there is a match, stack.pop() and discard both
• When there is no more token left
– If the stack is empty  WF
– If the stack is not empty  not WF
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
13
Recognizing WFE
[ { (
5/28/2016
{
} [
] )
CSE 250, SUNY Buffalo, © Hung Q. Ngo
} ]
14
Stack Application 2: Infix vs Postfix Expressions
• Infix:
5 + 4*5/2 - 3
4 5
4 5
5 4 5
5 4 5
5 20
5
5/28/2016
*
* 2 /
* 2 / +
* 2 / +
2 / +
10
+
15
12
3
3
3
3
CSE 250, SUNY Buffalo, © Hung Q. Ngo
-
Postfix expression
15
Stack Application 2: Infix vs Postfix Expressions
• Infix: 5 + 4*5/2 - 3
• Postfix: 5 4 5 * 2 / + 3 –
• Infix: (5+4)*5/2 - 3
• Postfix: 5 4 + 5 * 2 / 3 –
• In: ([6-2*3+1]/2 + 5)*4 – 3
• Post: 6 2 3 * - 1 + 2 / 5 + 4 * 3 5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
16
Postfix Expression Evaluation Algorithm
• Initialize an empty stack
• While (there is still a token to read)
– read the token t
– if t is an operand, push it onto the stack
– if t is an operator,
• pop two operands from the stack, compute the result (using
t)
// if there is division by zero, scream loud
• push the result back onto the stack
// if there is less than two operands, scream real loud
• In the end, if there is one number in the stack,
output it.
– // If there > 1 number in the stack, scream foul.
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
17
Postfix Evaluation
5 4
Bottom
5/28/2016
5
* 2
/
+
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
3
-
Top
18
Postfix Evaluation
4
5
* 2
/
+
3
-
5
Bottom
5/28/2016
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
19
Postfix Evaluation
5
5
Bottom
5/28/2016
* 2
/
+
3
-
4
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
20
Postfix Evaluation
* 2
5
Bottom
5/28/2016
4
/
+
3
-
5
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
21
Postfix Evaluation
2
/
+
3
-
*
5
Bottom
5/28/2016
4
5
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
22
Postfix Evaluation
2
/
+
3
-
20
5
Bottom
5/28/2016
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
23
Postfix Evaluation
/
5
Bottom
5/28/2016
20
+
3
-
2
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
24
Postfix Evaluation
+
3
-
/
5
Bottom
5/28/2016
20
2
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
25
Postfix Evaluation
+
5
Bottom
5/28/2016
3
-
10
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
26
Postfix Evaluation
3
-
+
5
Bottom
5/28/2016
10
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
27
Postfix Evaluation
3
-
15
Bottom
5/28/2016
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
28
Postfix Evaluation
-
15
Bottom
5/28/2016
3
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
29
Postfix Evaluation
15
Bottom
5/28/2016
3
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
30
Postfix Evaluation
5
Bottom
5/28/2016
Stack watching TV in bed
CSE 250, SUNY Buffalo, © Hung Q. Ngo
Top
31
How about Infix Expression?
• Shunting yard algorithm
– Invented by Dijkstra 1961.
– Used to convert infix to postfix
– Or, to evaluate infix expressions directly
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
32
Infix to Postfix: the Shunting Yard Algorithm
• Assume operands & binary operators only
• Initialize op_stack = empty.
• while (there is still next token t)
– If (t is an operand)
• Add its value to output
– Else If (t is an operator)
• while (precedence(t) ≤ precedence(stack.top())
– Add stack.top() to output
– stack.pop()
• stack.push(operator t)
• pop remaining operators from stack and add to
output one by one
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
33
Infix to Postfix: the Shunting Yard Algorithm
A + B * C / E - F
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
34
Infix to Postfix: the Shunting Yard Algorithm
+ B * C / E - F
A
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
35
Infix to Postfix: the Shunting Yard Algorithm
B * C / E - F
A
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
36
Infix to Postfix: the Shunting Yard Algorithm
* C / E - F
A B
Precedence(*) > Precedence(+)
op_stack.push(*)
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
37
Infix to Postfix: the Shunting Yard Algorithm
C / E - F
A B
*
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
38
Infix to Postfix: the Shunting Yard Algorithm
/ E - F
A B C
Precedence(/) ≤ Precedence(*)
Move stack.top() to output
*
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
39
Infix to Postfix: the Shunting Yard Algorithm
/ E - F
A B C *
Precedence(/) ≤ Precedence(*)
Move stack.top() to output
Precedence(/) > Precedence(+)
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
40
Infix to Postfix: the Shunting Yard Algorithm
E - F
A B C *
Precedence(/) ≤ Precedence(*)
Move stack.top() to output
Push / on top of the stack
/
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
41
Infix to Postfix: the Shunting Yard Algorithm
- F
A B C * E
/
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
42
Infix to Postfix: the Shunting Yard Algorithm
- F
A B C * E /
+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
43
Infix to Postfix: the Shunting Yard Algorithm
- F
A B C * E / +
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
44
Infix to Postfix: the Shunting Yard Algorithm
F
A B C * E / +
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
45
Infix to Postfix: the Shunting Yard Algorithm
A B C * E / + F
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
46
Infix to Postfix: the Shunting Yard Algorithm
A B C * E / + F -
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
47
Some Issues
• The shunting yard algorithm works if the input
is a valid infix expression
• But it is not an algorithm to detect whether
the input is a valid infix expression
– E.g., consider giving it A B C + - D *
– I’ll leave this as an exercise
• So far: no [], {}, () yet
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
48
Shunting Yard with Parentheses
[(A+B)*C–D*E+F]/G
When [, (, { is seen, push down
When ], ), } is seen, pop all operators
up until matching [, (, { is found
Remove matching pair
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
49
Shunting Yard with Parentheses
(A+B)*C–D*E+F]/G
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
50
Shunting Yard with Parentheses
A+B)*C–D*E+F]/G
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
51
Shunting Yard with Parentheses
+B)*C–D*E+F]/G
A
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
52
Shunting Yard with Parentheses
B)*C–D*E+F]/G
A
+
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
53
Shunting Yard with Parentheses
)*C–D*E+F]/G
A B
+
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
54
Shunting Yard with Parentheses
)*C–D*E+F]/G
A B
+
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
55
Shunting Yard with Parentheses
)*C–D*E+F]/G
A B +
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
56
Shunting Yard with Parentheses
)*C–D*E+F]/G
A B +
(
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
57
Shunting Yard with Parentheses
*C–D*E+F]/G
A B +
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
58
Shunting Yard with Parentheses
C–D*E+F]/G
A B +
*
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
59
Shunting Yard with Parentheses
–D*E+F]/G
A B + C
*
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
60
Shunting Yard with Parentheses
–D*E+F]/G
A B + C *
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
61
Shunting Yard with Parentheses
D*E+F]/G
A B + C *
−
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
62
Shunting Yard with Parentheses
*E+F]/G
A B + C * D
−
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
63
Shunting Yard with Parentheses
E+F]/G
A B + C * D
*
−
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
64
Shunting Yard with Parentheses
+F]/G
A B + C * D E
*
−
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
65
Shunting Yard with Parentheses
+F]/G
A B + C * D E *
−
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
66
Shunting Yard with Parentheses
+F]/G
A B + C * D E *-
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
67
Shunting Yard with Parentheses
F]/G
A B + C * D E *-
+
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
68
Shunting Yard with Parentheses
]/G
A B + C * D E *- F
+
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
69
Shunting Yard with Parentheses
]/G
A B + C * D E *- F
+
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
70
Shunting Yard with Parentheses
]/G
A B + C * D E *- F+
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
71
Shunting Yard with Parentheses
]/G
A B + C * D E *- F+
[
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
72
Shunting Yard with Parentheses
/G
A B + C * D E *- F+
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
73
Shunting Yard with Parentheses
G
A B + C * D E *- F+
/
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
74
Shunting Yard with Parentheses
A B + C * D E *- F+G
/
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
75
Shunting Yard with Parentheses
A B + C * D E *- F+G/
...
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
76
Evaluate Infix Directly: Rough Idea
• Use 2 stacks: operand stack, operator stack
• If t is an operand, push it on operand stack
• Else if t is one of + - * /
– while (precedence(t) ≤ precedence(stack.top())
• Evaluate stack.top()
– Push t on top of operator stack
• Else if t is one of ( [ {
– Push t on top of operator stack
• Else if t is one of ) ] }
– Evaluate operators on top until ( [ { seen, match up
5/28/2016
CSE 250, SUNY Buffalo, © Hung Q. Ngo
77