Token Definitions
One of the first steps to start with the project is to write a lexical analyzer. Your lexical
analyzer is supposed to convert a sequence of characters from the input file into a
sequence of tokens and verify if they are valid tokens or not.
Definition of some of the token is given below.
 Identifier: Any sequence of characters which may contain alphabets, digits from
0 to 9 or an underscore. The sequence must start with an alphabet or an
underscore.
 Integer: A sequence of characters which may contain any combination of digits
from 0 to 9.
 White Space: Any sequence of characters containing any combination of single
space, form feed, horizontal tab and vertical tab.
 Char: A sequence of three characters which contains a single character between
two single quotes. The single character between the two quotes can be any
character except a single quote itself. For example, ‘’’ is an invalid character.
 String: A sequence of characters which contains any number of characters
between two double quotes. The character sequence between the two double
quotes can be any character except a double quote itself. For example,
“Hello”how” is an invalid string.
 Comment: A sequence of characters which starts with { and ends with a }. This
type of comment can cross line boundaries.
Some of the pre-defined tokens in the language are given below:
No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Token
\n
program
var
const
type
function
return
begin
end
:=:
:=
output
if
then
else
while
do
case
of
..
Description
Newline
Start of Program
Variable
Constant
To define a data type
To define a function
return from function
start of a block
end of a block
swap
assignment operator
output an expression or string
keyword
keyword
keyword
keyword for loop
keyword for loop
keyword
keyword
dots for case expression
21
22
23
24
25
26
27
28
29
30
otherwise
repeat
for
until
loop
pool
exit
<=
<>
<
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
>=
>
=
mod
and
or
not
read
succ
pred
chr
ord
eof
{
}
:
;
.
,
(
)
+
*
/
keyword
keyword for repeat-until loop
keyword for loop
keyword for repeat-until loop
keyword for loop-pool loop
keyword for loop-pool loop
keyword
less than equal to binary operator
not equal to binary operator
less than binary operator
greater than equal to binary
operator
greater than binary operator
equal to binary operator
modulus binary operator
and binary operator
or binary operator
not unary operator
read an identifier
successor of an ordinal value
predecessor of a ordinal value
keyword for character function
keyword for ordinal function
keyword for end of file
begin comment
end comment
colon
semi colon
single dot
comma
Left parenthesis
Right parenthesis
plus
minus
multiply
divide