1. Lexical Analysis (Scanner): Reads the input from a file as a text stream and tries to divide the entire text into Tokens. The rules for creating these tokens are predefined for a given high-level language. Every token has a token type.

Example:

int x = 4;

| Token |      Token Type     |
|-------|---------------------|
| int   | keyword             |
| a     | identifier/variable |
| =     | operator/assignment |
| 5     | decimal number      |
| ;     | delimiter           |
|-------|---------------------|

The job of the scanner is to read the entire program as a text stream, and categorize each as a valid token. If it finds an invalid token, it should return an error.

After performing the lexical analysis, we should get as output a list of tokens. This list of tokens are then taken to the second stage of the compiler.

|--------------------|----|---------|----|----------------|----|--------|----|-----|----|------------------|
| High Level Program | -> | Scanner | -> | List of Tokens | -> | Parser | -> | AST | -> | Semantic Checker |
|--------------------|----|---------|----|----------------|----|--------|----|-----|----|------------------|

Lexical: anything related to words.
- Tasks in Compiler Design:
  i. Process the input characters that constitute a high level program into valid set of tokens.
  ii. It skips the comments and white spaces while creating these tokens.
  iii. If any erroneous input is provided by the user, the lexical analizer should correlate that error to a file source and line number.

- Terminology:
  i. Token: a set of input strings which are related through a similar pattern. Any word that starts with a letter and contains any number or letter in between is called an Identifier.
  ii. Lexeme: the actual input string which represents the token.
  iii. Pattern: rules that a lexical analyzer follows to create a token.

Ex.

x = x * (acc + 123)

|     Token     | Lexeme |
|---------------|--------|
| Identifier    | x      |
| Operator Eq   | =      |
| Identifier    | x      |
| Operator Mult | *      |
| L_Paren       | (      |
| Identifier    | acc    |
| Operator Plus | +      |
| Integer Const | 123    |

2. Syntax Checker (Parser): it tries to follow the rules of the grammar of a given language and creates a data structure, abstract syntax tree or parse tree.
                    G = (V, T, P, S)

A grammar is a four-tuple:
  - V: set of non-terminals.
  - T: set of terminals.
  - P: set of rules.
  - S: starting non-terminal.

                            S
                        /   |   \
                    V       V       V
                   / \     / \     / \
                  T   T   T   T   T   T

It follows the rules of the grammar to create this abstract syntax tree.It removes ambiguity by following association rules, and precedence rules.

Ex.

a + b * c

The operator * has a higher precendence, and hence b should be attached to * instead of +.

3. Semantic Checker: tries to follow certain semantic rules of the given language.

Ex. A variable is used before defined, but it is not defined, then it is a semantic error.

x = 10;

It is syntaxically correct, but semantically incorrect as it was never defined

end



Stuff to work on.......
1.  type data_type = PathType | StrType | IntType | BoolType | VoidType | DictType | ListType

2.  data_type is used in variable declaration
    maybe able to declare variable as void
    so might need to have separate, datatype and return type

3.  for is not correct… ours is the in version….

4.  our func type has local decl in-built
    may have problem because then we don't allow blocks of code inside function to have var decl
    def void main(){
      int a; // this works
      {
        int b; // this fails, because statement doesn't have declaration
      }
    }
    // if we make this change, then we will also need to update the typecheck and parser and fdl string // statements
5.  add Break to our list of statements
6.  SAST can be used to differentiate between overloaded operators, like + on int and + on strings ??
    should we add overload ??
7.  string_of_vtype is in typecheck and symboltable and fdl, need in a central place, maybe AST
8.  add symboltable and typecheck to makefile
9.  char* need to give it memory or max size,....
    can we use length of string from OCaml
10. preprocessor might make a mistake in case path has '.' other thatn file name eg: ./preprocessor/test.     fdl
11. can declare variables only at the start of the function, not after