The next call to the scanner will begin where the last ended, looking at that space. The machine will move from state 0 to state (say) 11 and copy 'f' to the buffer then move from state 11 back to state 11 and copy 'o' to the buffer - twice (two 'o'-s) then seeing the space it will move to some final state (say) 12, and return from the call, returning the object. Example: Suppose the scanner is called when the next few characters are 'foo '. Repeat until a final state is reached, then return an object identifying the type of lexical token and the value from the buffer. Using the lookup table, calculate the next state and 'consume' the next character. ).Д®ach call to the scanner follows this process: Begin in a common start state (e.g 0), looking at the next character.

This is (conceptually) implemented as a simple two-dimensional table, where rows are indexed by character values and columns are indexed by DFA state numbers (actually the rows are usually indexed by character classes - e.g 'letter' instead of 'a', 'b'. The scanner combines the simple DFAs for these expressions into a single DFA for their union. Each token type has its own structure as a character sub-sequence, representable as a regular expression which in turn is translated to a simple DFA for example: a name is (in most languages) a sequence of characters consisting of followed by zero or more. The scanner is implemented as a DFA which recognizes any token. The scanner is essentially a subroutine which the parser calls to read the raw character stream and extract & return the next lexical token. In principle, the parser can be written to inspect each character in the sequence directly, but the parser will be more comprehensible and efficient if built to inspect higher-level lexical tokens - names, numbers, quoted-strings, miscellaneous operator symbols, parentheses, etc.

The goal of any parser is to verify that a sequence of characters is a string in a specific language.

Finite state automata lexer