The scanner generator

Translates patterns into hierarchically-related deterministic FSMs
Produces state machines and a lexer
Execution and reduction of state machines guided by heuristics
- Reduce highest match in hierarchy
- Reduce longest match
- Prune paths with more than N tokens