Weekend project: Lexical analysis & expression trees
Last Friday I developed an itch. After watching a range of YouTube videos about programming languages I was reminded of the fact that I’ve never written a lexer. I’ve known the basic theory of lexers for years but I’ve never actually written one, despite coming across challenges in my work that could benefit from even the simplest lexical analysis. I decided to write a library that I could use in Cuterm to evaluate simple math expressions as well as in Novelty for conditional expressions.
So what is a lexer? Given an arbitrary string of characters a lexer produces a string of tokens: