@@ -97,11 +97,32 @@ This diagram shows the recogniser combinator with some of its constructors:
97 :height: 249px
97 :height: 249px
98 :alt: recogniser data structure
98 :alt: recogniser data structure
100+Whitespace and Comments
101+-----------------------
103+An important reason to have a lexer is to simplify the parser when whitespace
104+or comments are required. Without this the parser would require a match for
105+every combination, with and without, whitespace and comments.
107+In some languages the lexer needs to remove whitespace and comments so that
108+they don't appear in the token list. In some cases whitespace is significant
109+but a single whitespace token can stand for any number of spaces, tabs,
110+carriage returns and so on.
112+In some cases, such as when parsing the Idris language itself, whitespace
113+is important to indicate blocks. In this case we might want to output a
114+start-of-block token when the indent increases and an end-of-block token
115+when the indent decreases.
117+This topic needs to be more fully expanded here. Until someone does that, a
118+possible way to get more information is to see how it is done in Idris 2
119+_`here`: https://github.com/edwinb/Idris2/blob/master/src/Parser/Lexer.idr
100Simple Expression Example for Lexer
121Simple Expression Example for Lexer
101-----------------------------------
122-----------------------------------
103On this page we will implement a lexer to lex a very simple expression as
124On this page we will implement a lexer to lex a very simple expression as
104−a running example, on the next page we will go on to implement a parser for it.
125+a running example, on the next page, we will go on to implement a parser for it.
106First import the lexer and parser code:
127First import the lexer and parser code: