@@ -12,10 +12,12 @@ running example, we may want to make spaces significant in the future. For
1212example, we might want to implement function application expressed by
1313juxtaposition as in Haskell and Idris like this: ``f x``.
1414
15−Also, I'm not sure if it would be problem for error messages and so on if some
16−text does not have corresponding tokens.
15+So, on this page, we will add a token for whitespace and comments. We will
16+then consider two ways to process this:
1717
18−So, on this page, we will add a token for whitespace and comments.
18+- Filter all whitespace tokens from the lexer results before passing to
19+ the parser.
20+- Process the whitespace tokens in the parser.
1921
2022Whitespace and Comments in Lexer
2123--------------------------------
@@ -70,10 +72,31 @@ like this:
7072 spaces : Lexer
7173 spaces = some space
7274
75+If these whitespace tokens are not significant, that is, they can appear
76+anywhere and they are optional then we can filter them out before the parser
77+gets them like this:
78+
79+.. code-block:: idris
80+
81+ processWhitespace : (List (TokenData ExpressionToken), Int, Int, String)
82+ -> (List (TokenData ExpressionToken), Int, Int, String)
83+ processWhitespace (x,l,c,s) = ((filter notComment x),l,c,s) where
84+ notComment : TokenData ExpressionToken -> Bool
85+ notComment t = case tok t of
86+ Comment _ => False
87+ _ => True
88+
89+ calc : String -> Either (ParseError (TokenData ExpressionToken))
90+ (Integer, List (TokenData ExpressionToken))
91+ calc s = parse expr (fst (processWhitespace (lex expressionTokens s)))
92+
93+
7394Whitespace and Comments in Parser
7495---------------------------------
7596
76−Now that there are ``Comment`` tokens the parser needs to be able to handle them.
97+If we sometimes require whitespace to be significant then we can't filter
98+them out as above. In this case the ``Comment`` tokens are sent to the parser
99+which now needs to be able to handle them.
77100
78101.. code-block:: idris
79102
@@ -86,7 +109,7 @@ So far we don't have any syntax that requires spaces to be significant so we
86109need to define the grammar so that it will parse with, or without, spaces.
87110This needs to be done in a systematic way, here I have defined the grammar so
88111that there is an optional space to the right of every atom or operator.
89−First we add versions of ``intLiteral`` , ``openParen`` , ``closeParen``
112+First add versions of ``intLiteral`` , ``openParen`` , ``closeParen``
90113and ``op`` that allow optional spaces/comments to the right of them:
91114
92115.. code-block:: idris
@@ -104,7 +127,7 @@ and ``op`` that allow optional spaces/comments to the right of them:
104127 opC : String -> Rule Integer
105128 opC s = ((op s) <* commentSpace) <|> (op s)
106129
107−Then we just use these functions instead of the original functions:
130+Then just use these functions instead of the original functions:
108131
109132.. code-block:: idris
110133