@@ -12,10 +12,12 @@ running example, we may want to make spaces significant in the future. For
12example, we might want to implement function application expressed by
12example, we might want to implement function application expressed by
13juxtaposition as in Haskell and Idris like this: ``f x``.
13juxtaposition as in Haskell and Idris like this: ``f x``.
15−Also, I'm not sure if it would be problem for error messages and so on if some
15+So, on this page, we will add a token for whitespace and comments. We will
16−text does not have corresponding tokens.
16+then consider two ways to process this:
18−So, on this page, we will add a token for whitespace and comments.
18+- Filter all whitespace tokens from the lexer results before passing to
20+- Process the whitespace tokens in the parser.
20Whitespace and Comments in Lexer
22Whitespace and Comments in Lexer
21--------------------------------
23--------------------------------
@@ -70,10 +72,31 @@ like this:
70 spaces : Lexer
72 spaces : Lexer
71 spaces = some space
73 spaces = some space
75+If these whitespace tokens are not significant, that is, they can appear
76+anywhere and they are optional then we can filter them out before the parser
81+ processWhitespace : (List (TokenData ExpressionToken), Int, Int, String)
82+ -> (List (TokenData ExpressionToken), Int, Int, String)
83+ processWhitespace (x,l,c,s) = ((filter notComment x),l,c,s) where
84+ notComment : TokenData ExpressionToken -> Bool
85+ notComment t = case tok t of
89+ calc : String -> Either (ParseError (TokenData ExpressionToken))
90+ (Integer, List (TokenData ExpressionToken))
91+ calc s = parse expr (fst (processWhitespace (lex expressionTokens s)))
73Whitespace and Comments in Parser
94Whitespace and Comments in Parser
74---------------------------------
95---------------------------------
97+If we sometimes require whitespace to be significant then we can't filter
98+them out as above. In this case the ``Comment`` tokens are sent to the parser
76−Now that there are ``Comment`` tokens the parser needs to be able to handle them.
99+which now needs to be able to handle them.
78.. code-block:: idris
101.. code-block:: idris
@@ -86,7 +109,7 @@ So far we don't have any syntax that requires spaces to be significant so we
86need to define the grammar so that it will parse with, or without, spaces.
109need to define the grammar so that it will parse with, or without, spaces.
87This needs to be done in a systematic way, here I have defined the grammar so
110This needs to be done in a systematic way, here I have defined the grammar so
88that there is an optional space to the right of every atom or operator.
111that there is an optional space to the right of every atom or operator.
89−First we add versions of ``intLiteral`` , ``openParen`` , ``closeParen``
112+First add versions of ``intLiteral`` , ``openParen`` , ``closeParen``
90and ``op`` that allow optional spaces/comments to the right of them:
113and ``op`` that allow optional spaces/comments to the right of them:
92.. code-block:: idris
115.. code-block:: idris
@@ -104,7 +127,7 @@ and ``op`` that allow optional spaces/comments to the right of them:
104 opC : String -> Rule Integer
127 opC : String -> Rule Integer
105 opC s = ((op s) <* commentSpace) <|> (op s)
128 opC s = ((op s) <* commentSpace) <|> (op s)
107−Then we just use these functions instead of the original functions:
130+Then just use these functions instead of the original functions:
109.. code-block:: idris
132.. code-block:: idris