on master help sign in
Text.Token: utilities for token value conversion

This module contains: * `Token` - This record type is parameterized by some type `k`. It holds a value of type `k` representing the kind of token that was recognised, along with the text that was recognised for this particular `Token`. * `TokenKind` - Implementing this interface for some type `k` allows the user to specify how the recognised text from a token should convert into a value. There are two functions in the interface. * `tokType` - Determines the type that a recognised string should convert into for this kind of token. * `tokValue` - Given a kind and a string, convert the string into a value of the correct type. Using `Token k` as the token input to your grammar (and output of your lexer) allows for a couple advantages. 1. When a new kind of token is added, the compiler no longer accepts the program until it is accounted for in the `TokenKind` implementation. 2. The information about the recotnised text is kept for as long as possible. Since it's stored on the `Token` record, you don't lose information about the recognised text until a parse rule succeeds.

commitbc8da8d0a088994bd0695ab67d3f8526a3ec5402 authorMichael Morgan <morgan.michael@me.com> dateThu 9 Nov 2017, 14:22 (9 years ago) parent237f4b8
2 files changed+39
libs/contrib/Text/Token.idr+38 libs/contrib/contrib.ipkg+1
libs/contrib/Text/Token.idrnew file+38
@@ -0,0 +1,38 @@
1+module Text.Token
2+
3+%access public export
4+%default total
5+
6+||| For a type `kind`, specify a way of converting the recognised
7+||| string into a value.
8+|||
9+||| ```idris example
10+||| data SimpleKind = SKString | SKInt | SKComma
11+|||
12+||| TokenKind SimpleKind where
13+||| tokType SKString = String
14+||| tokType SKInt = Int
15+||| tokType SKComma = ()
16+|||
17+||| tokValue SKString x = x
18+||| tokValue SKInt x = cast x
19+||| tokValue SKComma x = ()
20+||| ```
21+interface TokenKind (k : Type) where
22+ ||| The type that a token of this kind converts to.
23+ tokType : k -> Type
24+
25+ ||| Convert a recognised string into a value. The type is determined
26+ ||| by the kind of token.
27+ tokValue : (kind : k) -> String -> tokType kind
28+
29+||| A token of a particular kind and the text that was recognised.
30+record Token k where
31+ constructor Tok
32+ kind : k
33+ text : String
34+
35+||| Get the value of a `Token k`. The resulting type depends upon
36+||| the kind of token.
37+value : TokenKind k => (t : Token k) -> tokType (kind t)
38+value (Tok k x) = tokValue k x
libs/contrib/contrib.ipkg+1
@@ -84,3 +84,4 @@ modules = CFFI
8484 , Text.PrettyPrint.WL
8585
8686 , Text.Quantity
87+ , Text.Token