3#+CAPTION: Bloodhound (dog)
6* Elasticsearch client and query DSL for Haskell
10Because you're tired of obnoxious errors like [[http://i.imgur.com/FKtZYIP.png][this]] and want types to guide your use of the API.
14Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but being compared to clients in other languages the story here so far is good.
24-- Formatted for use in ghci, so there are "let"s in front of the decls.
26-- if you see :{ and :}, they're so you can copy-paste
27-- the multi-line examples into your ghci REPL.
30import Database.Bloodhound
32import Data.Either (Either(..))
33import Data.Maybe (fromJust)
34import Data.Time.Calendar (Day(..))
35import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
36import Data.Text (Text)
37import GHC.Generics (Generic)
38import Network.HTTP.Conduit
39import qualified Network.HTTP.Types.Status as NHTS
41-- no trailing slashes in servers, library handles building the path.
42let testServer = (Server "http://localhost:9200")
43let testIndex = IndexName "twitter"
44let testMapping = MappingName "tweet"
46-- defaultIndexSettings is exported by Database.Bloodhound as well
47let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
49-- createIndex returns IO Reply
51-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
52response <- createIndex testServer defaultIndexSettings testIndex
61response <- deleteIndex testServer testIndex
63-- print response if it was a success
64Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
65 , responseVersion = HTTP/1.1
66 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
67 , ("Content-Length", "21")]
68 , responseBody = "{\"acknowledged\":true}"
69 , responseCookieJar = CJ {expose = []}
70 , responseClose' = ResponseClose}
72-- if the index to be deleted didn't exist anyway
73Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
74 , responseVersion = HTTP/1.1
75 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
76 , ("Content-Length","65")]
77 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
78 , responseCookieJar = CJ {expose = []}
79 , responseClose' = ResponseClose}
85**** Note, you *have* to do this if you expect to read what you just wrote
89resp <- refreshIndex testServer testIndex
91-- print resp on success
92Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
93 , responseVersion = HTTP/1.1
94 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
95 , ("Content-Length","50")]
96 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
97 , responseCookieJar = CJ {expose = []}
98 , responseClose' = ResponseClose}
108-- don't forget imports and the like at the top.
110data TweetMapping = TweetMapping deriving (Eq, Show)
112-- I know writing the JSON manually sucks.
113-- I don't have a proper data type for Mappings yet.
114-- Let me know if this is something you need.
117instance ToJSON TweetMapping where
118 toJSON TweetMapping =
120 object ["properties" .=
121 object ["location" .=
122 object ["type" .= ("geo_point" :: Text)]]]]
125resp <- createMapping testServer testIndex testMapping TweetMapping
133resp <- deleteMapping testServer testIndex testMapping
137** Document Operations
139*** Indexing Documents
143-- don't forget the imports and derive generic setting for ghci
144-- at the beginning of the examples.
147data Location = Location { lat :: Double
148 , lon :: Double } deriving (Eq, Generic, Show)
150data Tweet = Tweet { user :: Text
151 , postDate :: UTCTime
154 , location :: Location } deriving (Eq, Generic, Show)
156exampleTweet = Tweet { user = "bitemyapp"
158 (ModifiedJulianDay 55000)
159 (secondsToDiffTime 10)
160 , message = "Use haskell!"
162 , location = Location 40.12 (-71.34) }
164-- automagic (generic) derivation of instances because we're lazy.
166instance FromJSON Tweet
167instance ToJSON Location
168instance FromJSON Location
171-- Should be able to toJSON and encode the data structures like this:
172-- λ> toJSON $ Location 10.0 10.0
173-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
174-- λ> encode $ Location 10.0 10.0
175-- "{\"lat\":10,\"lon\":10}"
177resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
179-- print resp on success
180Response {responseStatus =
181 Status {statusCode = 200, statusMessage = "OK"}
182 , responseVersion = HTTP/1.1, responseHeaders =
183 [("Content-Type","application/json; charset=UTF-8"),
184 ("Content-Length","75")]
185 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
186 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
190*** Deleting Documents
194resp <- deleteDocument testServer testIndex testMapping (DocId "1")
202-- n.b., you'll need the earlier imports. responseBody is from http-conduit
204resp <- getDocument testServer testIndex testMapping (DocId "1")
206-- responseBody :: Response body -> body
207let body = responseBody resp
209-- you have two options, you use decode and just get Maybe (EsResult Tweet)
210-- or you can use eitherDecode and get Either String (EsResult Tweet)
212let maybeResult = decode body :: Maybe (EsResult Tweet)
213-- the explicit typing is so Aeson knows how to parse the JSON.
215-- use either if you want to know why something failed to parse.
216-- (string errors, sadly)
217let eitherResult = decode body :: Either String (EsResult Tweet)
219-- print eitherResult should look like:
220Right (EsResult {_index = "twitter"
225 , _source = Tweet {user = "bitemyapp"
226 , postDate = 2009-06-18 00:00:10 UTC
227 , message = "Use haskell!"
229 , location = Location {lat = 40.12, lon = -71.34}}})
231-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
233-- use the _source record accessor to get at your document
234λ> fmap _source result
235Right (Tweet {user = "bitemyapp"
236 , postDate = 2009-06-18 00:00:10 UTC
237 , message = "Use haskell!"
239 , location = Location {lat = 40.12, lon = -71.34}})
251-- exported by the Client module, just defaults some stuff.
252-- mkSearch :: Maybe Query -> Maybe Filter -> Search
253-- mkSearch query filter = Search query filter Nothing False 0 10
255let query = TermQuery (Term "user" "bitemyapp") Nothing
257-- AND'ing identity filter with itself and then tacking it onto a query
258-- search should be a null-operation. I include it for the sake of example.
259-- <||> (or/plus) should make it into a search that returns everything.
261let filter = IdentityFilter <&&> IdentityFilter
263-- constructing the search object the searchByIndex function dispatches on.
264let search = mkSearch (Just query) (Just filter)
266-- you can also searchByType and specify the mapping name.
267reply <- searchByIndex testServer testIndex search
269let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
271λ> fmap (hits . searchHits) result
272Right [Hit {hitIndex = IndexName "twitter"
273 , hitType = MappingName "tweet"
274 , hitDocId = DocId "1"
275 , hitScore = 0.30685282
276 , hitSource = Tweet {user = "bitemyapp"
277 , postDate = 2009-06-18 00:00:10 UTC
278 , message = "Use haskell!"
280 , location = Location {lat = 40.12, lon = -71.34}}}]
288let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
290-- mkSort is a shortcut function that takes a FieldName and a SortOrder
291-- to generate a vanilla DefaultSort.
292-- checkt the DefaultSort type for the full list of customizable options.
294-- From and size are integers for pagination.
296-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
298-- type Sort = [SortSpec]
299-- type TrackSortScores = Bool
303-- Search takes Maybe Query
309let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
315**** And, Not, and Or filters
317Filters form a monoid and seminearring.
321instance Monoid Filter where
322 mempty = IdentityFilter
323 mappend a b = AndFilter [a, b] defaultCache
325instance Seminearring Filter where
326 a <||> b = OrFilter [a, b] defaultCache
328-- AndFilter and OrFilter take [Filter] as an argument.
330-- This will return anything, because IdentityFilter returns everything
331OrFilter [IdentityFilter, someOtherFilter] False
333-- This will return exactly what someOtherFilter returns
334AndFilter [IdentityFilter, someOtherFilter] False
336-- Thanks to the seminearring and monoid, the above can be expressed as:
339IdentityFilter <&&> someOtherFilter
342IdentityFilter <||> someOtherFilter
344-- Also there is a NotFilter, it only accepts a single filter, not a list.
346NotFilter someOtherFilter False
354-- And'ing two Identity
355let queryFilter = IdentityFilter <&&> IdentityFilter
357let search = mkSearch Nothing (Just queryFilter)
359reply <- searchByType testServer testIndex testMapping search
365Similar to boolean queries.
369-- Will return only items whose "user" field contains the term "bitemyapp"
370let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
372-- Will return only items whose "user" field does not contain the term "bitemyapp"
373let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
375-- The clause (query) should appear in the matching document.
376-- In a boolean query with no must clauses, one or more should
377-- clauses must match a document. The minimum number of should
378-- clauses to match can be set using the minimum_should_match parameter.
379let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
387-- Will filter for documents that have the field "user"
388let existsFilter = ExistsFilter (FieldName "user")
392**** Geo BoundingBox Filter
396-- topLeft and bottomRight
397let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
399let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
401-- second argument is GeoFilterType, memory or indexed.
402let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
406**** Geo Distance Filter
410let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
412-- coefficient and units
413let distance = Distance 10.0 Miles
415-- GeoFilterType or NoOptimizeBbox
416let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
418-- SloppyArc is the usual/default optimization in Elasticsearch today
419-- but pre-1.0 versions will need to pick Arc or Plane.
421let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
425**** Geo Distance Range Filter
427Think of a donut and you won't be far off.
431let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
433let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
435let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
439**** Geo Polygon Filter
443-- I think I drew a square here.
444let points = [LatLon 40.0 (-70.00),
445 LatLon 40.0 (-72.00),
446 LatLon 41.0 (-70.00),
447 LatLon 41.0 (-72.00)]
449let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
453**** Document IDs filter
457-- takes a mapping name and a list of DocIds
458IdsFilter (MappingName "tweet") [DocId "1"]
468-- RangeFilter :: FieldName
469-- -> Either HalfRange Range
473let filter = RangeFilter (FieldName "age")
474 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
475 RangeExecutionIndex False
483let filter = RangeFilter (FieldName "age")
484 (Left (HalfRangeLt (LessThan 100000.0)))
485 RangeExecutionIndex False
501let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
502 RegexpAll (CacheName "test") False (CacheKey "key")
504-- RegexpFlags can be a combination of RegexpAll, Complement,
505-- Interval, Intersection, AnyString, and a combination of two options thereof.
509* Possible future functionality
511** Node discovery and failover
513Might require TCP support.
515** Support for TCP access to Elasticsearch
517Pretend to be a transport client?
519** Bulk cluster-join merge
521Might require making a lucene index on disk with the appropriate format.
525http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
527** Geohash cell filter
529http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
533http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
537http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
541http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
545http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
547** Script based sorting
549http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
551** Collapsing redundantly nested and/or structures
553The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
555** Runtime checking for cycles in data structures
557check for n > 1 occurrences in DFS:
559http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
561http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
565Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list