3#+CAPTION: Bloodhound (dog)
6* Elasticsearch client and query DSL for Haskell
10Because you're tired of obnoxious errors like [[http://i.imgur.com/FKtZYIP.png][this]] and want types to guide your use of the API.
14Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but being compared to clients in other languages the story here so far is good.
24-- Formatted for use in ghci, so there are "let"s in front of the decls.
26-- if you see :{ and :}, they're so you can copy-paste
27-- the multi-line examples into your ghci REPL.
30import Database.Bloodhound
32import Data.Either (Either(..))
33import Data.Maybe (fromJust)
34import Data.Time.Calendar (Day(..))
35import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
36import Data.Text (Text)
37import GHC.Generics (Generic)
38import Network.HTTP.Conduit
39import qualified Network.HTTP.Types.Status as NHTS
41-- no trailing slashes in servers, library handles building the path.
42let testServer = (Server "http://localhost:9200")
43let testIndex = IndexName "twitter"
44let testMapping = MappingName "tweet"
46-- defaultIndexSettings is exported by Database.Bloodhound as well
47let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
49-- createIndex returns IO Reply
51-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
52response <- createIndex testServer defaultIndexSettings testIndex
61response <- deleteIndex testServer testIndex
63-- print response if it was a success
64Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
65 , responseVersion = HTTP/1.1
66 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
67 , ("Content-Length", "21")]
68 , responseBody = "{\"acknowledged\":true}"
69 , responseCookieJar = CJ {expose = []}
70 , responseClose' = ResponseClose}
72-- if the index to be deleted didn't exist anyway
73Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
74 , responseVersion = HTTP/1.1
75 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
76 , ("Content-Length","65")]
77 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
78 , responseCookieJar = CJ {expose = []}
79 , responseClose' = ResponseClose}
85**** Note, you *have* to do this if you expect to read what you just wrote
89resp <- refreshIndex testServer testIndex
91-- print resp on success
92Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
93 , responseVersion = HTTP/1.1
94 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
95 , ("Content-Length","50")]
96 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
97 , responseCookieJar = CJ {expose = []}
98 , responseClose' = ResponseClose}
108-- don't forget imports and the like at the top.
110data TweetMapping = TweetMapping deriving (Eq, Show)
112-- I know writing the JSON manually sucks.
113-- I don't have a proper data type for Mappings yet.
114-- Let me know if this is something you need.
117instance ToJSON TweetMapping where
118 toJSON TweetMapping =
120 object ["properties" .=
121 object ["location" .=
122 object ["type" .= ("geo_point" :: Text)]]]]
125resp <- createMapping testServer testIndex testMapping TweetMapping
133resp <- deleteMapping testServer testIndex testMapping
137** Document Operations
139*** Indexing Documents
143-- don't forget the imports and derive generic setting for ghci
144-- at the beginning of the examples.
147data Location = Location { lat :: Double
148 , lon :: Double } deriving (Eq, Generic, Show)
150data Tweet = Tweet { user :: Text
151 , postDate :: UTCTime
154 , location :: Location } deriving (Eq, Generic, Show)
156exampleTweet = Tweet { user = "bitemyapp"
158 (ModifiedJulianDay 55000)
159 (secondsToDiffTime 10)
160 , message = "Use haskell!"
162 , location = Location 40.12 (-71.34) }
164-- automagic (generic) derivation of instances because we're lazy.
166instance FromJSON Tweet
167instance ToJSON Location
168instance FromJSON Location
171-- Should be able to toJSON and encode the data structures like this:
172-- λ> toJSON $ Location 10.0 10.0
173-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
174-- λ> encode $ Location 10.0 10.0
175-- "{\"lat\":10,\"lon\":10}"
177resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
179-- print resp on success
180Response {responseStatus =
181 Status {statusCode = 200, statusMessage = "OK"}
182 , responseVersion = HTTP/1.1, responseHeaders =
183 [("Content-Type","application/json; charset=UTF-8"),
184 ("Content-Length","75")]
185 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
186 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
190*** Deleting Documents
194resp <- deleteDocument testServer testIndex testMapping (DocId "1")
202-- n.b., you'll need the earlier imports. responseBody is from http-conduit
204resp <- getDocument testServer testIndex testMapping (DocId "1")
206-- responseBody :: Response body -> body
207let body = responseBody resp
209-- you have two options, you use decode and just get Maybe (EsResult Tweet)
210-- or you can use eitherDecode and get Either String (EsResult Tweet)
212let maybeResult = decode body :: Maybe (EsResult Tweet)
213-- the explicit typing is so Aeson knows how to parse the JSON.
215-- use either if you want to know why something failed to parse.
216-- (string errors, sadly)
217let eitherResult = decode body :: Either String (EsResult Tweet)
219-- print eitherResult should look like:
220Right (EsResult {_index = "twitter"
225 , _source = Tweet {user = "bitemyapp"
226 , postDate = 2009-06-18 00:00:10 UTC
227 , message = "Use haskell!"
229 , location = Location {lat = 40.12, lon = -71.34}}})
231-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
233-- use the _source record accessor to get at your document
234λ> fmap _source result
235Right (Tweet {user = "bitemyapp"
236 , postDate = 2009-06-18 00:00:10 UTC
237 , message = "Use haskell!"
239 , location = Location {lat = 40.12, lon = -71.34}})
251-- exported by the Client module, just defaults some stuff.
252-- mkSearch :: Maybe Query -> Maybe Filter -> Search
253-- mkSearch query filter = Search query filter Nothing False 0 10
255let query = TermQuery (Term "user" "bitemyapp") Nothing
257-- AND'ing identity filter with itself and then tacking it onto a query
258-- search should be a null-operation. I include it for the sake of example.
259-- <||> (or/plus) should make it into a search that returns everything.
261let filter = IdentityFilter <&&> IdentityFilter
263-- constructing the search object the searchByIndex function dispatches on.
264let search = mkSearch (Just query) (Just filter)
266-- you can also searchByType and specify the mapping name.
267reply <- searchByIndex testServer testIndex search
269let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
271λ> fmap (hits . searchHits) result
272Right [Hit {hitIndex = IndexName "twitter"
273 , hitType = MappingName "tweet"
274 , hitDocId = DocId "1"
275 , hitScore = 0.30685282
276 , hitSource = Tweet {user = "bitemyapp"
277 , postDate = 2009-06-18 00:00:10 UTC
278 , message = "Use haskell!"
280 , location = Location {lat = 40.12, lon = -71.34}}}]
292**** And, Not, and Or filters
294Filters form a monoid and seminearring.
298instance Monoid Filter where
299 mempty = IdentityFilter
300 mappend a b = AndFilter [a, b] defaultCache
302instance Seminearring Filter where
303 a <||> b = OrFilter [a, b] defaultCache
305-- AndFilter and OrFilter take [Filter] as an argument.
307-- This will return anything, because IdentityFilter returns everything
308OrFilter [IdentityFilter, someOtherFilter] False
310-- This will return exactly what someOtherFilter returns
311AndFilter [IdentityFilter, someOtherFilter] False
313-- Thanks to the seminearring and monoid, the above can be expressed as:
316IdentityFilter <&&> someOtherFilter
319IdentityFilter <||> someOtherFilter
321-- Also there is a NotFilter, it only accepts a single filter, not a list.
323NotFilter someOtherFilter False
331-- And'ing two Identity
332let queryFilter = IdentityFilter <&&> IdentityFilter
334let search = mkSearch Nothing (Just queryFilter)
336reply <- searchByType testServer testIndex testMapping search
342Similar to boolean queries.
346-- Will return only items whose "user" field contains the term "bitemyapp"
347let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
349-- Will return only items whose "user" field does not contain the term "bitemyapp"
350let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
352-- The clause (query) should appear in the matching document.
353-- In a boolean query with no must clauses, one or more should
354-- clauses must match a document. The minimum number of should
355-- clauses to match can be set using the minimum_should_match parameter.
356let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
364-- Will filter for documents that have the field "user"
365let existsFilter = ExistsFilter (FieldName "user")
369**** Geo BoundingBox Filter
373-- topLeft and bottomRight
374let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
376let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
378-- second argument is GeoFilterType, memory or indexed.
379let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
383**** Geo Distance Filter
387let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
389-- coefficient and units
390let distance = Distance 10.0 Miles
392-- GeoFilterType or NoOptimizeBbox
393let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
395-- SloppyArc is the usual/default optimization in Elasticsearch today
396-- but pre-1.0 versions will need to pick Arc or Plane.
398let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
402**** Geo Distance Range Filter
404Think of a donut and you won't be far off.
408let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
410let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
412let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
416**** Geo Polygon Filter
420-- I think I drew a square here.
421let points = [LatLon 40.0 (-70.00),
422 LatLon 40.0 (-72.00),
423 LatLon 41.0 (-70.00),
424 LatLon 41.0 (-72.00)]
426let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
430**** Document IDs filter
434-- takes a mapping name and a list of DocIds
435IdsFilter (MappingName "tweet") [DocId "1"]
445-- RangeFilter :: FieldName
446-- -> Either HalfRange Range
450let filter = RangeFilter (FieldName "age")
451 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
452 RangeExecutionIndex False
460let filter = RangeFilter (FieldName "age")
461 (Left (HalfRangeLt (LessThan 100000.0)))
462 RangeExecutionIndex False
478let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
479 RegexpAll (CacheName "test") False (CacheKey "key")
481-- RegexpFlags can be a combination of RegexpAll, Complement,
482-- Interval, Intersection, AnyString, and a combination of two options thereof.
486* Possible future functionality
488** Node discovery and failover
490Might require TCP support.
492** Support for TCP access to Elasticsearch
494Pretend to be a transport client?
496** Bulk cluster-join merge
498Might require making a lucene index on disk with the appropriate format.
502http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
504** Geohash cell filter
506http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
510http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
514http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
518http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
522http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
524** Script based sorting
526http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
528** Collapsing redundantly nested and/or structures
530The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
532** Runtime checking for cycles in data structures
534check for n > 1 occurrences in DFS:
536http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
538http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
542Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list