3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10* Elasticsearch client and query DSL for Haskell
14Search doesn't have to be hard. Let the dog do it.
18Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
28-- Formatted for use in ghci, so there are "let"s in front of the decls.
30-- if you see :{ and :}, they're so you can copy-paste
31-- the multi-line examples into your ghci REPL.
34import Database.Bloodhound
36import Data.Either (Either(..))
37import Data.Maybe (fromJust)
38import Data.Time.Calendar (Day(..))
39import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
40import Data.Text (Text)
41import GHC.Generics (Generic)
42import Network.HTTP.Conduit
43import qualified Network.HTTP.Types.Status as NHTS
45-- no trailing slashes in servers, library handles building the path.
46let testServer = (Server "http://localhost:9200")
47let testIndex = IndexName "twitter"
48let testMapping = MappingName "tweet"
50-- defaultIndexSettings is exported by Database.Bloodhound as well
51let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
53-- createIndex returns IO Reply
55-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
56response <- createIndex testServer defaultIndexSettings testIndex
67response <- deleteIndex testServer testIndex
75-- print response if it was a success
76Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
77 , responseVersion = HTTP/1.1
78 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
79 , ("Content-Length", "21")]
80 , responseBody = "{\"acknowledged\":true}"
81 , responseCookieJar = CJ {expose = []}
82 , responseClose' = ResponseClose}
84-- if the index to be deleted didn't exist anyway
85Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
86 , responseVersion = HTTP/1.1
87 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
88 , ("Content-Length","65")]
89 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
90 , responseCookieJar = CJ {expose = []}
91 , responseClose' = ResponseClose}
97**** Note, you *have* to do this if you expect to read what you just wrote
101resp <- refreshIndex testServer testIndex
109-- print resp on success
110Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
111 , responseVersion = HTTP/1.1
112 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
113 , ("Content-Length","50")]
114 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
115 , responseCookieJar = CJ {expose = []}
116 , responseClose' = ResponseClose}
126-- don't forget imports and the like at the top.
128data TweetMapping = TweetMapping deriving (Eq, Show)
130-- I know writing the JSON manually sucks.
131-- I don't have a proper data type for Mappings yet.
132-- Let me know if this is something you need.
135instance ToJSON TweetMapping where
136 toJSON TweetMapping =
138 object ["properties" .=
139 object ["location" .=
140 object ["type" .= ("geo_point" :: Text)]]]]
143resp <- createMapping testServer testIndex testMapping TweetMapping
151resp <- deleteMapping testServer testIndex testMapping
155** Document Operations
157*** Indexing Documents
161-- don't forget the imports and derive generic setting for ghci
162-- at the beginning of the examples.
165data Location = Location { lat :: Double
166 , lon :: Double } deriving (Eq, Generic, Show)
168data Tweet = Tweet { user :: Text
169 , postDate :: UTCTime
172 , location :: Location } deriving (Eq, Generic, Show)
174exampleTweet = Tweet { user = "bitemyapp"
176 (ModifiedJulianDay 55000)
177 (secondsToDiffTime 10)
178 , message = "Use haskell!"
180 , location = Location 40.12 (-71.34) }
182-- automagic (generic) derivation of instances because we're lazy.
184instance FromJSON Tweet
185instance ToJSON Location
186instance FromJSON Location
189-- Should be able to toJSON and encode the data structures like this:
190-- λ> toJSON $ Location 10.0 10.0
191-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
192-- λ> encode $ Location 10.0 10.0
193-- "{\"lat\":10,\"lon\":10}"
195resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
203Response {responseStatus =
204 Status {statusCode = 200, statusMessage = "OK"}
205 , responseVersion = HTTP/1.1, responseHeaders =
206 [("Content-Type","application/json; charset=UTF-8"),
207 ("Content-Length","75")]
208 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
209 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
213*** Deleting Documents
217resp <- deleteDocument testServer testIndex testMapping (DocId "1")
225-- n.b., you'll need the earlier imports. responseBody is from http-conduit
227resp <- getDocument testServer testIndex testMapping (DocId "1")
229-- responseBody :: Response body -> body
230let body = responseBody resp
232-- you have two options, you use decode and just get Maybe (EsResult Tweet)
233-- or you can use eitherDecode and get Either String (EsResult Tweet)
235let maybeResult = decode body :: Maybe (EsResult Tweet)
236-- the explicit typing is so Aeson knows how to parse the JSON.
238-- use either if you want to know why something failed to parse.
239-- (string errors, sadly)
240let eitherResult = decode body :: Either String (EsResult Tweet)
242-- print eitherResult should look like:
243Right (EsResult {_index = "twitter"
248 , _source = Tweet {user = "bitemyapp"
249 , postDate = 2009-06-18 00:00:10 UTC
250 , message = "Use haskell!"
252 , location = Location {lat = 40.12, lon = -71.34}}})
254-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
256-- use the _source record accessor to get at your document
257λ> fmap _source result
258Right (Tweet {user = "bitemyapp"
259 , postDate = 2009-06-18 00:00:10 UTC
260 , message = "Use haskell!"
262 , location = Location {lat = 40.12, lon = -71.34}})
274-- exported by the Client module, just defaults some stuff.
275-- mkSearch :: Maybe Query -> Maybe Filter -> Search
276-- mkSearch query filter = Search query filter Nothing False 0 10
278let query = TermQuery (Term "user" "bitemyapp") Nothing
280-- AND'ing identity filter with itself and then tacking it onto a query
281-- search should be a null-operation. I include it for the sake of example.
282-- <||> (or/plus) should make it into a search that returns everything.
284let filter = IdentityFilter <&&> IdentityFilter
286-- constructing the search object the searchByIndex function dispatches on.
287let search = mkSearch (Just query) (Just filter)
289-- you can also searchByType and specify the mapping name.
290reply <- searchByIndex testServer testIndex search
292let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
294λ> fmap (hits . searchHits) result
295Right [Hit {hitIndex = IndexName "twitter"
296 , hitType = MappingName "tweet"
297 , hitDocId = DocId "1"
298 , hitScore = 0.30685282
299 , hitSource = Tweet {user = "bitemyapp"
300 , postDate = 2009-06-18 00:00:10 UTC
301 , message = "Use haskell!"
303 , location = Location {lat = 40.12, lon = -71.34}}}]
311let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
312let search = mkSearch (Just query) Nothing
317**** Multi-Match Query
321let fields = [FieldName "user", FieldName "message"]
322let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
323let search = mkSearch (Just query) Nothing
331let innerQuery = QueryMatchQuery $
332 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
333let query = QueryBoolQuery $
334 mkBoolQuery (Just innerQuery) Nothing Nothing
335let search = mkSearch (Just query) Nothing
343let posQuery = QueryMatchQuery $
344 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
345let negQuery = QueryMatchQuery $
346 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
347let query = QueryBoostingQuery $
348 BoostingQuery posQuery negQuery (Boost 0.2)
352**** Rest of the query/filter types
354Just follow the pattern you've seen here and check the Hackage API documentation.
360let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
362-- mkSort is a shortcut function that takes a FieldName and a SortOrder
363-- to generate a vanilla DefaultSort.
364-- checkt the DefaultSort type for the full list of customizable options.
366-- From and size are integers for pagination.
368-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
370-- type Sort = [SortSpec]
371-- type TrackSortScores = Bool
375-- Search takes Maybe Query
381-- just add more sortspecs to the list if you want tie-breakers.
382let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
388**** And, Not, and Or filters
390Filters form a monoid and seminearring.
394instance Monoid Filter where
395 mempty = IdentityFilter
396 mappend a b = AndFilter [a, b] defaultCache
398instance Seminearring Filter where
399 a <||> b = OrFilter [a, b] defaultCache
401-- AndFilter and OrFilter take [Filter] as an argument.
403-- This will return anything, because IdentityFilter returns everything
404OrFilter [IdentityFilter, someOtherFilter] False
406-- This will return exactly what someOtherFilter returns
407AndFilter [IdentityFilter, someOtherFilter] False
409-- Thanks to the seminearring and monoid, the above can be expressed as:
412IdentityFilter <&&> someOtherFilter
415IdentityFilter <||> someOtherFilter
417-- Also there is a NotFilter, it only accepts a single filter, not a list.
419NotFilter someOtherFilter False
427-- And'ing two Identity
428let queryFilter = IdentityFilter <&&> IdentityFilter
430let search = mkSearch Nothing (Just queryFilter)
432reply <- searchByType testServer testIndex testMapping search
438Similar to boolean queries.
442-- Will return only items whose "user" field contains the term "bitemyapp"
443let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
445-- Will return only items whose "user" field does not contain the term "bitemyapp"
446let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
448-- The clause (query) should appear in the matching document.
449-- In a boolean query with no must clauses, one or more should
450-- clauses must match a document. The minimum number of should
451-- clauses to match can be set using the minimum_should_match parameter.
452let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
460-- Will filter for documents that have the field "user"
461let existsFilter = ExistsFilter (FieldName "user")
465**** Geo BoundingBox Filter
469-- topLeft and bottomRight
470let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
472let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
474-- second argument is GeoFilterType, memory or indexed.
475let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
479**** Geo Distance Filter
483let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
485-- coefficient and units
486let distance = Distance 10.0 Miles
488-- GeoFilterType or NoOptimizeBbox
489let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
491-- SloppyArc is the usual/default optimization in Elasticsearch today
492-- but pre-1.0 versions will need to pick Arc or Plane.
494let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
498**** Geo Distance Range Filter
500Think of a donut and you won't be far off.
504let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
506let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
508let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
512**** Geo Polygon Filter
516-- I think I drew a square here.
517let points = [LatLon 40.0 (-70.00),
518 LatLon 40.0 (-72.00),
519 LatLon 41.0 (-70.00),
520 LatLon 41.0 (-72.00)]
522let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
526**** Document IDs filter
530-- takes a mapping name and a list of DocIds
531IdsFilter (MappingName "tweet") [DocId "1"]
541-- RangeFilter :: FieldName
542-- -> Either HalfRange Range
546let filter = RangeFilter (FieldName "age")
547 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
548 RangeExecutionIndex False
556let filter = RangeFilter (FieldName "age")
557 (Left (HalfRangeLt (LessThan 100000.0)))
558 RangeExecutionIndex False
574let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
575 RegexpAll (CacheName "test") False (CacheKey "key")
577-- RegexpFlags can be a combination of RegexpAll, Complement,
578-- Interval, Intersection, AnyString, and a combination of two options thereof.
582* Possible future functionality
586Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
588** Function Score Query
590http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
592** Node discovery and failover
594Might require TCP support.
596** Support for TCP access to Elasticsearch
598Pretend to be a transport client?
600** Bulk cluster-join merge
602Might require making a lucene index on disk with the appropriate format.
606http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
610http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
612** Geohash cell filter
614http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
618http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
622http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
626http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
630http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
632** Script based sorting
634http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
636** Collapsing redundantly nested and/or structures
638The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
640** Runtime checking for cycles in data structures
642check for n > 1 occurrences in DFS:
644http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
646http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
650Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list