3#+CAPTION: Bloodhound (dog)
6[](https://travis-ci.org/bitemyapp/bloodhound)
8* Elasticsearch client and query DSL for Haskell
12Search doesn't have to be hard. Let the dog do it.
16Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
26-- Formatted for use in ghci, so there are "let"s in front of the decls.
28-- if you see :{ and :}, they're so you can copy-paste
29-- the multi-line examples into your ghci REPL.
32import Database.Bloodhound
34import Data.Either (Either(..))
35import Data.Maybe (fromJust)
36import Data.Time.Calendar (Day(..))
37import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
38import Data.Text (Text)
39import GHC.Generics (Generic)
40import Network.HTTP.Conduit
41import qualified Network.HTTP.Types.Status as NHTS
43-- no trailing slashes in servers, library handles building the path.
44let testServer = (Server "http://localhost:9200")
45let testIndex = IndexName "twitter"
46let testMapping = MappingName "tweet"
48-- defaultIndexSettings is exported by Database.Bloodhound as well
49let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
51-- createIndex returns IO Reply
53-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
54response <- createIndex testServer defaultIndexSettings testIndex
65response <- deleteIndex testServer testIndex
73-- print response if it was a success
74Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
75 , responseVersion = HTTP/1.1
76 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
77 , ("Content-Length", "21")]
78 , responseBody = "{\"acknowledged\":true}"
79 , responseCookieJar = CJ {expose = []}
80 , responseClose' = ResponseClose}
82-- if the index to be deleted didn't exist anyway
83Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
84 , responseVersion = HTTP/1.1
85 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
86 , ("Content-Length","65")]
87 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
88 , responseCookieJar = CJ {expose = []}
89 , responseClose' = ResponseClose}
95**** Note, you *have* to do this if you expect to read what you just wrote
99resp <- refreshIndex testServer testIndex
107-- print resp on success
108Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
109 , responseVersion = HTTP/1.1
110 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
111 , ("Content-Length","50")]
112 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
113 , responseCookieJar = CJ {expose = []}
114 , responseClose' = ResponseClose}
124-- don't forget imports and the like at the top.
126data TweetMapping = TweetMapping deriving (Eq, Show)
128-- I know writing the JSON manually sucks.
129-- I don't have a proper data type for Mappings yet.
130-- Let me know if this is something you need.
133instance ToJSON TweetMapping where
134 toJSON TweetMapping =
136 object ["properties" .=
137 object ["location" .=
138 object ["type" .= ("geo_point" :: Text)]]]]
141resp <- createMapping testServer testIndex testMapping TweetMapping
149resp <- deleteMapping testServer testIndex testMapping
153** Document Operations
155*** Indexing Documents
159-- don't forget the imports and derive generic setting for ghci
160-- at the beginning of the examples.
163data Location = Location { lat :: Double
164 , lon :: Double } deriving (Eq, Generic, Show)
166data Tweet = Tweet { user :: Text
167 , postDate :: UTCTime
170 , location :: Location } deriving (Eq, Generic, Show)
172exampleTweet = Tweet { user = "bitemyapp"
174 (ModifiedJulianDay 55000)
175 (secondsToDiffTime 10)
176 , message = "Use haskell!"
178 , location = Location 40.12 (-71.34) }
180-- automagic (generic) derivation of instances because we're lazy.
182instance FromJSON Tweet
183instance ToJSON Location
184instance FromJSON Location
187-- Should be able to toJSON and encode the data structures like this:
188-- λ> toJSON $ Location 10.0 10.0
189-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
190-- λ> encode $ Location 10.0 10.0
191-- "{\"lat\":10,\"lon\":10}"
193resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
201Response {responseStatus =
202 Status {statusCode = 200, statusMessage = "OK"}
203 , responseVersion = HTTP/1.1, responseHeaders =
204 [("Content-Type","application/json; charset=UTF-8"),
205 ("Content-Length","75")]
206 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
207 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
211*** Deleting Documents
215resp <- deleteDocument testServer testIndex testMapping (DocId "1")
223-- n.b., you'll need the earlier imports. responseBody is from http-conduit
225resp <- getDocument testServer testIndex testMapping (DocId "1")
227-- responseBody :: Response body -> body
228let body = responseBody resp
230-- you have two options, you use decode and just get Maybe (EsResult Tweet)
231-- or you can use eitherDecode and get Either String (EsResult Tweet)
233let maybeResult = decode body :: Maybe (EsResult Tweet)
234-- the explicit typing is so Aeson knows how to parse the JSON.
236-- use either if you want to know why something failed to parse.
237-- (string errors, sadly)
238let eitherResult = decode body :: Either String (EsResult Tweet)
240-- print eitherResult should look like:
241Right (EsResult {_index = "twitter"
246 , _source = Tweet {user = "bitemyapp"
247 , postDate = 2009-06-18 00:00:10 UTC
248 , message = "Use haskell!"
250 , location = Location {lat = 40.12, lon = -71.34}}})
252-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
254-- use the _source record accessor to get at your document
255λ> fmap _source result
256Right (Tweet {user = "bitemyapp"
257 , postDate = 2009-06-18 00:00:10 UTC
258 , message = "Use haskell!"
260 , location = Location {lat = 40.12, lon = -71.34}})
272-- exported by the Client module, just defaults some stuff.
273-- mkSearch :: Maybe Query -> Maybe Filter -> Search
274-- mkSearch query filter = Search query filter Nothing False 0 10
276let query = TermQuery (Term "user" "bitemyapp") Nothing
278-- AND'ing identity filter with itself and then tacking it onto a query
279-- search should be a null-operation. I include it for the sake of example.
280-- <||> (or/plus) should make it into a search that returns everything.
282let filter = IdentityFilter <&&> IdentityFilter
284-- constructing the search object the searchByIndex function dispatches on.
285let search = mkSearch (Just query) (Just filter)
287-- you can also searchByType and specify the mapping name.
288reply <- searchByIndex testServer testIndex search
290let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
292λ> fmap (hits . searchHits) result
293Right [Hit {hitIndex = IndexName "twitter"
294 , hitType = MappingName "tweet"
295 , hitDocId = DocId "1"
296 , hitScore = 0.30685282
297 , hitSource = Tweet {user = "bitemyapp"
298 , postDate = 2009-06-18 00:00:10 UTC
299 , message = "Use haskell!"
301 , location = Location {lat = 40.12, lon = -71.34}}}]
309let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
310let search = mkSearch (Just query) Nothing
315**** Multi-Match Query
319let fields = [FieldName "user", FieldName "message"]
320let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
321let search = mkSearch (Just query) Nothing
329let innerQuery = QueryMatchQuery $
330 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
331let query = QueryBoolQuery $
332 mkBoolQuery (Just innerQuery) Nothing Nothing
333let search = mkSearch (Just query) Nothing
341let posQuery = QueryMatchQuery $
342 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
343let negQuery = QueryMatchQuery $
344 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
345let query = QueryBoostingQuery $
346 BoostingQuery posQuery negQuery (Boost 0.2)
350**** Rest of the query/filter types
352Just follow the pattern you've seen here and check the Hackage API documentation.
358let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
360-- mkSort is a shortcut function that takes a FieldName and a SortOrder
361-- to generate a vanilla DefaultSort.
362-- checkt the DefaultSort type for the full list of customizable options.
364-- From and size are integers for pagination.
366-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
368-- type Sort = [SortSpec]
369-- type TrackSortScores = Bool
373-- Search takes Maybe Query
379-- just add more sortspecs to the list if you want tie-breakers.
380let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
386**** And, Not, and Or filters
388Filters form a monoid and seminearring.
392instance Monoid Filter where
393 mempty = IdentityFilter
394 mappend a b = AndFilter [a, b] defaultCache
396instance Seminearring Filter where
397 a <||> b = OrFilter [a, b] defaultCache
399-- AndFilter and OrFilter take [Filter] as an argument.
401-- This will return anything, because IdentityFilter returns everything
402OrFilter [IdentityFilter, someOtherFilter] False
404-- This will return exactly what someOtherFilter returns
405AndFilter [IdentityFilter, someOtherFilter] False
407-- Thanks to the seminearring and monoid, the above can be expressed as:
410IdentityFilter <&&> someOtherFilter
413IdentityFilter <||> someOtherFilter
415-- Also there is a NotFilter, it only accepts a single filter, not a list.
417NotFilter someOtherFilter False
425-- And'ing two Identity
426let queryFilter = IdentityFilter <&&> IdentityFilter
428let search = mkSearch Nothing (Just queryFilter)
430reply <- searchByType testServer testIndex testMapping search
436Similar to boolean queries.
440-- Will return only items whose "user" field contains the term "bitemyapp"
441let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
443-- Will return only items whose "user" field does not contain the term "bitemyapp"
444let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
446-- The clause (query) should appear in the matching document.
447-- In a boolean query with no must clauses, one or more should
448-- clauses must match a document. The minimum number of should
449-- clauses to match can be set using the minimum_should_match parameter.
450let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
458-- Will filter for documents that have the field "user"
459let existsFilter = ExistsFilter (FieldName "user")
463**** Geo BoundingBox Filter
467-- topLeft and bottomRight
468let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
470let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
472-- second argument is GeoFilterType, memory or indexed.
473let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
477**** Geo Distance Filter
481let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
483-- coefficient and units
484let distance = Distance 10.0 Miles
486-- GeoFilterType or NoOptimizeBbox
487let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
489-- SloppyArc is the usual/default optimization in Elasticsearch today
490-- but pre-1.0 versions will need to pick Arc or Plane.
492let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
496**** Geo Distance Range Filter
498Think of a donut and you won't be far off.
502let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
504let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
506let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
510**** Geo Polygon Filter
514-- I think I drew a square here.
515let points = [LatLon 40.0 (-70.00),
516 LatLon 40.0 (-72.00),
517 LatLon 41.0 (-70.00),
518 LatLon 41.0 (-72.00)]
520let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
524**** Document IDs filter
528-- takes a mapping name and a list of DocIds
529IdsFilter (MappingName "tweet") [DocId "1"]
539-- RangeFilter :: FieldName
540-- -> Either HalfRange Range
544let filter = RangeFilter (FieldName "age")
545 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
546 RangeExecutionIndex False
554let filter = RangeFilter (FieldName "age")
555 (Left (HalfRangeLt (LessThan 100000.0)))
556 RangeExecutionIndex False
572let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
573 RegexpAll (CacheName "test") False (CacheKey "key")
575-- RegexpFlags can be a combination of RegexpAll, Complement,
576-- Interval, Intersection, AnyString, and a combination of two options thereof.
580* Possible future functionality
584Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
586** Function Score Query
588http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
590** Node discovery and failover
592Might require TCP support.
594** Support for TCP access to Elasticsearch
596Pretend to be a transport client?
598** Bulk cluster-join merge
600Might require making a lucene index on disk with the appropriate format.
604http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
608http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
610** Geohash cell filter
612http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
616http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
620http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
624http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
628http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
630** Script based sorting
632http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
634** Collapsing redundantly nested and/or structures
636The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
638** Runtime checking for cycles in data structures
640check for n > 1 occurrences in DFS:
642http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
644http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
648Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list