3#+CAPTION: Bloodhound (dog)
8[[https://travis-ci.org/bitemyapp/bloodhound][https://travis-ci.org/bitemyapp/bloodhound.svg]]
10* Elasticsearch client and query DSL for Haskell
14Search doesn't have to be hard. Let the dog do it.
16** Version compatibility
18Elasticsearch >= 1.0 is recommended. Bloodhound mostly works with 0.9.x, but I don't recommend it if you expect everything to work.
22Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
24* Hackage page and Haddock documentation
26http://hackage.haskell.org/package/bloodhound
36-- Formatted for use in ghci, so there are "let"s in front of the decls.
38-- if you see :{ and :}, they're so you can copy-paste
39-- the multi-line examples into your ghci REPL.
42import Database.Bloodhound
44import Data.Either (Either(..))
45import Data.Maybe (fromJust)
46import Data.Time.Calendar (Day(..))
47import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
48import Data.Text (Text)
49import GHC.Generics (Generic)
50import Network.HTTP.Conduit
51import qualified Network.HTTP.Types.Status as NHTS
53-- no trailing slashes in servers, library handles building the path.
54let testServer = (Server "http://localhost:9200")
55let testIndex = IndexName "twitter"
56let testMapping = MappingName "tweet"
58-- defaultIndexSettings is exported by Database.Bloodhound as well
59let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
61-- createIndex returns IO Reply
63-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
64response <- createIndex testServer defaultIndexSettings testIndex
75response <- deleteIndex testServer testIndex
83-- print response if it was a success
84Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
85 , responseVersion = HTTP/1.1
86 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
87 , ("Content-Length", "21")]
88 , responseBody = "{\"acknowledged\":true}"
89 , responseCookieJar = CJ {expose = []}
90 , responseClose' = ResponseClose}
92-- if the index to be deleted didn't exist anyway
93Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
94 , responseVersion = HTTP/1.1
95 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
96 , ("Content-Length","65")]
97 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
98 , responseCookieJar = CJ {expose = []}
99 , responseClose' = ResponseClose}
105**** Note, you *have* to do this if you expect to read what you just wrote
109resp <- refreshIndex testServer testIndex
117-- print resp on success
118Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
119 , responseVersion = HTTP/1.1
120 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
121 , ("Content-Length","50")]
122 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
123 , responseCookieJar = CJ {expose = []}
124 , responseClose' = ResponseClose}
134-- don't forget imports and the like at the top.
136data TweetMapping = TweetMapping deriving (Eq, Show)
138-- I know writing the JSON manually sucks.
139-- I don't have a proper data type for Mappings yet.
140-- Let me know if this is something you need.
143instance ToJSON TweetMapping where
144 toJSON TweetMapping =
146 object ["properties" .=
147 object ["location" .=
148 object ["type" .= ("geo_point" :: Text)]]]]
151resp <- putMapping testServer testIndex testMapping TweetMapping
159resp <- deleteMapping testServer testIndex testMapping
163** Document Operations
165*** Indexing Documents
169-- don't forget the imports and derive generic setting for ghci
170-- at the beginning of the examples.
173data Location = Location { lat :: Double
174 , lon :: Double } deriving (Eq, Generic, Show)
176data Tweet = Tweet { user :: Text
177 , postDate :: UTCTime
180 , location :: Location } deriving (Eq, Generic, Show)
182exampleTweet = Tweet { user = "bitemyapp"
184 (ModifiedJulianDay 55000)
185 (secondsToDiffTime 10)
186 , message = "Use haskell!"
188 , location = Location 40.12 (-71.34) }
190-- automagic (generic) derivation of instances because we're lazy.
192instance FromJSON Tweet
193instance ToJSON Location
194instance FromJSON Location
197-- Should be able to toJSON and encode the data structures like this:
198-- λ> toJSON $ Location 10.0 10.0
199-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
200-- λ> encode $ Location 10.0 10.0
201-- "{\"lat\":10,\"lon\":10}"
203resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
211Response {responseStatus =
212 Status {statusCode = 200, statusMessage = "OK"}
213 , responseVersion = HTTP/1.1, responseHeaders =
214 [("Content-Type","application/json; charset=UTF-8"),
215 ("Content-Length","75")]
216 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
217 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
221*** Deleting Documents
225resp <- deleteDocument testServer testIndex testMapping (DocId "1")
233-- n.b., you'll need the earlier imports. responseBody is from http-conduit
235resp <- getDocument testServer testIndex testMapping (DocId "1")
237-- responseBody :: Response body -> body
238let body = responseBody resp
240-- you have two options, you use decode and just get Maybe (EsResult Tweet)
241-- or you can use eitherDecode and get Either String (EsResult Tweet)
243let maybeResult = decode body :: Maybe (EsResult Tweet)
244-- the explicit typing is so Aeson knows how to parse the JSON.
246-- use either if you want to know why something failed to parse.
247-- (string errors, sadly)
248let eitherResult = decode body :: Either String (EsResult Tweet)
250-- print eitherResult should look like:
251Right (EsResult {_index = "twitter"
256 , _source = Tweet {user = "bitemyapp"
257 , postDate = 2009-06-18 00:00:10 UTC
258 , message = "Use haskell!"
260 , location = Location {lat = 40.12, lon = -71.34}}})
262-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
264-- use the _source record accessor to get at your document
265λ> fmap _source result
266Right (Tweet {user = "bitemyapp"
267 , postDate = 2009-06-18 00:00:10 UTC
268 , message = "Use haskell!"
270 , location = Location {lat = 40.12, lon = -71.34}})
282-- exported by the Client module, just defaults some stuff.
283-- mkSearch :: Maybe Query -> Maybe Filter -> Search
284-- mkSearch query filter = Search query filter Nothing False 0 10
286let query = TermQuery (Term "user" "bitemyapp") Nothing
288-- AND'ing identity filter with itself and then tacking it onto a query
289-- search should be a null-operation. I include it for the sake of example.
290-- <||> (or/plus) should make it into a search that returns everything.
292let filter = IdentityFilter <&&> IdentityFilter
294-- constructing the search object the searchByIndex function dispatches on.
295let search = mkSearch (Just query) (Just filter)
297-- you can also searchByType and specify the mapping name.
298reply <- searchByIndex testServer testIndex search
300let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
302λ> fmap (hits . searchHits) result
303Right [Hit {hitIndex = IndexName "twitter"
304 , hitType = MappingName "tweet"
305 , hitDocId = DocId "1"
306 , hitScore = 0.30685282
307 , hitSource = Tweet {user = "bitemyapp"
308 , postDate = 2009-06-18 00:00:10 UTC
309 , message = "Use haskell!"
311 , location = Location {lat = 40.12, lon = -71.34}}}]
319let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
320let search = mkSearch (Just query) Nothing
325**** Multi-Match Query
329let fields = [FieldName "user", FieldName "message"]
330let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
331let search = mkSearch (Just query) Nothing
339let innerQuery = QueryMatchQuery $
340 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
341let query = QueryBoolQuery $
342 mkBoolQuery (Just innerQuery) Nothing Nothing
343let search = mkSearch (Just query) Nothing
351let posQuery = QueryMatchQuery $
352 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
353let negQuery = QueryMatchQuery $
354 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
355let query = QueryBoostingQuery $
356 BoostingQuery posQuery negQuery (Boost 0.2)
360**** Rest of the query/filter types
362Just follow the pattern you've seen here and check the Hackage API documentation.
368let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
370-- mkSort is a shortcut function that takes a FieldName and a SortOrder
371-- to generate a vanilla DefaultSort.
372-- checkt the DefaultSort type for the full list of customizable options.
374-- From and size are integers for pagination.
376-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
378-- type Sort = [SortSpec]
379-- type TrackSortScores = Bool
383-- Search takes Maybe Query
389-- just add more sortspecs to the list if you want tie-breakers.
390let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
396**** And, Not, and Or filters
398Filters form a monoid and seminearring.
402instance Monoid Filter where
403 mempty = IdentityFilter
404 mappend a b = AndFilter [a, b] defaultCache
406instance Seminearring Filter where
407 a <||> b = OrFilter [a, b] defaultCache
409-- AndFilter and OrFilter take [Filter] as an argument.
411-- This will return anything, because IdentityFilter returns everything
412OrFilter [IdentityFilter, someOtherFilter] False
414-- This will return exactly what someOtherFilter returns
415AndFilter [IdentityFilter, someOtherFilter] False
417-- Thanks to the seminearring and monoid, the above can be expressed as:
420IdentityFilter <&&> someOtherFilter
423IdentityFilter <||> someOtherFilter
425-- Also there is a NotFilter, it only accepts a single filter, not a list.
427NotFilter someOtherFilter False
435-- And'ing two Identity
436let queryFilter = IdentityFilter <&&> IdentityFilter
438let search = mkSearch Nothing (Just queryFilter)
440reply <- searchByType testServer testIndex testMapping search
446Similar to boolean queries.
450-- Will return only items whose "user" field contains the term "bitemyapp"
451let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
453-- Will return only items whose "user" field does not contain the term "bitemyapp"
454let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
456-- The clause (query) should appear in the matching document.
457-- In a boolean query with no must clauses, one or more should
458-- clauses must match a document. The minimum number of should
459-- clauses to match can be set using the minimum_should_match parameter.
460let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
468-- Will filter for documents that have the field "user"
469let existsFilter = ExistsFilter (FieldName "user")
473**** Geo BoundingBox Filter
477-- topLeft and bottomRight
478let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
480let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
482-- second argument is GeoFilterType, memory or indexed.
483let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
487**** Geo Distance Filter
491let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
493-- coefficient and units
494let distance = Distance 10.0 Miles
496-- GeoFilterType or NoOptimizeBbox
497let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
499-- SloppyArc is the usual/default optimization in Elasticsearch today
500-- but pre-1.0 versions will need to pick Arc or Plane.
502let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
506**** Geo Distance Range Filter
508Think of a donut and you won't be far off.
512let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
514let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
516let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
520**** Geo Polygon Filter
524-- I think I drew a square here.
525let points = [LatLon 40.0 (-70.00),
526 LatLon 40.0 (-72.00),
527 LatLon 41.0 (-70.00),
528 LatLon 41.0 (-72.00)]
530let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
534**** Document IDs filter
538-- takes a mapping name and a list of DocIds
539IdsFilter (MappingName "tweet") [DocId "1"]
549-- RangeFilter :: FieldName
550-- -> Either HalfRange Range
554let filter = RangeFilter (FieldName "age")
555 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
556 RangeExecutionIndex False
564let filter = RangeFilter (FieldName "age")
565 (Left (HalfRangeLt (LessThan 100000.0)))
566 RangeExecutionIndex False
582let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
583 RegexpAll (CacheName "test") False (CacheKey "key")
585-- RegexpFlags can be a combination of RegexpAll, Complement,
586-- Interval, Intersection, AnyString, and a combination of two options thereof.
590* Possible future functionality
594Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
596** Function Score Query
598http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
600** Node discovery and failover
602Might require TCP support.
604** Support for TCP access to Elasticsearch
606Pretend to be a transport client?
608** Bulk cluster-join merge
610Might require making a lucene index on disk with the appropriate format.
614http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
618http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
620** Geohash cell filter
622http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
626http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
630http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
634http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
638http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
640** Script based sorting
642http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
644** Collapsing redundantly nested and/or structures
646The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
648** Runtime checking for cycles in data structures
650check for n > 1 occurrences in DFS:
652http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
654http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
658Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list