3#+CAPTION: Bloodhound (dog)
8https://travis-ci.org/bitemyapp/bloodhound.svg
9https://travis-ci.org/bitemyapp/bloodhound
11* Elasticsearch client and query DSL for Haskell
15Search doesn't have to be hard. Let the dog do it.
19Bloodhound is alpha at the moment. The library works fine, but I don't want to mislead anyone into thinking the API is final or stable. I wouldn't call the library "complete" or representative of everything you can do in Elasticsearch but compared to clients in other languages, the usability is good.
29-- Formatted for use in ghci, so there are "let"s in front of the decls.
31-- if you see :{ and :}, they're so you can copy-paste
32-- the multi-line examples into your ghci REPL.
35import Database.Bloodhound
37import Data.Either (Either(..))
38import Data.Maybe (fromJust)
39import Data.Time.Calendar (Day(..))
40import Data.Time.Clock (secondsToDiffTime, UTCTime(..))
41import Data.Text (Text)
42import GHC.Generics (Generic)
43import Network.HTTP.Conduit
44import qualified Network.HTTP.Types.Status as NHTS
46-- no trailing slashes in servers, library handles building the path.
47let testServer = (Server "http://localhost:9200")
48let testIndex = IndexName "twitter"
49let testMapping = MappingName "tweet"
51-- defaultIndexSettings is exported by Database.Bloodhound as well
52let defaultIndexSettings = IndexSettings (ShardCount 3) (ReplicaCount 2)
54-- createIndex returns IO Reply
56-- response :: Reply, Reply is a synonym for Network.HTTP.Conduit.Response
57response <- createIndex testServer defaultIndexSettings testIndex
68response <- deleteIndex testServer testIndex
76-- print response if it was a success
77Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
78 , responseVersion = HTTP/1.1
79 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
80 , ("Content-Length", "21")]
81 , responseBody = "{\"acknowledged\":true}"
82 , responseCookieJar = CJ {expose = []}
83 , responseClose' = ResponseClose}
85-- if the index to be deleted didn't exist anyway
86Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}
87 , responseVersion = HTTP/1.1
88 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
89 , ("Content-Length","65")]
90 , responseBody = "{\"error\":\"IndexMissingException[[twitter] missing]\",\"status\":404}"
91 , responseCookieJar = CJ {expose = []}
92 , responseClose' = ResponseClose}
98**** Note, you *have* to do this if you expect to read what you just wrote
102resp <- refreshIndex testServer testIndex
110-- print resp on success
111Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}
112 , responseVersion = HTTP/1.1
113 , responseHeaders = [("Content-Type", "application/json; charset=UTF-8")
114 , ("Content-Length","50")]
115 , responseBody = "{\"_shards\":{\"total\":10,\"successful\":5,\"failed\":0}}"
116 , responseCookieJar = CJ {expose = []}
117 , responseClose' = ResponseClose}
127-- don't forget imports and the like at the top.
129data TweetMapping = TweetMapping deriving (Eq, Show)
131-- I know writing the JSON manually sucks.
132-- I don't have a proper data type for Mappings yet.
133-- Let me know if this is something you need.
136instance ToJSON TweetMapping where
137 toJSON TweetMapping =
139 object ["properties" .=
140 object ["location" .=
141 object ["type" .= ("geo_point" :: Text)]]]]
144resp <- createMapping testServer testIndex testMapping TweetMapping
152resp <- deleteMapping testServer testIndex testMapping
156** Document Operations
158*** Indexing Documents
162-- don't forget the imports and derive generic setting for ghci
163-- at the beginning of the examples.
166data Location = Location { lat :: Double
167 , lon :: Double } deriving (Eq, Generic, Show)
169data Tweet = Tweet { user :: Text
170 , postDate :: UTCTime
173 , location :: Location } deriving (Eq, Generic, Show)
175exampleTweet = Tweet { user = "bitemyapp"
177 (ModifiedJulianDay 55000)
178 (secondsToDiffTime 10)
179 , message = "Use haskell!"
181 , location = Location 40.12 (-71.34) }
183-- automagic (generic) derivation of instances because we're lazy.
185instance FromJSON Tweet
186instance ToJSON Location
187instance FromJSON Location
190-- Should be able to toJSON and encode the data structures like this:
191-- λ> toJSON $ Location 10.0 10.0
192-- Object fromList [("lat",Number 10.0),("lon",Number 10.0)]
193-- λ> encode $ Location 10.0 10.0
194-- "{\"lat\":10,\"lon\":10}"
196resp <- indexDocument testServer testIndex testMapping exampleTweet (DocId "1")
204Response {responseStatus =
205 Status {statusCode = 200, statusMessage = "OK"}
206 , responseVersion = HTTP/1.1, responseHeaders =
207 [("Content-Type","application/json; charset=UTF-8"),
208 ("Content-Length","75")]
209 , responseBody = "{\"_index\":\"twitter\",\"_type\":\"tweet\",\"_id\":\"1\",\"_version\":2,\"created\":false}"
210 , responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
214*** Deleting Documents
218resp <- deleteDocument testServer testIndex testMapping (DocId "1")
226-- n.b., you'll need the earlier imports. responseBody is from http-conduit
228resp <- getDocument testServer testIndex testMapping (DocId "1")
230-- responseBody :: Response body -> body
231let body = responseBody resp
233-- you have two options, you use decode and just get Maybe (EsResult Tweet)
234-- or you can use eitherDecode and get Either String (EsResult Tweet)
236let maybeResult = decode body :: Maybe (EsResult Tweet)
237-- the explicit typing is so Aeson knows how to parse the JSON.
239-- use either if you want to know why something failed to parse.
240-- (string errors, sadly)
241let eitherResult = decode body :: Either String (EsResult Tweet)
243-- print eitherResult should look like:
244Right (EsResult {_index = "twitter"
249 , _source = Tweet {user = "bitemyapp"
250 , postDate = 2009-06-18 00:00:10 UTC
251 , message = "Use haskell!"
253 , location = Location {lat = 40.12, lon = -71.34}}})
255-- _source in EsResult is parametric, we dispatch the type by passing in what we expect (Tweet) as a parameter to EsResult.
257-- use the _source record accessor to get at your document
258λ> fmap _source result
259Right (Tweet {user = "bitemyapp"
260 , postDate = 2009-06-18 00:00:10 UTC
261 , message = "Use haskell!"
263 , location = Location {lat = 40.12, lon = -71.34}})
275-- exported by the Client module, just defaults some stuff.
276-- mkSearch :: Maybe Query -> Maybe Filter -> Search
277-- mkSearch query filter = Search query filter Nothing False 0 10
279let query = TermQuery (Term "user" "bitemyapp") Nothing
281-- AND'ing identity filter with itself and then tacking it onto a query
282-- search should be a null-operation. I include it for the sake of example.
283-- <||> (or/plus) should make it into a search that returns everything.
285let filter = IdentityFilter <&&> IdentityFilter
287-- constructing the search object the searchByIndex function dispatches on.
288let search = mkSearch (Just query) (Just filter)
290-- you can also searchByType and specify the mapping name.
291reply <- searchByIndex testServer testIndex search
293let result = eitherDecode (responseBody reply) :: Either String (SearchResult Tweet)
295λ> fmap (hits . searchHits) result
296Right [Hit {hitIndex = IndexName "twitter"
297 , hitType = MappingName "tweet"
298 , hitDocId = DocId "1"
299 , hitScore = 0.30685282
300 , hitSource = Tweet {user = "bitemyapp"
301 , postDate = 2009-06-18 00:00:10 UTC
302 , message = "Use haskell!"
304 , location = Location {lat = 40.12, lon = -71.34}}}]
312let query = QueryMatchQuery $ mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
313let search = mkSearch (Just query) Nothing
318**** Multi-Match Query
322let fields = [FieldName "user", FieldName "message"]
323let query = QueryMultiMatchQuery $ mkMultiMatchQuery fields (QueryString "bitemyapp")
324let search = mkSearch (Just query) Nothing
332let innerQuery = QueryMatchQuery $
333 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
334let query = QueryBoolQuery $
335 mkBoolQuery (Just innerQuery) Nothing Nothing
336let search = mkSearch (Just query) Nothing
344let posQuery = QueryMatchQuery $
345 mkMatchQuery (FieldName "user") (QueryString "bitemyapp")
346let negQuery = QueryMatchQuery $
347 mkMatchQuery (FieldName "user") (QueryString "notmyapp")
348let query = QueryBoostingQuery $
349 BoostingQuery posQuery negQuery (Boost 0.2)
353**** Rest of the query/filter types
355Just follow the pattern you've seen here and check the Hackage API documentation.
361let sortSpec = DefaultSortSpec $ mkSort (FieldName "age") Ascending
363-- mkSort is a shortcut function that takes a FieldName and a SortOrder
364-- to generate a vanilla DefaultSort.
365-- checkt the DefaultSort type for the full list of customizable options.
367-- From and size are integers for pagination.
369-- When sorting on a field, scores are not computed. By setting TrackSortScores to true, scores will still be computed and tracked.
371-- type Sort = [SortSpec]
372-- type TrackSortScores = Bool
376-- Search takes Maybe Query
382-- just add more sortspecs to the list if you want tie-breakers.
383let search = Search Nothing (Just IdentityFilter) (Just [sortSpec]) False 0 10
389**** And, Not, and Or filters
391Filters form a monoid and seminearring.
395instance Monoid Filter where
396 mempty = IdentityFilter
397 mappend a b = AndFilter [a, b] defaultCache
399instance Seminearring Filter where
400 a <||> b = OrFilter [a, b] defaultCache
402-- AndFilter and OrFilter take [Filter] as an argument.
404-- This will return anything, because IdentityFilter returns everything
405OrFilter [IdentityFilter, someOtherFilter] False
407-- This will return exactly what someOtherFilter returns
408AndFilter [IdentityFilter, someOtherFilter] False
410-- Thanks to the seminearring and monoid, the above can be expressed as:
413IdentityFilter <&&> someOtherFilter
416IdentityFilter <||> someOtherFilter
418-- Also there is a NotFilter, it only accepts a single filter, not a list.
420NotFilter someOtherFilter False
428-- And'ing two Identity
429let queryFilter = IdentityFilter <&&> IdentityFilter
431let search = mkSearch Nothing (Just queryFilter)
433reply <- searchByType testServer testIndex testMapping search
439Similar to boolean queries.
443-- Will return only items whose "user" field contains the term "bitemyapp"
444let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp") False)
446-- Will return only items whose "user" field does not contain the term "bitemyapp"
447let queryFilter = BoolFilter (MustNotMatch (Term "user" "bitemyapp") False)
449-- The clause (query) should appear in the matching document.
450-- In a boolean query with no must clauses, one or more should
451-- clauses must match a document. The minimum number of should
452-- clauses to match can be set using the minimum_should_match parameter.
453let queryFilter = BoolFilter (ShouldMatch [(Term "user" "bitemyapp")] False)
461-- Will filter for documents that have the field "user"
462let existsFilter = ExistsFilter (FieldName "user")
466**** Geo BoundingBox Filter
470-- topLeft and bottomRight
471let box = GeoBoundingBox (LatLon 40.73 (-74.1)) (LatLon 40.10 (-71.12))
473let constraint = GeoBoundingBoxConstraint (FieldName "tweet.location") box False
475-- second argument is GeoFilterType, memory or indexed.
476let geoFilter = GeoBoundingBoxFilter constraint GeoFilterMemory
480**** Geo Distance Filter
484let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
486-- coefficient and units
487let distance = Distance 10.0 Miles
489-- GeoFilterType or NoOptimizeBbox
490let optimizeBbox = OptimizeGeoFilterType GeoFilterMemory
492-- SloppyArc is the usual/default optimization in Elasticsearch today
493-- but pre-1.0 versions will need to pick Arc or Plane.
495let geoFilter = GeoDistanceFilter geoPoint distance SloppyArc optimizeBbox False
499**** Geo Distance Range Filter
501Think of a donut and you won't be far off.
505let geoPoint = GeoPoint (FieldName "tweet.location") (LatLon 40.12 (-71.34))
507let distanceRange = DistanceRange (Distance 0.0 Miles) (Distance 10.0 Miles)
509let geoFilter = GeoDistanceRangeFilter geoPoint distanceRange
513**** Geo Polygon Filter
517-- I think I drew a square here.
518let points = [LatLon 40.0 (-70.00),
519 LatLon 40.0 (-72.00),
520 LatLon 41.0 (-70.00),
521 LatLon 41.0 (-72.00)]
523let geoFilter = GeoPolygonFilter (FieldName "tweet.location") points
527**** Document IDs filter
531-- takes a mapping name and a list of DocIds
532IdsFilter (MappingName "tweet") [DocId "1"]
542-- RangeFilter :: FieldName
543-- -> Either HalfRange Range
547let filter = RangeFilter (FieldName "age")
548 (Right (RangeLtGt (LessThan 100000.0) (GreaterThan 1000.0)))
549 RangeExecutionIndex False
557let filter = RangeFilter (FieldName "age")
558 (Left (HalfRangeLt (LessThan 100000.0)))
559 RangeExecutionIndex False
575let filter = RegexpFilter (FieldName "user") (Regexp "bite.*app")
576 RegexpAll (CacheName "test") False (CacheKey "key")
578-- RegexpFlags can be a combination of RegexpAll, Complement,
579-- Interval, Intersection, AnyString, and a combination of two options thereof.
583* Possible future functionality
587Beginning here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-span-first-query.html
589** Function Score Query
591http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
593** Node discovery and failover
595Might require TCP support.
597** Support for TCP access to Elasticsearch
599Pretend to be a transport client?
601** Bulk cluster-join merge
603Might require making a lucene index on disk with the appropriate format.
607http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
611http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html
613** Geohash cell filter
615http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geohash-cell-filter.html
619http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html
623http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html
627http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-filter.html
631http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html
633** Script based sorting
635http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_script_based_sorting
637** Collapsing redundantly nested and/or structures
639The Seminearring instance, if deeply nested can possibly produce nested structure that is redundant. Depending on how this affects ES perforamnce, reducing this structure might be valuable.
641** Runtime checking for cycles in data structures
643check for n > 1 occurrences in DFS:
645http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic.html
647http://hackage.haskell.org/package/stable-maps-0.0.5/docs/System-Mem-StableName-Dynamic-Map.html
651Photo from HA! Designs: https://www.flickr.com/photos/hadesigns/
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list